Efficient Multi-agent Communication via Self-supervised Information Aggregation
Authors: Cong Guan
Feng Chen
Lei Yuan
Chenghe Wang
Hao Yin
Zongzhang Zhang
Yang Yu
Venue: NeurIPS 2022 Accept (2022)
Keywords: Multi-agent reinforcement learning
Communication
State representation learning

Abstract: Utilizing messages from teammates can improve coordination in cooperative Multi-agent Reinforcement Learning (MARL). To obtain meaningful information for decision-making, previous works typically combine raw messages generated by teammates with local information as inputs for policy. However, neglecting the aggregation of multiple messages poses great inefficiency for policy learning. Motivated by recent advances in representation learning, we argue that efficient message aggregation is essential for good coordination in MARL. In this paper, we propose Multi-Agent communication via Self-supervised Information Aggregation (MASIA), with which agents can aggregate the received messages into compact representations with high relevance to augment the local policy. Specifically, we design a permutation invariant message encoder to generate common information aggregated representation from raw messages and optimize it via reconstructing and shooting future information in a self-supervised manner. Each agent would utilize the most relevant parts of the aggregated representation for decision-making by a novel message extraction mechanism. Empirical results demonstrate that our method significantly outperforms strong baselines on multiple cooperative MARL tasks for various task settings.

Efficient Multi-agent Communication via Self-supervised Information Aggregation
Cong Guan't Feng Chen', Lei Yuan 1.2, Chenghe Wang',
Hao Yin', Zongzhang Zhang', Yang Yu 1.2
National Key Laboratory for Novel Software Technology, Nanjing University Polixir Technologies
{guanc, chenf, yuanl, wangch, yinhyalamda, nju. edu, {zzzhang, yuy)enju. edu. en
Abstract
Utilizing messages from teammates can improve coordination in cooperative Multi- agent Reinforcement Learning (MARL.. To obtain meaningful information for decision-making, previous works typically combine raw messages generated by teammates with local information as inputs for policy. However, neglecting the aggregation of multiple messages poses great inefficiency for policy learning. Motivated by recent advances in representation learning, we argue that efficient message aggregation is essential for good coordination in MARL. In this paper, we propose Multi-Agen communication via Self supervieed Information Aggregation (MASIA), with which agents aggregate the received messages into compact representations with high relevance to augment the local policy. Specifically, we design a permutation invariant message encoder generate common information aggregated representation from raw messages and optimize via reconstructing and shooting future information in self-supervised manner. Each agent would utilize the most relevant parts of the aggregated representation for decision-making by novel message extraction mechanism. Empirical results demonstrate that our method significantly outperforms strong baselines on multiple cooperative MARL tasks for various task settings.
Introduction
has attracted widespread attention [17 g recently. achieving remarkable success many complex domains [ such as traffic signal control [
Multi-Agent Reinforcement Learning (MARL) 17
autonomous driving [59], and droplet control 24 For better coordination on further applications,
some issues like non-stationarity 33], scalability 3 remain be solved. solve the non-stationarity caused by the concurrent learning of multiple policies and scalability as the agent number increases, most recent works on MARL adopt the Centralized Training and Decentralized Execution (CTDE) 20 27
paradigm, which includes both value-based methods [42 36, 46 53) and policy gradient
methods [4 26 52]. Under the CTDE paradigm, however, the coordination ability the learned policies can be fragile due to the partial observability in the multi-agent environment, which is common challenge in many multi-agent tasks [29]. While recurrent neural networks could in principle relieve this issue by conditioning the policy oeciion-bseevattio history [16], the uncertainty of other agents (e.g.. states and actions) at execution time can result catastrophic miscoordination and
sub-optimality 48
"Equal Contribution Corresponding Author
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Communication shows great potential in solving these problems [55 60], with which agents can share information such as observations, intentions, or experiences stabilize the learning process, leading to better understanding of the environment (or the other agents) and better coordination as result.
Previous communication methods either focus on generating meaningful information [48] 19 57 for the message senders, or design techniques such as attention mechanism 4 [1, message gate 12, to filter the most relevant information on raw received messages. These approaches treat the received information black box and tacitly assume that policy networks can automatically extract the most critical information from multiple raw messages during policy learning. On this occasion, with the only signal given by reinforcement learning, the extraction process may be reasonably inefficient, especially in complex scenarios.
Motivated by recent advances in state representation learning 38 22 which reveals that auxiliary
representation objectives could facilitate policy learning [1], we aim at efficiently aggregating information as compact representations for policy by designing a novel communication framework Multi-Agent communication via Self-supervised Information Aggregation (MASIA). Specifically, representations are optimized through self-supervised objectives, which encourages the representa- tions to be both abstract of the true states and predictive of the future information. Since agents are guided towards higher cumulative rewards during policy learning, correlating representations with true states and future information could intensify the learning signals in policy learning. In this way, the efficiency of policy learning could be encouraged. Also, considering that permutation invariance of representations can also promote efficiency, we design a self-attention mechanism to maintain the invariance of obtained representations. We also design network that weighs the aggregated representation for individual agents to derive unique and highly relevant representation to augment local policies. To evaluate our method, conduct extensive experiments on various cooperative multi-agent benchmarks, including Hallway [48], Level-Based Foraging [4), Traffic Junction 4 and two maps from StarCraft Multi-Agent Challenge (SMAC) [48). The experimental results show that MASIA outperforms previous approaches, strong baselines, and ablations of our method.
Our main contributions are:
We propose novel framework that uses message aggregation network to extract from multiple messages generated by various teammates, with which we acquire a permutation invariant information aggregation representation. Agents can then novel focusing network to extract the most relevant information for decision-making
Two representation objectives are introduced to make the information representation compact and sufficient, including the state reconstruction and multi-step future states prediction.
Sufficient experimental results various benchmarks and communication conditions demonstrate that our proposed approach significantly improves the communication per- formance, and visualization results further reveal why it works.
2 Problem Formulation
This paper considers fully cooperative MARL communication problem, which can be modeled as Decentralised Partially Observable Markov Decision Process under Communication (Dec-POMDD Com) 32 and formulated as a tuple (N, A, P, O, R, M), where A = fl..... n) is the set of agents, S is the set of global states, AA is the set of actions, Ω is the set of observations, O is the observation function, represents the reward function, 7 E [0, 1) stands for the discounted factor, and M indicates the set of message. At each time step, due to partial observability, each agent ė E N can only acquire the observation o; E Ω drawn from the observation function O(s, i) with E S, each agent holds an individual policy <(a; Te, m,), where т; represents the history (o) a!.....
.a 1-1 ) of agent at current timestep f, and m, E M is the message received by the agent i. The joint action a = (aI.... ax) leads to next state s' ~ P(s' 5, a) and the global reward R(s, a). The formal objective is to find joint policy π T(T, a) to maximize the global value function Q (T, a) = Eg,a E Y R(ss a) 8p 8, ag = a, TT, with T = (m,,.... TT)) As each agent can behave learning useful message representation in the received end, and agents only use local information o; as message to share within the team.
of o
message sender well as message receiver, this paper considers
We optimize the policy by value-based MARL,, where deep Q-learning 30 implements the action- value function Q(s, a) with deep neural network Q(T, a; parameterized by 0. This paper follows

---

4T
(t,
Ci at)
Le
Mixing Network
K Kistep Rallout
State Decoder
2 g CLtR--1 D
st (T(T..))) at(
Information Extraction
Latent Model
Agent
Agent of
dluio W
of
Integration Network
+++1
Multilayer perceptron
gr+lit+K
Information Aggregation Encoder
Self-Attention Network
Information
Element-wise multiplication
Aggregation
Encoder
Concatenation
of
0in
EMA
0
1:m
++1:+K
(a)
(c)
update
(d)
Figure 1: Structure MASIA. (a) The overall architecture. (b) Information aggregation and
extraction. (c) Information aggregation optimization. (d) Transition model learning.
the CTDE paradigm. In the centralized training phase, deep Q-learning uses a replay memory D to store the transition tuple (T, r, We use Q(T, a; 0) to approximate Q(s, a; 0) to relieve the partial observability. Thus, the parameters A are learnt by minimizing the expected Temporal Difference (TD) error:
( ET (
where V (7': max. Q (T', a'; 0-) the expected future return of the TD target and 0 are parameters of the target network, which will be periodically updated with 0.
3 Method
In this paper, we propose efficient Multi-Agent communication via Self-supervised Information Aggregation (MASIA), novel multi-agent communication mechanism for promoting cooperation performance. Redundant communications could increase the burden of information processing for each agent to make decisions and pose challenges for information extraction since plenty of irrelevant information is contained in raw messages. To design an efficient communication mechanism, believe properties of vital importance sufficiency and compactness, where sufficiency means rich amount of information, and compactness calls for higher information density.
To meet the standard of sufficiency, global encoder, which we call Information Aggregation Encoder (IAE), is shared among agents to aggregate the information broadcasted by agents into a common representation. With proper training, this representation could reflect the global observation that each agent could obtain sufficient information from to make decisions. As for compactness, we first design auxiliary loss the global representation to correlate it with the policy learning process, and make each agent only focus on the part of the representation related to its performance and coordination by the designed focusing network through excluding the unrelated parts. The entire framework our method is shown in Fig. ! We introduce the aggregation and extraction process in Sec. 3. and the objectives for compactness and sufficiency in Sec. 3.2 Also, description of the training and execution process flow can be found Appendix A.6
3.1 Information Aggregation and Extraction
Information Aggregation. Believing that the true state should be reflected from combined messages, we design the aggregation encoder to be capable of subsuming the messages sent from agents. Also, the communication system in multi-agent systems is flexible and permutation invariant in nature,

---

which calls for permutation invariant structure for the aggregation encoder. Based on these beliefs, we apply self-attention mechanism aggregate multiple messages from different teammates:
Q, K, V Q...Q... (lof.... ................................ of . o,
(1)
QKT
H = softmax vd )
(2)
where the learnable matrices Q, K, and transform the perception from all agents into the corresponding query Q key K, and value V. which are the concepts defined in the atten- tion mechanism [4]. Specifically, each row vector of H can seen as a querying result of one agent for all available information, and the hidden state H will be fed into the sub- sequent integration network to finally obtain the output aggregated representation 2 A de- tailed discussion about the design of the integration network can be found in Appendix A.3 In the centralized training phase, we use the aggregated representation z' as extra inputs in addition to the indi- vidual observation to feed the value function. Since zt contains the information required to determine the true state, taking z' as extra inputs could reduce the uncer- tainty about the environment states and produce better estimations on the Q-value for value functions under any value-based policy learning algorithm.
100
-10
Mollan Win
-30
Information Extraction. Similar to the decision process of human beings, global messages are usually redundant for an individual agent to make good coordination in com- munication systems. For example,
-0
20
the task of Traffic 4. one natural idea is that the information of neighboring cars are more important for agents perceive than those distant ones, and the unrelated information in global message may sometimes even confuse the agents and impede the learning when the map is large. A toy ex- periment shown in Fig. supports this idea. We apply the QMIX algorithm in Traffic Junction tasks with different sight settings. The results show that the agents learn better policies when in small-sight setting (sight-1) 1) than both in a super- seneer-immtedd-iigee setting (sight-O) and full-sight setting (sight-full).
6:M
Timeetops
Junction
aeerry car
distant
c visible area
Figure A toy experiment for informa- tion redundancy the task of Traffic Junction.
To make each agent capable of deciding its own perceptive area, employ the focusing network to weigh the aggregated representations for each agent. The focusing network is designed as a Multi-Layer Perception (MLP) with the Sigmoid output activation function ensure that each dimension of ue is bounded between 0 and 1. By taking element-wise multiplication with z', a unique representation could be distilled for individual agents. In this way, if the focusing network produces higher weights on specific dimensions, changes in aggregated representations on these dimensions would be more significant and thus, the agent would be more sensitive to aggregated representation on these parts. On the contrary, if some near-zero weights are outputted on some dimensions, information on those dimensions would be filtered out. In particular, although the information extraction process can, to some extent, reflect the specificity of each agent, we stress the local information by feeding it into the subsequent network together with the extracted representation.
3.2 Information Aggregation Representation Optimization
As for the learning process of aggregated representation, we consider two typical objectives in global encoder training: reconstruction and multi-step prediction, which constrain the representations produced by the global encoder to be sufficient and compact, respectively. For the reconstruction objective, we employ an additional decoder, which aims to reconstruct the global state by the aggregated representation to allow self-supervision the global encoder. Specifically, the decoder is optimized together with the aggregation encoder by reconstructing global states s² from messages o²:
Lao (0, 1) = Eo 11. (z') - s²112, z' = fo(o'),
where fo. By denote the encoder network parameterized by 0 and the decoder network parameterized by n, respectively. This loss term resembles classical auto-encoder loss, while the decoder here is

---

not to reconstruct the input, but to recover the global state from representations instead. By utilizing this loss, we guide the encoder to extract observational features that can help infer the global state and let 2' [o(o²) be a sufficient representation.
As for the multi-step prediction objective, constrain the produced representation to be predictive of future information. Specifically, we design a transition model hw Z x A + Z parameterized by ei as auxiliary model, which predicts the aggregated representation 2 *1 on next step t + 1 through the aggregated representation 2' and joint action a' current step t. We regress the predicted aggregated representation after k-step rollout on the actual aggregated representation of future messages of + updating both the aggregation encoder and the auxiliary model via the multí-step prediction loss:
K
f. (0, 3/ = Eoo o' ""..a'....."*"""" ,00h o0
||
g1th
(4)
k=1
g(+1 hyy2',
(5)
5 ho a ( k= 2=.,..., K,
(6)
g*th = fo(o -= K.
To further stabilize the learning process, we apply the double network technique, which employs two networks with the same architecture but different update frequencies, for the aggeration encoder. The target network is updated via Exponential Moving Average (EMA) like in SPR 38 By forcing the aggregated representation to be predictive of its future states, the aggregated representation could be more correlated with the information required for its decision- making, which meets the compactness standard. Combining these two objectives allows the aggregation encoder to extract more helpful information for agents to coordinate better. To improve the capability of information extraction individual agents, we also enhance the learning process of these components with an RL objective. Specifically, we consider minimizing the TD loss:
() = E(T,TTT') ( max Qent (T',''; x-) Qtor a; (b) ) a'
mmax
(8)
where Qto is computed with individual Q-values. The computation of Q-values is actually dependent on the specific value-based learning algorithm. We apply it to prevalent methods, including VDN [42), QMIX [36). and QPLEX (46). Moreover, the updating of the focusing network is coupled to the RL objective, making the weights produced by the focusing network could be task sensisens,,,ee, which could also facilitate policy learning.
4 Experiment
We conduct experiments on various benchmarks with different communication request levels Specifically, aim answer the following questions in this section: 1) How does method perform when compared with multiple baselines in various scenarios (Sec. 4. ? 2) What kind of
knowledge has been learned by the information aggregation encoder (Sec. 4.2 ? 3) How can the
information extraction module extract the most relevant information for the individual from the learned embedding space (Sec. 4 ? 4) Can MASIA be applied to different value decomposition baselines to improve their coordination ability and robustness in various communication conditions (Sec. ? We compare MASIA against variety of baselines, including communication-fiee methods and some state-of-the-an communication approaches. QMIX (36) is a strong communication-Fnn free baseline, and we use the implementation by PyMARL for comparison, which has shown excellent performance on diverse multi- agent benchmarks [3]] TarMAC utilizes an attention mechanism to select messages according their relative importance. The implementation we used is provided by 48), denoted as TarMAC QMIX. NDQ [48] aims at learning nearly decomposable Q functions via generating meaningful messages and communication minimization. TMC [57] applies temporal smoothing technique at the message sender end to drastically reduce the amount of information exchanged between agents. For the ablation study, we design a baseline only different in the communication protocol, which adopts full communication paradigm, where each agent gets message from all other teammates at each timestep, denoted as FullComm.
'The codes are available at https://ghhuh..co//cccff//////// aiAAAI
"Our experiments all based on the PyMARL framework. which SC2.4.6.2.6922
5

---

g
(a) Hallway
(h) LBF
(c) Traffic Junction (TJ)
(d) SMAC
Figure Multiple benchmarks used in our experiments.
MASIA (Ours)
Fall-Comm
NDQ
TarMAC + QMIX
TMC
QMIX
100
40
Malba
20
LAN
CAM
CAM
1.2M
1AM
Timollep
(a) Hallway: 4x6x10 10
(b) LBF: 11x11 -ff-s---s
(c) TJ: medium
(d) SMAC: lo10b_vs_Ir
100
Midias Tea Wox
80
50
40
20
13M
1.M
2:51
0/M
0.40
1.5M
2.0
0.5
0.8M
3.04
1.30
1.2
(e) Hallway: 3x6-4x6x10 (0) LBF: 10x20-100-6f-3
(g) TJ: hard
(h) SMAC: 1o2r_vs vs 4r
Figure 4: Performance comparison with baselines on multiple benchmarks.
We evaluate our proposed method on multiple benchmarks shown in Fig. 3 Hallway [48 is cooperative environment under partial observability, where m agents are randomly initialized at different positions and required to arrive at the goal g simultancously. We consider two scenarios with various agents and different groups, and different groups have to arrive at different times. Level Based Foraging (LBF) 34 is another cooperative partially observable grid world game, where agents coordinate to collect food concurrently. Traffic Junction (TJ) [4] is a popular benchmark used to test move along two-way roads with
road junetions following the predefined routes, and we test on the medium and hard maps. Two maps named 1o2r_ 4r and lo10b_ _s______________________ Ir from SMAC [48] require the agents to cooperate and communicate to get the position of the enemies. For evaluation, all results are reported with median performance with 95% confidence interval 5 random seeds. Details about benchmarks, network architecture and hyper-parameter choices of our method are all presented in Appendices A. and A.3 respectively.
communication ability, where many
4.1 Communication Performance
We first compare MASIA against multiple baselines to investigate the communication efficiency on various benchmarks. As illustrated in Fig. 4 MASIA achieves the best performance with low variance on all benchmarks. indicating MASTA's strong applicability in scenarios with various difficulties. In Hallway (Fig. 4a & Fig. 4e). where frequent communications are required for good performance (method without communication such as QMIX fails), other communication methods such as TarMAC, NDQ, and TMC achieve low performance or even fail in this environment. This indicates that inappropriate message generation or message selection would injure the learning process. We believe the reason why FullComm succeeds is that there is hardly any redundancy in agents' observations Hallway. Our MASIA also succeeds this environment, showing superiority over others. The dominating performance of MASIA is even more significant in extended Hallway

---

Closing Phase
Battle Phase
Seoking Phase
Focusing Similarity
Score Curve
Similar Focusing
00 Pros SOODO oo
type type type
Agert
30
Time
Timeline
Agest
98g
20
30
:Ally Agents
Enemy Agents
aa)
(b)
Figure 5: (a) Information aggregation visualization. Each plotted dot represents aggregated representation. The three different colors respectively represent three different initialization situations for the enemy entities. For example, type shows the case when the enemies initialize at the lower right corner. To distinguish aggregated representations on different timesteps, we mark larger timesteps by darker shades of the dots. (b) Visualization of variations of selected agents' focus weight w/ in a single episode. We use the horizontal axes for timesteps in one single episode and vertical axes for dimensions of the aggregated representation. The weight is reflected through luminance, and the darker the cell, the larger the weight.
(Fig. 4e). where agents are separated into different groups. In this environment, MASIA can help agents extract information about their teammates and learn a coordination pattern more efficiently. MARL methods like NDQ, Fullcomm, and TarMAC struggle due to the sparsity of rewards, especially when the foods are more sparsely distributed (Fig. 4r In contrast to the performance of QMIX in Hallway, QMIX performs well in LBF, which is attributed to the fact that agents can observe the grids near them, and the mixing network of QMIX can help improve the coordination ability of the fixed group of agents in the training phase. Our method achieves comparable performance with QMIX and TarMAC, showing its strong coordination ability even in sparse reward scenarios. In Traffic Junction (Fig. 4c & Fig. 4g). TarMAC and NDQ have high variance due to the instability of their messages, while MASIA gains high sample efficiency and can generate steady messages since it aims to reconstruct the state. On the SMAC benchmarks (Fig. 4d & Fig. 4h). we test on two complex scenarios requiring communication to succeed, where one overseer is in active service to get the information of the enemies. Messages are demanded since the agents have limited sight, so other teammates need the overseer's messages to identify the enemies' positions. Our method MASIA can maintain the high efficiency of learning and always have competitive performance when converged, which is superior to other baselines.
In LBF (Fig. 4b] & Fig. 4f). existing communication-bart
4.2 Insights into Information Aggregation Encoder
To determine what kind of knowledge the encoder has learned through training, we conduct a visualization analysis on the map 1o2r_ vs_ 4r from SMAC to demonstrate the information contained in the aggregation representation z². We project the aggregation representation vectors into two- dimensional plane by t-SNE [43 in 55. We take trajectories from 3 scenarios of different types of initialization under task where agents have to seek the enemy at the start, discover the enemy, and finally battle with for better performance. It can be observed that (1) the aggregated representations could be well distinguished by phases. Projected representations in the seeking phase are far from those in the battle phase and closing phase. This implies that our learned representations could well reflect the true states. (2) the aggregated representations are first divergent in the seeking phase
7

---

when enemies have been initialized, but become increasingly interlaced later until the closing phase, when enemies have been wiped out after fierce battle. Since enemies are highly related decision making, such result verifies that the aggregated representations exploit also reward information. To sum up, the visualization results show that MASIA can extract valuable global information with these representations.
4.3 Study about Individual Information Extraction
To demonstrate the effectiveness of our information extraction mod- ule, we analyze the weights computed by the focusing network. Specifically, we select the TJ (medium) task for evaluation and compare the weights produced by two different agents. In this envi- ronment, agents could dynamically enter or leave the plane, making the agents staying in the environment flexible through time. It can be observed that agent 3 and agent 6 put focus on similar areas the aggregated representation. Especially after timestep 15, when agents 3 and 6 are in similar situations and distant from the inter- section, their focuses are nearly the same. This verifies that the global state information has been successfully extracted to indi- vidual agents. Also, on the top of the figure, we draw figure measure the relationship between the cosine similarity of weight vectors of different agents against timesteps. reveals that the sim- ilarity of their focus rises after these two agents begin to proceed in the same line (indicated by the render images posted on the lower parts of Fig. 5 (b). This also conforms to the intuition that similar messages should be extracted for similar observations.
Median T.al Win Bat: N.
100
SIA (19)
80
MASIA (18)
40
QNID (5)
40
QMIX (41)
20
9:0M
0.4M
0,534
1.M 1.00
2.00
Tincslope
Figure 6: Performance com- parison with varying sights, where sn means the sight range is n.
4.4 Generality of Our Method
We aim verify that the proposed approach is agnostic to various sight ranges and applied value-based MARL methods. We first con- duct experiments on the map 1o10b vs to show that MASIA could also generalize well on agents with limited observations. The results Sec. 4.. show the performance MASIA when the agents have sight range of 9. When we narrow the agents' sight ranges, shown in Fig. 6, by receiving and aggregating messages from team- mates, the performance of MASIA does not suffer from significant drop. Our information aggregation and extraction modules prevent the agent from forfeiting knowledge about the state when the sight range further limited.
MANMA
Arcemee Sost Wite Bate
694
UNIK
Figure 7: The increase of win- ning rates brought about by MASIA on map lo10b_vs_Ir
To show the generality of the MASIA framework, we also carry out experiments integrate MASIA with current baselines, including VDN, QMIX, and QPLEX. As illustrated in Fig. 7 when integrated with MASIA, the performance of these baselines can be vastly im- proved on the map 1o10b_vs Ir from SMAC. In this scenario, one overseer is in service to monitor the enemies. Without communication, the other agents have to search the map for the enemies exhaustively. While with reliable communication, they could communicate with each other and the overseer for better coordination. The results demonstrate that MASIA can efficiently aggregate the messages and improve the agents' coordination ability for these value-based MARL methods.
5 Related Work
Multi-agent reinforcement learning (MARL) has made prominent progress these years. Having emerged under the CTDE paradigm, many methods are designed to relieve the non- non-stationn stationarity issue, and have made noticeable progress these years. Most of them can be roughly divided into policy- based and value-based methods. Typical policy gradient-methods involves MADDPG (26). COMA 14, MAAC 18), SQDDPG [47), FOP (58). and HAPPO 21 which explore the optimization of multi-agent policy gradient methods, while value-based methods mainly focus on the factorization of the global value function 45 7j. VDN 42 applics simple additive factorization to decompose the

---

joint value function into agent-wise value functions. QMIX [36] structurally enforces the learned joint value function to be monotonic the agent's utilities, which can represent a more affluent class of value functions. QPLEX (46) further takes duplex dueling network architecture to factorize the joint value function, achieving a full expressiveness power of IGM [39]]
Multi-agent Communication Communication plays a promising role in multi-agent coordination under partial observability 60). Extensive researches have been made on learning communication protocols to improve performance on cooperative tasks 11
23
25
50). Previous
works can be divided into two categories. One focuses on generating a meaningful message for the message senders. The simplest way is to treat the raw local observation, or the local information history as message [T3] 40) VBC [56] and TMC [57] apply techniques, such as variance- based
control and temporal smoothing, in the sender end to make the generated messages meaningful
and valuable for policy learning. NDQ [48] generates minimized messages for different teammates to learn nearly decomposable value functions, and optimize the message generator based on two different information-ther www--tt-hh-seeeee regularizers achieve expressive communication. On the contrary, other works try to learn efficiently to extract the most useful message on the receiver end, and they design mechanisms to differentiate the importance of messages. 12C [6] and ACML [28] employ gate mechanism to be selective received messages. There are also works inspired by the broad application of the attention mechanism 12 3) TarMAC [4] achieves targeted communication via simple signature -ased soft-attention mechanism, where the sender broadcasts a key encoding the properties of the agents, then the receiver attends to all received messages for weighted sum of messages for decision marking. SARNet (35) and MAGIC [1] further remove the signature in TarMAC and leverage attention-based networks to learn efficient and interpretable relations between entities, decide when and with whom communicate.
To the best of our knowledge, none of the existing MARL communication methods explicitly consider how the multiple received messages can be optimized for efficient policy learning, Agents may be confused by redundant information from teammates, and simply augmenting the local policy with the raw message may burden the learning. Our proposed method applies message aggregation module to learn a compact information representation and extracts the most relevant part for decision makiin...
Conclusion and Future Work
In this paper, we investigate the information representation for multi-agent communication. Previous works either focus on generating meaningful message or designing a mechanism to select the most relevant message in a raw way, ignoring the aggregation of the message, resulting in low sample efficiency in complex scenarios. Our approach improves communication efficiency by learning a compact information representation to ground the true state and optimizing in a sefe-super way. Also, we apply a focusing network to extract the most relevant part for decision-making We conduct sufficient experiments in various benchmarks to verify the efficiency of the proposed methods, and more visualization results further reveal why our approach works. For future work, more results on image input and solving the scalability issue when facing environments with hundreds or thousands of agents by techniques like agent grouping would be of great interest.
Acknowledgements
This work is supported by the National Key Research and Development Program of China (2020AAA0107200), the National Science Foundation of China (61921006, 61876119, 62276126), the Natural Science Foundation of Jiangsu (BK20221442), and the program B for Outstanding PhD candidate of Nanjing University. We thank the anonymous reviewers for their useful suggestions.
References
[1] Lorenzo Canese, Gian Carlo Cardarilli, Luca Di Nunzio, Rocco Fazzolari, Daniele Giardino, Marco Re, and Sergio Spanò. Multi-agent reinforcement learning: A review of challenges and applications. Applied Sciences, 11(11):4948, 2021.
[2] Sneha Chaudhari, Varun Mithal, Gungor Polatkan, and Rohan Ramanath, An attentive survey of attention models. arXiv preprint 1xXiv:190..08874 2019.

---

[3] Filippos Christianos, Georgios Papoudakis, Muhammad A Rahman, and Stefano V Albrecht. Scaling multi-agent reinforcement learning with selective parameter sharing. In International Conference Machine Learning, pages 1989-1998, 2021.
[4] Abhishek Das, Théophile Gervet, Joshua Romoff, Dhruv Batra, Devi Parikh, Mike Rabbat, and Joelle Pineau. TarMAC: Targeted multi-agent communication. In International Conference on Machine Learning, pages 1538-1544 1546, 2019.
[5] Alama Santana Correia and Esther Luna Colombini. Attention, please! survey of neural attention models in deep learning. arXiv preprint arXiv:2103.16675,
2021. 2021.
[6] Ziluo Ding, Tiejun Huang, and Zongqing Lu, Learning individually inferred communication for multi-agent cooperation. In Advances in Neural Information Processing Systems, pages 22009922079, 2020.
[7] Zehao Dou, Jakub Grudzien Kuba, and Yaodong Yang. Understanding value decomposition algo- rithms in deep cooperative multi-agent reinforcement learning. arXiv preprint wwxi:2202.00888, 2022.
[8] Wei Du and Shifei Ding. A survey on multi-agent deep reinforcement learning: from the perspective challenges and applications. Artificial Intelligence Review, 54(5):3215-3238, 2021.
[9] Xin Du, Jiahai Wang, Siyuan Chen, and Zhiyue Liu. Multi-agent deep reinforcement learning with tentoal feature fusion for traffic signal control. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 470-485, 2021.
[10] Yali Du, Bo Liu, Vincent Moens, Ziqi Liu, Zhicheng Ren, Jun Wang. Xu Chen, and Haifeng Zhang. Learning correlated communication topology in multi-agent reinforcement learning. In International Conference on Autonomous Agents and MultiAgent Systems, pages 456 464, 2021.
[11] Tom Eccles, Yoram Bachrach, Guy Lever, Angeliki Lazaridou, and Thore Graepel. Biases for emergent communication in multi-agent reinforcement learning. In Advances In Neural Information Processing Systems, pages 13111-13121, 2019.
[12] Linus Ericsson, Henry Gouk, Chen Change Loy, and Timothy M Hospedales. Self- supervised representation learning: Introduction, advances and challenges. arXiv preprint arXiv:2110.09327,, 2021.
[13] Jakob Foerster, Ioannis Alexandros Assael, Nando De Freitas, and Shimon Whiteson. Learn- ing to communicate with deep multi-agent reinforcement learning. In Advances in Neural Information Processing Systems, pages 2137-2145, 2016.
[14] Jakob N. Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In AAAI Conference Artificial Intelligence, pages 2974-2982, 2018.
[15] C Lee Giles and Kam-Chuen Jim. Learning communication for multi-agent systems. In Workshop on Radical Agent Concepts, pages 377-390. Springer, 2002.
[16] Matthew J. Hausknecht and Peter Stone. Deep recurrent q-learning for partially observable mdps. In AAAI Conference Artificial Intelligence, pages 29-37, 2015.
[17] Pablo Hernandez-Leai Bilal Kartal, and Matthew E Taylor. A survey and critique of multiagent deep reinforcement learning. Autonomous Agents and Multi-Agent Systems, 33(6):750-797, 2019.
[18] Shariq Iqbal and Fei Sha. Actor-attention-eritte for multi-agent reinforcement learning. In International Conference Machine Learning, pages 2961-2970, 2019.
[19] Woojun Kim, Jongeui Park, and Youngchul Sung. Communication multi-agent reinforcement learning: Intention sharing, In International Conference on Learning Representations, 2020.
10

---

[20] Landon Kraemer and Bikramjit Banerjee. Multi-agent reinforcement learning as a rehearsal for decentralized planning. Neurocomputing, 190:82-94 2016.
[21] Jakub Grudzien Kuba, Ruiqing Chen, Munning Wen, Ying Wen, Fanglei Sun, Jun Wang, and Yaodong Yang. Trust region policy optimisation in multi-agent reinforcement learning, arXiv preprini arXiv:2109. 1251, 2021.
[22] Michael Laskin, Aravind Srinivas, and Picter Abbeel. CURL: Contrastive unsupervised repre- sentations for reinforcement learning. In International Conference on Machine Learning, pages 5639-5650, 2020.
[23] Angeliki Lazaridou and Marco Baroni. Emergent multi-agen communication in the deep learning era. arXiv preprint arXiv:2006.0244 2020.
[24] Tung-Che Liang, Jin Zhou, Yun-Sheng Chan, Tsung Yi Ho, Krishnendu Chakrabarty, and Cy Lee. Parallel droplet control in meda biochips using multi-agent reinforcement learning. In International Conference on Machine Learning, pages 6588-6599, 2021.
[25] Toru Lin, Jacob Huh, Christopher Stauffer, Ser Nam Lim, and Phillip Isola. Learning ground multi-agent communication with autoencoders. In Advances in Neural Information Processing Systems, pages 15230-15242, 2021.
[26] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed mooerative-compeeiiie environments. In Advances in Neural Information Processing Systems, pages 6379-6390) 2017.
[27] Xueguang Lyu, Yuchen Xiao, Brett Daley, and Christopher Amato. Contrasting centralized and decentralized critics in multi-agent reinforcement learning. arXiv preprint 20X.::2102.004400, 2021.
[28) Hangyu Mao, Zhengchao Zhang, Zhen Xiao, Zhibo Gong, and Yan Ni. Learning agent communication under limited bandwidth by message pruning. In AAAI Conference on Artificial Intelligence, pages 5142-51 1142-5149, 2020.
[29] Weichao Mao, Kaiqing Zhang, Erik Miehling, and Tamer Başar. Information state embedding in partially observable cooperative multi-agent reinforcement learning. In IEEE Conference on Decision and Control, pages 6124-6131, 2020.
[30] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. RUsu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Pe- tersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King. Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533 2015.
[31] Yaru Niu, Rohan Paleja, and Matthew Gombolay. Multi-agent graph-attention communication and teaming. In International Conference on Autonomous Agents and MultiAgent Systems, pages 964-973, 2021.
[32] Frans A Oliehock and Christopher Amato. A Concise Introduction to Decentralized POMDPs. Springer, 2016.
[33] Georgios Papoudakis, Filippos Christianos, Arrasy Rahman, and Stefano V Albrecht. Deal- ing with non-stationarity in multi-agent deep reinforcement learning. arXiv preprint arXiv: 2019 2019.
[34] Georgios Papoudakis, Filippos Christianos, Lukas Schäfer, and Stefano V Albrecht. Benchmark- ing multi-agent deep reinforcement learning algorithms in cooperative tasks. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, 2021.
[35] Murtaza Rangwala and Ryan Williams. Learning multi-agent communication through structured attentive reasoning. In Advances in Neural Information Processing Systems, pages 10088-10098. 2020.
11

---

[36] Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. QMIX: Monotonic value function factorisation for deep multi agent reinforcement learning. In International Conference on Machine Learning, pages -99444304, 2018.
[37] Mikayel Samvelyan, Tabish Rashid, Christian Schröder de Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip HS Torr, Jakob Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. In International Conference Autonomous Agents and MultiAgent Systems, pages 2186-2188, 2019.
[38] Max Schwarzer, Ankesh Anand, Rishab Goel, Devon Hjelm, Aaron Courville, and Philip Bachman. Data-efficient reinforcement learning with self- aeff-preiittve representations. In Interna- tional Conference on Learning Representations, 2020.
[39] Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi. QTRAN: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In International Conference on Machine Learning, pages 5887-5896, 2019.
[40] Sainbayar Sukhbaatar, Arthur Szlam, and Rob Fergus. Learning multiagent communication with backpropagation. Advances in Neural Information Processing Systems, pages 2244-2252, 2016.
[41] Yanchao Sun, Ruijie Zheng, Parisa Hassanzadeh, Yongyuan Liang, Soheil Feizi, Sumitra Ganesh, and Furong Huang. Certifiably robust policy learning against adversarial communication in multi-agent systems. arXiv preprint arXiv:2206.1011 10158, 2022.
[42] Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojeiech Marian Czamecki, Vinícius Flores Zambaldi, Max Jaderberg, Mare Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, and Thore Graepel. Value-decompositionn networks for cooperative multi-agent learning based team reward. In International Conference on Autonomous Agents and MultiAgent Systems, pages 2085 2085,2087,400 2018
[43] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of Machine Learning Research, 9(11)2579--265, 2008
[44] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages
2017.
[45] Jianhao Wang, Zhizhou Ren, Beining Han, Jianing Ye, and Chongjie Zhang. Towards under- standing cooperative multi-agent q-learning with value factorization. In Advances in Neural Information Processing Systems, pages 29142-291555 2021.
[46] Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjic Zhang. QPLEX: Duplex dueling multi-agent gelarnnnn. In International Conference Learning Representations, 2021.
[47] Jianhong Wang. Yuan Zhang, Tae- Kim, and Yunjie Gu. Shapley Q-value: A local reward approach solve global reward games. In AAAI Conference on Artificial Intelligence, pages 2020.
[48] Tonghan Wang, Jianhao Wang, Chongyi Zheng, and Chongjie Zhang. Learning nearly de- composable value functions via communication minimization. In International Conference Learning Representations, 2020.
[49] Yihan Wang, Beining Han, Tonghan Wang, Heng Dong, and Chongjie Zhang. DOP: Off- policy multi-agent decomposed policy gradients. In International Conference on Learning Representations, 2021.
[50] Di Xue, Lei Yuan, Zongzhang Zhang, and Yang Yu. Efficient multi-agent communication via shapley message value. In International Joint Conference on Artificial Intelligence, pages 578-584 2022.
12

---

[51] Wanqi Xue, Wei Qiu, Bo An, Zinovi Rabinovich, Svetlana Obraztsova, and Chai Kiat Yeo. Mis-spoke or mis-lead: Achieving robustness in aullti-aeent communicative reinforcement learning. In International Conference on Autonomous Agents and MulliAgent Systems, pages 1426, 2022.
[52] Jianing Ye, Chenghao Li, Jianhao Wang, and Chongjie Zhang. Towards global optimality in cooperative marl with sequential transformation. arXiv preprint arXiv:2207.11143 2022.
[53] Lei Yuan, Chenghe Wang, Jianhao Wang, Fuxiang Zhang, Feng Chen, Cong Guan, Zongzhang Zhang, Chongjie Zhang, and Yang Yu. Multi-agent concentrative coordination with decentral- ized task representation. In International Joint Conference on Artificial Intelligence, pages 599-605, 2022.
[54] Lei Yuan, Jianhao Wang, Fuxiang Zhang, Chenghe Wang, Zongzhang Zhang, Yang Yu, and Chongjie Zhang. Multi-agent incentive communication via decentralized teammate modeling. In AAAI Conference on Artificial Intelligence, pages 9466-9474, 2022.
[55] Mohamed Salah Zaïem and Etienne Bennequin. Learning to communicate in multi-agent reinforcement learning: A review. arXiv preprint arXiv: .05438, 2019.
[56] Sai Qian Zhang, Qi Zhang, and Jieyu Lin. Efficient communication in multi-agent reinforcement learning via variance based control. In Advances in Neural Information Processing Systems, pages 3230-3239, 2019.
[57] Sai Qian Zhang, Qi Zhang, and Jieyu Lin. Succinet and robust multi-agent communication with temporal message control. In Advances in Neural Information Processing Systems, pages 17271-17282, 2020.
[58] Tianhao Zhang, Yueheng Li, Chen Wang, Guangming Xie, and Zongqing Lu. FOP: Factorizing optimal joint policy of maximum-entropy multi-agent reinforcement learning. In International Conference on Machine Learning, pages 12491- 2021.
[59] Ming Zhou, Jun Luo, Julian Villela, Yaodong Yang, David Rusu, Jiayu Miao, Weinan Zhang, Montgomery Alban, Iman Fadakar, Zheng Chen, Aurora Chongxi Huang, Ying Wen, Kimia Hassanzadeh, Daniel Graves, Dong Chen, Zhenghang Zhu, Nhat M. Nguyen, Mohamed Elsayed, Kun Shao, Sanjeevan Ahilan, Baokuan Zhang, Jiannan Wu, Zhengang Fu, Kasra Rezaee, Peyman Yadmellat, Mohsen Rohani, Nicolas Perez Nieves, Yihan Ni, Seyedershad Banijamali, Alexander Imani Cowen-Rivers, Zheng Tian, Daniel Palenicek, Haitham Bou- Ammar, Hongbo Zhang, Wulong Liu, Jianye Hao, and Jun Wang. Smarts: Scalable multi-agent reinforcement learning training school for autonomous driving. arXiv preprint arXiv:2010.09776, 2020.
[60] Changxi Zhu, Mehdi Dastani, and Shihan Wang. A survey of multi-agent reinforcement learning with communication. arXiv preprint XXXx:2203.08973 2022.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 6
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
6
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] Our paper conforms to them.
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments...
13

---

(a) Did you include the code. data, and instructions needed to reproduce the main ex- perimental results (either the supplemental material or as URL)? [Yes] See our supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix A.3
(c) Did you report error bars (c.g., with respect to the random seed after running experi- ments multiple times)? [Yes] See the results in Section [
(d) Did you include the total amount compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix A.3
4. If you are using existing assets (e.g., code, data, models) or curating/releasng new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See Section
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
Our new assets are included the supplemental material.
(d) Did you discuss whether and how consent was obtained from people whose data you're
using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you deseribe any potential participant risks, with links Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "fg6KeMUgaE_", "reviewer": "Reviewer_P72Q", "summary": "The paper considers multi-agent RL (MARL), where agents share information at each step using communications.  The authors present a centralized training scheme where all the agents observations (sensing) are collected and processed / aggregated, and a focus of attention mechanism is also learned to weight the shared observations for each agent to collect only what is useful of the global information for each agent.  These can be added on to existing MARL methods (section 4.4).  The paper gives an overview the ideas, and then presents several numerical experiments and comparisons.  ", "questions": "Why not just start with a notion of neighbors, and proceed from there?\n\nThe comparisons are very difficult to interpret.  For example, “QMIX performs well in LBF, which is attributed to the fact that agents can observe the grids near them, and the mixing network of QMIX can help improve the coordination ability of the fixed group of agents in the training phase.”  So, QMIX knows the state of the neighbors (and communication with them is unnecessary)?  \n\nPerhaps some kind of table could be used to compare the different algorithms, including how they communicate or not, the observation model, and what kind of scenarios they fit (or not).  \n\nPlease clarify “updating of the focusing network is coupled to the RL objective, making the weights produced by the focusing network could be task-sensitive, which could also facilitate policy learning.”\n\n“other communication methods such as TarMAC, NDQ, and TMC achieve low performance or even fail in this environment.”  Why?  Please be specific here, beyond some intuition.\n\nAnd please clarify “why FullComm succeeds is that there is hardly any redundancy in agents’ observations in Hallway”.\n\n“TarMAC and NDQ have high variance due to the instability of their messages, while MASIA gains high sample efficiency and can generate steady messages since it aims to reconstruct the state.”  What is message instability?\n", "limitations": "No negative social impact is expected.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The idea of learning a focus of attention in the centralized learning phase is interesting.  Basically, what agents should be communicating. The focus of attention is apparently changing with global state (but not well explained).\n\nThe presentation is confusing and hard to read, and the algorithm(s) are never clearly laid out. There is undefined notation, and cases where variables are introduced and (perhaps) defined later.  For example, even in the basic model eqn (1), Q, K, and V aren’t clearly defined.\n\nThe presentation is weak on mathematical presentation, and many results are described in an intuitive but often confusing way.  There is no theoretical analysis.\n\nThe overall scalability is likely to be poor.  Collecting all observations and then reducing them is not scalable.  \n\nIt’s not quite clear if this work is intended to be model-based or model-free.  When a map is known, and the agent’s states are known, then for example it is straightforward in the global training to set up a local neighborhood for communications.  The algorithm seems to come back to this in the examples, which is not surprising.  In section 2 it is stated that “agents only use local information o_i as message to share within the team” so apparently this is an underlying assumption (but it isn’t clear if agents who are far away from each other but have useful information will be enabled to communicate).\n\nThere are a lot of parameters and hyper-parameters, and their selection in general is not considered. \n\nThe examples are useful, but also difficult to interpret without significant effort by the reader to examine all the cited works.  For example, it isn’t clear how each algorithm uses communications (or not), and why these are good for some problems and not others.  The conclusions about (local) communications are not surprising in each scenario given.  It seems that when the previously existing algorithm is well matched to the scenario, then there is little be gained with the proposed method.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "RDfD_DvueQG", "reviewer": "Reviewer_Edv9", "summary": "The authors propose MASIA, a method for improving convergence characteristics and communication in cooperative MARL settings.\n\nThe key motivation of their work is to produce compact and informative communication by aggregating information from different agents into a single representation, which individual agents can get relevant information from. To incentivize \"good\" overall representations (depicted as $z_t$ in the paper, the authors introduce a decoding loss (to predict the full state) and a predictive loss (to predict future states via rollout).\n\nIn experiments in several standard MARL domains, the authors show that MASIA integrates well with existing MARL training frameworks and typically trains agents faster and more stably than existing methods. Qualitative analysis of emergent representations indicates somewhat interpretable clustering and similarity of communication vectors learned by MASIA agents.", "questions": "1) How do the agents behave different during decentralized execution vs. centralized training? I understand the paradigm and have written code using such frameworks. I just want to understand, given the MASIA architectures, what information agents have access to during test time. In particular, I want to understand what feeds into the information aggregation encoder at test time, and if $z_t$ is used at test time as well. I really need to understand this part to assess the technical merits of this work.\n\n2) What are the inputs to the Information Aggregation Encoder generally? The authors use $o_i$ notation, which to me indicates different observations (including over time, with the $t$ superscript). So, for the traffic junction setting, for example, I assume that means the standard observations of things like the location, route id, and surrounding cells? Do the observations include communication, or is communication simply generated afresh as $z_t$ at every timestep based on the history of state observations?", "limitations": "The authors could dedicate more time to discussing limitations and societal impacts of their work. I understand that space within the 9-page limit is at a premium; at the very least, including a short discussion in an Appendix would be worthwhile. I simply think that it is an overstatement to say \"Besides, we do not see it has obvious negative societal impacts.\" I can easily list multiple hypothetical scenarios in which, by pooling communication, you allow for adversarial attacks that cover up particular agents' communication, or you regress to mean behaviors instead of allowing for individualized responses. I do not wish to imply that the authors have to address all these concerns, but dismissing them without any discussion in the main paper seems like an overstep.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths ##\n\nThe authors address an important problem: stably training emergent communication. Their proposed solution integrates several distinct ideas (in my interpretation, the autoencoding and rollout objectives represent different concepts) seemingly successfully to train agents that, in experiments, outperform existing methods.\n\nThe evaluation is particularly impressive, using a wide variety of settings from existing literature, and reporting median (great job using robust statistics!) values over 5 random seeds. \n\n\n## Weaknesses ##\n\nLiterature review: The authors appear to have missed some relevant related works. The paper could benefit from better framing by citing works on the challenges of emergent communication (e.g., Eccles et al on \"Biases for Emergent Communication in Multi-agent Reinforcement Learning\"). Most importantly, though, the authors do not appear to know of Lin et al.'s \"Learning to Ground Multi-Agent Communication with Autoencoders\" from NeurIPS last year. Although Lin's technique is certainly different from this paper's, there are deep underlying similarities, especially regarding the use of an autoencoding loss to promote informative communication. Their method would be an extremely relevant baseline.\n\nClaims about compactness: I am concerned about the authors' claims about compactness. The authors repeatedly note that they seek to induce \"compact and sufficient\" representations. I understand that an autoencoding loss incentivizes informative, or sufficient, representations by reward information content in representations. However, my understanding of compactness corresponds to notions of low information content, or compression. For example, information-bottleneck methods limit communication information, or using fewer discrete messages creates more \"compact\" representations. I do not see how the proposed work aligns with this notion. The authors write near line 173 that \"By forcing the aggregated representation to be predictive on its future states, the aggregated representation could be more correlated with the information required for its decision-making, which meets the compactness standard.\" I don't think I agree. Could $z_t$ not simply encode all information about observations and states? If so, that is certainly informative of future states. How does that imply compactness?\n\nAblation studies: MASIA involves multiple new training objectives over baselines, so I wish I had seen a more complete ablation study to reveal the importance of different components. The studies included in the appendices are a good start: in particular A.3 answers a question I had when reading the paper (although I do actually wonder about the significance of the results, as there seems to be a lot of overlap in error bars and it's only conducted for the very early timesteps). However, there is no such ablation study (as far as I can tell) for the autoencoding loss.  To me, studying the autoencoding loss contribution would be more important than the different pooling mechanisms.\n\nSome lack of clarity: Overall, I felt like I understood most, but not all, of the authors' proposed work. I elaborate on my questions in the next section. I do not mean to imply that the work was entirely unclear, only that there were parts that I did not understand from the main text. Depending on the answers to those questions, I will be better able to assess the technical contributions of the work. Because I am currently unsure about what information agents have access to, I'm not sure of how truly decentralized the agent setup is.\n\nAlthough certainly not disqualifying on their own, this paper contains some small typos that would have to be fixed for publication. Examples include the caption for Figure 3 \"ours\" vs. \"our\", line 221 \"injury\" should be a verb of some sort.\n\n## Summary ##\n\nOverall, I like the paper and think it could be an important contribution, but I need to make sure I understand aspects of the design better before I recommend acceptance. In other words, I am very willing to \"change my mind\" because my mind isn't currently made up. I look forward to engaging in followup discussion based on my questions.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "HwnmvITIuzB", "reviewer": "Reviewer_t3CW", "summary": "The authors proposed MASIA which aims to encourage efficient communication among agents within cooperative MARL settings. MASIA uses information aggregator and extractor and learns the message representations that are compact and sufficient for agents’ communication. Internally, self-attention mechanism and predictive representation are used for such representations, where additional model network is used for predictive representation. MASIA is shown to mostly outperform existing MARL baselines on the tasks where agents’ communication is crucial to the overall performance. There are further ablation studies that shows MASIA’s learned features and its compatibility with existing MARL algorithms. ", "questions": "1. DEC-POMDP is considered to be a theoretical framework for this work, which implies the state s is not directly observable. However, s should be used for the reconstruction loss for information aggregation, which makes me a bit confused. I believe s here seems a set of all agents’ observations under the CTDE, but I hope that the authors clarify this. \n\n2. How each loss affects the performance of MASIA is unclear. From the appendix, I could see that the loss is balanced through the weights $\\lambda_1$ and $\\lambda_2$. What would be the performance when (1) $\\lambda_1=0$ (2) $\\lambda_2=0$ (3) $\\lambda_1=\\lambda_2=0$?\n", "limitations": "There seems no negative societal impact of this research.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The authors’ intuitions and approaches are described well. Experiments are done in various perspectives to show the effectiveness of algorithms. Generalizability with existing algorithms is also a strong point.\n\nWeaknesses: The scalability for a large number of agents (e.g., tens or hundreds of agents) is questionable as most of CTDE frameworks has such restriction. I could not find major weaknesses of this work. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept