Learning Multi-Agent Communication with Contrastive Learning
Authors: Yat Long Lo
Biswa Sengupta
Jakob Nicolaus Foerster
Michael Noukhovitch
Venue: ICLR 2024 poster (2024)
Keywords: Multi-Agent Reinforcement Learning
Emergent Communication
Contrastive Learning
Primary area: reinforcement learning
TL;DR: A novel approach to learning communication in decentralized MARL based on a multi-view contrastive learning perspective by treating messages as agents' encodings of the state

Abstract: Communication is a powerful tool for coordination in multi-agent RL. But inducing an effective, common language is a difficult challenge, particularly in the decentralized setting. In this work, we introduce an alternative perspective where communicative messages sent between agents are considered as different incomplete views of the environment state. By examining the relationship between messages sent and received, we propose to learn to communicate using contrastive learning to maximize the mutual information between messages of a given trajectory. In communication-essential environments, our method outperforms previous work in both performance and learning speed. Using qualitative metrics and representation probing, we show that our method induces more symmetric communication and captures global state information from the environment. Overall, we show the power of contrastive learning and the importance of leveraging messages as encodings for effective communication.

Published as a conference paper at ICLR 2024
LEARNING MULTI-AGENT COMMUNICATION WITH CONTRASTIVE LEARNING
Yat Long Lo
Biswa Sengupta
Dyson Robot Learning Lab richie. lo@dysoo.n...o com
Imperial College London
biswa wwwwassnnuptaaaggge 1 com
Jakob Foerster
Michael Noukhovitch
FLAIR, University of Oxford
Mila
jakob foerstereeng ox ac
Université de Montréal
mnoukhoved gma i com
ABSTRACT
Communication is powerful tool for coordination in multi-agent RL. But inducing an effective, common language is difficult challenge, particularly in the decen- tralized setting. In this work, we introduce an alternative perspective where com- municative messages sent between agents are considered as different incomplete views of the environment state. By examining the relationship between messages sent and received, we propose to learn to communicate using contrastive learning maximize the mutual information between messages of a given trajectory. environments, our method outperforms previous work in both performance and learning speed. Using qualitative metrics and representa- tion probing, show that our method induces and captures global state information from the environment. Overall, we show the power of contrastive learning and the importance of leveraging messages encodings for effective communication.
communication-eseeni
symmetric communication
Dog
Encoder
Features
Proy at Botiom-Right
Encoder
Message
Augmentation
samples
Agent's
Positive camples
Encoder
Message
namplee
Negudive samples
Cal
Encoder
Features
Pray Bottom-Left
Encoder
Message
Augmentation
samples
Positive samples
Encoder
Features
Encoder
Message
Multi-view Contrastive Learning
CACL
Figure 1: In multi-view learning, augmentations of the original image "views" used positive samples for contrastive learning. In our proposed method, CACL, different agents' views of the same environment state are considered positive samples and messages contrastively learned as encodings of that state.
INTRODUCTION
Communication is key capability necessary for effective coordination among agents in partially observable environments. In multi-agent reinforcement learning (MARL) (Sutton & Barto 2018). agents their actions to transmit information (Grupen et al. 2020) but continuous discrete

---

Published as a conference paper at ICLR 2024
messages on a communication channel (Foerster al.) 2016), i.e., linguistic communication Lazarii dou & Baroni 2020), are more flexible and poweriul because they can convey more complex concepts. To successfully communicate, speaker and listener must share common language with shared understanding of the symbols being used (Skyrms) 2010 Dafoe et al. (020). Emergent communica- tion or learning common protocol (Wagner et al. 2003 Lazaridou & Baroni 2020), is thriving research direction but most works focus on simple, single-turn, sender-receiver games (Lazaridou al., [2188] Chaabouni et al. 2019). In more visually and structurally complex MARL environments (Samvelyan et al. 2019),, existing approaches often rely on centralized learning mechanisms by sharing models (Lowe et al. 2017) or gradients (Sukhbaatar et al.) (016))
However. a centralized controller is impractical in many real- woold environments (Mai et al. 202 Jung et al. 202T) where agents cannot easily synchronize and must act independently i.e. decentral- ized. Centralized training with decentralized execution (CTDE) (Lowe et al. (017) is middle- middle-ground between purely centralized and decentralized methods but may not perform better than purely decen- tralized training (Lyu et al. 2021 A centralized controller suffers from the curse of dimensionality; as the number of agents it must control increases, the amount communication between agents to process increases exponentially (Jin et al.) 2021). Furthermore, the fully decentralized setting is more flexible and requires fewer assumptions about other agents, making more realistic in many real-world scenarios (Li et al.) (2020). Hence, this work explores learning communicate to coordinate agents in the decentralized setting. In MARL, this means each agent will have its own model decide how act and communicate, and no agents share parameters or gradients.
Typical RL approaches to decentralized communication are known to perform poorly even in simple tasks (Foerster et al.) 2016) due to the large space of communication to explore, the high variance of RL, and a lack of common grounding on which to base communication (Lin el al. [202T). Earlier work leveraged how communication influences other agents (Jaques al., 2018 Eccles al.] (019)) to learn the protocol. Most recently, [in et al. (2021) proposed agents that autoencode their observations and use the encodings as communication, using the shared environment as the common grounding. We build on this work in using both the shared environment and the relationship between sent and received messages to ground a protocol. We extend the Lin et al. (202I) perspective that agents' messages are encodings and propose that agents in similar states should produce similar messages. This perspective leads to a simple method based on contrastive learning to ground communication.
Inspired by the literature in representation learning that uses different "views" of data sample (Bachman et al.] 2019). for a given trajectory, we frame an agent's 's observation as a "view" of the environment state. Thus, different agents' messages are encodings of different incomplete "views" of the same underlying state. From this perspective, messages from the same state should, generally, be more similar to each other than to those from distant states or other trajectories, as shown in Figure [ As with image augmentations, two agent observations may not necessarily overlap, but contrastive objective can generally lead to effective encodings. We propose Communication Alignment Contrastive Learning (CACL), where each agent separately uses contrastive learning between their own sent and received messages to learn a communication encoding.
We experimentally validate CACL in three communication caamumicatioonessenttall environments and show how CACL leads to improved performance and speed, outperforming state-of-the-ar decentralized MARL. communication algorithms. To understand CACL's success, we propose a suite of qualitative and quantitative metries. We demonstrate that CACL leads to more symmetric communication (i.c., different agents communicate similarly when faced with the same observations), allowing agents to be more mutually intelligible. By treating our messages as representations, we show that CACL's messages capture global semantic information about the environment better than baselines. Overall, we argue that contrastive learning is a powerful direction for multi-agent communication and has fundamental benefits over previous approaches.
2 RELATED WORK
Learning to coordinate multiple RL agents is challenging and unsolved task where naively applying single-agent RL algorithms often fails (Foerster et al., (016).. Recent approaches focus on neural network-based agents (Goodfellow communication protucol (Lazaridou & Baroni 2020). To handle issues of non-stationarity. some work focuses on centralized learning approaches that globally share models (Foerster al. 2016).
2016) with message channel to develop a common
2

---

Published as a conference paper at ICLR 2024
training procedures (Lowe et al.
2017). gradients (Sukhbaatar al., 2016) among agents. This improves coordination and can reduce optimization issues but results are often still sub-optimal in practise (Foerster et al. 2016 Lin et al. 2021 and may violate independence assumptions, effectively modelling the multi-agent scenario as single agent (Fecles et al. 2019).
This work focuses independent, decentralized agents and non-differentiable communication. In previous work, [aquus et al.] (2018) propose a loss to influence other agents but require explicit and complex models of other agents and their experiments focus on mixed cooperative-compcttive scenarios, [Eclee et al. (2019) add biases to each agent's loss function that separately encourage positive listening (i.e., the listener to act differently for different messages) and positive signaling (i.e.. the speaker to produce diverse messages in different situations). Their method is simpler but requires task-specific hyperparameter tuning to achieve reasonable performance and underperforms in sensory-rich environments (L.in et al.) 2021). Our work is closest to Lin et al. (2021), who leverage autoencoding as their method to learn message protocol in cooperative 2D MARL games. Agents learn to reconstruct their observations and communicate their autoencoding. It outperforms previous works while being algorithmically and conceptually simpler. Our method builds on this encoding perspective by considering other agents' messages to ground communication. Whereas agents in Lin et al.] 202I) can only learn to encode the observation, our approach leverages the relationship between different agents' messages to encode global state information. Empirically, our method is also more efficient as it requires no extra learning parameters whereas Lin et al] (202T) learn and discard their decoder network. Note that our setup uses continuous messages instead of discrete (Eccles et al. 2019; Lin et al. 2021), a standard choice in contrastive learning (Chopra et al. [2005]
He al. 2020, Chen et al. 2020a) and embodied multi-agent communication (Sukhbaatar et al.
2016 Jiang& & Lu 2018
Das et al. 2019).
Autoencoding is a form of generative self self-supe learning (SSL) (Doersch et al. 2015). We
propose to use another form of SSL, contrastive learning (Chen et al. (020a), as the basis for learning communication. We are motivated by recent work that achieves state-of-the-ard representation learning on images using contrastive learning methods (Chen et al. 2020b) and leverages multiple "views" of the data. Whereas negative samples are simply different images, positive samples are image data augmentations or "views" of the original image (Bachman et al. 2019). We treat agents' messages of the same state in trajectory as positives of each other, so we base our method on SupCon (Supervised Contrastive Learning) (Khosla et al. 2020) which modifies the classic contrastive objective to account for multiple positive samples. Relatedly, Dessi et al. (202l) use a two-agent discrete communication setup do contrastive learning images, we do the opposite and leverage contrastive learning to learn multi-agent communication in an RL. environment.
3 PRELIMINARIES
We base our investigations on decentralized partially observable Markov decision processes (Dec- POMDPs) with N agents to describe a fully cooperative multi-agent task (Oliehoek & Amato 2016). A Dec-POMDP consists of tuple G (S, A, P, R, 7, n, 7)- E S is the true state of the environment. At each time step, each agent E N chooses an action at E A' to form a joint action a E A = A A x A It leads to environment transition according to the transition function P(s"|s,a' ....a R(s,a) a): S A R. y [0, 1) is discount factor. As the environment is partially observable, each agent i receives individual observations = E Z based on the observation function 22((s) : 5 Z.
S x A x S + [o, 1]. All agents share the same reward function
We denote the environment trajectory and the action-observationn history (AOH) of an agent i as T: = 80, St. ar and o = (2') ( (..(s..) of E 7 = (Z x A)" respectively. A stochastic policy (aa (7) : T x + [0, 1] conditions on AOH. The joint policy π has corresponding action- value function Q (st, az) = Eee+ 10t [Re] st, az), where RG 20 Ei 2're+i is the discounted return. rt+ is the reward obtained at time + from the reward function R.
To account for communication, similar to Lin et al. (202I), at each time step t, an agent takes an action of and produces a message m² ''' (s)) after receiving its observation ('(st) including messages from the previous time step m where v is agent i's function to produce a message given its observation and m, refers to messages sent by agents other than agent i. The messages continuous vectors of dimensionality D.
3

---

Published as a conference paper at ICLR 2024
4 METHODOLOGY
We propose different perspective on the message space used for communication. At each time step t for a given trajectory T, a message mi of an agent can be viewed as an incomplete view of the environment state 8, because m; is a function of 8, formulated in section Naturally, messages of all the N agents are different incomplete perspectives of 8t. To ground decentralized communication, hypothesize that could leverage this relationship between messages from similar states to encourage consistency and similarity of the messages space across agents. Specifically, we propose maximizing the mutual information using contrastive learning which aligns the message space by pushing messages from similar states closer together and messages of different states further apart. Note that agents see partial view of the state from their observation, so they will inherently communicate different messages to reflect their partial knowledge. However, aligning their message spaces enables communicating these partial views of the state a more inttally-ttellliibble way.
As heuristic for state similarity, we consider window of timesteps within trajectory to be all similar states i.e. positive samples of each other. To guarantee dissimilar negative samples (Schroff) states from other trajectories negatives. Since each underlying state has
et al. 2015). multiple positive views (w steps, N agent messages each), we leverage the recent contrastive learning method SupCon (Khosla et al. 2020 We refer the contrastive SupCon objective across multiple MARL trajectories as Communication Aligument Contrastive Learning (CACL).
Let M be all the messages in batch of trajectories and M, be the messages in trajectory Let mi E M, the message of agent at time I. Thus, positives for a message mf given a timestep within that timestep window H (m/) = (mg Mr \ {m]] :'' [t w. t + w]}. Let all other messages K from all trajectories in the batch be K ((m))= M \ {m]]. Formally, the contrastive loss LOACE:
window
all other messages from the same trajectory
exp(m): . mp/n) log E ((mE)((n)) exp(m) mx/n))
Σ mfEM,
Σ
[mm)))
Where 77 Rt is scalar temperature and [m())] is the cardinality. Practically, each agent has replay buffer that maintains a batch of trajectory data collected from multiple environment instances. It contains messages received during training to compute the CACL loss. We use timestep window of size 5 for all the environments based on hyperparameter tuning of different window sizes. Following Khosla et al. (2020). messages are normalized before the loss computation and low temperature (i.e. ? = 0. is used as empirically benefits performance and training stability. The total loss for each agent is reinforcement learning loss LRL using the reward to learn a policy (but not message head) and a separate contrastive loss LCACL to learn just the message head, formulated as follows:
LRL + RLCACT
(2)
where K R+ is hyperparameter to scale the CACL loss. See Appendix A.l A.l for LRL details.
5
EXPERIMENTS AND RESULTS
5.1
EXPERIMENTAL SETUP
We evaluate our method on three multi-agent environments with communication channels. Given the limited information each agent observes, agents must meaningfully communicate in order to improve task performance. All results are averaged over 12 evaluation episodes and over 6 random seeds. More details of the environments and parameters can be found in appendix A.2
Traffic-Junctiin Proposed by Sukhbaatar et al. (2016), it consists of 4-way traffic junction with cars entering and leaving the grid. The goal is to avoid collision when crossing the junction. We use 5 agents with vision of 1. Though unnecessary, with limited vision in agents, communication could help to solve the task. We evaluate algorithms using their success rate avoiding all collisions during evaluation episodes.
4

---

Published as a conference paper at ICLR 2024
Traffic Junction
Find Gool
4COmm
OKL 25
00A
6.4
Figure 2: CACL (red) outperforms all other methods on Traffic-Junction (left), Predator- (left) and Find-Goal (right). Predator-Prey shows evaluation reward, higher is better. Traffic-Junction plots the percent of successful episodes, higher better. Find-Goal plots the episode length until the goal is reached, lower is better. The performance curves are smoothed by a factor of 0.5 with standard errors plotted as shaded areas.
Predator-Prey: A variant of the classic game (Benda et al., 1986; Barrett al. 2011) based on Koul (2019) where 4 agents (i.e. predators) have the cooperative goal to capture sandomly-moving prey by surrounding each prey with agents have to entirely surround a prey on all 4 sides to successfully capture it and they cannot see each other in their observations. Thus, agents must communicate their positions and actions in order to coordinate their attacks. We evaluate each algorithm with episodic rewards in evaluation episodes.
than predator. We devise a more difficult variation where
Find-Goal: Proposed by Lin et al. (2021), agents' goal to reach the green goal location fast as possible in grid environment with obstacles. We use 3 agents, each observes partial view of the environment centered its current position. Unlike in Lin al. (202I), we use field of view of 3 x 3 instead of 7 x 7 to make the problem harder. Each agent receives an individual reward of 1 for reaching the goal and an additional reward of when all of them reach the goal. Hence, it is beneficial for an agent to communicate the goal location once it observes the goal. As in Lin jet al. (2021), we measure performance using episode length. An episode ends quicker if agents can communicate goal locations to each other more efficiently. Hence, method performs better if it has shorter episode lengths.
5.2 TRAINING DETAILS
We compare CACL to the state-of-the-art independent, decentralized method, autoencoded commu- nication (AEComm: Lin et al., (02T). which grounds communication by reconstructing encoded observations. We also compare to baselines from previous work: independent actor critic without communication (IAC) and positive listening loss (PI.: PPL:EEclees et al. 2019) (See Appendix (A.)). We exclude the positive signalling loss (Fecles et al. 2019) as extending it to continuous messages is non-trivial but note that AEComm outperforms it in the discrete case (Lin et al., 202I). We also include DIAL (Foerster et al. 2016) which learns to communicate through differentiable messages to share gradients so is decentra lized but not independent.
All methods the same architecture based the IAC algorithm with n-step returns and asyn- chronous environments (Mnih et al.) (2016). Each agent has an encoder for observations and received messages. For methods with communication, each agent has communication head produce messages based on encoded observations. For policy learning, a GRU (Cho et al.] 2014) is used to generate hidden representation from history of observations and messages. Agents use the hidden state for their the policy and value heads, which are 3-layer fully-connected neural networks. We perform spectral normalization (Gogianu al. 202T) in the penultimate layer for each head to improve training stability. The architecture is shown in Figure g and hyperparameters are further described, both in Appendix
5.3 TASK PERFORMANCE
We run all methods on the three selected environments and plot results in Figure 2 Our proposed method CACL outperforms all baseline methods in both final performance and learning speed and,
5

---

Published as a conference paper at ICLR 2024
No-Prey
One- One-Prey
Two-Preys
MAC
36.67
15
48.33
DAL
63.33
31.34
50
5D
41.66
6.67
51.67
33.33
66.57
Capture Success Rate in Predator-Frey
Figure 3: Success rate in Predator- Prey: the percentage of final evaluation runs that captured no prey, one prey, or both prey. Average over 6 random seeds, each with 10 evaluation episodes. See Appendix for the same results with standard deviation.
10
-10
11ding Window
Figure 4: Predator-Prey ablation experiment on LCACI varying the sliding window size and k.
AEComm is the strongest baseline. The largest performance increase from CACL is in FindGioll where partial observability is most prominent due to necessary (hence why IAC performs
consistent with previous results (Lin et al. 2021
agents' small field-of-view which makes communication worst). These results show the effectiveness of self-supervised methods for learning communication in the mully-decentralized setting, they both outperform DIAL which, notably, backpropogates gradients through other agents. Furthermore, it demonstrates CACL's contrastive learning as more powerful alternative AEComm's autoencoding for coordinating agents with communication.
Improvement on Traffic-Junction is not as significant as others because communication is less essential for task completion, as shown by the strong performance of IAC. For Predator-Prey, results are clearly better than baselines but have high variance due to the difficulty of the task. The goal of Predatory- is to capture two moving prey and requires coordinating precisely to surround and attack prey the same time. Any slight miscoordination leads to sharp drop in rewards. For another metric of success, we compute the percentages of evaluation episodes that capture no, one, or two preys. Averaging over random seeds, show results in Figure 3 CACL does significantly better on the task, outperforms all baselines, and solving the complete task more robustly while failing less frequently. Find-Goal requires the most communication among the environments because the gridworld the largest and agents must clearly communicate the location of goal. Here, CACL significantly outperforms the baselines, demonstrating that the communicative task gets harder, CACL outperforms more.
We confirm the effectiveness of CACL with an ablation study of the key design decisions: sliding window and SupCon. CACL leverages the temporal nature of RL to treat a sliding window of timesteps as positive views of each other. We plot results for a range of window sizes run on Predator- Prey in Figure 4 No sliding window (size 1) performs poorly, demonstrating its necessity and that the choice of sliding window size is an important hyperparameter. Through the use of SupCon (Khosla et al. (020) we treat all sent and received messages in the sliding window as all positive views of each other, with many positives per batch. Creating a batch with just one positive view per message corresponds to SimCLR (Chen et al. 2020a) and results in much worse performance (1.36 +:2 9.46). We also run Predator-Pree and search across values of the CACL loss coefficient i: in Figure We used the best values (5-step window, ==0.5) across all the environments, demonstrating that the choice of CACL hyperparameters is robust. Overall, we show the issues in naively implementing contrastive learning for communication, and the clear, important design decisions behind CACL.

---

Published as a conference paper at ICLR 2024
Predaton-Srey
07
04
Loaking 300
200
100%
2006
Figure 5: Comparing CACL and AEComm with their respective variants when combined with DIAL. Variants with DIAL have generally worse performance.
5.4 AUGMENTING CACL WITH RL
The contrastive loss in the communication head of CACL is very performant without optimizing for reward, so natural question is whether we can achieve even better results if we learn the message using reward as well. To answer this, we add DIAL to both CACL and the next best method, ABComm, and evaluate in the three environments. This is equivalent to backpropogating LRL from Equation 2 through agents to learn the message head. In this way, both RL and SSL (contrastive or autoencoding) signals are used to learn the message head.
Figure compares the performance of CACL and AEComm with their DIAL-augmented variants. Our findings are consistent with Lin et al. (021). who find that mixing AEComm and RL objectives detrimental to performance. We observe that augmenting either AEComm CACL with DIAL performs generally worse, except in Find-Goal, where performances is similar but not better. We hypothesize that decentralized DIAL is a complex, and high-variance optimization that is difficult stabilize. DIAL's gradient updates may clash with CACL and result in neither useful contrastive representation, nor a strong reward-oriented one. It is also possible that CACL's messages would not be improved with reward-oriented gradients. As we show in Section 5.6 CACL already captures useful semantic information that other agents can effectively extract.
5.5 PROTOCOL SYMMETRY
We hypothesize that CACL's improved performance over the baselines is because it induces more consistent, mutually- communication protocol that is shared among agents. More specifically, consider consistency to be how similarly agents communicate (i.e., sending similar messages) when faced with the same observations. A consistent protocol can reduce the optimization complexity since agents only need to learn protocol for the whole group and it also makes agents more mutually intelligible.
Table 1: Protocol symmetry across environments, average and standard deviation over 10 episodes and 6 random seeds. CACL consistently learns the most symmetric protocol.
DIAL
PL
AEComm
CACL (Ours)
Predator-Prey
0.66 + 0.07
0.66 + 0.06
0.89 + 0.01
0.95 + 0.01
FindGoal
0.50 + 0.05
0.49 1 0.04
0.85 0.02
0.92 - 0.01
Traffic Junction
0.69 + 0.01
0.61 + 0.04
0.80 + 0.01
0.98 - 0.002
To evaluate consistency, we measure protocol symmetry (Graesser et al. 2019) so an agent swaps observations and trajectory with another agent, it should produce similar message as what the other agent produced. We extend this metric from previous work to the continuous, embodied case. We feed the same trajectory to all agents and measure the pairwise cosine similarities of the messages that they produce. Given a trajectory 7 and (t E T} as a set time steps of т, protocol symmetry (protocol_sym) written as:
7

---

Published as a conference paper at ICLR 2024
Other agent visible
Agent-own view
Goal visible
Agent-own view
Figure 6: DBSCAN (Ester et al. 1996) clustering results messages produced by CACL, after dimensionality reduction with t-SNE. We show four semantically meaningful clusters with their corresponding labels: messages sent when the goal is visible, when another agent is visible, and two clusters correspond to only the individual agents visible in the observation.
('() (D'(s)) (I)
M N
Σ JENY
N N
V (S)
KE IEN
Therefore, a more consistent protocol has higher symmetry. We swap agent trajectory and observations and compute this metric 10 evaluation episodes for 6 random seeds, and show results in Table [ The self-supervised methods (CACL and AEComm) clearly outperform the others (DIAL and PL) implying that SSL better for learning consistent representations in decentralized MARL. Furthermore, CACL's protocol is very highly symmetric, clearly outperforming all others. Each AEComm agent autoencodes their own observation without considering the other agents' messages, leading to the formation of multiple protocols between agents. In contrast, CACL induces a common protocol by casting the problem the multi-view perspective and implicitly aligning agents' messages. The possible correlation between protocol symmetry and overall performance and speed further indicates the benefits of learning a common protocol in the decentralized setting.
5.6 PROTOCOL. REPRESENTATION PROBING
To further investigate how informative protocols are, propose suite of qualitative and quantitative representation probing tests based on message clustering and classification, respectively. We perform these tests on the protocols learned in the Find-Goal environment.
Similar to Lin et al. 2021), we cluster messages generated from evaluation episodes to qualitatively assess how informative CACL's protocol is. The messages first compressed to dimension of using t-SNE (Van der Maaten & Hinton 2008) and then clustered using DBSCAN (Ester et al.) 1996). We look at each cluster's messages and their corresponding observations extract any patterns and semantics captured. As shown in Figure we observe a cluster of messages for observations when the goal visible and another one when another agent visible. Two clusters correspond agents sceing neither the goal nor another agent. Notably, the messages in these clusters can come from different agents in different episodes, demonstrating that agents can indeed communicate symmetrically. The clusters indicate that CACL learns to compress meaningful, global state information in messages, allowing other agents to reasonably learn this semantic information. To quantitatively evaluate the
Table Classification results of the two probing tests in Find-Goal. All methods perform similarly in the easier Goal Visibility Test while CACI. outperforms the baselines significantly in the more difficult Goal Location Test.
DIAL
PL
ABComm
CACL (Ours)
Goal Visibility
99. 45% + 2.68
98.87%
+ 0.67
99.75% + 0.04
97.75% + 0.69
Goal Location
68. 15% 1.76
78.31% 2.39
76.14% + 3.36
91.28% + 1.71
informativeness of learned protocols, propose treat messages representations and learn
8

---

Published as a conference paper at ICLR 2024
classifier top of the messages, following work in RL representation learning Lazaridou et al. Anand et al. 2019). Since FindGoal is focused on reaching a goal, intuitively. agents should communicate whether they have found the goal and, if so, where other agents should go to reach the goal. Thus, we propose to probe the goal visibility and goal location. The former uses the messages to classify whether the goal is visible in observations not (i.e. binary classification). The latter uses messages where the goal is visible in the observations to classify the general location of the goal (i.e., 5-class classification: Top-Left, Top-Right, Bottom-Left, Bottom-Right and Middle). Whereas goal visibility is easy for egocentric communication, goal location requires detailed spatial information and communicating the absolute location from their relative position. This tests whether the communication protocol can consider other agents" perspectives and give global information from an egocentric observation. We use 30 evaluation episodes per method to generate messages for our experiments but different methods may have different numbers of acceptable messages for our probing task (e.g. a limited number of messages where the goal is visible for predicting goal location). To ensure fair comparison, we choose an equal number of samples per class (i.e., positive/negative, 5-class location) for all methods and use a 70%/30% random split for training and testing. We use a 2 2-layer fully-connected neural network to test each method, as this corresponds to the same network that agents use to encode each others' messages as part of their observations.
Table shows the classification results for the two probing tests. Goal visibility is an easier task and all methods' messages can be effectively used to determine it. In the more difficult goal location task, all methods perform above chance (20%) but CACL's protocol significantly outperforms baselines. Contrastive learning across different agents' messages can enable CACL to learn more global understanding of location from their egocentric viewpoint. We further compare the similarity between messages sent when the goal is at location [1, 1] and when the goal is at other coordinates along the diagonal in Appendix A.7 which corroborates with the Goal Location Test in CACL's capability in encoding global information better. By encoding the goal's spatial information, CACL agents are more likely able to move directly towards it, and reduce episode length. If other methods simply communicate that goal is found, agents know to alter their search but are not as precise in direction. This explains why AEComm, PL, and DIAL perform better than IAC but worse than CACL, which also learns much quicker as shown in Figure 2 For completeness, also provide similar classification results with a one-layer (linear) probe in Appendix A.8
6 LIMITATIONS
Our work investigates fully-cooperative environments but learning to communicate in less cooperative settings, such as those with adversaries (Noukhovitch et al. (021)) is a harder optimization problem. CACL would likely need stronger regularization to be effective. Furthermore, our empirical testing has revealed that SSL objectives are ineffective with reward-oriented gradients, as demonstrated in section 5.4] Although this phenomenon is well known (Lin et al. 2021), it is still not fully understood and future work should aim combine the two objectives. Finally, this work evaluates agents that were trained together. A more challenging frontier is zero-shot communication, an extension of zero-shot cooperation (Hu et al., 2020), in which agents must communicate effectively with novel partners, unseen during training. In Appendix A we show how existing methods perform poorly in this settings and leave this challenging setup to future work.
7 CONCLUSION AND FUTURE WORK
This work introduces an alternative perspective for learning to communicate decentralized MARL based on the relationship between sent and received messages within a trajectory. Drawing inspiration from multi view learning. we ground communication using contrastive learning by considering agents' messages to be encoded views of the same state. We empirically show that method leads to better performance through a more consistent, common protocol and learns to communicate more global state information. We believe this work solidifies contrastive learning as effective perspective for learning to communicate and hope it invigorates research into contrastive methods for communication with a focus on consistency. Furthermore, by establishing the connection between multi-view SSL, which has traditionally focused on images, and communication in MARL, we hope to encourage more cross doms-n research. Finally, we see contrastive learning as a potential method for simulating human language evolution, and hope to inspire research in this direction.

---

Published as a conference paper at ICLR 2024
REFERENCES
Ankesh Anand, Evan Racah, Sherjil Ozair, Yoshua Bengio, Marc-Alexandre Côté, and R Devon Hjelm. Unsupervised State Representation Learning in Atari. In Advances in Neural Information Process- ing Systems, volume 32. Curran Associates, Inc., 2019. BtLLtttss://pppetee nips. cc/
paper/2019/hashn
Abstracct. .html
Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. Advances in neural information processing systems, 32, 2019.
Samuel Barrett, Peter Stone, and Sarit Kraus. Empirical Evaluation of Ad Hoc Teamwork in the Pursuit Domain. In AAMAS, pp. 8, 2011.
Miroslav Benda, V Jagannathan, and R Dodhiawala. On optimal cooperation of knowledge sources an empirical investigation. Technical report, Boeing Advanced Technology Center, Boeing Com- puting Services, 1986. URL (tR.htts::/ www. CS wtwxas www.eeaiwwaaaa/eeeeeeeeeeeeeeeee wtt.aaaaaaaaaaag//// html
b2hd-BendaJDis
Micah Carroll, Rohin Shah, Mark K. Ho, Thomas L.. Griffiths, Sanjit A. Seshia, Pieter Abbeel, and Anca Dragan. On the Utility of Learning about Humans for Human-Al Coordination. In Neural Information Processing Systems. arXiv, 2019. doi: [cs, stat].
(tt.://aawii.111.00755.. URL
attt:://axiv/ org/ wwwgg/abs//e / 05789 arXiv:1910.00
Rahma Chaabouni, Eugene Kharitonov, Alessandro Lazaric, Emmanuel Dupoux, and Marco Baroni.
Word-order biases in deep-agent emergent communication. In ACL, May 2019. URL http
l/arxiv.org// abs 905. 12330
arXiv: 1905.12330.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A Simple Framework for Contrastive Learning of Visual Representations, In arXiv:2002.0577 [cs, star], June 2020a. URL http://arxiv. org/ abs wttp::/aaaa.....oa...po/
arXiv: 2002.05709.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big Self-Supervised Models are Strong Semi-Superviised Learners In arXiv:2006. 10029 [cs, stat], arXiv: 2006. 10029.
October 2020b. URL
10029
Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio. On the Properties of Neural Machine Translation: Encoder--eecoder Approaches. In 8th Workshop on Syntax, Semantics and Structure in Statistical Translation, arXiv, October 2014. doi: 10.48550/arXiv. Ntt./aXXxx.114.1255555 1409. arXiv: 14e::110911220 1259 [cs, stat].
URL http:://axii.... org/ abs/1405 1259
Sumit Chopra, Raia Hadsell, and Yann LeCun. Learning a similarity metric discriminatively, with application to face verification. In 2005 IEEE Computer Sociery Conference on Computer Vision and Pattern Recognition (CVPR'0S) volume 1, pp. 539-546. IEEE, 2005.
Filippos Christianos, Lukas Schäfer, and Stefano Albrecht. Shared experience actor-critic for multi-agent reinforcement learning. Advances in neural information processing systems, 33: 10707-10717, 2020.
Allan Dafoe, Edward Hughes, Yoram Bachrach, Tantum Collins, Kevin R McKee, Jocl Z Leibo, Kate Larson, and Thore Graepel. Open problems in cooperative ai. arXiv preprint arXiv:20/2.08630, 2020.
Abhishek Das, Théophile Gervet, Joshua Romoff, Dhruv Batra, Devi Parikh, Michael Rabbat, and Joelle Pineau. TarMAC: Targeted Multi-Agent Communication. In ICML, 2019. URL
http:://ark http//axxvv..xaaaaaaaaa/ wwt.oogg//////.ooo http:/abs11111.11111 11187
Roberto Dessi, Eugene Kharitonov, and Marco Baroni. Interpretable agent communication from
scrttch(with a generic visual processor emerging on the side). In NewrIPS, 2021.
Carl Doersch, Abhinav Gupta, and Alexci A. Efros. Unsupervised Visual Representation Learning by Context Prediction. In ICCV, 2015. URL http: ptttt://arxivv wttpg//gg./......./// 05192
Tom Eccles, Yoram Bachrach, Guy Lever, Angeliki Lazaridou, and Thore Graepel. Biases for emergent communication multi-agent reinforcement learning. Advances in neural information processing systems, 32, 2019.
10

---

Published as a conference paper at ICLR 2024
Martin Ester, Hans-Pete Kriegel, Jörg Sander, Xiaowei Xu, et al. A density- based algorithm for discovering clusters in large spatial databases with noise. kdd, volume 96, pp. 1996.
Jakob Foerster, Ioannis Alexandros Assael, Nando De Freitas, and Shimon Whiteson. Learning to communicate with deep multi-agent reinforcement learning. Advances neural information processing systems, 29, 2016.
Florin Gogianu, Tudor Berariu, Mihaela Rosca, Claudia Clopath, Lucian Busoniu, and Razvan Pascanu. Spectral normalisation for deep reinforcement learning: an optimisation perspective. In International Conference on Machine Learning, pp- 3734-3744, PMLR, 2021.
lan Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016. http: 7/www.
org
Laura Graesser, Kyunghyun Cho, and Douwe Kiela. Emergent linguistic phenomena in multi-agent communication games. In EMNLP, 2019.
Niko A. Grupen, Daniel D. Lee, and Bart Selman. www-Banwwidth Communication Emerges Naturally Multi-Agent Learning Systems, December 2020. URI. http:://rxxlvorr// arXiv:2011. [cs].
Abs/2011.14880
Abhinav Gupta, Mare Lanctot, and Angeliki Lazaridou. Dynamic population-based meta-learning for multi-agent communication with natural language. Advances in Neural Information Processing Systems, 2021.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, pp. 9729-9738, 2020.
Hengyuan Hu, Adam Lerer, Alex Peysakhovich, and Jakob Foerster "Other-Play" for Zero-Shot Coordination. In ICML. arXiv, 2020. doi: 10.48550//rXii.2003..222 arg/abs/2003. 02979 arXiv:2003.029: 0.9999 [cs].
URL http://arxiv
Natasha Jaques, Angeliki Lazaridou, Edward Hughes, Caglar Gulcehre, Pedro A. Ortega, D. J. Strouse, Joel Z. Leibo, and Nando de Freitas. Social Influence as Intrinsic Motivation for Multi-Agent Deep Reinforcement Learning. In ICML. Xiv, September 2018. doi: 10.48550/arXiv.. 1 10008747. URL
org. abs /1100 08647 4rXiv:1810.08647 [es, stat].
Jiechuan Jiang and Zongqing Lu. Learning Attentional Communication for Multi-Agent Cooperation. In Neural information Processing Systems. arXiv, November 2018.
Chi Jin, Qinghua Liu, Yuanhao Wang, and Tiancheng Yu. V-Learning A Simple, Efficient, Decentralized Algorithm for Multiagent RL, October 2021. URL URL org.abs/.rg//b///
2110.14553 arXiv:2110. 10.1555 [cs, stat].
Soyi Jung, Won Joon Yun, Joongheon Kim, and Jae-Hyun Kim. Coordinated Multi-Agent Deep Reinforcement Learning for Energy-Aware UAV-Based Big-Data Platforms. Electronics, 10 (5):543, January 2021. ISSN 2079-9292, doi: w...3900eleccroniiss0000053.3 URL |https
com/ 0107999992/10//20 5443 Number: 5 Publisher: Multidisciplinary Digital
Publishing Institute.
Prannay Khosla, Piotr Teterwak, Chen Wang. Aaron Samaa, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in Neural Information Processing Systems, 13:18661-186733 2020.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.680 1412.6900 2014.
Anurag Koul. ma my-gym: Collection of multi- agent environments based on openai gym. https
//github. com htt.:/owllaaauuuuu wwwauuargaaaaaa gym 2019.
Angeliki Lazaridou and Marco Baroni. Emergent Multi-Agent Communication in the Deep Learning Era, July 2020.
11

---

Published as a conference paper at ICLR 2024
Angeliki Lazaridou, Karl Moritz Hermann, Karl Tuyls, and Stephen Clark. Emergence of Linguistic Communication from Referential Games with Symbolic and Pixel Input. In ICLR, April 2018.
httth:t/taaxiiiii org/ abs/ 1804. 03984 arXiv: 1804.03984.
Wenhao Li, Bo Jin, Xiangfeng Wang, Junchi Yan, and Hongyuan Zha. F2a2: Flexible fully-
decentralized approximate actor-critic for cooperative multi-agent reinforcement learning. arXiv preprint arXiv:2004.1 1.XX::2004.11445, 2020.
Toru Lin, Jacob Huh, Christopher Stauffer, Ser Nam Lim, and Phillip Isola. Learning ground multi- agent communication with autoencoders. Advances in Neural Information Processing Systems, 34, 2021.
Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-compent processing systems, 30, 2017.
environments. Advances in neural information
Andrei Lupu, Brandon Cui, Hengyuan Hu, and Jakob Foerster. Trajectory diversity for zero-shot coordination. In International Conference Machine Learning, pp. 7204-7213, PMLR, 2021.
Xueguang Lyu, Yuchen Xiao, Brett Daley, and Christopher Amato. Contrasting Centralized and Decentralized Critics in Multi-Agent Reinforcement Learning. In AAMAS, pp. 9, 2021.
Vincent Mai, Tianyu Zhang, and Antoine Lesage-Landry. Multi-agent reinforcement learning for renewable integration in the electric power grid. In Tackling Climate Change with Machine Learning Workshop at NeurIPS, 2021.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap. Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference machine learning, pp. 1928 1937. PMLR, 2016.
Michael Noukhovitch, Travis LaCroix, Angeliki Lazaridou, and Aaron Courville. Emergent Commu- nication under Competition. In Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS '21, pp. 974-982, Richland, SC, May 2021. International Foundation for Autonomous Agents and Multiagent Systems. ISBN 978-1-4503-8307-3.
Frans A Oliehoek and Christopher Amato. A concise introduction to decentralized POMDPs. POMDPP Springer, 2016.
Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim G. J. Rudner, Chia-Man Hung. Philip H. Torr, Jakob Foerster, and Shimon Whiteson. The StarCraft Multi-Agent Challenge, December 2019. URL http::://xxi...//o/ wwwp/va/rr/aaaaaooo 902
04043 190.:1902.04043 [cs, stat].
Florian Schroff, Dmitry Kalenichenko, and James Philbin. FaceNet: A Unified Embedding for Face Recognition and Clustering. In Conference on Computer Vision and Pattern Recognition (CVPR), pp. 813, June 2015. doi: 10. 1503. 03832)
[cs].
Amanpreet Singh, Tushar Jain, and Sainbayar Sukhbaatar. Learning when to communicate at scale in multiagent cooperative and competitive tasks. arXiv preprint arXiv:1812.0973: 2018.
Brian Skyrms. Signals: Evolution, Learning, & Information. Oxford University Press, Oxford, 2010.
Sainbayar Sukhbaatar, Rob Fergus, et Learning multiagent communication with backpropagation. Advances in neural information processing systems, 29, 2016
Richard Sutton and Andrew G Barto. Reinforcement Learning: An Introduction. MIT Press, 2018. bett www..ntttotoottebebooooooo
URL | http:httt:///conee et deeeedeas.c.m
2nd. html
Gerald Tesauro. Td-gammon, a self-teaching backgammon program, achieves master-level play. Neural computation, 6(2):215-219, 1994.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
Kyle Wagner, James A Reggia, Juan Uriagereka, and Gerald S Wilkinson. Progress in the simulation of emergent communication and language. Adaptive Behavior, 11(1):37-66 2003.
12

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1 REINFORCEMENT LEARNING LOSS
In Equation 2 we use LRL to denote the RL loss. As our investigation is orthogonal to the choice of RL algorithm. this term can be any RL algorithm dependent on which algorithm is being used. In this work, we use Independent Asynchronous Actor Critic (IA2C) as our base algorithm, denoted as IAC in our experiments (Christianos et al. 2020). Given an agent E N, it has policy Th and value function v parameterized by parameters and respectively. The policy loss for agent is defined as:
L(ö)
(4)
with the value function minimizes:
L( = [IV - (re (
(5)
These two losses are denoted as I.r. due to space limit. the case with communication, the local observation of an agent is simply augmented with received messages.
A.2 ENVIRONMENT DETAILS
Figure A.2 provides visual illustration of the environments used.
One way
Car leaving
Goal is to reach the green block
Car entering
Figure 7: Visual illustration of the environments used. Left: Predator Prey, taken from Koul (2019) Middle: Find-Goal, taken from Lin et al. (2021). Right: Traffic-Juccton, taken from Singh et al. (0018)
A.2.1 PREDATOR PREY
We modify the Predator-Prey implementation by Koul (2019). Our Predator-Pree has a higher com- munication and coordination requirement than the original Predator Prey environment. Specifically, for prey to be captured, it has to be entirely surrounded (i.e. the prey cannot move to another grid position in any actions).
Here, an 7x7 gridworld. In each agent's 's observation, it can only see the prey if it is within the field view (3x3) and cannot see where other agents are. A shared reward of 10 given for successful capture and penalty of -0.5 is given for failed attempt. A -0.01 step penalty is also applied per step. Each agent has the actions of LEFT, RIGHT, UP. DOWN and NO-OP. The prey has the movement probability vector of (0.175. 0.175, 0.175, 0.175, 0.3 with each value corresponding to the probability of each action taken.
All algorithms trained for 30 million environment steps with maximum of 200 steps per episode.
13

---

Published as a conference paper at ICLR 2024
A.2.2 FIND-GOAL
We use the Find-Goal environment implementation provided by Lin et al. (2021). The agents have the goal to find where the goal ís in 15x15 grid world with obstacles.
Unlike in Lin al. (202T), each agent has 3x3 field view (instead of 7x7) to make the task more difficult. Each agent receives reward of 1 for reaching a goal and an additional reward of 5 if all agents reach the goal. We use a step penalty of -0.01 and an obstacle density of 0.15.
All algorithms are trained for 40 million environment steps with maximum of 512 steps per episode.
A.2.3 TRAFTIC-JUNCTION
We use the Traffic-Junction environment implementation provided by |inghh et al. (2018). The gridworld is 7x7 with traffic junction. The rate of cars being added has minimum and maximum of 0.1 and 0.3. We use the easy version with two arrival points and 5 agents. Agents are heavily penalized if a collision happens and have only two actions, namely gas and brake.
All algorithms are trained for 20 million environment steps with maximum of 20 steps per episode.
A.3 ARCHITECTURE AND HYPERPARAMETERS
LCACL LAEComm
Message Head
Oungolig message- +
Grounding Module
Hidden
To sent to other agents
Observation
Observation Encoder
GRU
Policy Head
Encoded message
Message
Incoming messages
Encader
Value Head
LRL
Figure 8: Architectural illustration for algorithms with communication. To remove communication, the message head is disabled. Grounding module is only relevant to CACL and AEComm. The former is a loss function and the latter is a decoder to reconstruct the encoded observation. Green boxes denote components where gradients of I.nt applied to. Red boxes denote components where gradients of LCACE and LAEComm are applied to. LAECom refers to the loss function of AEComm (Lin al. 2021).
Figure iillustrates the components of the architecture used in this work, similar (Lin et al., 2021). A message head is only used for algorithms with communication, namely CACL, AEComm, PL and DIAL. The Grounding Module refers to mechanisms to ground the messages produced by the message head, used in CACL and AEComm. Colored boxes denote components where gradients are applied to for a particular loss function. In the case of DIAL, gradients would flow from another agent to the agent that sends message through the message head and message encoder. Unless specified otherwise, we fix all hidden layers to a size of 32.
We experimented with using the output of the GRU, or hidden state, condition the message head. Empirically we found that directly conditioning on the observation encoding, as in Lin et al. 2021 led to more stable learning dynamics.
The observation encoder output values of size 32. For Predator-Pres Prey and Traffic Junction, a one-layer fully-connected neural network is used as observation encoder. For Find-Goal, same Lin al.
14

---

Published as a conference paper at ICLR 2024
Table 3: Success rate in Predator- Prey: the percentage of final evaluation that captured prey, one prey, or both prey. Average and standard deviation over 6 random seeds.
No-Prey 36.67% + 7.50
One-Prey
Two-Preys
IAC
15.00%
3.57
48.33% + 6.83
DIAL
63.33% + 7.56
3.33%
1.24
33.34% + 7.86
PL.
50.00% + 8.33
0.00% = 0.00
50.00% + 8.63
AEComm
41.66% 7.48
6.67% = 1.84
51.67% = 7.60
CACL (Ours)
33.33% = 7.86
0.00% 0.00
66.67% 7.86
2021 two-layer convolutional neural network followed by 3-layer fully-connected neural network.
For the message encoder, it outputs values of size 8 in Serdator-Prey and Find-Goal with hidden layer. It outputs values of size 16 in Traffic-Junction with two hidden layers. These configurations selected based on the best performance of the baseline communication learning algorithm used- DIAL. Messages received are concatenated before passing to message encoders. For all the methods with communication, they produce messages of length (D 4) with sigmoid function as activation. All models are trained with the Adam optimizer (Kingma & Ba 2014).
Table 4 lists out the hyperparameters used for all the methods.
A.4 CACL CONTRASTIVE Loss CURVE
Contrastive Loss Over Time Predator-Prey CAC
7000
6500
6000
5500
updates
1004
300k
500k
Figure 9: CACL's contrastive loss over time during training for Predator-Prey
Figure 9 shows the CACL's contrastive loss over time during training for Predator-Prey, This illustrates training convergence of the loss and improved separation of positive and negative samples between the start and end of training.
A.5 PREDATOR- CAPTURE RATE
each method's s success rate in capturing preys for Predator-Prey. CACL outperforms the baselines by capturing the most preys of the evaluation episodes.
A.6 POSITIVE LISTENING
This section describes the loss function we implemented for positive listening, based on Eccles et al. (0019). Given two policies n' and मं agent i where the latter the policy with messages zeroed in the observations, and trajectory of length T, the positive listening loss is written as:
15

---

Published as a conference paper at ICLR 2024
Learning Rate
0.0003
Epsilon for Adam Optimizer
0.001
0.99
Entropy Coefficient
0.01
Value Loss Coefficient
0.5
Gradient Clipping
2500
1) for CACL
0.1
for CACL
0.5
Loss Coefficient for PL
0.01
Number of Asynchronous Processes
12
N-step Returns
5
Table 4: Table for hyperparameters used across methods
T
- (alT))) + (al()) log (ogr)) (TT(F) Σ -
LP1
(6)
T
LaeA
where in inner summation, the first term is the L1 Norm and the second term is the cross entropy loss.
A.7 MESSAGE SIMILARITY FOR DIFFERENT GOAL LOCATIONS IN FIND-GOAL
0.5
(21)
0.00
Accorm
2AN ww0--40- paritt Cosire SRI1MN
0.75
0.70
0.5
060
235
0.0
10
16
Distance [1,
Figure 10: In Find-Goal, average pairwise cosine similarity between messages sent when the goal at [1, 1] and when the goal is in one of these coordinates [5,5], [9, 9], [13, 13). Similar to the representation probing test in section and filter for messages with the goal visible.
generate rollouts from the corresponding algorithms
To further support that CACL can encode global information better, we measure the message similarity between messages sent when the goal is in [1, 1] and when the goal is other coordinates along the diagonal of the gridworld. As shown in Figure TO CACL shows clear downward trend in similarity as the distance from [1, 1] increases unlike AFComm. This means CACL, learns to encode global locations differently despite learning only with egocentric views. This cooroborates with the Goal Location Test in Table that CACL is able to encode global information better.
A.8 PROTOCOL REPRESENTATION PROBING: 1-LAYER
Table 5 shows the same results for the two probing tests in section except here we use 1-layer neural network instead 2 layers. We observe significant dips in performance across all methods. Particularly, CACL becomes worse than the baselines in the easier Goal Visibility Test. However, CACL remains superior in the more difficult Goal Location test by an even bigger margin than the
results in table 2
16

---

Published as a conference paper at ICLR 2024
Table 5: Classification results of the two probing tests in the Find-Goal environment, comparing all methods with communication. 1-layer neural networks are used for probing AEComm
DIAL
PL.
CACL
Goal Visibility Test
94.21% % 2.68
96.93% + 3.14
96.27% + 3.98
87.65% +: 3.86
Goal Location Test
52.29% 5.25
53.65% + 9.60
48.16% + 7.34
79.18%
A.9
ZERO-SHOT CROSS-PLAY
Table 6: Zero-shot cross-play performance in Predator-Pres Prey. Best per- method results are bolded.
CACL
AFComm
PL.
DIAL
CACL (Ours)
-77200
28..9
2.78
5.77
28.78 4 3.99
AlCComm
-37.86 7.20
-31.56 3.766
-29733
3.66
PL.
2.99
-22.89
3.98
DIAL
22.85 1: 2.04
Table 7: Zero-shot cross-play performance in Find-Goal. Best per-method
results bolded.
CACL
AEComm
PL
DIAL
(Ours
468.75 15.32
471.66
13.54
487.56 = 8.61
488.28 + 16.60
AFCoom
14.96
472.85
PL
192,08 5.67
486.41 + 10.46
DIAL
476.07 + 15.89
advanced form of coordination working with partners you have not seen during training (uu et al. 2020). Previous work has focused on coordination through actions (Carroll et al.) 2019 Lupu ctt al. 2021 or pre-test grounding with a common dataset (Gupta et al. 2021 but to our knowledge, previous work has succeeded in learning a linguistic communication protocol that is robust to zero-shot partners. To assess this advanced robustness, we take trained agents from different methods and random seeds and evaluate them with each other (i.e., zero-shot cross-play) in Predator-Pree Prey and Find-Goal. Given two communication learning methods, my and m2, we sample two agents from each method for Predator- and for Find-Goal, method and one agent from the other and vice-versa. For intra-method cross-play, mi = m2, we evaluate agents that were trained with the same method but from different random seeds, so they have not been trained with each other. For inter-method cross-play, m, * m2, we sample agents from two different methods and pair them with each other. Each pairing is evaluated for 10 random seeds each with 10 evaluation episodes. Given that agents are trained in self-play (Tsauuro 1994) without regard for cross-play, we expect severe performance dips.
average over sampling two agents from one
We show mean and standard deviation across random seeds for Predator-Prey and Find-Goal in Tables andd 7 respectively. As expected, all pairings take significant dip in performance when compared with the main results. Inter-method gross-play performance is particularly bad across all algorithms. However, notably, CACL outperforms other methods in intra-method cross-play, indicating that the protocols learned by CACL are generally more robust even across random seeds. In general, zero-shot linguistic communication is incredibly difficult and our results are far from optimal. Still, CACL shows promise and demonstrates that contrastive SSL methods can lead to better zero-shot communication and coordination.
10 BROADER IMPACT
More multi-agent learning systems will be deployed in the real-world as further progress is made in fields of multi-agent learning like MARL. We expect communication to play an essential role in these systems given how well-wolld problems are inherently complex and partially observable in most cases. Our focus on the decentralized communication setting contributes to the capability of learning more effective and consistent communication protocols. Having more consistent protocols improve mutual intelligibility and pave the way to multi-agent systems which agents can communicate with unseen agents humans.
17

---

Published as a conference paper at ICLR 2024
On the other hand, increasing adoption of such multi-agent learning systems could exacerbate certain risks. For instance, this could increase unemployment by multiple humans like warchouses are replaced with multi-robot learning systems. It could also contribute to more advanced automated weaponry. In particular, given that our method explicitly considers messages sent from other agents in our protocol learning algorithm, it could encourage adversarial attacks which would lead to harmful behaviors and miscommunication. especially in mission-critical systems.
significant scale systems operated
18


Reviews:
[{"review_id": "PV6Y37BhBb", "reviewer": "Reviewer_sueF", "summary": "This paper presents a novel approach to guide multi-agent communication learning via contrastive learning in a decentralized MARL scenario. The intuition is that under similar circumstances the agents should emit similar messages, and vice versa. Hence the authors employ contrastive learning to maximize the mutual information between messages of a given trajectory and minimize other cases. The authors claimed their method has outperformed exisiting approaches on several benchmarks.", "questions": "More comprehensive comparison and analysis are expected:\n1. It is encouraged for the authors to demonstrate the scalability of the proposed approach, like in a continuous state/action space environments which may involve a large number of agents with quite dense communication load. In such cases, would the proposed scheme be better than the most recent communication-based work like ATOC[1], MF-MARL[2], TarMAC[3], I2C[4], ToM2C[5]?\n2. Besides showing the similarity of messages among multiple states, the exact improvement from such a contrastive learning method should be analyzed. For instance, it is better to compare the adjacency/disparity of positive/negative sample pairs before/after the proposed training.\n\n[1] Jiang, J., & Lu, Z. (2018). Learning attentional communication for multi-agent cooperation. Advances in neural information processing systems, 31.\n[2] Yang, Y., Luo, R., Li, M., Zhou, M., Zhang, W., & Wang, J. (2018, July). Mean field multi-agent reinforcement learning. In International conference on machine learning (pp. 5571-5580). PMLR.\n[3] Das, A., Gervet, T., Romoff, J., Batra, D., Parikh, D., Rabbat, M., & Pineau, J. (2019, May). Tarmac: Targeted multi-agent communication. In International Conference on Machine Learning (pp. 1538-1546). PMLR.\n[4] Ding, Z., Huang, T., & Lu, Z. (2020). Learning individually inferred communication for multi-agent cooperation. Advances in Neural Information Processing Systems, 33, 22069-22079.\n[5] Wang, Y., Zhong, F., Xu, J., & Wang, Y. (2021). Tom2c: Target-oriented multi-agent communication and cooperation with theory of mind. arXiv preprint arXiv:2111.09189.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "The idea delivered by this work is clear and somewhat grounded. Indeed it would be worthwhile for agent to learn a guidance of its message during multi-agent communication. And the intuition of enforcing messages under similiar state to be alike with each other is a straightforward motivation, for which contrastive learning might be one of the most popular method to achieve.", "weaknesses": "However, after going through the whole paper, It is easy to find that the proposed idea is less sufficiently proved and there are many flaws in the manuscript. There are a few such perspectives:\n1. In section 4, the negative samples are defined as from outside the current time window or other trajectories. This is not technically sound since it would be possible for agents to encounter similar states at different trajectories (which would be considered as negative by the proposal). It is suggested that the authors should discuss such cases in detail and figure out more solid principle to decide positive/negative samples for contrastive learning.\n2. The selected benchmark for comparison is kind of limited. Public MARL evaluation platforms like SMAC[1] or MATE[2] which involves higher complexity should be considered for more pursuasive comparison. In addition, in the compared task of Traffic Junction, the improvement seems to be marginal.\n3. The compared baseline methods are sort of obselete. Newer published works in recent 3 years should get included. (especially new work in 2022-2023).\n\n[1] Samvelyan, M., Rashid, T., De Witt, C. S., Farquhar, G., Nardelli, N., Rudner, T. G., ... & Whiteson, S. (2019). The starcraft multi-agent challenge. arXiv preprint arXiv:1902.04043.\n[2] Pan, X., Liu, M., Zhong, F., Yang, Y., Zhu, S. C., & Wang, Y. (2022). Mate: Benchmarking multi-agent reinforcement learning in distributed target coverage control. Advances in Neural Information Processing Systems, 35, 27862-27879.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hBAJMWOHTd", "reviewer": "Reviewer_64q7", "summary": "This paper proposes a new method for fully independent communication in MARL, CACL. The proposed method leverages the power of contrastive learning to ground communication and learning efficient communication for MARL tasks.", "questions": "* I have questions about whether it is reasonable to evaluate the similarity of messages of different agents by simply looking at a window of a few timesteps in the trajectory. The observations corresponding to the generated messages can be different in important aspects from one timestep to the other, and thus would require distinct messages that could be biased by the contrastive loss. I am unsure whether this would scale to more complex cases, since it could not capture these differences in the observations.\n* The experimented environments seem a bit simple and model scenarios where the observations can indeed be more similar to each other in some cases. Have the authors tested in more complex scenarios where there can be stronger variations on the observations such as SMAC? It would be interesting to see the performance in such complex environments.\n* The authors mention that the setting followed is a fully decentralised setting where the agents do not share parameters or gradients (section 1); does this apply to the message encoder? I.e., do the agents share the same message encoder or does each one of them use a separate encoder to generate messages? \n* I believe another potential direction for further work would be to investigate how to make methods such as the proposed one work together with the reward. I.e., in section 5.5 it was shown that currently it is detrimental. Yet, have the authors thought whether the method can be improved in any way in order to take advantage of the reward as well?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper tackles the problem of communication for fully independent learners, which is a very important topic in MARL and it is often underexplored. Also, mixing contrastive learning with MARL is interesting. Generally, the paper is well organised and well written.", "weaknesses": "Overall, this paper is interesting and investigates an important topic in MARL. However, I still have some concerns and questions that I would like the authors to comment on. Please find my comments below and questions ahead.\n\n* The example of predator prey in figure 1 (right) is a bit confusing. I would not agree that the given examples correspond to similar views; for example, the first view (counting from the top) seems more similar to the third view rather than to the second view.\n* In section 1, the authors mention: \"we propose that an agent’s observation is a “view” of the environment state. Thus, different agents’ messages are encodings of different incomplete “views” of the same underlying state.\". This is in fact the premise of a dec-pomdp; Observations are tipically local perceptions of the environment's state; in other communication methods in MARL where observation encodings are used as messages, I would say that the same logic is followed: individual perceptions of the environment are being shared as message encodings to the others. As described by the authors is section 3, the observations come from a function of the state.\n* In the loss function, the RL loss is not defined. It would be good to have it for clarity purposes.\n* A more detailed diagram of the architecture of this method could be beneficial to get a better understanding of the approach; the one presented in figure 8 in the appendix seems very simple and lacks detail and we cannot clearly understand how the gradients flow; this can be important since the authors are dealing with fully independent learning, without sharing parameters.\n\nMinor:\n* In section 3, do the authors mean $m_{t-1}^{-i}$ instead of $m_{t-1}^{-1}$?\n* In section 3 \"At each time step, each agent $i \\in N$ chooses an action $a ∈ A^i$\": shoud be $a^i$ since ahead $a$ is defined as the joint action.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MjRMb6TllG", "reviewer": "Reviewer_aKdr", "summary": "The authors describe CACL, a contrastive learning approach for inducing communication among multiple agents. There are close parallels to classic contrastive learning methods in vision, for example, but the authors apply their technique to \"emergent communication\" to allow teams of agents to communicate with each other. \"Postive\" examples are grouped based on a window of recent timesteps, and, as in standard constrastive learning, agents learn to encode positive examples near each other.\n\nIn experiments, the authors show that CACL outperforms numerous baselines, including the SOTA AEComm method (which in some ways is similar in that it is a non-reward-based mechansim for inducing emergent communication).", "questions": "I have no outstanding questions. Overall, this was a clear paper.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "I like this paper. It presents a simple idea that works well.\n\n## Originality\nApplying contastive losses to emergent communication is somewhat novel. (I know other works have also come out in this area, but they remain different in some important ways).\n\n## Quality\nThe work is well-scoped and presented, with good results backing up claims.\n\n## Clarity\nI find the paper quite clear. Some figures could likely be redone to present the same information better (e.g., Figure 3), but mostly these are small changes.\n\n## Significance\nI think this work, should it be published, would be an important baseline for future emergent communication work.", "weaknesses": "Overall, this is a strong paper. To further improve the paper the authors could\n\n1) Conduct further experiments to fill in Figure 4 in more detail (instead of just 3 or 4 checkpoints along the curve)\n\n2) Run more trials, especially in the traffic junction where variance is high and not all methods seem to have converged.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5zuAvwivqn", "reviewer": "Reviewer_Y1JN", "summary": "This paper studies multi-agent communication by contrastive learning. It is motivated by the fact that communicated messages based on local observation can be viewed as incomplete views of the global state. From this perspective, a contrastive learning based approach is proposed, where states from close time steps are considered as positive samples, and those from distant time steps or episodes are considered as negative samples. The proposed algorithm is tested on several multi-agent benchmarks.", "questions": "- The policy $\\pi$ defined in section 3 seems to be only conditional on the local observation $\\tau^i$. Should it also be conditional on the communicated messages?\n- If some local observations miss important information, the approximated global states reconstructed by the message encoders may be very different from the true ones. Will this make the contrastive learning not meaningful?\n- Does CACL require all-to-all communications, i.e., each agent communicate to all the other agents? If so, CACL is not scalable with large number of agents.\n- During a time step, each agent receives multiple approximations of the global state from the communicated messages. This seems to include redundant information if the messages are not selectively received. How does CACL handle redundant information?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper has an innovative perspective on the multi-agent communications, which motivates the use of contrastive learning.\n- The paper is well-written, and the proposed contrastive learning framework is easy and clean to implement.\n- There are various ablation studies over different components of the proposed algorithm and visualizations over the learned communicated messages.", "weaknesses": "- The improvement over the baselines is not obvious, especially in Traffic Junction in Figure 2. The standard error is also too large with a lot of overlaps, so it may need more seeds of experiments.\n- The proposed algorithm CACL is only tested on three tasks. The paper could benefit from additional experiments on some more challenging tasks with partial observability where communications are intuitively beneficial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)