Learning to Communicate using Contrastive Learning 
Authors: Yat Long Lo
Biswa Sengupta
Jakob Nicolaus Foerster
Michael Noukhovitch
Venue: Submitted to ICLR 2023 (2023)
Keywords: Reinforcement Learning
Multi-Agent Reinforcement Learning
Multi-Agent Communication

Abstract: Communication is a powerful tool for coordination in multi-agent RL.  Inducing an effective, common language has been a difficult challenge, particularly in the decentralized setting. In this work, we introduce an alternative perspective where communicative messages sent between agents are considered as different incomplete views of the environment state. Based on this perspective, we propose to learn to communicate using contrastive learning by maximizing the mutual information between messages of a given trajectory. In communication-essential environments, our method outperforms previous work in both performance and learning speed. Using qualitative metrics and representation probing, we show that our method induces more symmetric communication and captures task-relevant information from the environment. Finally, we demonstrate promising results on zero-shot communication, a first for MARL. Overall, we show the power of contrastive learning, and self-supervised learning in general, as a method for learning to communicate.

Under review as a conference paper at ICLR 2023
LEARNING TO COMMUNICATE USING CONTRASTIVE LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Communication is a powerful tool for coordination in multi-agent RL. Inducing effective, common language has been difficult challenge, particularly in the decentralized setting. In this work, we introduce an alternative perspective where communicative messages sent between agents considered different incom- plete views of the environment state. Based on this perspective, we propose to learn communicate using contrastive learning by maximizing the mutual infor- mation between messages of given trajectory. In communication-esse -ssential envi- ronments, our method outperforms previous work in both performance and learn- ing speed. Using qualitative metrics and representation probing, we show that our method induces more symmetric communication and captures task-relevant information from the environment. Finally, we demonstrate promising results on zero-shot communication, first for MARL. Overall, we show the power of con- trastive learning, and sulf-super learning in general, as method for learning to communicate.
INTRODUCTION
Encoder
Features
Bottom-Right
Encoder
Message
Agents
Augmentation
Similar
Similar
Encoder
Features
Encoder
Message
Dissimilar
Dissimilar
Cat
Encoder
Bottom Lett
Encoder
Message
Augmentation
Similar
Agent's
Similar
Encoder
Features
Encoder
Message
Multi-view Contrastive Learning
CACL
Figure 1: Multi-view contrastive learning and CACL, contrastive learning for multi-agent communi- cation. In multi-view learning, augmentations the original image or "views" are positive samples contrastively learn features. CACL, different agents' views of the same environment states are considered positive samples and messages are contrastively learned as encodings of the state.
Communication between agents is key capability necessary for effective coordination among agents in partially observable environments. In multi-agent reinforcement (MARL) (Sutton & Barto 2018). agents can use their actions transmit information (Cruuen al.) 2020) but continuous or discrete messages
communication channel (Foerster et al. 2016)) also known as linguistic communication (Lazaridou & Baroni 2020), is more flexible and powerful. To successfully com- municate, speaker and listener must share common language with shared understanding of the symbols being used (Skyrms) 2010 Dafoe et al.) 2020). Learning a common protocol, or emergent communication /Wagner et al. 2003 Lazaridou & Baroni 2020), is thriving research direction but many works focus on simple, single-turn, sender-receiver games (Lazaridou et al.] 2018 Chaabouni
al. 2019). In more visually and structurally complex MARL environments (Samvelyan et al.

---

Under review as a conference paper at ICLR 2023
2019
existing approaches often rely centralized learning mechanisms by sharing models (Lowe
ct al. 2017) or gradients (Sukhbaatar et al., 2016).
However, centralized controller is impractical in many real-world environments MMai et al. Jung et al, 202T and centralized training with decentralized execution (CTDE) (owe et al. 2017
2021
may not perform better than purely decentralized training (Lyu et al. (02T). Furthermore, the decen- tralized setting is more flexible and requires fewer assumptions about other agents, making it more realistic in many real-world scenarios (Li et al.) 2020). The decentralized setting also scales better, as a centralized controller will suffer from the curse of dimensionality: as the number of agents it must control increases, there is an exponential increase in the amount of communication between agents to process (Jin et al. (02T). Hence, this work explores learning to communicate in order to coordinate agents in the decentralized setting. In MARI., this means each agent will have its own model to decide how to act and communicate, and no agents share parameters or gradients.
Normal RL approaches to decentralized communication are known to perform poorly even in sim- ple tasks (Foerster et al. 2016). The main challenge lies in the large space of communication to explore. the high variance of RL, and lack of common grounding to base communication on (Lin) et al. 2021). Earlier work leveraged how communication influences other agents (Jaques et al., 2018 Eccles et al. 2019) to learn the protocol. Most recently, [in et al. (2021) proposed agents that autoencode their observations and simply use the encodings as communication, using the shared en- vironment as the common grounding. We propose to use the shared environment and the knowledge that all agents are communicating to ground a protocol. If, like Lin et al. (2021), consider agents' messages to be encodings of their observations then agents in similar states should produce similar messages. This perspective leads a simple method based on contrastive learning to ground communication.
Inspired by the literature in representation learning that uses different "views" of a data sample (Bachman et al. 2019),, for a given trajectory, we propose that an agent's observation is a "view" of some environment states. Therefore, different agents' messages are encodings of different "views" of the same underlying state. From this perspective, messages within a trajectory should be more similar to each other than to messages from another trajectory. We visually show our perspective in Figure 1 We propose that each agent use contrastive learning between sent and received messages to learn to communicate, which we term Communication Alignment Contrastive Learning (CACL).
We experimentally validate our method in three communication-essente environments and empir- ically show how our method leads to improved performance and speed, outperforming state-of-the- art decentralized MARL communication algorithms. To understand CACL's success, we propose suite of qualitative and quantitative metrics. We demonstrate that CACL leads to more symmetric communication, allowing agents be more mutually intelligible. By treating our messages as rep- resentations, we show that CACL's messages capture task-relevant semantic information about the environment better than baselines. Finally, we look at zero-shot cooperation with partners unseen at training time, a first for MARL communication. Despite the difficulty of the task, we demon- strate the first promising results in this direction. Overall, we argue that supervupe learning a powerful direction for multi-agent communication.
2 RELATED WORK
Learning to coordinate multiple RL agents is challenging and unsolved task where naively applying agent RL algorithms often fails (Foerster et al. 2016). Recent approaches focus on agents parameterized by neural networks (Goodfellow et al., 2016) augmented with a message channel so that they can develop common communication protocol (Lazaridou & Baroni, (020). To solve issues of non-stationarity. some work focuses on centralized learning approaches that globally share models (Foerster et al.] 2016). training procedures (Lowe et al.) 2017). gradients (Sukhbaatar ctt al. 2016) among agents. This simplifies optimization issues can still be sub-optimal (Foerster et al. 2016 Lin et al. 2021). This also violates independence assumptions, effectively modelling the multi-agent scenario as a single agent (Eccles et al. 2201)).
This work focuses on independent, decentralized agents and non-differentiable communication. In previous work, Daques et al.] (2018) propose a loss to influence other agents but require explicit and complex models of other agents and their experiments focus on mixed cooperative-compertive
2

---

Under review as a conference paper at ICLR 2023
scenarios. Eccles al. (2019) build this and add biases to each agent's loss function that sep- arately encourage positive listening and positive signaling. Their method is simpler but requires task-specific hyperparameter tuning to achieve reasonable performance and still underperforms in sensory-rich environments (Lin et al., 2021). Our work is closest to Lin et al. (2021) who leverage autoencoding as their method to learn message protocol. Agent learn to reconstruct their obser- vations and simply communicate their autoencoding. The authors find that it outperforms previous methods while being algorithmically and conceptually simpler. Our setup differs from Eecles et al. 2019); [in et al. (202T) by using continuous instead of discrete messages. This choice is standard in contrastive learning (Chopra et al. 2005 He et al. [220 Chen et al. 2020a) and common in em- bodied multi-agent communication (Sukhbaatar et al. [0166] |ingh et al.) 2018 Jiiang & Lu 2018
Das et al. 2019). As well, our representation learning task requires no extra learning parameters that are discarded at test time, whereas Lin et al. (202T) discard their decoder network.
Autoencoding is a form of generative self-supervised learning (SSL) (Doersch et al.] 2015). We propose to use another form of SSL, contrastive learning (Chen et al. 2020a), the basis for learn- ing communication. We are motivated by recent work that achieves state-of-the-ar representation learning on images using contrastive learning methods (Chen et al. 2020b) and leverages multi- ple "views" of the data. Whereas negative samples are simply different images, positive samples "views" of the original image (Bachman et al.) 2019). Since our setup includes supervised labels, we base our method on SupCon (Supervised Contrastive Learning) Khosla et al. 2020) which modifies the classic contrastive objective to use account for multiple positive samples. Also related is Dessi et al. (2021) who propose discrete two-agent communication as a contrastive learning task, we do the opposite and leverage contrastive learning for multi-agent communication.
image data augmentations
3 PRELIMINARIES
We base investigations decentralized partially observable Markov decision processes (Dec- POMDPs) with N agents to describe a fully cooperative multi-agent task (Oliehoek & Amato 2016). A Dec-POMDP consists of tuple G AS P. R, Z, n,)) E S is the true state of the environment. At each time step, each agent E N chooses an action a E A' to form joint action a E A A² x A A It leads an environment transition according the transition function P(s'|8, 1 ....aN R(s, a): SxA R. 7 E [0, 1) is discount factor. As the environment is partially observable, each agent i receives individual observations = E Z based on the observation function (2'(s) : S + Z.
S A S - [0, 1]. All agents share the same reward function
We denote the environment trajectory and the action-observation history (AOH) of an agent i as = Sp. Ay, St. Oe and T/- = (20))s) ...S5' of E T = (Z x A)" respectively. A stochastic policy T(a') (r) : T A + [0, 1] conditions on AOH. The joint policy has a corresponding action- value function Q" (s,, a,) = E Ex t JR, (s,, 0z), where R, = Σi Y're+i i is the discounted return. rt+z is the reward obtained at time + from the reward function R.
To account for communication, similar to Lin et al. (202T), at each time step f, an agent i takes an action of and produces a message m; (12 () after receiving its observation Ω2 (s;)) and messages sent at the previous time step m7-1. where Yp is agent i's function to produce a message given its observation and m refers to messages sent by agents other than agent i. The messages are continuous vectors of dimensionality D.
4 METHODOLOGY
We propose a different perspective on the message space used for communication. At each time step t for a given trajectory т, a message m; of an agent i can be viewed as an incomplete view of the environment state s; because it is a function of the environment state as formulated in section 3 Naturally, messages of all the agents d, are different incomplete perspectives of To ground decen- tralized communication, we hypothesize that we could leverage this relationship between messages from similar states to encourage consistency and proximity of the messages across agents. Specif- ically, we propose maximizing the mutual information using contrastive learning which aligns the message space by pushing messages from similar states closer together and messages of different
3

---

Under review as a conference paper at ICLR 2023
states further apart. As a heuristic for state similarity, we consider window of timesteps within trajectory to be all similar states i.e. positive samples of each other. To guarantee dissimilar negative samples (Schroff et al. 2013). we states from other trajectories negatives.
We extend the recent supervised contrastive learning method SupCon (Khosla et al. 2020) to the MARL setting by considering multiple trajectories during learning. We refer to this loss formulation as Communication Aligument Contrastive Learning (CACL). In this case, we consider messages within a trajectory to be different views of the same data sample with the same label.
Let H be a batch of trajectories with messages M. Let M, be the messages in trajectory т. For an agent i, let mi M, be its message at time and other messages in trajectory be A- = {m' E M, P(m)) {mg E (m()) : E w, + w]}. Formally, the contrastive loss is:
# m.}. Therefore, positives for message m given a timestep window w are
exp(m mp/n)
LCACL
Σ mEM
log
P(m)]
Σ Lm. exp(m ma/
mpEP(mn
Where IR. is scalar temperature and |P(m] is the cardinality.
Practically, each agent has a replay buffer that maintains batch of trajectory data containing mes- sages received during training to compute the CACL loss. Following Khosla et al. (2020), messages
are normalized before the loss computation and a low temperature (i.e. 77 0.1) is used as
empirically benefits performance and training stability. The total loss for each agent is a reinforce-
ment learning loss T.RI. using the reward to learn policy (but not message head) and a separate contrastive loss LCACL to learn just the message head, formulated follows:
== = + KLCACL where K E R+ is hyperparameter scale the CACL loss.
(2)
5 EXPERIMENTS AND RESULTS
5.1 EXPERIMENTAL SETUP
We evaluate method on three multi-agent environments with communication channels. Given the limited information each agent observes themselves, agents are encouraged to mcaningfully communicate in order to improve task performance.
Predator-Prey: A variant of the classic game (Benda et al. 1986 Barrett et al. 2011) 1) based on Koul (2019) where 4 agents (i.e. predators) have the cooperative goal to capture 2 randomly-mov prey by surrounding each prey with more than one predator. We devise more difficult variation where agents are required to entirely surround prey on all four sides for it be captured and they cannot see each other in their fields of view. Therefore, it is essential for agents to communicate their positions and actions order coordinate their attacks. We evaluate each algorithm with episodic rewards during evaluation episodes.
Find-Goal: Proposed by Lin et al. 2022 agents' goal reach the green goal location as fast as possible in a grid environment with obstacles. We use agents and .at each time step, each agents observes a partial view of the environment centered its current position. Unlike in al. (202T), we use field of view of 3 x instead of 5 x to make the problem harder. Each agents receives an individual reward of for reaching the goal and an additional reward of 5 when all of them reach the goal. Hence, it is beneficial for agent to communicate the goal location once it observes the goal. As in Lin et al. (2021). we measure performance using episode length. An episode ends quicker agents can communicate goal locations each other more efficiently. Hence, method has better performance it has shorter episode lengths.
Traffic-Junction Proposed by Sukhbaatar et al. (016)) it consists of A 4-way traffic junction with cars entering and leaving the grid. The goal is to avoid collision when crossing the junction. We use 5 agents with vision of 1. Although not necessary, given the limited vision in agents,
4

---

Under review as a conference paper at ICLR 2023
communication could help in solving the task. We evaluate each algorithm with success rate during evaluation episodes.
All results averaged over 12 evaluation episodes and over 6 random seeds. More details of the environments and parameters can be found appendix A.1
5.2 TRAINING DETAILS
We compare CACL to the state-of-the-art independent, decentralized method, autoencoded com- munication (AEComm: [in et al. 202T). which grounds communication by reconstructing encoded observations. We also compare to baselines from previous work: independent actor critic without communication (IAC) and positive listening (PL:(Eccles et al. (2019) which encourages agents act differently when receiving different messages. We do not include the positive signalling loss (Eccles) al. 2019) because extending it to continuous messages is non-trivial but note that AEComm out- performs it in the discrete case (Lin et al. (2021). We also compare to DIAL (Foerster et al. (016) which learns to communicate through differentiable communication and therefore decentralized but not independent.
All methods use the same architecture based on the IAC algorithm with n-step returns and asyn- chronous environments (Mnih et al. (0016). Each agent has encoder for observations and received messages. For methods with communication, each agent has a communication head to produce messages based encoded observations. For policy learning, GRU (Cho al. 2014) is used to generate a hidden representation from a history of observations and messages. Agents use the hidden state for their the policy and value heads, which 3-layer fully-connected neural networks.
We perform spectral normalization (Gogianu et al. [222 in the penultimate layer for each head to improve training stability. The architecture is shown Figure 6 and hyperparameters are further described, both in Appendix
5.3
TASK PERFORMANCE
Predooor Pray
Traffe-lunction
Find Gool
AACESs OOL -L 206
SAC 334 142
30
203
Figure 2: CACL (red) outperforms all other methods on Predator-Prey (left), Traffic-Junction (cen- ter) and Find-Goal (right). Predator-Prey shows evaluation reward, higher is better. Jrantic-Junction plots the percent of successful episodes, higher is better. Find-Goal plots the length the episode until the goal reached, lower is better. Standard errors are plotted shaded areas and the perfor- mance curves are smoothed by factor of 0.5.
We run all methods on Predator-Prey, Find-Goal and Traffic-Junction and plot results in Figure 2 Our proposed method CACL outperforms all baseline methods in terms of both final performance and learning speed and, consistent with previous results (Lin al. 2021) D. AEComm is the strongest baseline. The largest performance increase from CACL is in FindGoal where partial observability most prominent because of agents' small field-of-view which makes communication more neces- sary (hence why IAC performs worst). These results show the effectiveness self-supervised methods for learning communication in the dilly-decentralized setting, as they both outperform DIAL which, notably, backpropogates gradients through other agents. Furthermore, it demonstrates CACL's trastive learning as a more powerful alternative to AEComm's 's autoencoding for coordinating agents with communication.
5

---

Under review as a conference paper at ICLR 2023
Table 1: Success rate in Predator- Prey: the percentage of final evaluation runs that captured prey, one prey, or both prey, Average and standard deviation over 6 random seeds.
No-Prey
One-Prey
Two-Preys
IAC
36.67% + 7.50
20.00% t 4.70
48.33% = 6.83
DIAL
63.33% + 7.56
3.33% = 1.24
33.33% + 7.86
PL.
50.00% + 8.33
0.00% = 0.00
51.67% + 8.63
AEComm
41.67% = 7.48
11.67%
2.79
51.67% = 7.60
CACL (Ours)
33.33% = 7.86
0.00% = 0.00
68.33% = 8.07
To give practical context to our reward curves, we assess the algorithms from the perspective of task completion. In Predatory-PPee Prey, compute the percentages of evaluation episodes that capture prey, one-prey and two-preys, where capturing two preys is a successful. We average over 6 random seeds and, as shown in Table [ CACL outperforms all baselines and not only solves the complete task more robustly, but also completely fails less frequently.
5.4 PROTOCOL SYMMETRY
To explain CACL's improved performance over the baselines, we hypothesize that it induces more consistent, communal language that is shared among agents. More specifically, we consider a lan- guage's consistency to be how similarly agents communicate (i.e., sending similar messages) when faced with the same observations. A consistent protocol can reduce the optimization complexity since agents only need to learn one protocol for the whole group and it also makes agents more mutually intelligible.
To evaluate consistency, we measure protocol symmetry (Graesser al. 2019) so if agent swaps observations and trajectory with another agent, it should produce a similar message as what the other agent produced. We extend this metric from previous work the continuous, embodied case and measure the pairwise cosine similarities of messages sent by different agents for the same observa- tion. Let (N) denote the set of all k--aeett subset of set of N agents, Given a trajectory and {t E T} as a set time steps of r, protocol symmetry (protocol is written as:
protocol.symm17
Σ J.Ke
mj my |
mx]
Therefore, more consistent protocol has higher symmetry. We swap agent trajectory and observa- tions and compute this metric over 10 sampled evaluation episodes for 6 random seeds, and show results in Table
Table 2: Protocol symmetry across environments, average and standard deviation over 10 episodes and 6 random seeds. CACL consistently learns the most symmetric protocol.
DIAL
PL
AEComm
CACL (Ours)
Predator-Prey
0.66 + 0.07
0.66 + 0.06
0.89 = 0.01
0.95 - 0.01
FindGioa
0.50 + 0.05
0.49 + 0.04
0.85 + 0.02
0.92 + 0.01
Traffic Junction
0.69 + 0.01
0.61 + 0.04
0.80 + 0.01
0.98 + 0.002
The self-supervised methods (CACL and AEComm) clearly outperform the others (DIAL and PL. implying that SSL is better for learning consistent representations in decentralized MARL. Further- more, CACL's protocol very highly symmetric, clearly outperforming all others. Each ABComm agent autoencodes their own observation without considering the messages of other agents, leading to the formation of multiple protocols between agents. contrast, CACL induces a common proto- col by casting the problem in the multi-view perspective and implicitly aligning agents' messages. The possible correlation between protocol symmetry and overall performance and speed further indicates the benefits of learning common language in the decentralized setting.

---

Under review as a conference paper at ICLR 2023
Other agent visible
Nothing visible
Goal visible
Nothing visible
Figure 3: DBSCAN Ester et al. 1996 clustering results of messages produced by CACL after reduced in dimensions using t-SNE (an der Maaten & Hinton 2008). Exemplary clusters are shown with their corresponding observational patterns. Specifically, two clusters correspond to messages sent when the goal visible and another agent is visible respectively. The other two clusters of when only individual agents are visible.
5.5 PROTOCOL REPRESENTATION PROBING
To further investigate how informative protocols are, propose suite of qualitative and quan- titative representation probing tests based on message clustering and classification, respectively. We perform these tests on the protocols learned in the Find-Goal environment.
Similar to Lin et al. 2221), } we perform message clustering on messages generated from 10 evalua- tion episodes to qualitatively assess whether CACL can learn an informative protocol. The messages are first compressed to a dimension of 2 using (-SNE (Van der Maaten & Hinton| (008) and then clustered using DBSCAN (Ester et al., 1996). We look each cluster's messages and their corre- sponding observations to extract any patterns and semantics captured. As shown in Figure B] we observe a cluster of messages for observations when the goal is visible and a cluster for observa- tions when another agent is visible. Two clusters correspond to agents seeing neither the goal nor another agent. This indicates that CACL, learns to compress meaningful, task-relevant information in messages and allows agents reasonably learn this semantic information.
Classification results of the two probing tests in the Find-Goal environment, comparing all methods with communication. All methods perform similarly in the casier Goal Visibility Test while CACL outperforms the baselines significantly in the more difficult Goal Location Test.
Table 3:
DIAL
PL
ABComm
CACL (Ours) 97.75% + 0.69
Goal Visibility
99.45% + 2.68
98.87% = 0.67
99.75% - 0.04
Goal Location
68.15% + 1.76
78.31% = 2.39
76.14% + 3.36
91.28% + 1.71
To quantitatively evaluate the informativeness of learned protocols, we propose to treat messages as representations and follow literature in representation learning for RL (Lazaridou et al.] 2018 Anand et al. 2019) to learn a classifier on top of the messages. Since FindGoal is focused on reaching a goal, intuitively, agents should communicate whether they have found the goal and, if so, where other agents should go to reach the goal. Therefore, we propose to probe the goal visibility and goal location. The former uses the messages to classify whether the goal is visible in observations or not
7

---

Under review as a conference paper at ICLR 2023
(i.e. a binary classification). The latter uses messages where the goal is visible in the observations to classify the general location of the goal (i.e. 5-class classification: Top-Left, Top-Right, Bottom- Left, Bottom- Right and Middle). Goal location is more difficult to predict than goal visibility as requires detailed, spatial information. We use 30 evaluation episodes per method to generate messages for experiments but different methods may have different numbers of messages that are acceptable for our probing task (e.g. a limited number of messages where the goal is visible for predicting goal location). To ensure fair comparison, we choose an equal number of samples per class (i.e. -pssitive/negative, 5-class location) for all methods and use 70%/30% random split for training and testing. We use a two-layer fully-connected neural network to test each method, this corresponds to the same network that agents use to encode each others' messages as part of their observations.
Table shows the classification results for the two probing tests. For goal visibility, the easier task, all methods' messages can be effectively used to interpret whether a goal visible in the observations or not. In the more difficult prediction of goal location, all methods perform above chance (20%) but CACL's protocol significantly outperforms baselines. Contrastive learning across different agents' messages can enable CACL to learn more global understanding of location. By encoding the goal's spatial information, CACL agents are more likely able to move directly towards it, and reduce episode length. If other methods simply communicate that a goal is found, agents know to change their search but are not as precise in direction. This explains why AFComm, PL,, and DIAL perform better than IAC but worse than CACL, which also learns much quicker as shown in Figure 2 For completeness, we also provide classification results with a one-layer (linear) probe with similar results in Appendix A.4
5.6 ZERO-SHOT CROSS-PLAY
Table 4: Zero-shot closs-play performance in Predator-Pres Prey. Intra-method results are bolded.
CACL
AEComm
PL.
DIAL
CACL (Ours)
17.20 5.14
-28.49 1 2.78
-24.61 1 5.77
-28.78 3.99
AEComm
-37.86 + 7.20
-31.56 + 3.76
-29.73 3.66
PL
-27.07 2.94
-22.89 = 3.98
DIAL
22.85 + 2.04
Table 5: Zero-shot cross-play performance in Find-Goal. Intra-method results are bolded.
CACL
AEComm
PL
DIAL 488.28 16.60
CACL (Ours)
468 75 = 15.32
471.66 13.54
487.56 8.61
AEComm
479.96 - 14.96
440.18 + 23.04
472.85 = 16.77
PL
492.08 + 5.67
486.41 + 10.46
DIAL
476.07 = 15.89
An advanced form of coordination is working with partners you have not seen during training (uu let al.] 2020). Previous work has focused on coordination through actions (Carroll et al.) 2019 Lupu al. 2021) but our knowledge, no previous work has succeeded in learning linguistic communication protocol that is robust to zero-shot partners. To assess this advanced robustness, we take trained agents from different methods and random seeds and evaluate them with each other (i.e., zero-shot cross-play in Tredator-Prey and Find-Goal. Given two communication learning methods, mi and m2, we sample two agents from each method for Predator- Prey and for Find Goal, we average over sampling two agents from one method and one agent from the other and vice-versa. For intra-method cross-play, mi = m2, we evaluate agents that were trained with the same method but from different random seeds, so they have not been trained with each other. For inter-method cross-play, mi # m2, we sample agents from two different methods and pair them with each other. Each pairing is evaluated for 10 random seeds each with 10 evaluation episodes. Given that agents are trained in play (Tesauro) 1994) without regard for cross- cross-play, play, we expect severe performance dips.
We show mean and standard deviation across random seeds for Predator-Pree Prey and Find-Goal in Tables and 5 respectively. As expected, all pairings take significant dip in performance when
8

---

Under review as a conference paper at ICLR 2023
compared with the main results. Inter-method cross-play performance is particularly bad across all algorithms. However, notably, CACL outperforms other methods in intra-method cross-play, indicating that the protocols learned by CACL are generally In general, zero-shot linguistic communication incredibly difficult and our results are quite weak. Still, CACL. shows promise and demonstrates that contrastive SSL methods can lead better zero- shot communication and coordination.
robust across random seeds.
5.7 PROTOCOL REPRESENTATION LEARNING WITH REINFORCEMENT LEARNING
Predato-Paay
Traffic-anciion
Find Goal
$01
A00cm
Leae
200
2004
Figure 4: Comparing CACL and AEComm with their respective variants when combined with DIAL. Variants with DIAL have generally worse performance.
Given the overall improved performance of our method, natural question is whether we can achieve even better results if we use the reward to optimize our message well. To answer this question, add DIAL both CACL and the next best method, AEComm, and evaluate in the three envi- ronments. This is equivalent to backpropogating T.RL from Equation 2 through agents to lear the message head. In this way, both RL and SSL. (contrastive or autoencoding) signals are used to learn the protocol.
Figure 3 compares the performance of CACL and AEComm with their DIAL-augmented variants. We observe that augmenting either SSL method with DIAL performs generally worse, except in Find-Goal, where performances similar but not better. These findings are consistent with Lin al. 2021), who find that mixing SSL and RL objectives hypothesize that decentralized DIAL a complex, and high-variance optimization that is difficult to stabilize. DIAL's gradient updates may clash with CACL and result in neither useful contrastive representation, nor a strong reward-oriented one.
detrimental performance. We
6 CONCLUSION AND FUTURE WORK
In this work, we introduce an alternative perspective in learning to communicate in decentralized MARL by considering the relationship between messages and received within trajectory. Inspired by multi-view learning, we propose to ground communication using contrastive learning by considering agents' messages be encoded views of the same state. First, we empirically show that our method leads to better performance and more consistent, common language among agents. Then, we qualitatively and quantitatively probe our messages as learned representations to show that our method more consistently captures task-relevant information. We also test our method on zero- shot pross-play, a first for MARL communication, and demonstrate promising results. Finally, we show that our SSL objective is not improved by further optimizing with RL, in line with previous work. We believe this work solidifies SSL as an effective method for learning to communicate in decentralized MARL. Furthermore, has been focused on images and communicative MARL. We hope this inspires more investigation the intersection of these research directions.
have demonstrated a link between multi-view SSL which

---

Under review as a conference paper at ICLR 2023
REFERENCES
Ankesh Anand, Evan Racah, Sherjil Ozair, Yoshua Bengio, Marc-Alexandre Cóté, and R Devon Hjelm. Unsupervised State Representation Learning in Atari. In Advances in Neural Information Processing Systems, volume 32. Curran Asso-
ciates, Inc., 2019. URL bttss::/papprs. nips. wttpppap////11///aa///////
html
Philip Bachman, Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. Advances in neural information processing systems, 32, 2019.
Samuel Barrett, Peter Stone, and Sarit Kraus. Empirical Evaluation of Ad Hoc Teamwork in the Pursuit Domain. In AAMAS, pp. 8, 2011.
Miroslav Benda, Jagannathan, and R Dodhiawala. On optimal cooperation of knowledge sources an empirical investigation. Technical report, Bocing Advanced Technology Center, Boeing Com- puting Services, 1986. URL/nttps: wwww/wwww..ooo cs ut exas. edu/ wwwiaraai readings b2hd-BendaJDI 986 9tm.mmmm..mmm...mm.mmmmmm.mmmmm.
Micah Carroll, Rohin Shah, Mark K. Ho. Thomas L. Griffiths, Sanjit A. Seshia, Picter Abbeel, and Anca Dragan. On the Utility of Learning about Humans for Human-AI Coordination. In Neural Information Processing Systems. arXiv, 2019. doi: (88550/arXii.1110.05577 http://arxiv. owwoorggaao abs /9900 05789 arXiv:
URL
[cs, stat]. g
Rahma Chaabouni, Eugene Kharitonov, Alessandro Lazaric, Emmanuel Dupoux, and Marco Baroni. Word-order biases in deep-agent emergent communication. In ACL, May 2019. URL http:
//axxiv.org/abbt /105 12330
arXiv: 1905.12330. I
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A Simple Framework
for Contrastive Learming of Visual Representations. In arXiv:2002.0557 05000 [cs, stat], June 2020a.
URL http:://txxxi...oot org/ aws/aaas..aaa/ 05709 arXiv: 2002.05709, 3
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big Self-Supervised Models are Strong Semi-Supervised Learners. In arXiv:2006. 10029 [cs, stat), 10029
October 2020b. httphtttp///axii> http:/awwwwv.rraa////bbaaaaaaaaa
arXiv: 2006. 10029. 3
Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio. On the Properties of Neural Machine Translation: Encoder-Decoder Approaches. In 8th Workshop on Syntax, Se- mantics and Structure in Statistical Translation. arXiv, October 2014. doi: 10.48550/arXiv. 1409. 1259. URL http:://a// http:////wwwwopo//////////////// httg:bb//1/9.1122/2/ 1259 MrXiv:1149.. 1259 [cs, stat].
Sumit Chopra, Raia Hadsell, and Yann LeCun. Learning a similarity metric discriminatively, with application to face verification. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05), volume 1, pp. 539-546. IEEE, 2005. 3
Allan Dafoe, Edward Hughes, Yoram Bachrach, Tantum Collins, Kevin R McKee, Joel Z. Leibo, Kate Larson, and Thore Graepel. Open problems in cooperative ai. arXiv preprint arXiv:2012.086300 2020. I
Abhishek Das. Théophile Gervet, Joshua Romoff, Dhruv Batra, Devi Parikh, Michael Rabbat, and Joelle Pineau. TarMAC Targeted Multi-Agent Communication. In ICML, 2019, URL Sthhtt:
//arxiv.org/abs/lluu
Roberto Dessi, Eugene Kharitonov, and Marco Baroni. Interpretable agent communication from scratch(with generic visual processor emerging on the side). In NeurlPS, 2021.
Carl Doersch, Abhinav Gupta, and Alexei A. Efros. Unsupervised Visual Representation Learning by Context Prediction. In ICCV, 2015. URL. http:tttpp//ii..../.// www.:org/aaa. orggabss/105 051 192
Tom Eccles, Yoram Bachrach, Guy Lever, Angeliki Lazaridou, and Thore Graepel. Biases for emergent communication multi-agent reinforcement learning. Advances in neural information processing systems, 32, 2019.
3
14
10

---

Under review as a conference paper at ICLR 2023
Martin Ester, Hans-Peter Kriegel, Jörg Sander, Xiaowei Xu, et al. A dlensitybbased algorithm for discovering clusters in large spatial databases with noise. In kdd, volume 96, pp. 226-231, 1996.
Jakob Foerster, Ioannis Alexandros Assael, Nando De Freitas, and Shimon Whiteson. Learning to communicate with deep multi-agent reinforcement learning. Advances in neural information processing systems, 29, 2016. 2
Florin Gogianu, Tudor Berariu, Mihaela C Rosca, Claudia Clopath, Lucian Busoniu, and Razvan Pascanu. Spectral normalisation for deep reinforcement learning: an optimisation perspective. In
International Conference on Machine Learning, pp. 3734-3744, PMLR, 2021. S
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016. http:
7/wwws www. www.deeppeearnngboo.. ozq
Laura Graesser, Kyunghyun Cho, and Douwe Kiela. Emergent linguistic phenomena in multi-agent communication games. In EMNLP. 2019.
Niko A. Grupen, Daniel D. Lee, and Bart Selman. Low-Bandwidth Communication Emerges Nat- urally in Multi-Agent Learning Systems, December 2020. URL http://arxiv. wwwplvv.rrgaaba. /bs// 2011. 14890 #rXiv:2011.14890 [cs]. [
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, pp. 9729-9738, 2020.
Hengyuan Hu, Adam Lerer, Alex Peysakhovich, and Jakob Foerster. "Other-Play" for Shro-Shot Coordination, In ICML. arXiv, 2020. doi: (0.48550/rXxi.2000..002 Sxgg/bs//2e 02979 2XXi:20033002999 [cs]. g
URLhttp: /tattp///arivv
Natasha Jagues, Angeliki Lazaridou, Edward Hughes, Caglar Gulcehre, Pedro A. Ortega, D. J. Strouse, Joel Z. Leibo, and Nando de Freitas. Social Influence as Intrinsic Motivation for Multi- Agent Deep Reinforcement Learning In ICML. arXiv, September 2018. doi: 10.48550/arXiv. 1810.08647.
abaaabaa/1111 08647 arXiv:1810.08647 [cs, stat]. P
Jiechuan Jiang and Zongqing Lu. Learning Attentional Communication for Multi-Agent Coopera- tion. In Neural Information Processing Systems. arXiv, November 2018. 3
Chi Jin, Qinghua Liu, Yuanhao Wang, and Tiancheng Yu. V-Learning A Simple, Efficient, De-
centralized Algorithm for Multiagent RI., October 2021. URL http::://errvii.rrr//sss.
2110.14553
arXiv:2110. 14555 [cs, stat]. 2
Soyi Jung. Won Joon Yun, Joongheon Kim, and Jac-Hyun Kim. Coordinated Multi-Agent Deep Reinforcement Learning for Ewwrg UAV-Based Big-Data Platforms. Electronics, 10(5): 543, January 2021. ISSN 2079-9292. doi: 10.33900clectroniics0003333 10050543. URL https: www. wdwi. com/ 207 979992922/00///44 Number: 5 Publisher: Multidisciplinary Digital
Publishing Institute.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in Neural Information Processing Systems, 33:18661-18673, 2020. 3
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint
orXiv:1412.2 148.0690...6....... 2014. 14
Anurag Koul. ma-gym: Collection of multi-agent environments based on openai gym. https
//github. com wtw.:uu..cooooookauaaaaaaaaaa wttp///kuouaaannnnnaaaa gym. 2019.
13
Angeliki Lazaridou and Marco Baroni. Emergent Multi-Agent Communication in the Deep Learning
Era, July 2020.
Angeliki Lazaridou. Karl Moritz Hermann, Karl Tuyls, and Stephen Clark. Emergence of Linguistic Communication from Referential Games with Symbolic and Pixel Input. In ICLR, April 2018. httthtt:://arat www.oorgg/..... 03984 arXiv: 1804.03984.

---

Under review as a conference paper at ICLR 2023
Wenhao Li, Bo Jin, Xiangfeng Wang, Junchi Yan, and Hongyuan Zha. F2a2: Flexible fully- decentralized approximate actor-critic for cooperative multi-agent reinforcement learning. arXiv preprint wrXhv:2004.11444 2020. 2
Toru Lin, Jacob Huh, Christopher Stauffer, Ser Nam Lim, and Phillip Isola. Learning to ground multi-agent communication with autoencoders. Advances in Neural Information Processing Sys- tems, 34, 2021. 2
14
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-comptttive environments. Advances in neural information processing systems, 30, 2017. 2
Andrei Lupu, Brandon Cui, Hengyuan Hu, and Jakob Foerster. Trajectory diversity for zero-shot coordination. In International Conference on Machine Learning, pp. 7204-7213. PMLR, 2021.
Xueguang Lyu, Yuchen Xiao, Brett Daley, and Christopher Amato. Contrasting Centralized and Decentralized Crities in Multi-Agent Reinforcement Learning. In AAMAS, pp. 2021.
Vincent Mai, Tianyu Zhang, and Antoine Lesage-l.andry. Multi-agent reinforcement learning for renewable integration in the electric power grid. In Tackling Climate Change with Machine Learn- ing Workshop at NeurIPS, 2021. 2
Volodymyr Mnih Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning, In International conference on machine learning, pp. 1928-1937. PMLR, 2016. 5
Frans A Olichoek and Christopher Amato. A concise introduction to decentralized POMDPs.
Springer, 2016.
Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim G. J. Rudner, Chia-Man Hung, Philip H. S. Torr, Jakob Foerster, and Shimon Whiteson. The StarCraft Multi-Agent Challenge, December 2019. URL htttp://axxiv
org/ abs / 902. 04043 arXiv:1902.04045 [cs, stat]. D
Florian Schroff, Dmitry Kalenichenko. and James Philbin. FaceNet: A Unified Embedding for Face Recognition and Clustering. In Conference on Computer Vision and Pattern Recognition (CVPR), pp.
June 2015. doi: N0.110/CCPR22205..29666
URL httpp:////axxx
arg/abs/1500 03832
arXiv: 1.Xiv:150...0 50333333322222 [es]. 4
Amanpreet Singh, Tushar Jain, and Sainbayar Sukhbaatar. Learning when to communicate at scale in multiagent cooperative and competitive tasks. arXiv preprint arXiv:1812.0975: 2018. 3 1
Brian Skyrms. Signals: Evolution, Learning, Information. Oxford University Press, Oxford, 2010. 0
Sainbayar Sukhbaatar, Rob Fergus, al. Learning multiagent communication with backpropaga- tion. Advances in neural information processing systems, 29, 2016. 3
Richard Sutton and Andrew G Barto. Reinforcement Learning: An Introduction. MIT Press, 2018. URL httt::/t///t http://iinooppeeeeeeeeee htw.:ooppeeeeeeeeee..eeeee/oooo. www.neee///ootbeeee book 2nd. html
Gerald Tesauro. Td-gammon, a self-teaching backgammon program, achieves master-level play.
Neural computation, 6(2):215-219, 1994.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008. 7
Kyle Wagner, James A Reggia, Juan Uriagereka, and Gerald S Wilkinson. Progress in the simulation of emergent communication and language. Adaptive Behavior, 11(1):37-69 2003. [
12

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 ENVIRONMENT DETAILS
Figure provides a visual illustration of the environments used.
One way
Car leaving
Goal to reach the green block
entering
Figure 5: Visual illustration of the environments used. Left: Predator-Pre Prey, taken from Koul (0119)
Middle: Find-Goal, taken from Lin et al. (202I). Right: Traffic-Junction, taken from Singh et 2018)
A.1.1 PREDATOR-PREY
We modify the Predator-Pre implementation by Koul (2019). Our Predator-Prey has higher com- munication and coordination requirement than the original Predator-Prey environment. Specifically, for prey to be captured, has to be entirely surrounded (i.e. the prey cannot move another grid position in any actions).
Here, we use an 7x7 gridworld. In each agent's observation, it can only see the prey it is within the field of view (3x3) and cannot see where other agents are. A shared reward of 10 is given for successful capture and A penalty of -0.5 is given for failed attempt. A -0.01 step penalty is also applied per step. Each agent has the actions of LEFT, RIGHT, UP. DOWN and NO-OP. The prey has the movement probability vector of (0.175, 0.175, 0.175, 0.175, 0.3] with each value corresponding to the probability of each action taken.
algorithms are trained 30 million environment steps with a maximum of 200 steps per episode.
A.1.2 FIND-GOAL
We use the Find-Goal environment implementation provided by Lin et al. (2021). The agents have the goal to find where the goal is in 15x15 grid world with obstacles.
Unlike in Lin al. (202T), each agent has 3x3 field view (instead of 7x7) make the task more difficult. Each agent receives reward of I for reaching goal and additional reward of if all agents reach the goal. We use a step penalty of -0.01 and an obstackle density of 0.15.
All algorithms are trained for 40 million environment steps with maximum of 512 steps per episode.
A.1.3 TRAFFIC-JUNCTION
We the Traffic-Juncton environment implementation provided by Singh et al. 2018). The grid- world is 8x8 with traffic junction. The rate cars being added has a minimum and maximum of 0.1 and 0.3. We use the easy version with two arrival points and 5 agents. Agents heavily penalized a collision happens and have only two actions, namely gas and brake.
All algorithms trained for 20 million environment steps with maximum of 20 steps per episode.
13

---

Under review as a conference paper at ICLR 2023
A.2 ARCHITECTURE AND HYPERPARAMETERS
observation
iacoming mressages
Obsarvation
Message Encoder
messaget
GRU
Value Head
Policy Head
Message Head
outgoing message
Grounding Module
Grounding module ourput
Figure 6: Architectural illustration for algorithms with communication. To remove communication, the message head is disabled. Grounding module is only relevant to CACL and AEComm. The former is loss function and the latter is a decoder to reconstruct the encoded observation.
Figure
fillutttil illustrates the components the architecture used in this work, similar to (Lin et al. 2021) A message head is only used for algorithms with communication, namely CACL, AEComm, PL and DIAL. The Grounding Module refers to mechanisms to ground the messages produced by the
message head, used in CACL and AEComm. Unless specified otherwise, we fix all hidden layers to be a size of 32.
We experimented with using the output of the GRU, hidden state, condition the message head. Empirically we found that directly conditioning on the observation encoding, as in Lin et all (02T)) lead to more stable learning dynamics.
The observation encoder output values of size 32. For Predator-Prey and Traffic Junction, a one- layer fully-connected neural network is used as observation encoder. For Find-Goal, same Lin et al. (2021). we use two-layer convolutional neural network followed by a 3-layer fully-connected neural network.
For the message encoder, it outputs values of size 8 in Predator- Prey and Find-Goal with one hidden layer. outputs values size in TrafficJunction with two hidden layers. These configurations are selected based on the best performance of the baseline communication learning algorithm used DIAL. Messages received are concatenated before passing to message encoders. For all the meth- ods with communication. they produce messages of length 4 (D 4) with sigmoid function as activation. All models are trained with the Adam optimizer (Kingma & Ba 2014).
Table lists out the hyperparameters used for all the methods.
A.3 POSITIVE LISTENING
This section describes the loss function we implemented for positive listening, based on Eccles et al. 2019). Given two policies and n¹ of agent where the latter is the policy with messages zeroed in the observations, and trajectory of length T, the positive listening loss is written as:
14

---

Under review as a conference paper at ICLR 2023
Learning Rate
0.0003
Epsilon for Adam Optimizer
0.001
0.99
Entropy Coefficient
0.01
Value Loss Coefficient
0.5
Gradient Clipping
2500
1) for CACL
0.1
for CACL
0.5
Loss Coefficient for PL
0.01
Number of Asynchronous Processes
12
N-step Returns
5
Table 6: Table for hyperparameters used across methods
T Σ
LP1 =
Σ
(alT,)) + ( (alT)) log (Tg()) (alr))
(4)
T J
LaE
where in inner summation, the first term is the LI Norm and the second term is the cross entropy loss.
A.4 PROTOCOL REPRESENTATION PROBING: 1-LAYER
Table 7: Classification results of the two probing tests in the methods with communication. 1-layer neural networks are used for probing
environment, comparing all
DIAL
PL.
AEComm
CACL
Goal Visibility Test
94.21 % + 2.68
96.93% + 3.14
96.27% + 3.98
87.65% 3.86
Goal Location Test
52.29% + 5.25
53.65% 9.60
48.16% + 7.34
79.18% 5.63
Table shows the same results for the two probing tests in section 5.3 except here use 1-layer neural network instead of 2 layers. We observe significant dips in performance across all methods. Particularly, CACL becomes worse than the baselines in the easier Goal Visibility Test. However, CACL remains superior in the more difficult Goal Location test by an even bigger margin than the results in t
15


Reviews:
[{"review_id": "HWhDcwY8AhZ", "reviewer": "Reviewer_Tpss", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "this paper proposes to develop linguistic communication between decentralized agents via multiple view contractive learning. which is an interesting problem. it is suggested to combine the contractive loss with the RL loss with a predefined hyperparameter. ", "summary_of_the_review": "there is not linguistics/linguistic-based communication. the objective is standard. there is no general  principles/conclusions.", "strength_and_weaknesses": "Strength:\nThe problem of linguistic communication between decentralized agents is important \n\nWeaknesses:\n1.  \"Linguistics studies language and its structure, including morphology, syntax, phonetics, and semantics\". I do not see any of these in the paper, even though \"linguistic communication\", linguistics and similar terms are used to describe the advantages of this paper and novelty. There is not language and not linguistics, but the standard multiply view constrictive learning of observations.\n\n2. contrastive (and multi-view) losses can be useful in many scenarios. this paper shows one more such scenario. I am not sure this is enough for a full paper in ICLR. In my opinion the contribution to the knowledge and understanding of linguistic communication for decentralized RL is minor. \n\n3. The current comparison between different approaches is indirect evidence only, because a) there can be different number of parameters in DNNs, (which is not mentioned), b) with more powerful harsher one can achieve better results (more training..). \n\n4. there are no conclusions but rather a summary of the paper. \n\n\n\na few minor comments regarding the phrasing:\n\n1. \"..but many works focus on simple, non-MDP environments.\" what simple non-MDP environments do you mean? \n2. \"...requires fewer assumptions about other agents...\" which assumptions about other are assumed in centralized and decentralized setting? that will help to understand \"..fewer..\". It is desired to provide evidence.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "BTOlGHCdxh", "reviewer": "Reviewer_dj4f", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In the paper, the authors propose to communicate between different agents using contrastive learning in a decentralized setting. \n\nAs their main contribution of making agents communicate based on contractive learning for multi-agent reinforcement learning in the decentralized setting, they evaluate the proposed model in different environments and obtain promising results. ", "summary_of_the_review": "It is an exciting job to make agents communicate using contrastive learning in a decentralized setting. However, the current version is not suitable for ICLR.\n", "strength_and_weaknesses": " The paper proposes a communication method based on supervised contractive learning with sufficient experiments to evaluate the proposed method.\n\nThe weakness of this paper:\n1. The problem investigation in Section III is confusing, especially in the usage of symbols with lower and upper case, such as s and S for real states. \n2. Meanwhile, is it right for the transition function P->SxAxS?\n3. Some citations are in the wrong formation, such as the caption of Fig. 3 for DBSCAN and t-SNE.\n4. Compared with the workshop paper “LEARNING TO GROUND DECENTRALIZED MULTIAGENT COMMUNICATION WITH CONTRASTIVE LEARNING”,  the current version is updated less in the method.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "BMQQ0UuzDh", "reviewer": "Reviewer_xPcm", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a self-supervised learning objective using a contrastive loss to learn a medium/language for communication between agents to complete a task (in a decentralized training setting). Similar to Lin et al. (2021), the messages here are proposed to be encodings of observations. To learn these encodings/ messages, the contrastive objective uses positive anchors sampled from the current trajectory around a window and negative samples from other messages in the batch. The authors test their method (CACL) on three MARL tasks: Predator-Prey, Find-Goal, and Traffic Junction. Through different analyses for probing representations and testing for zero-shot coordination, the authors present their method's effectiveness.", "summary_of_the_review": "The paper is thorough in comparing with baselines and probing to understand the representations learned by the proposed method. The paper lacks a proper discussion/justification for the setting, the impact of the method used, its implications for different tasks or the emergence of language. ", "strength_and_weaknesses": "### Strengths:\n\n- The authors provide a thorough analysis of their proposed method through various embedding probing techniques:\n    - qualitative plots for different scenarios,\n    - quantitative measures of similarities for messages,\n    - transfer of message encodings to auxiliary tasks\n- The authors also do a great job of comparing their method to well-known techniques in the literature and show how their approach is better than the others for a wide range of measures\n- The section on zero-shot crossplay and the associated analysis was quite interesting to read!\n- **************************Clarity:************************** The paper, the methods, the metrics were quite easy to read and understand.\n\n### Weaknesses:\n\n- I think the paper's main weakness lies in justifying the approach's motivation for the relevant problems. I found the introduction and discussion relatively weak in their description of the broader motivations of the work, particularly those relevant to emergent communication: applications/ transfers to the real world and understanding human communication.\n- More importantly, similar to Lin et al. (2021), it wasn't clear if this was actually communication; sending the complete representation of the observation without accounting for the other agent as there is no bottleneck. A potential baseline that could test this could use a contrastive loss to learn observation embeddings (w/o multiagent dynamics), freeze that and train agents to communicate by using these. It would help us understand how the setting of sending messages and communicating changes representations.\n- **************************Novelty:************************** The approach essentially combines Lin et al. (2021) and with a contrastive info-NCE objective to learn representations.\n\n### Questions:\n\n- In the crossplay setting (Table 4), the interplay agent outperforms the intraplay agent for CACL-AEComm and CACL-PL. Why do the authors think that the interplay agent outperforms agents specifically trained to communicate with each other?\n- Is there a systematic split between the training tasks for the agents and the evaluation ones?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject