Learning to Cooperate and Communicate Over Imperfect Channels
Authors: Jannis Weil
Gizem Ekinci
Heinz Koeppl
Tobias Meuser
Venue: Submitted to ICLR 2023 (2023)
Keywords: multi-agent systems
deep reinforcement learning
emergent communication
imperfect communication channels

Abstract: Information exchange in multi-agent systems improves the cooperation among agents, especially in partially observable settings. This can be seen as part of the problem in which the agents learn how to communicate and to solve a shared task simultaneously. In the real world, communication is often carried out over imperfect channels and this requires the agents to deal with uncertainty due to potential information loss. In this paper, we consider a cooperative multi-agent system where the agents act and exchange information in a decentralized manner using a limited and unreliable channel. To cope with such channel constraints, we propose a novel communication approach based on independent Q-learning. Our method allows agents to dynamically adapt how much information to share by sending messages of different size, depending on their local observations and the channel properties. In addition to this message size selection, agents learn to encode and decode messages to improve their policies. We show that our approach outperforms approaches without adaptive capabilities and discuss its limitations in different environments.

Under review as a conference paper at ICLR 2023
LEARNING TO COOPERATE AND COMMUNICATE OVER IMPERFECT CHANNELS
Anonymous authors Paper under double-bling review
ABSTRACT
Information exchange in multi-agent systems improves the cooperation among agents, especially in partially observable settings. This can be seen as part of the problem in which the agents learn how to communicate and solve a shared task simultaneously. In the real world, communication is often carried out over imperfect channels and this requires the agents to deal with uncertainty due potential information loss. In this paper, we consider a cooperative multi-agent system where the agents act and exchange information in decentralized manner using limited and unreliable channel. To cope with such channel constraints, we propose a novel communication approach based on independent Q-lcarning. Our method allows agents dynamically adapt how much information to share by sending messages of different size, depending on their local observations and the channel properties. In addition to this message size selection, agents learn encode and decode messages to improve their policies. We show that our approach outperforms approaches without adaptive capabilities and discuss its limitations different environments.
INTRODUCTION
In multi-agen systems, cooperation and communication are closely related. Whenever a task requires agents with partial views cooperate, the exchange of information about one's 's view and intent can help to reduce uncertainty and allows for more well-founded decisions. Communication allows agents to solve tasks more efficiently, and can even be necessary to achieve acceptable results (Singh et al., 2019). As an example, consider safety-critica autonomous driving scenario (Li et al.) (2211) l). By letting the cars exchange sensor data or abstract details about detected objects in the scene, occluded objects can be considered in the planning processes of all cars and reduce the risk of collisions.
Multi-agent reinforcement learning (MARL) comprises learning methods for problems where multiple agents interact with a shared environment (Buşoniu et al. 2010 Hernandez-Leal et al. 2019). The goal is to find an optimal policy for the agents that maximizes the outcome of their actions with respect to the environment's 's reward signal. Key challenges in MARL include non-stationarity (Papoudakis et al. 2019), the credit assignment problem (Zhou et al.) (020) and partial observability (Oroojlooyjadid & Hajinezhad] 2019). We focus on cooperative environments with partial observability.
As communication is essential in cooperative environments, many works include a predefined information exchange between agents (Melo et al. [00 Schneider al. 2021). Additionally, there is ongoing research to include learnable communication into MARL approaches. Pioneering work gave first empirical evidence that communication between agents can be learned with deep MARL (Foerster et al. 2016; Lowe et al. 2017 Sukhbaatar et al. 2016). This enhances the performance on existing environments and allows to address a new class of problems that require communication between agents. Building upon these ideas, many researchers proposed methods to improve the performance and stability of these approaches (Gupta et al. 2020 Jiang Lu 2018 Li ct al. 202I). While related work investigates effects of using different fixed message sizes (Li et al.) 2022) and multiple communication rounds (Das et al. 2019), selectively sending messages (Singh et al. 2019). and sending messages only to other agents in their proximity (Jiang & Lu 2018), most of these approaches are designed for communication channels without capacity limitations or message losses. Recent approaches started to investigate such settings, e.g. by learning central controllers for coordinated

---

Under review as a conference paper at ICLR 2023
access to communication channel (Kim al. 2019). To the best of our knowledge, there studies on message size adaptation to improve mullti-gennt communication over imperfect channels.
In work, address this gap by investigating cooperative MARL setting in which agents communicate over an unreliable and limited channel. We focus on agents learning when, what and how much to communicate over imperfect channels in a decentralized manner. The key challenge here is to determine how to utilize the limited capacity efficiently and cope with the lack of reliability, in order maximize the benefit for the cooperative multi-agent problem.
With this paper, we provide a novel approach to address this challenge. Our contributions are as follows: (i) we propose novel communication approach that allows for an adaptive message size selection while learning the message encoders and decoders, (ii) we introduce discrete communication trained with the pseudo-gradient method, (iii) we analyze the effect of different message types and message sizes, (iv) we introduce POMNIST, a fast MNIST- benchmark environment for communication, (v) we show that agents adapt to the given communication channels in POMNIST and show limitations of our approach in the traffic junction environment.
2 RELATED WORK
Agents in MARL can exchange information a) with implicit communication, and b) with explicit messages that are forwarded between the agents. Implicit communication refers to exchange of information without separate communication actions, e.g. through the agents' regular actions and observations (Foerster et al. 2019) or with a joint policy (Berner et al. 2019).
Within the scope of this paper, we focus explicit communication. This can further be divided into i) continuous communication with real-valued messages and ii) discrete communication with a finite set of messages. In the context of deep learning, exchanging continuous messages allows for back- propagation across different agents (Sukhbaatar et al.] (016). This results in significant performance improvements in partially observable environments, where agents can benefit from coordination or the exchange of local information. Recent approaches include restricting communication to agent groups (Jiang & 2018) and specific topologies (Du et al., 202I), deciding when to send messages (Liu et al. 2020; Singh et al. (219)) and estimating the importance of messages with attention (Das)
ct al. 2019 Li et al. 2021] Rangwala & Williams 2020).
Discrete communication with finite message sets allows for more fine-grained control of the used data rate in limited communication scenarios and is the focus of this paper. In order to facilitate backpropogation for discrete communication, Foerster et al. (2016) regularize continuous messages with noise during training and discretize them during evaluation. In their experiments, this yields better results than extending the action space with communication actions. Differentiability can also be retained by sampling messages from a numbel-softmaal distribution (Jang et al. (017)) instead of a categorical distribution (Gupta et al.) [2000] Lowe et al. (017). [i et al. (2022) aim to compensate for the message discretization with skip connections. Their results also suggest that the message size has neglectable effect on continuous and significant effect discrete communication.
Our work combines learnable communication with deep Q-learning to adaptively select the message size based on the observations given at each step. We consider an uncoordinated channel of limited capacity and demonstrate how agents can benefit from message size selection in such settings. Related works consider limited communication via centrally controlled channel of limited capacity (Kim
et al. ).19), by pruning messages (Mao et al. 2020) or with regularizations based on the length of messages (Freed et al.) 2020). Hu et al. (202)) show empirically that controlling whether to send messages improves communication in slotted p-CSMA channel. It is unclear how and if agents can choose from different message sizes to improve their communication efficiency in unreliable channels of limited capacity. We address this research gap with our work.
Although not the focus of this work, efficient use of imperfect channels can also be improved by coordinating the agents' access to the channel. For example, Kim et al. (2019) and Wang et al. (2020) consider this by learning a centralized scheduler for multi-agent communication. The classical commu- nication literature compromises multitude of sophisticated mechanisms for medium access control (Huang et al., 2013 Kumar et al. 2018). We note that such schemes can be used in conjunction with adaptive message size selection and leave further exploration of this combination to future work.
2

---

Under review as a conference paper at ICLR 2023
3 ADAPTIVE COMMUNICATION
In this section, we formulate the cooperative multi-agent problem with message size selection and describe our adaptive communication approach. Fig. summarizes the information flow in the multi-agent system with adaptive communication.
3.1 PROBLEM FORMULATION
We consider a network of N agents in a cooperative and partially observable stochastic game (Hansen) al., 2004).. Each agent i at time step based on the state E S. As none of the agents has direct access to the state, it is essential communicate with others in order to make better decisions. However, the agents are not provided with any prior information on how to communicate. During training, they must leam to exchange meaningful and beneficial information and correctly decode the incoming messages.
{1 [l,..., ...., N has private and partial observation of O (oj 8t)
We consider stepwise communication where each agent sends a single message mi E MM of size o E C Ng to all agents. Note that a message of size zero corresponds to not sending any message. Each agent can choose different size o in each step, but we omit superscript and subscripts in this section for notational simplicity. The message space .M can be continuous M = [-1, 1] or discrete M {0, 1}. Once all agents have created message, the messages are transferred using given channel. The success of sending a message depends on the channel model, which receives all messages at the current timestep as input and outputs the set of successfully transmitted messages M, S {m!,... m. }. In each step, agents receive messages that have been successfully transmitted in the previous step, except for their own message Mi M,-1 {mf_1)- Each agent takes action a' E A sampled from its stochastic action policy n² based its observation of and incoming messages, of ~ n² <'(aj oj, Mi-1). The actions are then applied in the environment which causes transition to the next state based on the transition probabilities. $(+1 P(s St, a²) with a, =
a, ). In parallel to taking an action, each agent chooses a message size from the given set of sizes & using its stochastic message size policy Tyg. conditioned on the current observations and incoming messages, ip T ( of, M)-1). The agent then creates message mi M of size yp with specispe message encoder m; fú (o), MI-1). The messages are broadcasted to all other agents using the given communication channel and have no direct influence on the environment.
depending on the state of the environ- ment 8t and the joint action The discounted return of agent i for step t is defined as ax) with time horizon T N and discount factor y [0, 1]. We consider cooperative environments in this paper. Agents cooperate by choosing actions that increase the rewards of other agents and by exchanging local information via messages. Each agent i optimizes their action policy to maximize their return G This is complemented with finding message size policies क f क and message encoders fo Uier,pea ( that jointly maximize the
Each agent receives individual rewards R' (St, az)
G >
expected mean discounted return of all agents for a given channel, i.e. max E
ENN G
N
3.2 ARCHITECTURE
With adaptive communication (see Fig. []. we propose a deep MARI. architecture to jointly learn action policies, message size policies and message encoders, only based on the reward signal. Our approach comprises two modules to learn communication: (a) a message-size selector and (b) a message encoder. This is complemented with action selector to perform actions in the environment. We first describe how the agent's 's input is processed and then explain the modules.
Input processing The input consists of the observation from the environment of and the successfully transmitted messages from the last step, excluding the agent's is own message M An environment- specific observation decoder maps the observation to a vector fixed length. The message decoder maps all incoming messages to vector of fixed length by taking the mean of all messages padded to the same length, including a one-hot encoding of the message's 's length. The vectors from the observation and message decoders are concatenated and processed by core module. Details regarding the concrete architecture can be found in the appendix in Sec. A.1 The core's 's output zi is the input to the following modules.
3

---

Under review as a conference paper at ICLR 2023
perform step
agent step 4 ++3
M
input
observation decoder
message decoder
Environment
new ++1
aggregate Og
observation
of
action
message selector nr
Communican cuuuuuicaiion
Agent i
Adaptive
messages
M
message
Mi
(m. m
successtully
irput
action selector
message
try send
encoder
Communication
drapped
losses
Figure 1: Overview of the multi-agent system with adaptive communication. At each step, agents messages. Based on this input, they select an action and create
receive an observation and message of selected size Φ. Actions are used to perform steps in the environment, messages distributed stepwise according to a given channel model. The channel defines whether messages are dropped transmitted correctly. The gradients from the action and message size selector losses of an agent that receives a message are backpropagated to the sender this message.
Message size selector The message size selector employs independent deep Q-networks (Tampuu et al., 2017), where each agent i independently and simultaneously learns its own Q-function. In our case, we want to select the message size that leads to the highest expected discounted returns for the receiving agents. The message-size-valee network Q'xx), so; 0z)) conditions the agent's 's individual core output o; and returns value for each message size Φ. is parameterized by 0g.
The message size selection cannot have any effect the rewards of the current step, as the created messages will be received in the next step. Therefore, we propose to use an offset of one step in the target for the message-size values. Additionally, the sending agent does not receive its message in the next step, the sent message cannot have any effect on this reward either. Based on these considerations, we define the message-value target of agent in step / for message size yo
Di(P)) = E
Σ
G
R¹ R5 azt1))
po -
We train the message-size-value network to approximate the target Eq, bby iteratively minimizing the mean squared eтror multiple sampled training episodes for all agents, Eq.
(b()) = = (Q' 0o) Di((!)))
(2)
The agent's message selection policy TT. uses Boltzmann softmax exploration with decreasing temperature + 0 (Sutton & Barto 2018) based on the predicted message-size values, i.e. message size io Ø is chosen with probability proportional exp(Q' (x), so; fz)/n).
Message encoder After the message size yp is selected, the agent creates its message m using message encoder network m = fg (ot, m:-1i parameterized by Ep. For a given set of available message sizes Φ, the agent learns to encode messages each message size yp E & {0). This is achieved through multi-headed neural network instead of separate networks, which allows learning both shared and private parameters for different message sizes while reducing the complexity of the model. It should be noted that the agents learn how form messages for all message sizes during training, which is the key idea leading to adaptivity in this work. We describe the message types that define the output of the message encoder in Sec. 3.3
4

---

Under review as a conference paper at ICLR 2023
Action selector The action selector represents the agent's policy x' and is jointly learned with message selection and encoding, but it is conceptually independent and not the focus of this paper. We evaluate approach with (a) an é-greedy action selector based Q-values that trained with the discounted monte carlo return G; as target and (b) a stochastic action selector similar to (Singh al. 2019) that is trained via REINFORCE (Williams 1992), including learned value baseline and an entropy bonus loss to encourage exploration. Each action selector has its own loss function 2 to maximize the agent's individual return. This loss is backpropagated through the core train the observation and message decoders. Through the message decoder, the gradients are passed to the messages' senders if supported by the message type. This enables the sending agents to adapt the content of their messages to improve the actions of the recipients.
Parameter sharing We employ centralized learning and decentralized execution (Foerster et al. 2016 Kraemer & Banerjee, 2016 Oliehoek al. 2008 Rashid et al. 2018) minimize the mean losses Li and L over all agents. During learning, the trainer has access to the reward of all agents to optimize the message-size size policies. These policies are represented by a single model. However, the execution is performed in a fully decentralized manner. The agents make the decisions for actions and message size solely based on their individual observations and the incoming messages. We allow agents to learn different behavior by adding agent-specific features to the observations. This approach reduces the number of parameters and is computationally more efficient.
3.3 MESSAGE TYPES
Messages are either continuous or discrete and the choice of encoding method can affect the commu- nication. We refer to the combination of message space and encoding method as message type. In this section, we present the message types that we consider in conjunction with our approach.
Discrete communication with Q-values Communication can be seen as an extension to the agent's 's action space. Accordingly, standard RL methods like Q-learning can be applied to learn communication actions (Foerster et af., 2016). We use the mean discounted return of the receiving agents equivalent to our message size selection to learn Q-values for each message value. Therefore, this message type is not differentiable and the agents have to learn how to encode messages purely by trial and error. To avoid confusion, the loss for this message type is omitted in Fig. [ This method does not scale well, as increasing the message size leads to an exponential increase the number of messages values and message Q-vluus that have to be learned.
Continuous communication With this message type, agents send real-valued messages to each other. As they are used as imput to networks representing different agents, real-valued messages allows gradient flow across agents (Foerster et al.] 2016). as indicated on the right side of Fig. [ This serves as direct feedback on messages from the recipients to the sender and allows the message encoder network to learn from this feedback. Since the real-valued messages are the most expressive message type used in this work, we expect it to achieve the best results and to serve as baseline.
Discrete communication with pseudo-gradient method The exchange of discrete messages with Q-values limits the agents, since the discrete communication channel is non-differentiable and the backpropagation algorithm cannot be applied for end-to-end training across agents. The pseudo- gradient method was first proposed for recurrent neural networks with discrete activations (Zeng) ctt al., 1993) and later applied to multi-layer neural networks (Goodman & Zeng 1994). This method approximates the gradient of the discrete activation function using the true gradient of an analog activation function as a heuristic hint. In our work, we employ the pseudo-gradd gradient method with (G(m) = 2 1 ((tnh((m) (m) > 0}- for message features m in the forward pass and directly pass the gradients to tanh in the backward pass. This allows to benefit from end-to-end backpropogation.
Discrete communication with DRU The discretise/regulariss unit (DRU) proposed by Foerster et al. (Foerster et al. 2016) aims at retaining differentiability with discrete messages. During training, the DRU regularizes continuous messages m by adding noise DRU (m) = (ogisticcNV(mm o²)) from a normal distribution N N(m, o?) with mean and standard deviation The authors argue that noise with o > effectively regularizes the message to a single bit. During execution, the DRU then discretizes the message using element-wise threshold DRUexecution (m) 1 (m > 0}.
5

---

Under review as a conference paper at ICLR 2023
3.4 LIMITED COMMUNICATION CHANNEL
Communication resources are limited in practice. We propose to combine the problem of learning how to communicate with limited and lossy communication channel. While greater message sizes typically yield better performance in an unlimited scenario, always selecting the highest message size might lead to congestion and dropped packets in real networks. Missing and delayed messages could have a negative impact on the cooperation. Instead of assuming perfect communication channel like most related work, we consider a limited channel with stochastic collisions between messages.
We use a slotted channel model that is parameterized by channel size C' E N defining the available slots (o.... C' 1}. The communication synchronized with the environment's steps. If an agent decides to send message of size vp $ {0). this message is inserted into contiguous slots using a fixed stochastic channel access mechanism. Messages are inserted into the channel independently and the starting slot is chosen uniformly from (0, yp ..p.... _ into the communication channel, i.e. yp > C, are dropped. After the insertion of all messages, if two or more messages shares at least one slot, collision occurs and all involved messages are dropped.
(p). Messages that do not fit
We have chosen this slot assignment over a completely uniform assignment to reduce the number of collisions in the channel. However, note that the channel is still highly unreliable and the expected throughput is way lower than the channel size, depending on the agent's message sizes. This is known phenomenon of early decentralized slotted channel access models with uncoordinated clients (Tanenbaum & Wetherall 2011). More details are provided in the appendix in Sec. A.5
This simple channel model suffices to study the behavior of learning agents with limited and unreliable communication, as we will show in the experiments section. More sophisticated channels and channel access mechanisms could be considered in future work.
4
EXPERIMENTS
In this section, we discuss the effectiveness of adaptive communication in different environments. We conduct all experiments with two Intel Xeon Silver 4214 and NVIDIA GeForce RTX 2080 Ti.
4.1 PARTIALLY OBSERVABLE MNIST (POMNIST)
We first evaluate our approach with cooperative digit prediction based on the MNIST dataset (LeCun al. 1998). It consists of grayscale images of hand-written digits with 28 28 pixels and their corresponding labels. It is split into train and test sets with 60,000 and 10,000 examples respectively.
Environment We introduce the partially observable MNIST (POMNIST) en-
agent 0
agent 1
vironment that splits each MNIST sample into partial views of the same size (see Figure 2p. These views are fixed during training and evaluation, and each agent is assigned to one particular view. Each episode consists of two steps: In the first
step, agents observe their local view and broadcast message all other agents. In the second step, the agents independently predict the digit based on their obser- vation and the received messages. Agents receive a reward of if their prediction is correct and 11 otherwise. The agent's reward in the first step is set to zero. The mean return can mapped to accuracy with accuracy = (mean return + 1)/2.
agent 2 agent 3
Figure 2: A digit is split into non- overlapping views for four agents.
POMNIST allows to construct environments of varying difficulty and reliance on communication. The higher the number views and agents, the less information is included in each agent's 's observation. Therefore, they have to increasingly rely on communication to make correct predictions.
Training We use the Q-value action selector and train our approach for 2000 iterations using the Adam optimizer (Kingma & Ba 2015) and learning rate 0.001. The core is a dense layer with a skip connection. We train with 2048 parallel environments, resulting in batch size of 2048 for each agent in each iteration. During training, we randomly draw samples with replacement from the train set. For testing, run exactly episode for each sample in the test set. A complete training run takes around 2 minutes without communication to 4 minutes with adaptive communication. The message-size selection begins with exploration phase until iteration 1200, details are in Sec. A.1

---

Under review as a conference paper at ICLR 2023
Cont nusas Ponstolindeett
3.3
Dacils
5.
Moin ugent
0.75
Contuuut
0.79
DRU
0.5
0.00
(9)
(1) 64) 01 0.2.2
(4) (c)
(0. (0.3,2443
Number of berattom:
(a)
(b)
Figure 3: Results for training (a) and testing (b, c) in POMNIST with 4 agents and channel size 8. (a) shows mean return during training, (b) mean return of different message types with fixed and
adaptive message sizes, and (c) shows channel metrics for and bar shows the mean over runs, the transparent area and the whiskers represent the standard deviation. The black dots in (c) show the metrics' values for random message size selection.
messages. Each line
Results A single agent with full view achieves a mean return of 0.979 + 0.002 on the test set. We consider a configuration with one horizontal and one vertical split, resulting in agents. The mean return on the test set for 4 agents without communication is 0.633 + 0.003. We focus on message sizes C {0, 1, 2, 4} and explain this selection in the appendix, see Sec. A.2
To investigate how adaptive communication performs in limited channels and how it compares to fixed message sizes, we consider a case example with channel of size 8. Fig. 3h shows the mean return of adaptive approach with message sizes Ø = {0, 1, Confirming our initial expectations, the continuous messages show the best performance, followed by discrete communication with the gecudo-gradient method. The DRU shows the lowest performance.
2,2,4 4] during training for different message types.
Fig. (b) and (c) show the results on the test set after training. Fig. 3p shows that adaptive com- munication achieves higher returns than the approaches with fixed message sizes. For the analysis of the channel metrics in Fig. Be. we focus on the pseudo pseudo-gradient gradient message type as achieved the highest return among the discrete communication methods. The left bars show the mean number of dropped messages in each step and the right bars show the throughput. The throughput is defined as the average number of slots occupied by messages that are not colliding. The number dropped messages increases proportionally to the message size for single message sizes. The throughput increases from message size to 2, but then decreases from to due to the high number of collisions in the channel. It can be seen that the number of drops and the throughput for adaptive communication are comparable with the results for message size 1. When compared to random selection of message sizes in {0, 1, 2, 4). shown as black dots, we can see that adaptive communication decreases the number of drops on average and increases the throughput. is noticeable that the standard deviations for the mean number of drops and throughput in Fig. 33 are considerably higher for adaptive commu- nication. This can be seen as an indication that individual runs might learn different strategies for the message size selection, while still outperforming the approaches with single message sizes.
Next, we analyze the effect of different channel sizes by ex- periments with adaptive message sizes Ø- {0, 1, 4} using the pseudo-gradient method, Fig. 4 visualizes the through- put during training and Tab. [ shows the performance metrics on the test set. In all cases, the agents are able to increase the throughput with adaptive communication, compared to the exploration phase. When using an unlimited channel, the throughput quickly reaches the highest value after the initial exploration phase. As expected, this leads to the highest return. By introducing the limited communication channel, the prob- ability of successful transmission decreases with the channel size. This can be observed both from the increasing number of dropped messages and from the decreasing throughput. The limited communication ability is also reflected the decreased mean return, proportional the channel size.
Moantoo wwrpopoo
3
Sto
Number of Deeerentioon
Figure 4: Mean throughput during training with different channel sizes. Each line shows the mean over 5 runs, the transparent area represents the standard deviation.
7

---

Under review as a conference paper at ICLR 2023
Table Performance metrics for pseudo-gradient messages with different channel sizes over runs.
Channel size C
inf
32
8
mean return
0.94 =0.00
0.88 #0.00
0.83 #0.01
0.80 +0.03
# of drops
0.00
0.63 10.05
1.23 10.23
1.36 10.42
pos listening
0.26 10.01
0.17 10.01
0.14 10.00
0.13 1:003
pos signaling
0.66 10.05
0.75 10.02
0.83 10.03
0.69 10007
throughput
11.84 1.1.1
6.55 10.35
2.60 1.010
1.55 100.99
mean msg size
2.68 =0.31
2.16 #0.14
1.62 +0.29
1.39 +0.15
The positive listening metric (see Sec, A.3) quantifies the impact of communication on an agent's behavior. As expected, this impact is highest for the unlimited channel, since the agents send messages of higher sizes more frequently and are able to share more information. The positive signaling metric (see Sec. (e.) quantifies the correlation between an agent's outgoing messages and its actions. Interestingly, positive signaling increases with decreasing channel size, except for channel size 4. As the agents send fewer messages as the channel size decreases, this could lead them sending messages that are more specific to their local information. However, upon inspection of our data, we see that the agents restrict themselves to sending messages of size and 2 with channel size 4. These message sizes might not be enough to encode specific situations, which could explain the decrease in the positive signaling for channel size 4. We observe that even in an unlimited channel, the mean throughput is below 12 instead of its maximum value of 16. The mean message size in Tab. shows that the agents do not always send messages of the maximum size 4. This could indicate that agents learn judge the relevance of their local information, and that they avoid confusing other agents by sending shorter messages when this local information cannot contribute to the success of the team. We also hypothesize that the agents encode information in the selected message sizes. The following ablations support this such that the agents perform better with adaptive message sizes and zero content compared to randomly selected sizes.
0.90
We show an ablation study of our approach in Fig. 5 The bar none {0). The others are adaptive
the baseline with message size communication with message sizes + = (0, 1, 1, 4). C 8 and the following modifications: random selects random message sizes but learns the message encoders, zeros learns the message size selection but the encoders always return zero, adaptive is our approach that jointly learns the message size selection and message encoder. The increased return from none to random shows the benefit of learning message encoders and the increment from random to adaptive shows the benefit of the adaptive message size selection. Surprisingly, sending messages with content zero of different sizes outperforms the random mode. This indicates that the message size itself can be used to share information between the agents. As there are four different message sizes, selecting a message size can be seen as discrete communication with 2 bits of information. However, the information about a message's s size lost when messages are dropped.
0.85
Meen agent return
0.80
0.75
0.70
0.05
0.60
Communication method
Figure 5: Mcan agent return of the pseudo- gradient method with ab- lations over 5 runs.
4.2 TRAFFIC JUNCTION
The traffic junetion environment by Sukhbaatar et al. (2016) has been used
in several works with slight changes (Das et al. 2019 Singh et al. 2019) to
study communication in MARL. We use the version by Singh et al. (0019)
Agent
Environment the traffic junetion environment (see Fig. 6), agents control cars that move along predefined routes in gridworld. Their goal is to reach the end of the route without colliding with other agents. The cars are controlled by discrete action that is either gas or break. Action gas advances the car to the next cell in its route, action break leaves the car where is. If the current number of cars is lower than a predefined number of agents, new cars spawn with probability p at the beginning of each lane. When a car spawns, it is assigned predefined route that starts this position. An active
Figure 6: Traffic june- tion with two agents that control two cars.
8

---

Under review as a conference paper at ICLR 2023
car is removed when it reaches the end of the route. The reward for agent with an active at step t is defined as -0.017, 10C;, where Tj is the number of steps the car of agent i has been active for and C₁ is the number of cars that are involved in collision with agent at step t. Agents that control an active car observe their last action, the route id and information about the agent's position, including the number of cars at this position. They do not observe positions of other agents and need to communicate to prevent collisions. We focus on a two-lane scenario with one route each and up to 5 agents with spawn probability p= = 0.3. Each episode ends after 20 steps.
Training Our configuration is based on Singh et al. 2019). We use the stochastic action selector and train for 2000 iterations with 128 parallel environments using the Adam optimizer and learning rate 0.001. The core is extended by a gated recurrent unit (Cho et al. 2014). A curriculum increases p from 0.1 to 0.3 between training iterations 250 and 1250. A training run takes around 30 minutes without communication to I hour with adaptive communication. Details are in Sec. A.1
Results The performance in traffic junction is measured in terms of success rate that is defined as the percentage of episodes without any crashes. Singh et al. (2019) report success rate of to 30.2 0.4 without communication and 9.7 when agents communicate their continuous-valued hidden state of size 128. We consider message sizes & C (0. 32, ]28] and discrete communication method. We reproduce the results for the baseline without communication
with the pseudo-gradie with a mean success rate of 30.27 + 1.34. In an unlimited channel, we slightly outperform the baseline with a mean success rate of 97.04 + 1.75. The results with communication are shown in Tab. 2] we focus on the highlighted parts. Communication greatly improves the success rate compared to the baseline without communication, even with messages of size 32 in a limited channel. The success rate for message size 128 under a limited channel declines compared message size 32, while showing an increasing number of drops. This suggests that reliable communication is critical in this environment. Our approach achieves higher mean success rate than random message size selection, but the standard deviation of its success rate, throughput and mean message size are very high. We conclude that our approach unstable in this environment. We think this could be improved with alternative formulations of the message- size value, e.g. by reducing the variance of the target.
Table 2:
Results for escudo-gradien messages. Mean over 5 runs with 2048 evaluation episodes.
channel size C' message sizes db selection method
512 {32} fixed
512 {128) fixed
512 {0, 32, 128} adaptive
512 {0,32, 128] random
OC
{128) fixed
success rate
87.33 11.41
64.67
11.17
82.78 19.71
63.17 11.37
97.04
1.77
# of drops
0.48 10.01
1.54
10.01
0.64 10.49
0.63 10.01
0.00
10.00
throughput
78.63 =0.47
175.78
+0.53
93.69 #40.68
98.69
#0.71
391.54
#5.17
mean msg size
32.00
=0.00
128.00 +0.00
55.97 865.8
53.37 +0.08
128.00 =0.00
5 CONCLUSION
With this work, we investigate communication over unreliable and limited channels in MARL. We propose an adaptive communication mechanism that allows agents to choose different message sizes depending on their local observations and incoming messages. We introduce the POMNIST environment and show that adaptive communication achieves higher returns than non-adaptive approaches in a limited and unreliable channel. We compare different message types and find that continuous communication performs best, followed by discrete communication with the pscudo- gradient method. The traffic junction environment indicates limitations and open challenges of adaptive communication. Although our approach is better than random message size selection, it is unstable and on average inferior to choosing fixed-size messages in this environment. Future work could investigate the applicability of adaptive communication to different domains and improve its stability, e.g. with alternative formulations of the message-siae sizze selection target. Possible extensions with respect to the communication channel include shared medium access control mechanisms and the consideration of more sophisticated channel models. Another possible direction is the interpretation of the message content. A comparison with traditional machine learning methods would also be valuable.
9

---

Under review as a conference paper at ICLR 2023
Reproducibility Statement We provide our implementation with the supplementary material together with a README file that describes how to set up and run our code. The included experiment configuration files allow readers reproduce all experiments that reported in the main paper and the appendix. Additional details regarding the used models and training procedure are in Sec. A.
REFERENCES
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Christopher Hesse, Rafal Józefowicz, Scott Gray, Catherine Olsson, Jakub Pachocki, Michael Petrov, Henrique Pondé de Oliveira Pinto, Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, llya Sutskever, Jie Tang, Filip Wolski, and Susan Zhang. Dota with large scale deep reinforcement learning. arXiv preprint 1rI.v:1912.060.
[cs.LG]] 2019.
Lucian Buşoniu, Robert Babuška, and Bart De Schutter. Multi-agent reinforcement learning: An overview. Innovations in multi-agent systems and applications-I, pp. 183-221, 2010.
Kyunghyun Cho, B van Merrienboer, Caglar Gulcehre, F Bougares, H Schwenk, and Yoshua Bengio. Learning phrase representations using ron encoder-decoder for statistical machine translation. In Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1724-1734, 2014.
Abhishek Das, Théophile Gervet, Joshua Romoff, Dhruv Batra, Devi Parikh, Mike Rabbat, and Joelle Pineau. Tarmac: Targeted multi-agent communication. In International Conference on Machine Learning, pp. 1538-1546, 2019.
Yali Du, Bo Liu, Vincent Moens, Ziqi Liu. Zhicheng Ren, Jun Wang, Xu Chen, and Haifeng Zhang. Learning correlated communication topology multi-agent reinforcement learning. In Proceedings of the 20th International Conference on Autonomous Agents and MultiAgen gent Systems, pp. 456 464, 2021.
Jakob Foerster, Ioannis Alexandros Assael, Nando De Freitas, and Shimon Whiteson. Learning to communicate with deep multi-agent reinforcement learning. In Advances in Neural Information Processing Systems, pp. 2137-2145, 2016.
Jakob Foerster, Francis Song, Edward Hughes, Neil Burch, Iain Dunning, Shimon Whiteson, Matthew Botvinick, and Michael Bowling. Bayesian action decoder for deep multi-agent reinforcement learning. International Conference on Machine Learning 29, pp. 1942-1951, 2019.
Benjamin Freed, Rohan James, Guillaume Sartoretti, and Howie Choset. Sparse discrete communica- tion learning for multi-agent cooperation through backpropagation. In IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pp. 7993-7998, 2020.
R.M. Goodman and Zheng Zeng. learning algorithm for multi-layer perceptrons with hard-limiting threshold units. In Proceedings of IEEE Workshop on Neural Networks for Signal Processing, pp. 219-228, 1994.
Shubham Gupta, Rishi Hazra, and Ambedkar Dukkipati. Networked multi-agent reinforcement learning with emergent communication. In Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems. pp. 1858-18
1858-1860, 1860, 2020.
Eric A Hansen, Daniel S Bernstein, and Shlomo Zilberstein. Dynamic programming for partially observable stochastic games. In Proceedings of the AAAI Conference Artificial Intelligence, volume 4, pp. 709-715, 2004.
Pablo Hernandez-Leal. Bilal Kartal, and Matthew E Taylor. A survey and critique of multiagent deep reinforcement learning. Autonomous Agents and Multi-Agent Systems, 33(6):750-797, 2019.
Diyi Hu, Chi Zhang, Viktor K. Prasanna, and Bhaskar Krishnamachari. Intelligent communication over realistic wireless networks in multi-agent cooperative games. In 21st International Conference Autonomous Agents and Multiagent Systems (AAMAS), pp. 1627-1629, 2022.
10

---

Under review as a conference paper at ICLR 2023
Pei Huang, Li Xiao, Soroor Soltani, Matt W. Mutka, and Ning Xi. The evolution of MAC protocols in wireless sensor networks: A survey. IEEE Communications Surveys Tutorials, 15(1):101-120, 2013.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. In Proceedings of the 5th International Conference on Learning Representations (ICLR), pp. 1-13, 2017.
Jiechuan Jiang and Zongging Lu. Learning attentional communication for multi-agent cooperation. In Advances in Neural Information Processing Systems 31, pp. 7254-7264, 2018.
Daewoo Kim, Sangwoo Moon, David Hostallero, Wan Ju Kang, Taeyoung Lee, Kyunghwan Son, and Yung Yi. Learning to schedule communication in multi-agent reinforcement learning. In Proceedings of the 7th International Conference on Learning Representations (ICLR), 2019.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference on Learning Representations (ICLR), 2015.
Landon Kraemer and Bikramjit Banerjee. Multi-agent reinforcement learning as a rehearsal for decentralized planning. Neurocomputing, 190:82-94, 2016.
Arun Kumar, Ming Zhao, Kai-Juan Wong, Yong Liang Guan, and Peter Han Joo Chong. A compre- hensive study of loT and WSN MAC protocols: Research issues, challenges and opportunities. IEEE Access, 6:76228-7622 2018.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-2324, 1998.
Sheng Li, Yutai Zhou, Ross Allen, and Mykel Kochenderfer. Learning emergent discrete message communication for cooperative reinforcement learning. In International Conference Robotics and Automation (ICRA), pp. 5511-5517, 2022.
Yiming Li. Shunli Ren, Pengxiang Wu, Siheng Chen, Chen Feng, and Wenjun Zhang. Learning dis- tilled collaboration graph for multi-agent perception. Advances in Neural Information Processing Systems 34, 2021.
Yen-Cheng Liu, Junjiao Tian, Nathaniel Glaser, and Zsolt Kira. When2com: Multi-agent perception via communication graph grouping. In Proceedings of the IEEE/CVF Conference computer vision and pattern recognition, pp. 4106-4115, 2020.
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperatiee-compettivv environments. In Advances in Neural Information Processing Systems 30, pp. 6379-6390, 2017.
Ryan Lowe, Jakob Foerster, Y-Lan Boureau, Joelle Pineau, and Yann N. Dauphin. On the pitfalls of measuring emergent communication. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems (AAMAS), pp. 693-701, 2019.
Hangyu Mao, Zhengchao Zhang, Zhen Xiao, Zhibo Gong, and Yan Ni. Learning agent communication under limited bandwidth by message pruning. Proceedings of the AAAI Conference on Artificial Intelligence, (1(04)):512-51444 2020.
Francisco S Melo, Matthijs TJ Spaan, and Stefan Witwicki. Querypomdp: Pomdp-based commu- nication in multiagent systems. In European Workshop on Multi-Agent Systems, pp. 189-204. Springer, 2011.
Frans A Oliehoek, Matthijs TJ Spaan, and Nikos Vlassis. Optimal and approximate Q-value functions for decentralized pomdps. Journal of Artificial Intelligence Research, 32:289-323 353, 2008.
Afshin Oroojlooyjadid and Davood Hajinezhad. A review of cooperative multi-agent deep reinforce- ment learning. arXiv preprint arXiv: neee::10800099 [cs.LG] 2019.
Georgios Papoudakis, Filippos Christianos, Arrasy Rahman, and Stefano V. Albrecht. Dealing with non-stationarity in multi-agent deep reinforcement learning. arXiv preprint arXiv: 1906.04737 [cs.LG), 2019.
11

---

Under review as a conference paper at ICLR 2023
Murtaza Rangwala and Ryan Williams. Learning multi-agent communication through structured attentive reasoning. In Advances in Neural Information Processing Systems 33, pp. 00088-10098 2020.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shi- mon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. International Conference on Machine Learning, pp. 4295 -404,, 2018.
Stefan Balthasar Schneider, Haydar Qarawlus, and Holger Karl. Distributed online service coor- dination using deep reinforcement learning. In IEEE International Conference on Distributed Computing Systems (ICDCS), pp. 539-549, 2021.
Amanpreet Singh, Tushar Jain, and Sainbayar Sukhbaatar. Learning when to communicate at scale in multiagent cooperative and competitive tasks. In Proceedings of the 7th International Conference on Learning Representations (ICLR), 2019.
Sainbayar Sukhbaatar, Arthur Szlam, and Rob Fergus. Learning multiagent communication with backpropagation. In Advances in Neural Information Processing Systems 29, pp- 2244 22422,25, 2016.
Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Ardi Tampuu, Tambet Matiisen, Dorian Kodelja, Ilya Kuzovkin, Kristjan Korjus, Juhan Aru, Jaan Aru, and Raul Vicente. Multiagent cooperation and competition with deep reinforcement learning. PLOS ONE, 12())201, 2017.
Andrew S. Tanenbaum and David Wetherall. Computer networks, 5th Edition. Pearson, 2011. ISBN 0132553171.
Rundong Wang, Xu He, Runsheng Yu, Wei Qiu, Bo An, and Zinovi Rabinovich. Learning efficient multi-agent communication: An information bottleneck approach. In Proceedings of the 37th International Conference on Machine Learning (ICML), pp. 9908-9918, 2020.
Ronald Williams. Simple statistical following algorithms for connectionist reinforcement learning. Machine Learning, 8(3):229-255) 256. 1992.
Zheng Zeng, Rodney M. Goodman, and Padhraic Smyth. Learning Finite State Machines With Self-Clustering Recurrent Networks. Neural Computation, 5(6))976-990, 1993.
Meng Zhou, Ziyu Liu, Pengwei Sui, Yixuan Li, and Yuk Ying Chung. Learning implicit credit assignment for cooperative multi-agent reinforcement learning. Advances in Neurul Information Processing Systems 33, pp. 1 11853-1 11664, 2020.
12

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 TRAINING AND ARCHITECTURE DETAILS
The input of the model consists of the agent's observation and the received messages. Environment- specific observation decoders map observations to a vector of length 128. The observation decoder for POMNIST is multi-layer convolutional neural network. The first convolutional layer convolves the input with 16 filters of size 3 x with stride length of 1 followed by a rectifier nonlinearity. The second layer convolves 32 filters of the same size and stride length. This is followed by 2 x 2 max ppolinn layer. The last layer is a fully connected layer with 128 rectifier units, followed by dropout layer with drop probability 0.5. The observation decoder for traffic junction is single- layer fully connected network with 128 rectifier units. The message decoder maps all incoming messages to vector of fixed length. It takes the mean of all messages padded to the same length, including a one-hot encoding of the message's s length. First, each message m of size is padded with zeroes to the maximum message size max 4. Then we append a mask that indicates the size of the message as a one-hot encoding of y. The output of the message decoder is the mean of these vectors m @ (mma 4 @ on)- In practice, we define one-hor(O) == 0 to indicate empty or dropped messages and mask them during aggregation.
The vectors from the observation and message decoders are concatenated and processed by a core network. The input size of the core network huore depends on t and is given as heore 128 + max t (p/. In POMNIST, we also append an one-hot encoding of the agent's id, i.e. core extended by N. The core module of POMNIST is fully connected layer with lcore rectifier units and a skip connection. We append a single-layer gated recurrent unit with Loore hidden features for traffic junction. The output of the core network is the input to the following networks.
The message-size selector is a single-layer feedforward network with |&]| neurons to predict the message-size values. For exploration, 400 to 1200. The message encoder is a multi-headed network with a shared layer of foore neurons and a separate head for each message size &o Φb both followed by tanh activation functions. As mentioned in Sec. 3.2 the action selector is conceptually independent from the rest of the architecture and can be trained using different reinforcement learning algorithms. For POMNIST, we use single-layer feedforward network to predict the Q-value for each action and an e-greedy action selector with 0.01. For traffic junction, we use a stochastic policy that determines the action probabilities with a single-layer feedforward network followed by softmax. We train this network with REINFORCE and use separate linear layer to predict the state-value baseline. We encourage exploration by adding an entropy bonus to the loss that is weighted by a factor that decays linearly from 2 to 0.1 over the first 1400 iterations and is 0.1 thereafter.
exponential e-decay from to 0.01 between iterations
We jointly train these networks by minimizing aLy, + (1-x) o)) Lr, over all agents. The mixing
coefficient a set to 0.5 in POMNIST and 0. traffic junction to mitigate the instability. In traffic junction, we gradient clipping with maximum 2-norm of 0.1. For both environments, we use the Adam optimizer with learning rate 0.001 and a discount factor of 1.
A.2 POMNIST ENVIRONMENT CONFIGURATION
Number of agents The most influential parameters in the POMNIST environment are the number of splits along the horizontal and vertical axes. They determine the number of agents in the environment. The full view of size 28 x 28 is divided according to the number of splits along each axis and we assign one agent to each view. For example, with O vertical and 1 horizontal splits, we get two agents with views of size 28 x 14 each. The more splits, the higher the number of agents and the smaller the view of each agent. The results for different splits without communication are shown in Tab. 3 As expected, the agents' performance decreases with an increasing number of splits. Note that random guessing corresponds to mean return of 1- 0.1 1. 0.9 = -0.8. Even the agents with (3,3) 3) splits learn a policy that is much better than random guessing. We decided to continue with 11, splits, as the computational overhead of sending messages will be lower than for 8 or 16 agents and the return is already much lower than the return of the baseline (0, 0). We show that agents can use communication improve their return in the following.
13

---

Under review as a conference paper at ICLR 2023
Table 3: Mean return and standard deviation without communication over 5 runs with different splits.
(vertical, horizontal) splits
(0,0)
(0, 1)
(1,0)
(1,1)
(1.3)
(3, 1)
(3,3)
number of agents
1
2
2
4
8
8
16
mean agent return
0.978
0.863
0.909
0.633
0.125
0.029
-0.315
0.001
0.001
0.001
0.003
0.012
0.004
0.011
Message sizes Next, we have to choose the message sizes & for our four agents. Higher message sizes allow them to encode and transmit more information, but also require more resources. This can be problematic in limited and unreliable channels. We hypothesize that the benefit of higher message sizes depends on the considered environment and that the resulting performance saturates with an increasing message size.
With continuous messages as our baseline, this performance saturation is already visible for relatively small message sizes in POMNIST. Tab, 4 shows the mean evaluation performance after training with selected message sizes between O and 256. While the return increases by more than 0.2 between message sizes 0 and the step-wise improvements diminish with greater message sizes. While the difference between 2 and 4 is still greater than 0.03, doubling the message size from this point on results in improvements smaller than 0.01. Based on these results, we choose message sizes 4 S {0, 1, 2, 4 for our main experiments. Message size 0 represents not sending any message. Agents can choose this size to keep the channel free if they do not have valuable information share. Message size 1 allows for minimum of coordination and message sizes 2 and 4 are trade-off between a comparably small message size and a high return.
We hypothesize that this saturation effect will also depend on the concrete message type. A continuous messages of size with a 32-bit float can carry significantly more information than a 1-bit discrete message. While this is true in theory, the experiments in our main paper show that agents can perform well even with discrete messages of small sizes.
Table Mean return and standard deviation over 5 runs with fixed-sized continuous messages, i.e. g 1, the test dataset after training for 2000 iterations.
+
(0}
[1}
(2)
(4)
[8}
{16}
(32)
(64)
(128)
(256)
return
0.633
0.867
0.923
0.957
0.966
0.969
0.972
0.972
0.974
0.974
+
0.003
0.008
0.007
0.001
0.001
0.001
0.000
0.002
0.000
0.002
The optimal choice of message sizes depends on the used hardware and environment-specific factors, e.g. balancing the training overhead and the diminishing improvements in return for higher message sizes. Therefore, this design choice should be made on case-by-case basis.
Next, we analyze how the return changes depending on the number of received messages with four agents. Tab. [5] shows the mean agent return for message sizes 1, 2 and 3 when an unlimited communication channel only forwards messages from certain agents and blocks all other messages.
Table 5: Mean return and standard deviation over 5 runs with fixed-sized continuous messages and different sender subsets on the test dataset after training for 2000 iterations.
senders
{0}
{0,1)
{0, 1,
{0, 1,
{1}
0. 0.632+0.000 004
0.774+0.003
0.835=0.000 003
0.858+0.013
0.856+0.000
Φ [2]
0.63440.001
0.792=0.002
0.877=0.008
0.916+0.000 000
0.923=0.001
Φ- = {4}
0.606
0.001
0.9.0.003
0.946:10.002
0.957
The gradual increment in the reward for increasing number of transmitted messages is similar to Tab. 4 There is a big improvement from not transmitting any messages { to transmitting one message (0), but the benefit of transmitting more messages diminishes with the number of senders.
14

---

Under review as a conference paper at ICLR 2023
Especially for message size 1 (Φ (1}), the difference between transmitting 3 messages from agents {0, 1.2] and messages from all agents {0, 1,2,3) 3) is neglectable. This suggests that the messages beneficial to receive messages
contain redundant information and it might not be necessary from all agents at all times. Although the agent's 's return increases slightly from 3 to 4 sending agents for message sizes 2 and 4, the higher utilization might lead to additional collisions and thus reward detonations in imperfect channels.
A.3 POSITIVE LISTENING METRIC
In order to quantify the impact of communication on an agent's behavior, we use a positive listening metric (Lowe al. 2019). We define positive listening the rate at which an agent makes wrong decision and corrects it after receiving messages. We compute this metric in POMNIST by exploiting the two-step execution explained in Sec. 4.1] The messages are considered the second step of the decision making, while in the first step the agent takes only its observation into account. According to this, we measure the effect of messages by comparing the chosen action in the first and second step. It should be noted that the models used for the computation of this metric must satisfy the condition E & and have no internal state, i.e. the agents are trained to make the decision without any messages.
A.4 POSITIVE SIGNALING METRIC
To quantify positive signaling in discrete action and message spaces, Lowe et al. suggest to use the mutual information between selected actions and sent messages (Lowe al. 2019):
p(a,
M)A:M)= =
Σ AEA,
pla, m) log p(a)p(m)
They treat the action space and discrete message space M as random variables that take on values E A and m E .M with probabilities p(a) and p(m) respectively. In practice, these probabilities are approximated empirically by normalizing the action and message frequencies.
In adaptive communication, agents can choose from different message sizes 3 E T Messages for each size are chosen independently and the entropy of the corresponding message distributions can differ. Additionally, messages of certain sizes could be solely sent in conjunction with certain actions. As the mutual information is non-negative and bound by the minimum entropy of actions and messages, i.e. 0 < I(A; S (A)), (MM), we can normalize the mutual information across different message sizes.
Based on these considerations, express positive signaling jointly for all messages sizes
Σ p(y) T(Ag:M9) (E (()( Min H[M+)}
PS
where p((p) is the probability of choosing a message of size oo E & and M is the set of messages with this size. The random variable Ap conditions actions on message size s. If 0 ş D define p(0) = O. Note that empty messages of size (0) are excluded from the metric, i.e. an agent that communicates only when taking specific action in specific state and stays silent otherwise will have a positive signaling value of 1 instead of O.
Note that our positive signaling metric is defined for discrete messages. As continuous messages can take any value in range [-1, 1] they are often unique. This would lead to mutual information and positive signaling values of while not allowing for any conclusions regarding the usage of the message space with respect to the taken actions. Positive signaling metrics for continuous messages could e.g. be based on clustering methods.
A.5 COMMUNICATION CHANNEL
In this section, provide further details regarding the communication channel introduced in Sec. 3.4
15

---

Under review as a conference paper at ICLR 2023
First, describe example illustrate why the expected throughput (see Sec. 4.1 is usually lower than the channel size with the given model. Recall that the starting position for a message of size yp is chosen randomly from {0, s, 2...... [و with channel size C' and that the message occupies slots. We call this channel model stochastic channel with spacing. Consider 4 agents that try to send messages of size 4 simultaneously in channel of size C' 8. 'hee message of agent then placed into slots (0, 1, 2,3) 3) or {4, 5, 6, 7, each with probability 0.5. It is transmitted correctly without collisions iff the messages of all remaining agents are placed into the other slots. With four agents, these are 2- 4 = 8 events with probability 0.5*. The probability of successfully transmitting any message therefore 8 0.5 0.5. As all messages have size 4, the throughput 0.5 4 4= 2.
When 4 agents send messages with random sizes vp Φ {0, 1, 2, 4} over channel of size 8, we empirically get throughput of 2.297 over million steps. The drop probabilities for different message sizes are shown in Tab. 6 The table also includes the drop probabilities for the stochastic channel, where a message of size yp randomly placed into {0, 2,,....... slots. The results show that this channel access scheme results in a lower throughput of 1.579 and higher drop probabilities for higher message sizes.
....,,C pp and occupies yp
Table 6: Message drop probabilities for individual message sizes and throughput using a random message size selection in different channel models.
Channel model
Drop probabilities
Throughput
yp
=
G = 4
stochastic with spacing stochastic
0
0.524
0.578
0.756
2.297 slots
0.512
0.682
0.886
1.579 slots
Based on these results, we decided to use the stochastic channel with spacing for our main experiments. Further experiments, e.g. comparing the learned behavior for different channel models, are left for future work. An active placement of messages in the channel could also be investigated, e.g. agents could listen on the channel and decide when to send messages to avoid collisions.
A.6 MESSAGE SIZES
In this section take closer look the mean message sizes at the experiments with 4 agents and message sizes {0, 1, 2,. 4). Fig. shows the mean message size dur- ing training. We observe that, for small channel sizes 4 and 8, the agents do not exploit the channel and decrease their mes- sage sizes to avoid collisions. However, with higher channel sizes (C 2 16), the agents first try to exploit the mean message size increases from the random selection. As expected, they learn to decrease their message size to avoid collisions later in the training. Therefore, the mean message size decreases and converges to a message size that is higher than the random selection. The message sizes after training for 2000 iterations are reported on Tab[[ the main paper for different channel sizes.
Moun thozeh
resources and
1253 1750
Figure 7: Mean chosen message size during training with different chan- nel sizes. Each line shows the mean over 5 runs, the transparent area rep- resents the standard deviation.
16


Reviews:
[{"review_id": "qVKBth8G1fX", "reviewer": "Reviewer_hSti", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a communication policy for multi-agent RL when the communication channel is imperfect. The idea is to adapt the size of communication data based on observations from the communication channel. Then, the performance of the proposed policies was evaluated using two datasets. The results show that the proposed algorithms outperform the alternatives.", "summary_of_the_review": "Overall, this is a timely topic. However, the contribution of this paper sounds limited. On the theory side, there is no theoretical analysis of the proposed algorithms. On the experimental side, there could be more comparison with the state-of-the-art and also more insights on the results. ", "strength_and_weaknesses": "**Strengths**\n- The topic of communication-efficient cooperative learning is a timely topic. \n- The focus of this paper on considering the imperfect communication channels is new in the context of MARL.\n-  The paper is well-written and easy to read.\n- The experiments are given using two datasets. \n\n**Weaknesses**\n- While the imperfect channel issue makes sense and is important, the paper does not provide any theoretical understanding and analysis of the proposed cooperative policies. \n- Hence, this paper should be evaluated as an empirical study. Then, focusing just on experiments, it seems that the paper fails to compare with prior cooperative policies. In the current presentation, it is unclear whether the algorithms are from other papers or are different versions of the algorithms proposed in this paper. \n- In addition, the authors can provide some insights into the observations from the experiments. For example, for Fig. 3a, the authors just state which algorithm is the best, and there are no further justifications for the reasoning of these results. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "TglK92yF4BZ", "reviewer": "Reviewer_JWRp", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a reinforcement learning based approach to multi-agent communication over noisy or limited channels. Agents therefore adapt to share messages of varying lengths and content depending on the channel characteristics and the local environment. The approach is demonstrated on a partially observable version of MNIST (POMNIST) in which each agent is allowed access to a mutually exclusive and jointly exhaustive subset of the image, and on a traffic junction problem. The results illustrate the ability adapt to different characteristics of channels and successfully communicate to achieve shared goals. ", "summary_of_the_review": "The paper addresses the problem of communication about partially observed worlds over unknown channels. The proposed approach uses reinforcement learning to dynamically adapt to the features of the world and the channel so that agents may cooperate toward shared goals. While the problem and approach seem to be novel, the contribution appears to be modest. ", "strength_and_weaknesses": "Strengths: \n- Multiagent communication and cooperation are important questions that are topics of broad interest to the community. \n- The problem of limited and noisy channels is an important one in practice. (At least noisy channels is for humans.) \n- The proposed approach extends prior work toward an interesting direction. \n- The POMNIST problem is fun. \n- It is a positive that the traffic junction problem reveals limitations of the approach. \n\nWeaknesses: \n- The problem itself, while obviously necessary, is not so interesting. The novelty contribution seems relatively modest. \n-  There is no theoretical contribution. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "-R5eQZtGlT8", "reviewer": "Reviewer_rgRs", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper discusses a solution for communication in multi-agent learning in terms of message size (and implicitly when to transmit). The paper is evaluated by means of an environment developed by the authors, which is based on a classification task of the MNIST dataset based on partial views of the images themselves. Each agent \"decides\" which information is sent to the others in order to execute the classification task end-to-end.", "summary_of_the_review": "Overall, the reviewer struggles to see a sufficient contribution in this paper. The reviewer has concerns about the motivation, generalizabilty and design of the architecture. For this reason, the reviewer is not in a position to recommend this work for publication.", "strength_and_weaknesses": "This is an interesting paper about a problem of general applicability. The main concern is that the problem itself is evaluated in a very specific environment. The main novelty of the work resides probably in framing the general problem of learning the type of messages exchanged by the agents in presence of noise. The actual solution per se is rather standard (end-to-end training with local encoders that generate messages that are exchanged among a group of agents in order to execute a task).\nMore specifically, the reviewer wonders if the actual task used for the evaluation might have a strong impact on the results. In fact, at the end, the local \"encoding\" might be seen as a problem of classification. Each agent \"encodes\" the most probable \"digits\" that are present in the image and then this is used for the overall classification. There might be different types of encoding given different available sizes. From the results presented by the authors, there is a minimal amount of information that is necessary to transmit. The problem of traffic in this case can be framed as communication over a noisy channel, but then, again, a \"trivial\" solution might be to have a sufficient amount of information sent. It would be interesting to see what happens when just one or two agents transmit their messages, just to check if the results are independent from the fact that all (or most of) the agents are involved. Having said that, I think that the authors present an interesting set up for testing their model. The question is if it is possible to have sufficient evidence using only that environment in terms of validation. Probably different tasks that are not related to classification are necessary for a full-evaluation of this work. This is quite important, since the solution is evaluated only experimentally.\n\nThere is also a question about the message that is sent - the actual selection of the size of the message should be clarified in the paper.\n\nThere is also a question about which actual component has the strongest influence/impact on the obtained results. I understand that this is kind of lost in an end-to-end training, but I think it is a really important aspect. It seems that the size is important, but probably mostly importantly the encoding efficiency (given the probability of transmission error ) is also a key factor.\n\nIn general, the paper only briefly discusses the limitations of the proposed solution. A major limitation of the work appears to be the fact that the authors consider only one rather specific task (distributed image classification). It is quite difficult to generalize from the experimental results they present, even if they are valuable.\n\nOther open points about this paper are the following:\n\n- The reviewer wonders if the results we observe might be considered as strictly related to the problem/environment under consideration. How can you say that the results reported in the paper might be generalized to other environments?\n\n- What is the real impact of the encoding, etc. when you have redundancy in terms of information? It seems to me that this work applies to cases where every agent is essentially performing the same action. Is it possible to extend this work to situations in which the different agents are involved in different tasks and communication is needed for coordination?\n\n- An interesting aspect to investigate is the fact that you have overlapping \"states\" (different views of the image). Is it possible to say that this is not a factor contributing to the results presented in the paper or is it?\n\n\n\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject