Learning to Communicate Through Implicit Communication Channels
Authors: Han Wang
Binbin Chen
Tieying Zhang
Baoxiang Wang
Venue: ICLR 2025 Poster (2025)
Keywords: implicit communication
multi-agent reinforcement learning
the Hanabi challenge
Primary area: reinforcement learning
TL;DR: We propose the Implicit Channel Protocol (ICP) framework, which allows agents to construct implicit communication channels similar to the explicit ones.

Abstract: Effective communication is an essential component in collaborative multi-agent systems. Situations where explicit messaging is not feasible have been common in human society throughout history, which motivate the study of implicit communication. Previous works on learning implicit communication mostly rely on theory of mind (ToM), where agents infer the mental states and intentions of others by interpreting their actions. However, ToM-based methods become less effective in making accurate inferences in complex tasks. In this work, we propose the Implicit Channel Protocol (ICP) framework, which allows agents to communicate through implicit communication channels similar to the explicit ones. ICP leverages a subset of actions, denoted as the scouting actions, and a mapping between information and these scouting actions that encodes and decodes the messages. We propose training algorithms for agents to message and act, including learning with a randomly initialized information map and with a delayed information map. The efficacy of ICP has been tested on the tasks of Guessing Numbers, Revealing Goals, and Hanabi, where ICP significantly outperforms baseline methods through more efficient information transmission.

Published as a conference paper at ICLR 2025
LEARNING TO COMMUNICATE THROUGH IMPLICIT COMMUNICATION CHANNELS
Han Wang
Binbin Chen
The Chinese University of Hong Kong, Shenzhen xwanghan& gma ccom
ByteDance Inc.
chenbinbin. 196@bytedance.com
Tieying Zhang
ByteDance Inc.
tieying. zhangebytedance.cm com
Baoxiang Wang"
The Chinese University of Hong Kong, Shenzhen Vector Institute
bxiangwang@euhk, edu. en
ABSTRACT
Effective communication is an essential component in collaborative multi-agent systems. Situations where explicit messaging not feasible have been common in human society throughout history, which motivate the study of implicit com- munication. Previous works on learning implicit communication mostly rely on theory of mind (ToM), where agents infer the mental states and intentions of others by interpreting their actions. However, ToM-based methods become less effective in making accurate inferences in complex tasks. In this work, we propose the Implicit Channel Protocol (ICP) framework, which allows agents communicate through implicit communication channels similar to the explicit ones. ICP lever- ages a subset of actions, denoted as the scouting actions, and mapping between information and these scouting actions that encodes and decodes the messages. We propose training algorithms for agents to message and act, including learn- ing with randomly initialized information map and with a delayed information map. The efficacy of ICP has been tested on the tasks of Guessing Numbers, Re- vealing Goals, and Hanabi, where ICP significantly outperforms baseline methods through more efficient information transmission.
INTRODUCTION
Effective communication is pivotal in collaborative multi-agent systems, especially in environments characterized by incomplete information (Panait & Luke [003] Busoniu al.) 2008 Tuyls & Weiss 2012). Communication acts vital conduit, enabling agents to exchange private information, co- ordinate joint actions, and infer real-world states (Wang et al. 2021). These processes synergistically foster tighter cooperation and enhance collective performance (Li et al. 2002 Cao et al. 2012). We focus on multi-agent reinforcement learning (MARL) methods for communication, where commu- nication is broadly categorized into explicit and implicit strategies (Dafoe et al. 2020).
Explicit communication uses direct channels independent of the environment dynamics (Sukhbaatar et al., 2016)] Foerster al. 2016 Jiang & Lu 2018), allowing agents to transmit observations. inten- tions, and advice to facilitate cectsiionmmaingg and coordination (Zhu et al.) [002 Qu et al. 2021). This approach, analogous human language verbal exchanges (Havrylov & Titov] 2017 Baker et al. 1999), has been widely employed in MARL to enhance collaboration. However, dependence on direct channels introduces significant computational and memory overheads (Roth al. 2006).
"Corresponding to: Baoxiang Wang

---

Published as a conference paper at ICLR 2025
which makes it challenging to implement in certain scenarios. Examples are tasks without commu- nication channels or decentralized frameworks (Olichock et al. 2008 Kraemer & Banerjee 2016).
Situations where explicit messaging is not feasible have been common in human society throughout history. From early humans engaging in hunting and gathering through silent cooperation (Klein 2009 Tomasello & Vaish 2013), to modern military operations using gestures and codes for covert communication 2008), and even in everyday social interactions where intentions are conveyed through expressions, tone, and body language (Pease) 1984] Duncan Jr. 1969). Implicit communi- cation has established an effective mechanism for information sharing without explicit language.
Learning methods for implicit communication have been investigated by the MARL community. A prominent method is the theory of mind (ToM) (Premack & Woodruff] 1978). where agents infer the mental states and intentions of others by interpreting their actions (Heider & Simmel 1944). By modeling the beliefs, desires, and intentions of other agents. ToM enables agents to coordinate in variety simple tasks (Baker et al., 2017] Zhao et al., 2023] Nguyen et al.) 2020). However, ToM- based approaches face significant challenges, including the difficulty of making accurate inferences and the high computational complexity involved modeling other agents. These issues become particularly pronounced in dynamic environments where agents must constantly update their models
based on limited or ambiguous information.
To address these challenges associated with ToM methods, we introduce Implicit Channel Protocol (ICP), a novel framework that allows agents to communicate through communication protocols in implicit communication likewise how it was done in explicit communication. ICP leverages sub- set of actions, denoted as the scouting actions, which have no or uniform effects on environment dynamics. A centralized mapping between information and these scouting actions is established to encode and decode the messages. Agents exchange information by deliberately taking scouting actions, forming an implicit communication channel. We further demonstrate how agents' strate- gies are trained on this channel, including training with a randomly initialized information map and training with delayed information map.
We validate the effectiveness of ICP through comprehensive experiments on the tasks of Guessing Numbers, Revealing Goals, and Hanabi (Bard et al. 2020). These environments share common characteristic: they lack direct communication but agents must collaboratively make decisions to achieve shared rewards. This setting introduces significant challenges, including sparse and delayed reward feedback, along with difficulty in credit assignment both temporally and among agents. De- spite these hurdles, our experiments on Guessing Numbers and Revealing Goals demonstrate that ICP significantly enhances performance, through more efficient information transmission, compared to baseline methods. In Hanabi, which is a popular card game played by humans, our approach achieved an average score of 24.91 out of 25, which surpasses the best available learning algorithm which obtains 23.81.
2 BACKGROUND AND RELATED WORK
Decentralized Partially Observable Markoy Decision Process The cooperative multi-agent problem can be formulated as a Dec-POMDP game (Bernstein et al. 2002). which is described by the tuple G = (S, U, N, T, O, R, 7), where S represents the true global state of the environment. At each discrete time step I, every agent: i E N := .I.....n] selects an action uy from its action space U4. The state transition function T(s' [s, u) :S SS x U S P(S) governs the transition of states, where u = (u,,.... un) denotes the joint action. In POMDP, the global state remains in- accessible, and each agent i only perceive its individual observation o, through the observation function (s) : 5 x N + O. (s, u4): S U, + R represents the reward function for each agent In the cooperative scenarios, all agents share a common reward function R(s, u) S U Rt known as team reward. The objective for each agent is to maximize the expected return, which makes effective cooperation among all agents necessary.
Learning to Communicate Communication among agents is critical for effective collaboration in MARL. Most researches in this field focused on explicit communication protocols (Tucker et al.
2022
Peng et al. 2017; Kong al.] 2017 Pesce & Montanal 2020; Kim et al. 2019 Wang et al. 2020 Freed et al. 2020b al Gupta et al. [203)
[ Foerster et al. 2016). where agents exchange mes-
sages containing critical information. Additionally, techniques like attention mechanisms or graph-
2

---

Published as a conference paper at ICLR 2025
based methods that used to build more effective communication connection (Jiang & Lu 2018 Das et al. 2019 Jiang et al. 2018 Sukhbaatar et al. 2016 Chen et al. 2024 Tucker et al. 2022) and in- tention sharing theory of mind (ToM) reasoning to transmit more useful information (Wang al.
2021 Kim et al. 2020 Qu et al. 2021). further enhancing the effectiveness of MARI. systems.
Explicit communication direct channels often incurs significant communication overhead and may not be available in environments with limited bandwidth and high communication costs. Im- plicit communication, where agents convey information through their behaviors, has been explored as an alternative approach eLi al. 2024 TTia et al. 2023 Li et al. 202 Shaw et al. 20222 Grupen al., (022). ToM-based methods represent one main research strand for implicit communication, where agents infer the intentions and beliefs of others to anticipate actions and adjust strategies ac- cordingly. Methods in [Rabinowitz al.] (2018); Tian et al. (2020); Nguyen al. (2020); Zhao et al. (2023) aim to enhance coordination by modeling and predicting others' behavior based on observed actions. However, ToM-based approaches can computationally intensive and may struggle to accurately infer intentions in complex environments.
SETTING
Consider in a fully cooperative Dec-POMDPs, where no direct communication channel is available. Each agent maintains a joint action-observation history Tt {oo,i; tgg Fl,...;'' "..,,,,,,,,,. 0t,a) and makes decisions and executes actions based on it. Within this setting, we define a subset of actions as scouting actions U which have no or uniform effects on the state st or reward rr and mainly affect the observation function (s{s)) The remaining actions are defined as regular actions U".
In an idealized situation, scouting actions serve to influence the observation mapping function, which allows the agents to gather critical information from the environment to improve their decision- making and coordination. scouting action might reveal information about an agent's surroundings or the state of other agents without changing the environment unpredictably. For instance, in the game of Hanabi, Hint actions reveal information to other players while consuming an information token, thereby uniformly altering the game state. Similarly, in StarCraft II, scouting the map with Scan from Terrans requires depleting energy, which impacts the environment and provides new information in the agent's 's observations.
We are interested in this setting because agents generally need to employ scouting actions to collect sufficient information before taking other actions to obtain rewards. Both the information collection phase and the reward acquisition phase may require cooperation among agents. For example, agents will only receive positive rewards if one agent scouts the correct information for another agent, and the latter correctly utilizes this information. In this context, sparse and delayed reward feedback, along with the credit assignment problem both temporally and among agents, poses significant chal- lenges for agents to learn optimal strategies.
INFORMATION CARRIED BY SCOUTING ACTIONS
Scouting actions carry two types of information.
1. Information reflected through the environment. When an agent performs a scouting action, it influences the observation function, and the new observations provide information. This type of information depends on the observation function, the states, and the joint actions of all agents.
2. Information reflected through the choice of the scouting action. The specific scouting action chosen by an agent can carry intentional meaning. This type of information depends only on other agents' scouting action policy. If the receiver agent cannot understand the sender agent's ss intention, no information will be transmitted.
General multi-agent reinforcement learning (MARL) methods tend to focus more the first type of information. In Pec-POMDPs without modeling other agents' strategies, both partial observation of states and other agents' policies introduce uncertainty. In the exploration phase of online MARL training, the second type of information cannot be utilized due to the unknown policies of others. Therefore, MARL methods mostly learn use the information reflected through the environment.
3

---

Published as a conference paper at ICLR 2025
For the second type of information which is not limited by observation function and states, when agents' intention is understood correctly, it is more fixable and could be more useful for decision- making and cooperation. Theory of mind (ToM) methods utilize the second type of information by modeling strategies or inferring the intention of other agents. By estimating the internal states of other agents, agent can interpret the intentional choices behind their scouting actions. One approach is to build an explicit belief model that infers the state from observed actions to achieve communication. However, this method incurs a computational cost that grows exponentially with the size of the game and the number of agents. Moreover, the precise state distribution information often difficult to fully utilize. Another approach involves training a neural network to map actions to states intentions, which requires exponentially more data compared to action policy training. Since this method does not produce explicit inferences, the resulting information inherently bi- ased, which could negatively impact the agent's decision-making process.
We propose a simpler and more effective solution to take advantage of the second type of information conveyed by scouting actions while also mitigating the computational overhead and inaccuracies of ToM methods. By establishing a shared communication protocol where all agents follow the same strategy for selecting scouting actions, the intended information be easily decoded from the chosen action. This approach eliminates the need for state estimation and intention inference required in ToM methods. In our setting, this communication protocol has the following benefits.
1. Low Computational Overhead and Accurate Information Transmission. By having all agents follow the same strategy for selecting scouting actions, there is need for additional computational inference or training of a separate inference model to deduce intentions. The inverse of the strategy can be used to directly interpret the information, resulting in low compu- tational overhead and accurate transmission of intentions.
2. Efficient Communication Enabled by Broadcasted Information. Scouting actions are observ- able by all agents in the system, allowing any information conveyed through these actions to be inherently broadcasted. This characteristic facilitates the use of efficient embedding techniques, such as the hat mapping method, to achieve more effective communication.
3. Independence from Observation-Base Information. When different embeddings used, the information conveyed by altering the observation function may differ, making the information carried by scouting actions not necessarily correlated with the environment-basee information. Both types of information can be optimized concurrently to enhance cooperation and decision- making.
5 STRATEGY TRAINING ON CONSTRUCTED IMPLICIT CHANNEL
To establish shared communication protocol, we propose Implicit Channel Protocol (ICP), new framework for agents' implicit communication. In this framework, at each step, agents choose whether to send information. If an agent chooses not to send information, a regular action u" will be taken according to the action policy. If otherwise an agent wishes to convey message m. E M,, it will output a scouting action u² = P(m.)) which maps m; through the centralized mapping mecha- nism P. This mechanism P needs to be decodable, which provides output messages from scouting actions. By allowing agents to send information through encoded scouting actions and other agents to decode the messages from the globally observed scouting actions, implicit communication channel is constructed.
As agents need to decide whether send information, the new action space becomes U" = {U U*, send info). then agents sample actions u, E U" from new action policy u, ~ T.(- (- Tr.s, m. where m are decoded messages transmitted from other agents. For message m,, it is selected by the message strategy m4 ~ @z(. Tr,s, m.)). Based on this process, we can formulate agents' value
functions as VT.AE (s) E x² --rr Uu s(+1) $1 s), for St+1 T(: St, un) and ur = (U,, U,
u² = P(m, ~ Ls(. m-)) if send info,
and U
(1)
w. T.(. Te,s
else.
4

---

Published as a conference paper at ICLR 2025
Equation highlights three essential components of the framework: the agents' action policy n, the message strategy d. and the centralized mapping mechanism P. To develop these components, investigate two different methodologies. The first one is training with random initial informa- tion map and the second one is training with a delayed information map. Both approaches construct an implicit communication channel within the environment, with which the agents exchange infor- mation and achieve implicit coordination. In the rest of the section, we will discuss the details of these approaches.
5.1 STRATEGY TRAINING WITH RANDOM INITIAL MAP
In this approach, we first establish a one-to-one mapping between information and actions using randomly initialized embedding, that makes the size of the message the same as the size of the scouting action space {I,... [0 Once this one-to-one mapping is set, the inverse of will (uf)
obtain the original message
In our implementation, we use value-based method for training the action policy, incor- porating parameter sharing and value decomposition techniques to facilitate effective coor- dination among agents. For the information strategy, we use the communication gradient method (Foerster al.] 2016), which is designed to optimize communication protocols within a predefined channel. Both the action policy and the information strategy are parameter- ized as Q-networks, for action policy (T(T),) Given the value function Val.o.. (s), the Q-functions
argmax Q ( ut), and message strat-
egy (T,,)) argmax Q (Tt,;; defined by Q (Te,i, uz) = Em (ri (St, ut 8141) + TV and (Tr, m.) Eu prr ue, s1+1) + yVT.867)) where π (Ti,... nn), (Ø1, ..., On) and (mi,... ma,
Parameter Sharing and Value Decomposition Parameter sharing enables different agents with the same observation and action spaces to learn from a shared network. Despite using the same network, agents evolve different hidden states and receive distinct observations, allowing them to behave differently. We incorporate the agent index into the network input, allowing for specializa- tion through rich representations in deep Q-networks.
To further enhance learning efficiency, apply value decomposition (Sunehag et al. 2018) for the action policies и. The joint action-value function Qua u) is expressed as the sum of individual value functions: (T, (T) simplifies learning by allowing the gradients to propagate through individual agents' policies while maintaining shared reward structure, which accelerates the overall training process.
Di=1 (T u,; 01.)). where (T...... TN). This decomposition
Communication Gradient Since message space is M (1..... [ which is discrete and non-binary, we introduce the Gumbel-Softmin technique (Havrylov & Titov 2017|Jang et al. (0166) to sample one-hot vector messages m. The Humbel-Softmax method enables crete samples that are differentiable, facilitating end-to-end training with gradient descent. Specif- ically, we sample the one-hot vector message ni such that nū, if arg maxjeM. (log(o<) Te,á, m-)) + gi)) and n, = 0 otherwise, where = - log( log(u;)) and uj U 1).
generate dis-
This method ensures continuous relaxation between and n. As result of this relaxation, the game becomes fully differentiable and can be trained using the backpropagation algorithm. By enabling gradients flow smoothly through the communication process, this approach facilitates more effective and efficient training of agents. Comparing the communication gradient implemen- tation in DIAL (Foerster et al. 2016), we adopt this Eumbel-SSffmax technique to replace the Discretize/Regulariz Unit (DRU). This substitution allows agents to send discrete and non-binary messages during the learning phase, thereby enhancing the richness and accuracy the information exchanged.
5.2 STRATEGY TRAINING WITH DELAYED MAP
In the training process with delayed map, we begin by learning the information strategies and ac- tion policies using a direct communication channel. This initial phase provides agents with flexible environment that allows for direct communication and enable them to develop their strategies more
5

---

Published as a conference paper at ICLR 2025
effectively. Once these strategies are established, we determine the mapping mechanism that maps the messages to actions. Finally, we fine-tune both the action policies and information strategies in the ICP framework to ensure optimal performance within the constraints of the final environment.
Given that the information strategies were initially trained using a direct communication channel, there may be discrepancies between this channel and the implicit communication channel con- structed by scouting actions, particularly in terms of capacity and whether the channel is discrete or noiseless. To minimize any potential performance loss resulting from these differences, we can limit the capacity of the direct channel or apply efficient embedding techniques (such as the hat map- ping method) during the message-to-acee to-aciinn mapping process. These steps help to align information strategies developed in the direct channel with the constraints of the implicit channel and ensure messages are decodable.
During implementation, we utilize standard explicit communication algorithms designed for direct channels, such as RGMComm, to pre-train the information strategy. Then we use two kinds of map- ping methods, including simple one-to-one mapping and the hat mapping method (Brown & Tanton 2009 Bushi 2012 Butler et al.] 2009; Feige, 2004, Havil 2011; Winkler 2002) to obtain new in- formation strategies. The one-to-one mapping ensures that each message corresponds directly to an action, while the hat mapping allows an agent to communicate efficiently with multiple receivers. Finally, we fine-tune both the information strategy and the action policy optimize performance within the implicit communication framework.
Pre-training with Direct Channel In the pre-training phase, we utilize a direct communication channel to facilitate the learning of the communication strategies. During this phase, the message size can be larger than the scouting action space, allowing for more complex and detailed communi- cation between agents. This flexibility enables the agents to explore a wider range of communication strategies and coordination behaviors, which are crucial for solving complex tasks in partially ob- servable environments. However, after pre-training, the message strategy needs to transition to the implicit communication framework and it is necessary to compress these messages without loss of information. For one-to-one mapping, this mapping requires that the message size matches the scouting action space exactly. It ensures a direct and consistent mapping between the learned mes- sages and the corresponding actions.
Hat Mapping Implementation When the local observations of different agents' have large over- laps, the hat principle-basee mapping method utilizes broadcast channels achieve a single trans- mission of messages to multiple receivers. The method is inspired by the multi-color hat guessing game, where players use logical deduction based on limited communication maximize team suc- cess. In this game, players can observe the hat colors of others but not their own, and they must guess their own hat color using pre-defined strategy. A well-known strategy involves each player leveraging the sum of the hat colors they observe, modulo the total number of possible colors. For example, in an 8-color version of the game, each player can compute the sum of the colors they ob- serve (assigned numerical values) and use a pre-determined rule, such as calculating the sum modulo 8, to make a logical deduction about their own hat color.
In our implementation, at each step, each agent computes local messages intended for every other agent. These local messages are derived from the shared information strategy and the observations that the target agent cannot but are overlapped with other agents-minn mucch like the "hat" in the hat guessing game. When an agent sends information, it transmits a structured public message that combines all these local messages using sum modulo operation. As a result, each receiving agent can also use the sum modulo operation to combine the received public message with the local messages of all agents except themselves and the sender. This allows them to infer their own specific local message from the combined public message. This method enables a single broadcast to convey individualized information to multiple agents simultaneously, enhancing communication efficiency even when explicit communication is not possible.
EXPERIMENTS
In this section, we explore the application of the ICP across various environments where no direct communication is available. ICP plays vital role in these settings by constructing implicit

---

Published as a conference paper at ICLR 2025
aicuoro pecping
Reveal
stop
Maxi
Gool
Gool
6
Gool
Reveal
Gool
Time
Time Hd
Figure I: Left: Guessing Numbers Environment. Agents cannot see their own digits, but can reveal others' segments' states by collaboratively giving hints. They can deduce their own digit and obtain shared rewards. Right: Revealing Goals Environment. The agents positioned in random spots in a grid world and are assigned a unique target. However, they can only observe others' targets which do not include themselves. By revealing each other's targets in the nearby grid, they can eventually find their own targets and reach them.
communication channel that enhances the agents' ability to share and interpret information. We will demonstrate how ICP facilitates effective collaboration and decision-making in 3 tasks, namely Guessing Numbers, the Revealing Goals, and the Hanabi card game. The first two tasks are designed by us and can be reused by future works testing environments. The Hanabi game is a popular card game played by humans, and is described as The Hanabi Challenge (Bard et al.) 2020) by the community. Each of these environments presents unique challenges, and we will show how ICP optimizes the use of scouting actions to improve overall performance and strategy.
6.1 GUESSING NUMBERS
In the Guessing Numbers game, we present a collaborative, turn-based multi-agent reinforce- ment learning environment involving N agents. Each agent is uniquely assigned a digit de {0, 1.... 9}, which is visible to all other agents # i but remains unknown to agent i themselves. The primary objective for each agent is to deduce and correctly guess their own digit d,. During each turn, an agent can choose to either guess their own digit by selecting d, E {0, (,..,...., 9} or provide a hint about another agent's digit using the Radioland Slim font-a seven-segment digital display representation. Specifically. hints involve revealing the state ("on" or "off"" of one of the seven seg- ments of another agent's digit, and these hints are public and observable by all agents. Each agent's action space comprises 10 + x (N - actions: 10 options for guessing their own digit and 7 hint options for each of the N other agents. The game encourages strategic collaboration, as agents must balance between gathering information to deduce their own digit and assisting others through informative hints, aiming for the collective success of all agents correctly guessing their digits.
In this game, successfully guessing their digit rewards an agent with 10, while performing a hint action incurs small penalty of -0.1 encourage efficient communication and collaboration. The game imposes limit on the number of hint actions, and each agent is allowed only guess. The game concludes when all agents have made their guesses, with the goal being accurately guess all digits within these constraints.
In the Guessing Numbers experiment, we evaluate the performance of approaches: VDN-on- policy, VDN-off-policy ICP with the random initial map approach (ICN-DIAL-RMD) ICP with the delayed map approach (ICN-DIAL-DM), and a cheating approach where a direct communication channel is available (DIAL-Cheat) Each approach is evaluated over 1k episodes with 6 random seeds, and rumning on a Linux metal machine with 256 GB RAM and 3090Ti GPU for 36 hours.
For VDN-offf we begin by warming up the replay buffer until its size exceeds the batch size. During each training step, we add 10 episodes to the replay buffer and randomly sample a batch of episodes from the buffer for training. In contrast, for VDN-on-policy and our proposed method, utilize a vectorized environment to sample a batch of episodes at each training step and use these samples for training.
Despite differences in network architecture, these approaches share similar hyperparameters. Specif- ically, set the hidden size of the MI.P and GRU to 256, use 2 layers in the GRU, and set the
7

---

Published as a conference paper at ICLR 2025
Perfarmance Guezsing Number
/ewrage Tpleade Length Gomaing humber
04
CL OL0ALEM
150
200
Updatt Each Steps
Figure 2: (a): The training curves of Guessing Numbers over total of 100k steps with N 3,1 11. (b): Average episode length running by different algorithms in Guessing Numbers.
learning rate to 5 10-4, batch size to 256. The target network update rate is set 10, y set to 0.99, is set to 0.1, and we apply gradient clipping with a threshold of 10.
The results are presented in Figure 2 All algorithms are capable eventually guessing their own digit correctly, but the differences of number of hit actions needed is significant among the algo- rithms. By examining the saved models and averaging over 10k episodes, we find that ICN-DIAL- RM has an average episode length of approximately 8.63, while VDN-on-pollic and VDN-off- policy have average episode lengths of around 12.4. This observation further demonstrates that ICN achieves more efficient information transfer by using scouting actions in implicit channels.
6.2 REVEALING GOALS
In the Revealing Goals task, we present a non-sequential collaborative that emphasizes information sharing among agents. The environment consists of N agents operating within H x H grid world. Each agent i starts at random position and is assigned a unique goal location 9: that is at least two grid units away from its starting position. Critically, agents cannot perceive their own goal locations; they can only observe the goal locations of other agents j F i. At each time step, agents select from a fixed action space of eight actions: moving up, down, left, or right, and reveal information about the adjacent grid cells in these directions. When agent performs reveal action, the adjacent grid cell in the specified direction becomes revealed, and all agents gain information about whether their own goals are located in that cell. The grid world features wrap-around edges, creating toroidal topology where, for example, moving left from cell (0, y) leads to cell (H - 1,
An agent observes the locations of all agents, the goal points of other agents, and any goals the revealed grids. Since they cannot directly perceive their own goals, agents must rely on information revealed by themselves and others to infer the locations of their own goals. The objective is for each agent to navigate to its designated goal location 9i. upon which all agents receive a shared reward of but is at least two grid units
When an agent reaches its goal, new goal is randomly assigned away from its current position. The game proceeds for total of T time steps. This environment encourages strategic collaboration, as agents must balance between exploring the grid find their own goals and assisting others by revealing grid cells that may contain teammates' goals. The agents will need communicate implicitly, using the fixed action space of size 8.
In the experiments for the Revealing Goals task, we use the same training hyperparameters and model architecture as in the Guessing Numbers experiments. We also compare the performance of approaches: DIAL-Cheat, ICN-DIAL-DM, ICN-DIAL-RM VIN-on-policy, and VDN-offf-ollicy The experimental results are shown in Figure aa). ICN-DIAL-RM significantly outperforms the baseline methods by achieving an average score 2.17 times that of the baselines. One reason for its effectiveness is that, in each round, each of the N agents can engage in broadcast communi- cations through implicit channels, which significantly enhances information efficiency compared to relying solely on environmental feedback. This result further validates the effectiveness of ICP in more complicated grid world tasks.
8

---

Published as a conference paper at ICLR 2025
Performance Revealing Goals
hanati
25-
63 201y
0
Thury
202
Update Each
1030 upciss for cliipd 5V/339 accnis cooppd
(a)
(b)
Figure 3: (a): The training curves of Revealing Goals total 100k train steps with N 4, 5, = 50. (b): The training curves of 4-players Hanabi with on-policy algorithms take around 150
hours and off off-policy algorithms take around 20 hours.
6.3 HANABI
To demonstrate ICP's broader applicability, we also evaluate in the game of Hanabi. Hanabi is fully cooperative card game where players work together play cards in specific order to form sequences, aiming to achieve the highest possible score. When it is played by humans, people will mute themselves and coordinate only through taking and observing actions. This game requires effective implicit communication and strategic information sharing among the agents.
Because the game rule of Hanabi is quite involved, refer it to popular website https: // hanabi. github. 1o/ that discusses both rules and human-play strategies. The action space for each agent includes three options: playing card out of their 5-card hand, discarding card, or giving hint towards another player's 5-card hand. The hint could reveal all cards in player's hand of specific color rank. Since they cannot see their own cards, agents must rely on hints from others to infer what cards they hold. The success of the game heavily depends on the agents' capacity to use these hints effectively to coordinate their actions and play the correct cards in order.
In our experiments, we focus on the 4-payer full version of Hanabi. Since hint actions in Hanabi are natural scouting actions, we select them to serve as information carriers for our ICP implementa- tion. We implement several baseline algorithms, including VDN-off-policy and the original version of SAD (Hu & Foerster 2019), as well as a variant of SAD with the hint action masks removed. For the ICP algorithm, explore several variations, including CEN-DIAL-RM-OO conpolicy, which in- volves strategy training with a random initial map using DIAL's communication gradient technique resulting in on-policy learning method. We also implement LLL----L--M------rr-a policy, similar to MIN-DIAL-RMM-nnPPe
but with the communication gradient chain cut off to enable off-policy learning. Additionally, we use CCN-RGMCCmm-DD with a delayed map using RGMComm (Chen et al.) 2024), an off-policy MARL communication algorithm, and (CN-RGMComm--De ----ooom----aaa-mappal-- hat--ppitip extension of ICN-RGMComm-DDA that uti- lizes the hat mapping method for improved communication.
which is strategy training
Our presented results are based on the best runs selected from more than three random seeds. From the results shown in Figure 3b), we observe that the ICP method using the hat mapping approach, ICN-RGMCommn Comm-DD--lee -M---a----maaa hat-mappinng achieved the best performance. The score achieved by ICP is 24.91, which surpasses the best available learning algorithm (SAD, 23.81 points) and vastly outperforms theory mind-based methods (Fuchs et al. (2021) 19.13 points) and average human players (17 points, reported in Kantack (2021)). It is worth remarking that the state-of- the-art strategy in the Hanabi game is search-basee based algorithm, WTFFWhattseare an averaged score of 24.96 points (Lerer et al. 2020). With the maximum score of the game being 25, and there are certain deals that prevent the players from winning 25 points in any way, both ICP and serahh close to fully solving the game.
which reports

---

Published as a conference paper at ICLR 2025
For additional details implementation, please refer to Appendix A A comparison between ICP and Hanabi human conventions is provided Appendix B along with detailed analysis of the Hanabi results in Appendix C
DISCUSSIONS
7.1 COMPATIBILITY OF ICP WITH OTHER TRAINING METHODS
In the ICP framework, action policy allows for the use of various algorithms for training. For instance, the action policy can be trained using value-based methods like VDN (Sunehag et al.) 2018) or policy-based methods such as MADDPG (Lowe et al., 2017). This compatibility ensures that ICP can tailored various problem settings and agent dynamics. One could choose the most suitable algorithm depending on the specific characteristics of the environment and the agents.
The communication strategy within the ICP framework is also versatile. can be implemented using a variety of direct channel communication algorithms that support discrete channels. These include well-established methods like DIAL (Foerster al.) 2016) and RGMComm (Chen et al.) 2024), which are designed to optimize communication under different constraints and requirements. The flexibility in choosing both the action policy and the communication algorithm highlights the comparability and broad applicability of the ICP framework, making it a general approach for multi- agent reinforcement learning scenarios.
7.2 POTENTIAL OF FURTHER UTILIZING THE ENVIRONMENT INFORMATION
As described in Section scouting actions carry two types of information. The ICP framework utilizes the information reflected through the choice of the scouting action (choice information) to establish an implicit communication channel. However, even when leverage this information, the information reflected through the environment (environment information) remains useful. Although most cases the constructed implicit channel transmits more stable and useful information, some situations the environment information is also significant. For example, in the Revealing Goals environment, a revealing action intended to transmit embedded information might also reveal other agents' goals, which helps agents make better decisions.
To verify the impact of this environmental information on performance, we randomly shuffle different embeddings the MIM---GMMommhehe saa----ht-m-ppinn policy saved from the previous Hanabi experiment. We then tue-tune the action policy while keeping the information strategy fixed. The results are shown in Figure 4 in the appendix, where, under fixed information strategy, some randomly shuffled embeddings enjoy improved performance. This suggests that one could further utilize the environment information for more effective communication in future works.
CONCLUSION AND FUTURE WORK
The Implicit Channel Protocol (ICP) introduced in this paper is an advancement in implicit commu- nication through multi-agent reinforcement lcarning (MARL). By mapping information to scouting actions construct implicit channels and optimizing both action and communication strategies, ICP facilitates communication protocols to be established even without explicit channels. The experi- mental results validate the protocol's effectiveness across various MARL scenarios and demonstrate its capacity to significantly improve both coordination among agents and performance in the tasks.
possible direction for future work is extending ICP to environments that lack inherent scouting actions. In such settings, identifying appropriate actions to serve as information carriers becomes challenging, as these actions must both convey information effectively and interact with the environ- ment beneficially. Future research could focus developing algorithms that dynamically identify or design such actions, balancing effective communication with minimal impact on the environment. These advancements would enhance the applicability of ICP across wider range of scenarios, mak- ing it a more versatile tool in multi-agent communication.
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
The code and our designed environments are freely available in the supplementary material. The de- tails of the implementation can be found in Appendix A The hyper-parameters used in experiments are also mentioned in Section 6
ACKNOWLEDGEMEET
Han Wang and Baoxiang Wang are partially supported by the National Natural Science Foundation of China (62106213 72394361), an extended support project from the Shenzhen Science and Tech- nology Program, Longgang District Key Laboratory of Intelligent Digital Economy Security, and ByteDance.
REFERENCES
Chris L Baker, Julian Eitinger, Rebecca Saxe, and Joshua Tenenbaum, Rational quantitative attribution of beliefs, desires and percepts in human mentalizing. Nature Human Behaviour, 1(4): 0064, 2017.
Michael Baker, Tia Hansen, Richard Joiner, and David Traum. The role of grounding in collaborative learning tasks. Collaborative learning: Cognitive and computational approaches, 31:63, 1999.
Nolan Bard, Jakob Foerster, Sarath Chandar, Neil Burch, Marc Lanctot, H Francis Song, Emilio Parisotto, Vincent Dumoulin. Subhodeep Moitra, Edward Hughes, et al. The hanabi challenge: A new frontier for ai research. Artificial Intelligence, 280: 103216, 2020.
Daniel S Bernstein, Robert Givan, Neil Immerman, and Shlomo Zilberstein. The complexity of decentralized control of markov decision processes. Mathematics of operations research, 27(4): 819-840, 2002.
Ezra Brown and James Tanton. A dozen hat problems. Math Horizons, 16(4):22-25 2009.
Jaime Bushi. Optimal strategies for hat games. Society for Industrial and Applied Mathematics, 2012.
Lucian Busoniu, Robert Babuska, and Bart De Schutter. comprehensive survey of multiagent rein- forcement learning. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 38(2):156 -721 2008
Steve Butler, Mohammad Hajiaghayi, Robert D Kleinberg, and Tom Leighton. Hat guessing games. SIAM review, 51(2):399-413, 2009.
Yongcan Cao, Wenwu Yu, Wei Ren, and Guanrong Chen. An overview of recent progress in the study of distributed multi-agent coordination. IEEE Transactions on Industrial informatics, 9(1): 427-438. 2012.
Jingdi Chen, Tian Lan, and Carlee Joe- Joe-Wong. Rgmcomm: Return gap minimization via discrete communications in multi-agent reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, 2024.
Junyoung Chung, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks sequence modeling. In NIPS 2014 Workshop Deep Learning, December 2014, 2014.
Allan Dafoe, Edward Hughes, Yoram Bachrach, Tantum Collins, Kevin R McKee, Joel Z Leibo, Kate Larson, and Thore Graepel. Open problems in cooperative ai. arXiv preprint arXiv:20/2.086330 2020.
Abhishek Das, Théophile Gervet, Joshua Romoff, Dhruv Batra, Devi Parikh. Mike Rabbat, and Joelle Pineau. Tarmac: Targeted multi-agent communication. In International Conference on machine learning, pp. 1546- PMLR, 2019.

---

Published as a conference paper at ICLR 2025
Starkey Dunean Jr. Nonverbal communication. Psychological bulletin, 72(2):1 1969.
Uriel Feige. You can leave your hat (if you guess its color). Technical report, Technical Report MCS04-03, Computer Science and Applied Mathematics, The ..., 2004.
Jakob Foerster, Ioannis Alexandros Assacl, Nando De Freitas, and Shimon Whiteson. Learning to communicate with deep multi-agent reinforcement learning. Advances in neural information processing systems, 29, 2016.
Benjamin Freed, Rohan James, Guillaume Sartoretti, and Howie Choset. Sparse discrete communi- cation learning for multi-agent cooperation through backpropagation. In 2020 JEEF/RSJ Interna- tional Conference on Intelligent Robots and Systems (IROS), pp. 7993-7998. IEEE, 2020a.
Benjamin Freed, Guillaume Sartoretti, Jiaheng Hu, and Howie Choset. Communication learning via backpropagation in discrete channels with unknown noise. In Proceedings of the AAAI conference on artificial intelligence, 2020b.
Andrew Fuchs, Michael Walton, Theresa Chadwick, and Doug Lange. Theory of mind for deep reinforcement learning in hanabi. arXiv preprint 1XXi:22001.09328, 2021.
Niko Grupen, Daniel D Lee, and Bart Selman. Multi-agent curricula and emergent implicit signal- ing. In Proceedings of the 2/ International Conference on Autonomous Agents and Multiagent Systems, pp. 553-561, 2022.
Nikunj Gupta, Srinivasaraghavan, Swarup Mohalik, Nishant Kumar, and Matthew Taylor. Ham- mer: Multi-level coordination of reinforcement learning agents via learned messaging. Neural Computing and Applications, pp. 1-16, 2023.
John Harsanyi. Games with incomplete information played by "bayesian" players, i-iii part i. the basic model. Management science, 14(3):159-133 1967.
Matthew Hausknecht and Peter Stone. Deep recurrent q-learning for partially observable mdps. In 2015 aaai fall symposium series, 2015.
Julian Havil. Impossible?: Surprising Solutions to Counterintuitive Conundrums. Princeton Uni- versity Press, 2011.
Serhii Havrylov and Ivan Titov. Emergence of language with multi-agent games: Learning com- municate with sequences of symbols. Advances in neural information processing systems, 30, 2017.
Fritz Heider and Marianne Simmel. experimental study of apparent behavior. The American journal of psychology, 57(2):243-259, 1944.
Hengyuan Hu and Jakob N Foerster.
Simplified action decoder for deep multi-agent reinforcement learning. International Conference on Learning Representations, 2019.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax, arXiv preprint arXiv: /6/ 1.0/ 144, 2016.
Jiechuan Jiang and Zongqing Lu. Learning attentional communication for multi-agent cooperation. Advances in neural information processing systems, 31, 2018
Jiechuan Jiang, Chen Dun, Tiejun Huang, and Zongqing Lu. Graph convolutional reinforcement learning. In International Conference Learning Representations, 2018.
Nicholas Kantack. Reinforcement learning human decision models for uniquely collaborative ai teammates, arXiv preprint arXiv:2111.09800 2021.
Woojun Kim, Myungsik Cho, and Youngchul Sung. Message-dropoutt An efficient training method for multi-agent deep reinforcement learning, In Proceedings of the AAAI conference artificial intelligence, 2019.
Woojun Kim, Jongeui Park, and Youngchul Sung. Communication in multi-agent reinforcement learning: Intention sharing. In International conference learning representations, 2020.
12

---

Published as a conference paper at ICLR 2025
Richard G Klein. The human career: Human biological and cultural origins. University of Chicago Press, 2009.
Vijay Konda and John Tsitsiklis. Actor-critic algorithms. Advances in newral information processing systems, 12, 1999.
Xiangyu Kong, Bo Xin, Fangchen Liu, and Yizhou Wang. Revisiting the master-slave architecture in multi-agent deep reinforcement learning. arXiv preprint arXiv: 2017.
Landon Kraemer and Bikramjit Banerjee. Multi-agent reinforcement learning a rehearsal for decentralized planning. Neurocomputing, 190:82-94, 2016.
Adam Lerer, Hengyuan Hu, Jakob Foerster, and Noam Brown. Improving policies via search in cooperative partially observable games. In Proceedings of the AAAI conference artificial in- telligence, 2020.
Dapeng Li, Zhiwei Xu, Bin Zhang, Guangchong Zhou, Zeren Zhang, and Guoliang Fan. From explicit communication to tacit cooperation: A novel paradigm for cooperative marl. In Proceed- ings of the 23rd International Conference on Autonomous Agents and Multiagent Systems, pp. 1360-23362 2024.
Ling Li, Alcherio Martinoli, and Yaser Abu-Mostafa. Emergent specialization in swarm sys- tems. In Intelligent Data Engineering and Automated Leurning-IDEAL. 2002: Third Interna- tional Conference Manchester, UK, August 12-14, 2002 Proceedings 3, pp. 261-266. Springer, 2002.
Sheng Li, Jayesh K Gupta, Peter Morales, Ross Allen, and Mykel J Kochenderfer. Deep implicit coordination graphs for multi-agent reinforcement learning. In Proceedings of the 20th Interna- fional Conference on Autonomous Agents and MultiAgent Systems, pp. 764-772, 2021.
Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi- agent actor-critic for mixed nooperatve-coompeiie environments. Advances in neural informa- tion processing systems, 30, 2017.
Dung Nguyen, Svetha Venkatesh, Phuoc Nguyen, and Truyen Tran. Theory of mind with guilt aversion facilitates cooperative reinforcement learning. In Asian conference machine learning, pp. 33-48. PMLR, 2020.
Frans A Oliehoek, Matthijs TJ Spaan, and Nikos Vlassis. Optimal and approximate q-value func- tions for decentralized pomdps. Journal Artificial Intelligence Research, 02:289-353, 2008
Liviu Panait and Sean Luke. Cooperative multi-agent learning: The state of the art. Autonomous agents and multi-agent systems, 11:87-4334 2005.
Allan Pease. Body language: How to read others' thoughts by their gestures. Sheldon press, 1984.
Peng Peng, Ying Wen, Yaodong Yang, Quan Yuan, Zhenkun Tang, Haitao Long, and Jun Wang. Multiagent bidireetionall--coodnnntt nets: Emergence of buman-level coordination learning to play stareraft combat games. arXiv preprint 173iv: 10069, 2017.
Emanuele Pesce and Giovanni Montana. Improving coordination in small-scale multi-agent deep inforcement learning through memory-driven communication. Machine Learning, 109(7):1277- 1747, 2020.
David Premack and Guy Woodruff. Does the chimpanzee have a theory of mind? Behavioral and brain sciences, 144)5115-526, 1978.
Chao Qu, Hui Li, Chang Liu, Junwu Xiong, james zhang, Wei Chu, Weiqiang Wang, Yuan Qi, and Song. Intention propagation for multi-agent reinforcement learning, 2021. URL. https: //openzeviww. wwt.//eetuut// wwwenrevvew.net//oouuni....
Neil Rabinowitz, Frank Perbet. Francis Song, Chiyuan Zhang, SM Ali Eslami, and Matthew Botvinick. Machine theory of mind. In International conference on machine learning. pp. 4218- 4227. PMLR, 2018.
13

---

Published as a conference paper at ICLR 2025
Maayan Roth, Reid Simmons, and Manuela Veloso. What to communicate? execution-time decision in multi-agent pondps. In Distributed autonomous robotic systems 7, pp. 177-186. Springer, 2006.
Samuel Shaw, Emerson Wenzel, Alexis Walker, and Guillaume Sartoretti. Formie: Foraging via multiagent rl with implicit communication. IEEE Robotics and Automation Letters, 7(2):4877- 4884, 2022.
Sainbayar Sukhbaatar, Rob Fergus, al. Learning multiagent communication with backpropaga- tion. Advances in neural information processing systems, 29, 2016.
Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojeiech Marian Czarnecki, Vinieius Zambaldi, Max Jaderberg, Mare Lanctot, Nicolas Sonnerat, Joel Leibo, Karl Tuyls, et al. Value-decomposition networks for cooperative multi-agent learning based team reward. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, pp. 2085-2087, 2018.
Kai Tian, Athanasios Tzigieras, Chongfeng Wei, Yee Mun Lee, Christopher Holmes, Matteo Leonetti, Natasha Merat, Richard Romano, and Gustav Markkula. Deceleration parameters as implicit communication signals for pedestrians' crossing decisions and estimations of automated vehicle behaviour. Accident Analysis Prevention, 190: 1071737,, 2023.
Zheng Tian, Shihao Zou, Ian Davies, Tim Warr, Lisheng Wu, Haitham Bou Ammar, and Jun Wang. Learning communicate implicitly by actions. In Proceedings of the AAAI Conference Arti- ficial Intelligence, 2020.
2020.
Michael Tomasello and Amrisha Vaish. Origins of human cooperation and morality. Annual review of psychology, 64())3331- 2013.
Mycal Tucker, Roger Levy, Julie Shah, and Noga Zaslavsky. Trading off utility, informativeness, and complexity in emergent communication. Advances in neural information processing systems, 01-22214-22228 2022.
Karl Tuyls and Gerhard Weiss. Multiagent learning: Basics, challenges, and prospects. Ai Magazine, 33(3):41-41, 2012.
Sun Tzu. The art of war. In Strategic Studies, pp. 63-91. Routledge, 2008
Rundong Wang, Xu He, Runsheng Yu, Wei Qiu, Bo An, and Zinovi Rabinovich. Learning efficient multi-agent communication: An information bottleneck approach. In International Conference on Machine Learning, pp. 9908-9918 18. PMLR, 2020.
Yuanfei Wang, Jing Xu, Yizhou Wang, et al. Tom2c: Target-oriented multi-agent communication and cooperation with theory of mind. In International Conference Learning Representations, 2021.
Peter Winkler. Games people don't play, Puzzler's Tribute: a Feast for the Mind, pp. 301-313, 2002.
Zhuoya Zhao, Feifei Zhao, Yuxuan Zhao, Yi Zeng, and Yinqian Sun. A brain-inspired theory of mind spiking neural network improves multi-agent cooperation and competition. Patterns, 4(8), 2023.
Changxi Zhu, Mehdi Dastani, and Shihan Wang. survey of aulti-agent reinforcement learning with communication. arXiv preprint a7Xiv:2203.08975 2022.
14

---

Published as a conference paper at ICLR 2025
A TRAINING DETAILS
A.1 ICP IMPLEMENTATION WITH DIAL AND VDN
In our implementation, we employ a 2-head input and 2-head output Implicit Channel Net (ICN) to parameterize all functions. Specifically, each time step t, for agent i, observation- pairs (or,s, 44-1) are input to one Multilayer Perceptron (MLP) and Rectified Linear Unit (ReLU) layer. Additionally, the last message vectors from other agents (mi,, where k max(j mi E M)) for E (1,...,N] N) are stacked together along with each message sender's ID, then input to another MLP and ReLU layer. The outputs of these two layers are summed and input to a Gated Recurrent Unit (GRU) network (Chung et al., (2014). This process approximately achieves the Q function input with action-observation history r and other agents' messages m- by utilizing hidden states and inputs at each time step.
Subsequently, the output of the GRU directed to two distinct heads: an action head and a message head. The action head generates logits for actions, with an output dimension of IU U + 1, while the message head is specialized to handle the agent's communication behavior, producing logits for messages. Specifically, the output dimension of the message head is set to |t + 1, where the additional dimension represents the NOOP (no operation) action. This design ensures that the agent refrains from sending message when the action sampled from the action logits does not correspond to send send-info. info. To enforce this behavior, a mask is applied to the output of the message head. When the action sampled from the action logits is send info, the mask restricts Gumbel-Softman sampling to select the NOOP action. If the agent decides to send a message, the sampled one-hot vector of messages m directly maps the j-th information action v where
During centralized training sessions, action sampling follows c-greedy policy ensure explo- ration, while we maintain the gradient of the one-hot vector of messages mi to ensure gradients pass through the communication process. However, since gradient information becomes inaceurate for updated policies, we can only use sampled episodes once, making our method on-policy. In de- centralized execution sessions, simply utilize the shared parameter network manage greedy actions and one-hot vector messages sampled from Gumbel- Sumbel-Softmax.
A.2 ICP IMPLEMENTATION WITH RGMCOMM AND VDN
In the pre-training phase of this approach, we used RGMComm, which utilized Regularized Infor- mation Maximization loss (Chen et al.) (024) to generate discrete messages. This method represents the latest state-of-the-arr explicit communication algorithm in multi-agent reinforcement learning (MARL).
RGMComm itself is based on the MADDPG (Multi-Agent Deep Deterministic Policy Gradient) framework (Lowe et al., 2017). Initially, a full observability centralized actor-critic model (Konda & T'sstsiklis) 1999) is trained to develop a policy with access to global information. This centralized policy serves reference to guide the communication strategy for each agent. During this stage, the agents use a centralized critic with complete view of the enviromment to assess the action- functions and optimize the communication strategy. After training this full-observability policy, RGMComm samples different information sets based on local observations. It then clusters these sampled information sets (Harsanyi 1967) to assign specific messages to each cluster. By doing so, RGMComm derives a discrete communication protocol tailored to each agent's local observations, optimizing their joint policy under partial observability.
In our implementation, instead of using MADDPG, we employed RDQN (Recurrent Deep Q- Network) (Hausknecht & Stone, 2015) to train the policy with full observability. For example, in environments like Hanabi. the difference between the full observation (global view) and the local observation is minimal, with only additional knowledge of the agent's own hand being considered. Using this full-observation policy, we performed clustering on the information sets, similarly to the RGMComm approach, to obtain the message strategy.
Following the pre-training phase, we proceed to the fine-tuning phase. This phase involves using VDN (Value Decomposition Network) to train the action policy. During this process, the communi- cation strategy is embedded into the scouting action space and is fine-tuned to adapt to the environ-
15

---

Published as a conference paper at ICLR 2025
Algorithm 1: ICP implementation with DIAL and VDN
Input: environment eny, max train siep, max episodee length, lurget update rale, ", exploration constant c, learning rale selection of
information action space Viaoo
Oulput: Trained model puramelers
Initialize shared RNN network weights (9), 02) for each agent and centralized Mapping Mechanism P. target net weights
(0 (0) +- 02),
for Train step 1,2,...... maxx-rati--iiep do Reset environment, oo eav rrset(): buffer
Initialize empty onc-bot messages ma target empty empty one-bet messages m/ RNNs' hidden (h1, (2)) target RNNS' hidden states (KG. AG):
for time step 1 = 1, 2, mar eatee--piideeeeee longhh do Initialize empty for coch agent do
messages m:;;
Sample message update hidden state: Memppi h2 = GumbelSofmax((Fa, (0s, maaeii---1110 1, b2]):
10
Sample a random action Mnmden E Unsittoo
Compute greedy action update hidden Mgpcady; hi arg (TT,, MR.-,,,, h.)))) e-greody sample action Me,,
MMandmm (1 Mymady.
Get action Me, gt, Qay (Me,, I 0e,6, 1,,, A
14
if se. = Unemago (send info) then
Update action
Ninfo P(Mmample, Mug Te,s);
Update message m.,, Masgle:
17
Sample tagett message and update hidden state: i maanpie 112 GumbolSoftmxx(ppg m/-,,,,11,-1.. 14))): Compute target greedy action's ' and update hidlden state: Ve hi = max, (Tog (oL.s mi - 1.-5 N()): if arg maax (mpg (0e, -i Update message mi,; minnii
18
Unnage (send_info) then
20
21
Step environment, 0241.
env stepst)
22
Store (q. o/. re) into buffer B:
23
Use samples in buller to compute sum-q's ss 'DD-eroor 6 = + (,411,)))) E,(rah Update network weights 6s. O2 using Adam Optimizer with loss 65,z if Train N mod torget-upddae--au Update target network weights (01, 81) + (01,
24
25
then
26
ment. By freezing the CGMComm-genente communication strategy during the initial stages of fine-tuning, we ensure stability and later adjust the embedding to better align with the action policy.
A.3 IMPLEMENTATION DIAL-CHEAT AND (CN-DIAL-DM
DIAL with Direct Communication Channel (DIAL-Cheat)): In this approach, we added a discrete direct channel to the original environment, with the message size matching the size of the scouting action space. This channel allows agents broadcast information while simultaneously taking actions.
ICP Implemented with the Delayed Map Approach ((CN-DIAL-DM): For this method, we first removed scouting actions from the environment and introduced discrete direct channel (with the same message size as the scouting action space). Using this modified environment, we pre-trained the agents with DIAL to learn the information strategy. The learned information strategy then transferred back to the original environment for fine-tuning, where both the information strategy and action policy were optimized.
B HUMAN CONVENTION IN HANABI
Hanabi serves as compelling platform for studying cooperation and ad-hoc teamwork. particularly within the MARL community, while also being an engaging game in its own right. Beyond its academic appeal, many players have developed highly sophisticated strategies, as detailed in the referenced H-Group conventions T These conventions outline framework that enables players to achieve high scores by adhering to agreed-upon strategies.
For more detailed information on these strategies, please refer to the H-Group's conventions and learning path at hantts::/haaaae .aa.bbiggithti. github. 1o/
16

---

Published as a conference paper at ICLR 2025
4 players Full henabi score with different shuffled embedding
24.00
24.00
Figure 4: By shuffling the embedding of information into scouting actions, even if the information strategy stays the same, environment information will also change. After fine-tuning, the perfor- mance of implementation with the same information strategy but different embedding varies.
Interestingly, some of these strategies embed elements of ToM reasoning into the conventions them- selves. For example, techniques like Delayed Play Clues explicitly define the information that cer- tain actions are intended to convey when teammates perform specific moves. Even the Basic and Advanced Strategies, such as misleading bluffs, leverage second-order ToM reasoning to facilitate more effective collaboration.
In manner similar to ICP, the H-Group conventions construct stable implicit communication channel by explicitly codifying the ToM reasoning into the actions. When all players follow these conventions, actions communication channel within the game.
reliable carriers of information, ensuring consistent and effective
Compared with H-Group conventions, ICP framework differs in that the communication strat- egy itself is learned as part of the training process, rather than being predefined. The communication through informative actions becomes akin to following learned convention where agents develop their own patterns for encoding and transmitting information during the training phase. This allows agents to use scouting or informative actions explicit communication mechanisms, while regu- lar actions are selected through an online decision- decision-making policy during gameplay. Notably, our method achieves score of approximately 24.91 in the Hanabi game, which significantly surpasses the average human score of around 17 (Kantack 202T). Thus, ICP offers dynamic communica- tion framework where the convention communication emerges from the agents' learning process, enabling more flexible and adaptive strategies depending on the environment and task at hand.
C ANALYSIS OF HANABI RESULTS
In Figure 3 b), present a comparison between proposed method and SAD, focusing on dif- ferences in average scores. While the difference in average score appears minimal the figure, detailed analysis provided below reveals that our method achieves substantial improvement in overall performance.
According the results reported in the original SAD paper, SAD achieved average score of 23.81/25 with a win rate of 41.45% %. In contrast, our proposed method achieved an average score of 24.91/25 with a win rate of 91 %. This represents more than twofold increase in the win rate. The relatively small difference in average scores, despite the significant improvement in win rates, can be attributed the score ceiling of 25 in Hanabi.
Hanabi imposes inherent constraints scoring due to its game structure. Certain starting hands, such as those with multiple high-value cards (e.g.. 4s and 5s), can make achieving the maximum score of 25 impossible. Considering these limitations, achieving win rate of 91 101%%-alll and scores close in the remaining 9% of games demonstrates that our method performs near the theoretical maximum.
17


Reviews:
[{"review_id": "TLmcABHknl", "reviewer": "Reviewer_W1kG", "summary": "This paper presents the **Implicit Channel Protocol (ICP)** framework, an approach for enabling implicit communication in collaborative multi-agent reinforcement learning (MARL) environments where explicit communication is unavailable or costly. ICP introduces a subset of actions termed \"scouting actions\" that allow agents to communicate indirectly by encoding and decoding messages through their choice of these actions. By leveraging these actions, ICP builds an implicit channel similar to explicit communication protocols. The framework is evaluated on tasks such as Guessing Number, Revealing Goals, and the Hanabi card game, where it demonstrates effective information transmission and improved performance over baseline methods.", "questions": "1. In experiments, for agents trained with VDN, is the action space the combination of “regular actions” and “scouting actions“?  \n2. In figure 3b, why are those methods not trained to the same length?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-structured and easy to follow. The algorithm is clearly-explained, with clear definitions of necessary notations. The experiments are well-designed and comprehensive, with sufficient implementation details.", "weaknesses": "1. The ICP framework requires pre-identified scouting actions that can serve as indirect communication channels. This dependency limits its applicability in environments where such actions are not readily available or are difficult to define.\n\n2. It can help the readers to better understand the method if the authors can include a diagram of the algorithm pipeline.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KIDhgonp7V", "reviewer": "Reviewer_D4Qe", "summary": "This paper proposes a communication framework for a multi-agent reinforcement learning system. Efficient and targeted communication in the form of query-key pairs have been explored under previous works. Moreover, in order to address the challenges of a non-stationary environment, prior works have used Theory of Mind (ToM) methods to infer the intentions/states of the other agents in the environment to make more informed decisions. Developing models of other agents, add complexity to the training of multi-agent systems. This paper proposes an Implicit Communication Protocol, where each agents actions are supplemented with a *communication/scouting* action, that controls whether an agents sends a scouting action/query. Unlike the attention mechanism, where all agents receive 'N' messages from all 'N' agents, this paper proposes a common channel that aggregates all the information into one message vector. This paper additionally uses a gating mechanism similar to IC3Net, but with a Gumbel-Softmax relaxation that allows it to encode it as a binary classifier that functions as ATOC's (Jiang et al, 2018) \"initiator\" gating mechanism. \n\nThe paper utilizes the non-differentiable communication aggregator mechanism RGMComm. Additionally, standard end-to-end differentiable communication networks can instead be used. A lookup table of the local observation of each agent with the respected broadcasted messages are then constructed for all agents to discretize the communication channel. Moreover, they use the hat-mapping technique where agents can infer their own targeted messages from the common message. The messages are then passed along with the hidden states of the agents to get the updated action.", "questions": "The results for Hanabi are quite remarkable, however,  game rules indicate, the hints are only constrained towards revealing either the color or the number. Since ICP implicitly encodes the local observation of each agent (both for DIAL and RGMComm) into message vectors, I believe each agent observes in essence have access to the complete global state, which inherently breaks the rule of Hanabi, unless I am mistaken. I would like more clarification on this.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The proposed method is interesting, particularly toward the foundational problem of efficient multi agent communication by generating a information table with respect to the observation and message. Additionally, research into reducing computational complexity for intention inference techniques of agents will be highly valuable to the large-scale multi-agent systems.", "weaknesses": "1. The paper does not discuss the impact of scalability or impact of heterogeneous agents to the proposed framework for multi agent systems, as you increase the number of agents to say up to 10, 20, 50 agents.\n\n2. The paper does not compare results for common communication architectures such as CommNet, TarMAC, SARNet etc for their environments but only with value decomposition networks (VDN), that do not perform communication as part of their actions. \n\n3. The paper does not discuss limitations on the size of the action spaces or performance with respect to more dynamic environments such as predatory-prey, cooperative navigation.\n\nI believe more comprehensive experimental results are needed for the proposed framework.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fOkmtgUFnO", "reviewer": "Reviewer_avMG", "summary": "This paper looks into communication in collaborative multi-agent system under the formalism of multi-agent reinforcement learning. Specifically, the focus is on implicit communication for situations where explicit messaging is not possible. The paper proposes the Implicit Channel Protocol (ICP) framework. Unlike common implicit communication approaches like theory of mind which requires belief modeling of other agents, ICP uses a subset of actions (scouting actions) to broadcast in formation. It uses a mapping between the scouting actions and information that is learned either with a randomly initialized map or a delayed information map. The latter first learned with explicit communication before fixing the mapping. ICP is evaluated against baseline methods on 3 environments, including Guessing Number and Revealing Goals which are designed by the authors. Experimental results show ICP’s effectiveness in transmitting information more efficiently.", "questions": "- Isn’t broadcasting an assumption rather than a benefit? This is not possible in many practical scenarios \n- What do you mean by ‘efficient embedding techniques’ in line 268?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The effectiveness of the method is evaluated against two newly designed benchmarks and results show ICP being superior to baselines", "weaknesses": "- I fail to comprehend how the framework is a form of implicit communication. My understanding of implicit communication is the use of environment actions to communicate information (e.g., learning a protocol such that walking forward means yes and walking backward means no). ICP proposes add actions to the action space that maps information into certain ‘embeddings’ to be broadcasted to other agents. How is this not explicit communication? \n- Unless I am missing something, the channels are provided so the agents are not learning to construct these channels, they are simply learning to use those channels (in an explicit manner). This makes the paper title confusing\n- Results in Hanabi with only 3 random seeds are not enough. SAD reported results with 13 random seeds. Figure 2b is also missing error bar\n- Writing clarity can be improved. For instance, phrases like ‘commmunication gradient method’ and ‘hat mapping method’ are often used and assumed with a lot of context before defining it.\n- Some statements are not well justified. For instance, in line 251-254, it is unclear to me how this method specifically simplifies the learning process and promotes better coordination and communication among the agents\n\nMinor writing issues:\n- Line 122 changing environment unpredictable->unpredictably\n- Line 149 partial observe -> partial observation", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0LMzLcuueQ", "reviewer": "Reviewer_wGjX", "summary": "This work presents techniques for communicating through implicit channels (i.e. using environment actions instead of a dedicated communication channel). Specifically, by distinguishing between \"regular\" actions and \"scouting\" actions, agents can send messages through scouting actions. The first proposed technique uses the Gumbel-Softmax trick to have a fully differentiable communication pipeline through discrete actions, allowing a communication map between messages and scouting actions. The second proposed techniques adds a direct communication channel for pre-training and then uses a \"hat mapping\" strategy to encode and decode messages within scouting actions. These implicit communication techniques are effective at outperforming baselines in environments that require implicit communication, like Hanabi.", "questions": "- How does the delayed map approach perform in Guessing Number and Revealing Goals?\n- Can the proposed technique be used in situations where there is no clear delineation between \"scouting\" and \"regular\" actions?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The problem setting of constructing implicit communication channels is very important to multi-agent RL, and this paper takes important steps to tackling this challenge.\n- The hat mapping strategy is a quite smart application of the classic logic problem to a broader space of communication challenges.\n- The techniques and environments are easy to understand.", "weaknesses": "- All of the environments studied in this work have the same quirk as Hanabi, namely that agents cannot view the information they need but they can view the information for all other agents and have to communicate that information to other agents. This work would be more convincing if the key communication challenge between settings were more unique.\n- The task of learning an implicit communication channel in this paper does not seem too different from learning an explicit discrete communication channel, with the only major difference being that the \"scouting\" actions actually have some information prior whereas discrete channels are typically arbitrary. I would've liked to see how algorithms for explicit discrete communication channels compare to the proposed techniques in this paper as baselines. Furthermore, DIAL should be added as an explicit baseline instead of just comparing with VDN baselines. \n- Although two techniques are presented (random initial map and delayed map), the two techniques are only compared in the Hanabi game. Readers should be able to see the performance of both techniques across all environments due to the significant differences between the two.\n\n\nMinor note:\n- There are many grammatical errors throughout the paper, especially mixing up the use of singular and plural nouns and incorrect exclusions of definite articles (\"the\").", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)