Learning to Communicate Through Implicit Communication Channels

Authors: Han Wang, Binbin Chen, Tieying Zhang, Baoxiang Wang
Primary area: reinforcement learning
Keywords: implicit communication, multi-agent reinforcement learning, the Hanabi challenge

Abstract:
Effective communication is an essential component in collaborative multi-agent systems. Situations where explicit messaging is not feasible have been common in human society throughout history, which motivate the study of implicit communication. Previous works on learning implicit communication mostly rely on theory of mind (ToM), where agents infer the mental states and intentions of others by interpreting their actions. However, ToM-based methods become less effective in making accurate inferences in complex tasks. In this work, we propose the Implicit Channel Protocol (ICP) framework, which allows agents to communicate through implicit communication channels similar to the explicit ones. ICP leverages a subset of actions, denoted as the scouting actions, and a mapping between information and these scouting actions that encodes and decodes the messages. We propose training algorithms for agents to message and act, including learning with a randomly initialized information map and with a delayed information map. The efficacy of ICP has been tested on the tasks of Guessing Numbers, Revealing Goals, and Hanabi, where ICP significantly outperforms baseline methods through more efficient information transmission.

Decision: accept (poster)
Reviewer scores: 8, 3, 6, 8

Reviews and discussion:
<thread>
creation date:
1729993594289

thread_type:
official_review

<thread_object>
creator: reviewer_wgjx

<content>
summary:
This work presents techniques for communicating through implicit channels (i.e. using environment actions instead of a dedicated communication channel). Specifically, by distinguishing between "regular" actions and "scouting" actions, agents can send messages through scouting actions. The first proposed technique uses the Gumbel-Softmax trick to have a fully differentiable communication pipeline through discrete actions, allowing a communication map between messages and scouting actions. The second proposed techniques adds a direct communication channel for pre-training and then uses a "hat mapping" strategy to encode and decode messages within scouting actions. These implicit communication techniques are effective at outperforming baselines in environments that require implicit communication, like Hanabi.

strengths:
- The problem setting of constructing implicit communication channels is very important to multi-agent RL, and this paper takes important steps to tackling this challenge.
- The hat mapping strategy is a quite smart application of the classic logic problem to a broader space of communication challenges.
- The techniques and environments are easy to understand.

weaknesses:
- All of the environments studied in this work have the same quirk as Hanabi, namely that agents cannot view the information they need but they can view the information for all other agents and have to communicate that information to other agents. This work would be more convincing if the key communication challenge between settings were more unique.
- The task of learning an implicit communication channel in this paper does not seem too different from learning an explicit discrete communication channel, with the only major difference being that the "scouting" actions actually have some information prior whereas discrete channels are typically arbitrary. I would've liked to see how algorithms for explicit discrete communication channels compare to the proposed techniques in this paper as baselines. Furthermore, DIAL should be added as an explicit baseline instead of just comparing with VDN baselines. 
- Although two techniques are presented (random initial map and delayed map), the two techniques are only compared in the Hanabi game. Readers should be able to see the performance of both techniques across all environments due to the significant differences between the two.


Minor note:
- There are many grammatical errors throughout the paper, especially mixing up the use of singular and plural nouns and incorrect exclusions of definite articles ("the").

questions:
- How does the delayed map approach perform in Guessing Number and Revealing Goals?
- Can the proposed technique be used in situations where there is no clear delineation between "scouting" and "regular" actions?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730631533061

thread_type:
official_review

<thread_object>
creator: reviewer_avmg

<content>
summary:
This paper looks into communication in collaborative multi-agent system under the formalism of multi-agent reinforcement learning. Specifically, the focus is on implicit communication for situations where explicit messaging is not possible. The paper proposes the Implicit Channel Protocol (ICP) framework. Unlike common implicit communication approaches like theory of mind which requires belief modeling of other agents, ICP uses a subset of actions (scouting actions) to broadcast in formation. It uses a mapping between the scouting actions and information that is learned either with a randomly initialized map or a delayed information map. The latter first learned with explicit communication before fixing the mapping. ICP is evaluated against baseline methods on 3 environments, including Guessing Number and Revealing Goals which are designed by the authors. Experimental results show ICP’s effectiveness in transmitting information more efficiently.

strengths:
- The effectiveness of the method is evaluated against two newly designed benchmarks and results show ICP being superior to baselines

weaknesses:
- I fail to comprehend how the framework is a form of implicit communication. My understanding of implicit communication is the use of environment actions to communicate information (e.g., learning a protocol such that walking forward means yes and walking backward means no). ICP proposes add actions to the action space that maps information into certain ‘embeddings’ to be broadcasted to other agents. How is this not explicit communication? 
- Unless I am missing something, the channels are provided so the agents are not learning to construct these channels, they are simply learning to use those channels (in an explicit manner). This makes the paper title confusing
- Results in Hanabi with only 3 random seeds are not enough. SAD reported results with 13 random seeds. Figure 2b is also missing error bar
- Writing clarity can be improved. For instance, phrases like ‘commmunication gradient method’ and ‘hat mapping method’ are often used and assumed with a lot of context before defining it.
- Some statements are not well justified. For instance, in line 251-254, it is unclear to me how this method specifically simplifies the learning process and promotes better coordination and communication among the agents

Minor writing issues:
- Line 122 changing environment unpredictable->unpredictably
- Line 149 partial observe -> partial observation

questions:
- Isn’t broadcasting an assumption rather than a benefit? This is not possible in many practical scenarios 
- What do you mean by ‘efficient embedding techniques’ in line 268?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730657891098

thread_type:
official_review

<thread_object>
creator: reviewer_d4qe

<content>
summary:
This paper proposes a communication framework for a multi-agent reinforcement learning system. Efficient and targeted communication in the form of query-key pairs have been explored under previous works. Moreover, in order to address the challenges of a non-stationary environment, prior works have used Theory of Mind (ToM) methods to infer the intentions/states of the other agents in the environment to make more informed decisions. Developing models of other agents, add complexity to the training of multi-agent systems. This paper proposes an Implicit Communication Protocol, where each agents actions are supplemented with a *communication/scouting* action, that controls whether an agents sends a scouting action/query. Unlike the attention mechanism, where all agents receive 'N' messages from all 'N' agents, this paper proposes a common channel that aggregates all the information into one message vector. This paper additionally uses a gating mechanism similar to IC3Net, but with a Gumbel-Softmax relaxation that allows it to encode it as a binary classifier that functions as ATOC's (Jiang et al, 2018) "initiator" gating mechanism. 

The paper utilizes the non-differentiable communication aggregator mechanism RGMComm. Additionally, standard end-to-end differentiable communication networks can instead be used. A lookup table of the local observation of each agent with the respected broadcasted messages are then constructed for all agents to discretize the communication channel. Moreover, they use the hat-mapping technique where agents can infer their own targeted messages from the common message. The messages are then passed along with the hidden states of the agents to get the updated action.

strengths:
The proposed method is interesting, particularly toward the foundational problem of efficient multi agent communication by generating a information table with respect to the observation and message. Additionally, research into reducing computational complexity for intention inference techniques of agents will be highly valuable to the large-scale multi-agent systems.

weaknesses:
1. The paper does not discuss the impact of scalability or impact of heterogeneous agents to the proposed framework for multi agent systems, as you increase the number of agents to say up to 10, 20, 50 agents.

2. The paper does not compare results for common communication architectures such as CommNet, TarMAC, SARNet etc for their environments but only with value decomposition networks (VDN), that do not perform communication as part of their actions. 

3. The paper does not discuss limitations on the size of the action spaces or performance with respect to more dynamic environments such as predatory-prey, cooperative navigation.

I believe more comprehensive experimental results are needed for the proposed framework.

questions:
The results for Hanabi are quite remarkable, however,  game rules indicate, the hints are only constrained towards revealing either the color or the number. Since ICP implicitly encodes the local observation of each agent (both for DIAL and RGMComm) into message vectors, I believe each agent observes in essence have access to the complete global state, which inherently breaks the rule of Hanabi, unless I am mistaken. I would like more clarification on this.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730702674926

thread_type:
official_review

<thread_object>
creator: reviewer_w1kg

<content>
summary:
This paper presents the **Implicit Channel Protocol (ICP)** framework, an approach for enabling implicit communication in collaborative multi-agent reinforcement learning (MARL) environments where explicit communication is unavailable or costly. ICP introduces a subset of actions termed "scouting actions" that allow agents to communicate indirectly by encoding and decoding messages through their choice of these actions. By leveraging these actions, ICP builds an implicit channel similar to explicit communication protocols. The framework is evaluated on tasks such as Guessing Number, Revealing Goals, and the Hanabi card game, where it demonstrates effective information transmission and improved performance over baseline methods.

strengths:
The paper is well-structured and easy to follow. The algorithm is clearly-explained, with clear definitions of necessary notations. The experiments are well-designed and comprehensive, with sufficient implementation details.

weaknesses:
1. The ICP framework requires pre-identified scouting actions that can serve as indirect communication channels. This dependency limits its applicability in environments where such actions are not readily available or are difficult to define.

2. It can help the readers to better understand the method if the authors can include a diagram of the algorithm pipeline.

questions:
1. In experiments, for agents trained with VDN, is the action space the combination of “regular actions” and “scouting actions“?  
2. In figure 3b, why are those methods not trained to the same length?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731772323275

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
### General Response

We thank the reviewers for their insightful comments. Here, we address three main aspects that may have caused confusion. Additionally, writing issues and the title have been updated in the revised version of the PDF.

---

#### 
**Q1: Clarification of Our Work's Background**

**A1:**
For our setting, which may cause reviewers to be confused, we focused on cooperative multi-agent reinforcement learning (MARL) **without any direct communication channels**. As mentioned in the third paragraph of Section 1, throughout history, humans have used observable actions to convey information indirectly to achieve implicit communication in this situation. In this work, we propose a novel communication protocol framework that enables agents to learn to transmit information through actions. Compared to traditional methods for achieving implicit communication, such as Theory of Mind (ToM)-based approaches, our framework allows for more stable and accurate information transmission. This stability and accuracy are significant enough to support the use of strategies originally designed for stable, direct communication channels to learn communication policies within this framework. Thus, we refer to this framework that enables agents to learn a stable and accurate action-based communication as 'Construct an Implicit Communication Channel'.

---

####
**Q2: Clarification of Section 4**

**A2:**
In this paper, by considering which actions are more suitable as carriers for indirectly transmitting information, we defined *scouting actions* in **Section 3 —a subset of the action space** that have either no or uniform effects on the state $s_t$ or the reward \( r_t \) but primarily influence the observation function $O_i(s_t)$. For scouting actions, we analyzed in Section 4 that they carry two types of information: (1) information reflected through the environment and (2) information reflected through the choice of the scouting action itself. Implicit communication methods like ICP and Theory of Mind (ToM) leverage the second type of information—*information reflected through the choice of the scouting action*. In contrast, methods like VDN utilize only the first type, which leads to the challenges mentioned in the third paragraph of **Section 3**. 

---

#### 
**Q3: Clarification of ICP's Action Space** 

**A3:**
One thing we want to clarify is that the action space of VDN and ICP are same, for VDN only used one action policy to choose both *regular actions* and *scouting actions*, while ICP use action policy to choose {*regular actions*, *send\_info*}, when *send\_info* is chosen, information strategy combined with mapping mechanism will select *scouting actions*.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes a multi-agent RL approach to learn implicit communication for multi-agent cooperation. Essentially, agents can use predefined scouting actions (actions that have either no or uniform effects on the states) instead of explicit messages to communicate with one another. The proposed method is technically solid and showed good performance. Overall, the proposed method is a good contribution to RL-based methods for solving implicit communication problems for multi-agent systems.