Multi-Agent Reinforcement Learning with Communication-Constrained Priors
Authors: Guang Yang
Tianpei Yang
Jingwen Qiao
Yanqing Wu
Jing Huo
Xingguo Chen
Yang Gao
Venue: NeurIPS 2025 poster ()
Keywords: multi-agent reinforcement learning
mutual information
lossy communication
communication-constrained prior
Primary area: reinforcement_learning

Abstract: Communication is one of the effective means to improve the learning of cooperative policy in multi-agent systems. However, in most real-world scenarios, lossy communication is a prevalent issue. Existing multi-agent reinforcement learning with communication, due to their limited scalability and robustness, struggles to apply to complex and dynamic real-world environments. To address these challenges, we propose a generalized communication-constrained model to uniformly characterize communication conditions across different scenarios. Based on this, we utilize it as a learning prior to distinguish between lossy and lossless messages for specific scenarios. Additionally, we decouple the impact of lossy and lossless messages on distributed decision-making, drawing on a dual mutual information estimatior, and introduce a communication-constrained multi-agent reinforcement learning framework, quantifying the impact of communication messages into the global reward. Finally, we validate the effectiveness of our approach across several communication-constrained benchmarks.

Multi-Agent Reinforcement Learning with Comnunication-Constriiee Priors
Guang Yang'
Tianpei Yang 12+ Jingwen Qiao² Yanqing Wu²
Jing Huo
Xingguo Chen Yang G20
State Key Laboratory for Novel Software Technology, Nanjing University
2School of Intelligence Science and Technology, Nanjing University
Accepted "School of Network Security and Information Technology. YiLi Normal University
*Nanjing University of Posts and Telecommunications
{yangg.jingeen jingwenniiiooo yww.ingnnngnueeeemm (amaill catimmail.jj...ee.. njj.edu.ccm (tianpei. yang, huo jing,ing, gaoy) (aay)an Anju. senju.edu.ccm edu. cn chenxgonjupt edu. cn
Abstract
Communication is one of the effective means to improve the learning of coopera- tive policy in multi-agent systems. However, most real-world scenarios, lossy communication is prevalent issue. Existing multi-agent reinforcement learning with communication, due to their limited scalability and robustness, struggles to apply complex and dynamic weal-world environments. To address these chal- lenges, we propose a generalized communication-comm constrranne model to uniformly characterize communication conditions across different scenarios. Based this, we utilize it as a learning prior to distinguish between lossy and lossless messages for specific scenarios. Additionally, we decouple the impact of lossy and lossless messages on distributed decision-making, drawing on dual mutual information multi-agent reinforcement
estimatior, and introduce communication-constaaiii learning framework, quantifying the impact of communication messages into the global reward. Finally, validate the effectiveness of our approach across several coanuumication-conttaingee benchmarks.
Introduction
In multi-agent reinforcement learning (MARL) with partial observations, collaboration poses significant challenge [19]. Communication is one of the effective measures improve the learning of cooperative policy in MARL (31, widely applied in scenarios such as autonomous driving 4 24 and cooperative drones B 12). However, real-world scenarios are far from ideal and communication between agents often faces various constraints, specifically: (1) limited communication bandwidth, meaning only limited amount of message can be transmitted, and (2) lossy communication, where transmitted message may be subject to interference, delay, loss, and other issues.
Much of the research focuses on the limited bandwidth issue [22] 14 10 30). In this setting, communication channel is ideal, which means that transmission is real-time and lossless. Therefore, these methods typically only need to focus on how to effectively allocate communication resources (e.g., communication bandwidth, communication medium) to promote cooperation among agents. For example, compressing the communication information to extract and transmit parts beneficial cooperation [16], and systematically allocating communication mediums to avoid competition. among other problems (II).
multi-agent reinforcement learning (MARL)
assumed that the
*Corresponding Author: Tianpei Yang: Yang Gao
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

However, in most real-world scenarios, communication links are uncertain, and lossy communication more prevalent. In this setting. existing work typically addresses two types of issue: noise interference and communication delay. Approaches to addressing noise interference typically involve modeling the unknown noise distribution and constructing learnable processes to adaptively optimize cooperative policy 77] Another more common issue is communication delay, which frequently occurs in wireless network environments, which refers to the non-real- transmission of message, It has been shown to impact the performance of multi-agent behaviors (29 27 Approaches to addressing communication delay focus primarily on how to remove the impact of delayed message on cooperative policy, such as constructing communication buffer [29] and determining when to communicate [27]. Nevertheless, the above methods are all based on ideal assumptions about communication delay, and may not be applicable to more complex and unknown real- world scenarios such as underwater and caves. The reasons are as follows: (1) These methods lack scalability due to the lack of consideration of the common characteristies of lossy communication in different unknown scenarios. (2) These methods lack robustmess due to the lack of consideration of the dilemma of promoting the relevance of effective communication messages and suppressing the relevance of lossy communication information.
In order to overcome the above challenges, we first propose a generalized model of lossy com- munication to uniformly characterize the communication conditions in different scenarios, such as underwater, caves, and wireless networks. Based on this model, for specific scenarios, we use the lossy communication model learning prior to differentiate between lossy and lossless messages. Fur- thermore. for the second issue, we decouple the impact of different types of messages on distributed Decision-making borrowed from dual mutual information estimatior. On one hand, enhance the positive impact of lossless messages on decisionnmaking by maximizing the lower bound of mutual in- formation. On the other hand, we reduce the negative impact of lossy messages decision-making by minimizing the upper bound of mutual information. Finally, we propose a communication connttrrnnnnot MARL framework and validate its robust performance in two communication-constaaiin scenarios serving as benchmarks: Markov Model-Basee and Distance-Based communication constraints.
2
Related Work
The challenge of multi-agent collaboration within communication-constie environments has undergone extensive scholarly investigation. Existing research predominantly addresses two key dimensions: one concerns the optimization of interactions bandwidth-constren scenarios, and the other focuses enhancing system robustness under conditions of message loss.
2.1 MARL with Communication Bandwidth Constraints
Research on communication bandwidth constraints has primarily focused on how achieve efficient interaction among multiple agents under limited bandwidth resources. Zhang et al. (28) proposed the VBC method, which reduces communication overhead by constraining the variance of information exchanged between agents during the training phase, thereby eliminating noise from the messages. Kim al. [T] addressed the issue of medium contention in the information transmission process, proposing the SchedNet framework. By incorporating a MAC protocol from the wireless commu- nication field, this framework alleviates bandwidth pressure and addresses the scheduling problem of constrained agents. Mao et al. [8] introduced the GaLe algorithm, which uses gating mechanism to filter messages. It adaptively prunes unnecessary information by determining whether message is beneficial using threshold, thus reducing the number of messages exchanged. Hu et al. 9 proposed the ETCNet framework, which converts limited bandwidth into a penalty threshold for event-triggered strategies, improving communication efficiency in multi-agent systems by sending messages only when necessary.
These methods essentially perform effective compression on messages, which aligns closely with the information bottleneck (2I) principle-elimieeee eliminating task-irrelevant redundant information and extracting task-relevant refined information. Such compression approaches have been validated to exhibit a certain degree of generalization capability in similar tasks 255.. However, compression not equivalent to robustness 6 23 In environments with unstable or lossy communication, these
compressed methods may not be effective. For instance, unstable communication is highly prone to losing critical messages, leading to a significant degradation in the quality of policies.

---

2.2 MARL with Lossy Communication
Research on lossy communication has primarily focused on the robustness of multi-agent systems under non-ideal communication conditions, such as environmental noise, transmission delays, and packet loss. Freed et al. 77] proposed novel diffecrentiable communication method using random- ized message encoding scheme, where it mathematically equates discrete communication channels to simulated channels with additive noise, enabling gradient backpropagation through these channels. Kim et al. [13] proposed the Message-Dropout training method, based on the concept of Dropout, which randomly drops communication messages from other agents during training to improve robust- ness against communication errors during execution. Zhang et al. [29] introduced the TMC method, which leverages temporal locality reduce redundant messages and incorporates a message buffering mechanism to enhance robustness against packet loss, making it suitable for bandwidth-constran and packet-loss pross- network environments. Yuan et al. [27 proposed DACOM, an adaptive delay- aware multi-agent communication model in which agents can learn to schedule waiting times for messages from other agents, which suitable for delay-sensitive tasks and high-latency scenarios.
Nevertheless, the aforementioned studies predominantly remain confined to communication- constrained problems in specific scenarios. While these research approaches help to analyze particular issues in depth, its conclusions may not fully apply to other more complex and dynamic real-world application scenarios. To better align with real-world scenarios, in the next section, we have further refined the formalization of MARL with communication constraints.
3 Problem Formulation
Considering fully collaborative multi-agent task where each agent is in a partially observable and communicative environment, it can be modeled as decentralized partially observable Markov decision process POMDP) M with communication, which is defined by the following 9-tuple, G S, O, A, R, M, 7) N == (1...... N) is denoted by the set of N agents. S is denoted by the set of environmental states. O {o] is observation set of all agents, and of an observation set for agent i, which is determined by observation function Z(s, i). A A' x x... x A is the set of agents' joint action space, where A'' is denoted by the action space of agent i. r = R(s, a) the global reward signal shared by the agents. T A + [0. 1] is denoted by the transition probability. y E [0, 1) is the discount factor.
Furthermore, M is denoted by the message space, and m M is denoted by the message sent by agent to agent j. To uniformly characterize the communication conditions in different environments, we further introduce notation ((*) jäzj where /d E (0, 1) is denoted by the communication link status when agent i sends message to agent j. where indicates effective communication, while O indicates lossy communication. The set of message received by the agent can be defined by M m p 1 by each agent is different.
j==1- In this case where the communication link is dynamic, the information received
Then, given observation of and message M' at the time step f,, each agent i uses stochastic policy T-('''', M) to choose actions. We denote the joint policy as = { =,, x~ } E 11, where lI the joint policy space. In cooperative MARL, the collaborative team aims to find a joint policy to maximize the total expected discounted return J ((T) E 12 rrr...
4 Robust Learning with Communication-Conttaainn Priors
To overcome two challenges of multi-agent collaborative policy learning with lossy communication. this section will introduce our algorithmic framework, which specifically includes: (1) modeling
communication-constainne. priors to capture the dynamics of communication links; (2) estimating messages' behavioral impacts to characterize the correlation between different communication messages and agent behaviors; and (3) proposing a communiiation-ccontrriinn MARL approach to optimize and enhance the robustness of policy learning across diverse communication euvironments.
3

---

4.1 Conmunication--onstaaaiie Priors Modeling
MARL, modeling constrainted communication in unknown scenarios is crucial and challenging. It is necessary not only to abstract the common problems that affect multi-agent policy learning as much as possible, but also to generalize them in different real-world scenarios. Therefore, we first propose binary communication link parameter to characterize message reliability. addition, for different scenarios, the communication link can be further formalized as follows,
In
ت = Jo.
(1)
where 0e is the parameter determined by the environment, and s is the part of the state most relevant to agents and j. For different environments, fo. can be defined manually or obtained through pre- training of binary classification tasks. In the context of specific learning, there are several approaches: (1) When addressing specific and stable coammuicationnconstaaiinn
environments, one can estimate the priors of communication links through sampling or empirical data, enabling policy learning to adapt as closely as possible to these specific conditions. (2) In the face diverse and non-stable communication--cnsttaiinn environments, designing more diverse priors for communication links can allow policy learning to cover multiple exceptional scenarios. Specifically, message-dropout [3] is considered case of this prior modeling, where messages are randomly masked with certain probability to adapt to communication constraints. Therefore, incorporating such priors helps distinguish more effectively between lossy and lossless messages.
4.2 Messages' Behavioral Impacts Estimating
Building on the previous section, a natural goal is to maximize the utilization of lossless messages while minimizing the adverse effects of lossy messages. It is critical to measure the correlation be- tween messages and agent behaviors, where we will specifically characterize via mutual information.
4.2.1 MT between Messages and Behaviors
MI is often used to improve multi-agent collaboration in MARL 22 15 It is a theory that measures the correlation between different variables. When extended to communication-constriinn basic requirement is to measure the correlation between messages and agent behaviors. So we have
MARL,
T(m a²) H(a²) - H(alIm") ),j +
(2)
where H-))) and (--l-) denote the entropy and conditional entropy respectively, and m denote the message transmissed from agent j to agent i. (a') describes the ability to explore various behaviors of agent i, which could help generate diverse trajectories and avoid policy collapse when maximized. If (a" (a) measures the behavioral uncertainty of agent i, which encourages agent to behave deterministically given message m² when minimized.
4.2.2 Du-MIE for Constrained Communication
Note that the basic definitions of wwwpmiiicatooooonsttnnnnne prior and MI, we thus formulate natural objective propose for communncationn-contrriine MARL: maximize the MI between lossless messages and agent behaviors (to enhance their positive correlations) while minimizing the MI between lossy messages and agent behaviors (to mitigate their negative correlations). However, exact computation of MI is highly challenging, as it necessitates simultancously determining both the joint probability distribution and the marginal distributions of the variables involved (2). Inspired by the Dual Mutual Information Estimatior (Du-MIE) [5]. we construct Du-MIE for constrained commu- nication estimate the above objective. The
Cessen-Shanoon divergence (JSD) g estimates the lower bound of I T(m?;; a²) for maximization and the Contrastive Log-ratio Upper Bound (CLUB) (5) estimates the upper bound of I [mm"; a²) for minimization, based on replay buffer with lossless messages M and lossy messages M respectively. Specifically, JSD can be defined
MI estimator based
IIsp(m"):'" = Ep Ma [ssp (-To, (m²² ] - Ep NI 8P ssp (To, (m )
-c)1(01)
where IA is message-action joint distribution for agent i, P. @ PA is the product of the marginals. sp(x) = llog11 + e²) is the softplus function. Te, is a discriminator function modeled by

---

Soc(0r
Communiction Unksowa
JSD esiimate
envinoment
Real onviremoi
Istinate +
Ts (nfl
(e.(o)))))) ) Do (oc))))
Communication
Maseycssttion Encader
Action Proliction Network 6,
Mcssges
coosiege
Low movigg
9
Agoor
Butfer
Trajecoory
M6-
Gel, MfL-l)
4
Mulli-sgent
Figure 1: The overall framework for communication-consttiinn MARL. can be divided into three main steps: ① Distinguishing between lossy and lossless messages by constructing communication link priors; @ Shaping the global reward through learning Du-MIE for constrained communication; @ Stably optimizing multi-agent policies based on MARL algorithms.
neural network with parameters 01 E O my and at are obtained by joint sampling from the replay buffer D with lossless messages M while dj is sampled independently from the same buffer. By minimizing the loss (#(()), ), we can make Jssn (m="; a²) closely approximate the lower bound of [mm at)
On the contrary, CLUB can be defined as
IcLuB(me" aa) = =Ep log Top (a]lm?")
-Ep ME A
[log To, (ak mzf²)
(0))
where Te, is a variational approximation modeled by a neural network with parameters 02 E O. The samples are sampled the same way JSD, but are based on the replay buffer D with lossy messages M Similarly, by minimizing the loss (02),, we can make ICLUB (m3 a²) closely approximate the upper bound of [(m*";a")
Note that the training losses of the JSD and CLUB estimators are (B, and L (82) respectively, then, the overall training loss of Du-MIE for constrained communication is defined
Lcc(O1,0)) 02) = ΣΣ cO) (01) + (1- (1
(5)
The architecture of Du-MIE for constrained communication are illustrated in 1 After training, the ISD and CLUB can used as learning signals to guide the behavior of the agent towards lossless communication and away from lossy communication, which will be introduced in the next subsection.
4.3
Coammunication-Constaaiiin MARL
In order to characterize the impact of the upper and lower bounds of mutual information multi-agen policy learning, inspired by reward shaping 15 we reshape the global reward as follows:
7, Tt ΣΣ ae' (ss(mm";; a')) 3(1 (C))B a')
(6)
j
where a, 8 are weight coefficients. Then, a new goal in the communicationnccostriiied MALR is to maximize the expected discounted return J(T) )) = E, /
This reward shaping can be combined with different MARI algorithms. Taking CTDE-based MARL algorithms as an example, the results of the combination are presented in Algorithm T Both policy- based 17 and value-based [20) methods can derive a global temporal difference (TD) loss, which is
5

---

presented as follows:
LMARL( = a (ooa (st, ar) (f:+ ($1+1, (5(+).0+1)))
where Qoq is the parameters of the global Q-value network and Qo is the parameters of the corresponding target network. For value-based MARL,, the joint actions 4++1 are obtained via greedy policies 0++1 arg maXa4+1 2og (se+1: 0++1). For policy-based MARL, the joint actions
a++1 1 are obtained via target policies 4'+1 To.. (+10++1: M2+1), and additional policy loss (in actor network) must be incorporated, as follows:
[MARL = Es,,,rDD | (st,
(8)
Lines 5-13 present the sampling process of trajectories. Compared to traditional MARL,, the samples here not only include additional communication messages and the communication links but also reshape the global reward. Line 14 shows the Du-MIE training according to the equation ( Finally, in line 15, to adapt different MARL algorithms, the loss function should be correspondingly adjusted, as described equations and
Algorithm Communication Contsiiinn--n MARL
1: Input: maximum episode length T,, hyperparameters a and to balance the effects of MI, update frequency & for Du-MIE,
www.iiiccctooonnnsttnnnnne priors fo,
2: Initialize: main network parameters in MARL lq. 0x,, corresponding target networks 0f and
0 JSD parameters 0,, CLUB parameters 0z.
3: Initialize: experience replay buffer D.
4: repeat
5: fort== to 7 do
6:
Get patial observation or {o]
7: Get message M, {M]]
8:
Predict communication link status I, (4) 1+11=1> 4 -1" Jopi-
9:
Execute joint actions
{aj) via sampling at TT' (-10 M2).
10:
Receive 0++1 (o/ J M++1 {M²+ // and team reward Tt.
11:
Calculate the shaping reward Fr. according to the equation (6).
12:
end for
13:
Store = (o,, Mc, It, ot+1: M,41, (1, to to
14:
Update Du-MIE with replay buffer D every k steps, according to the equation
15:
Update network parameters in MARL, 0q. On, according to the equations (7) and
16:
until reaching maximum training steps
Experiments
In this section, we evaluate the algorithm's 's effectiveness from three aspects: overall performance, the impact of communication-consstaaa specific evaluation contents are as follows:
priors (CCPs), and the role of Du-MIE for messages. The
(1) Overall Performance: This aims to validate the proposed algorithm's performance under different communication-consteee
scenarios.
(2) Impact of Communication Priors: This focuses on verifying the performance and properties of the proposed method when using different communication priors.
(3) Role of Du-MIE for Messages: Through ablation experiments, this evaluation seeks to determine how this module impacts the learning of multi-agent policies under communication constraints.
5.1 Experimental Setup
In the experimental setup, we integrate the proposed algorithm framework with MADDPG 17 to form Communication-Consttaee MADDPG (CC-MADDPG) the primary validation target. It is

---

then compared with four baselines: MAIC (26). coll-Communiaation MADDPG (FC-MADDPG), Dropout-MAn MADDDG, and the standard MADDPG, operating without inter-agent communication. We adopt the Multi-Agent Particle Environments (MPEs) [ as benchmarks, To simulate communica- tion constraints, we employ the following two distinct models,
Markov-Bass -Base Communication (MBC); The Markov model [29] assumes that the state of a system any time determined by state transition probability matrix, which includes one noiseless state and multiple lossy states-the more the number of lossy states, the higher the loss probability. In this experiment, the transition probability matrices are set with dimensions of 3, and 8, corresponding to loss levels of light, medium, and heavy, respectively.
Distance-Basse Communication (DBC): This approach [27 simulates signal attenuation in real- world environments (e.g.. underwater or cave-like) based on inter-agent distances. Specifically, it sets distance threshold to determine the degree of communication constraint, where smaller distances lead to higher message loss rates. In this experiment, the distance threshold are set with 5, 3 and 1, corresponding to loss levels of light, medium and heavy, respectively.
5.2 Results and Analysis
5.2.1
Performance Evaluation
This section systematically evaluates the overall performance and robustness of the CC-MADDPG algorithm by analyzing average episode cumulative rewards across different MPE task scenarios and coamuuicattion-costraanne testing environments.
compares the performance (mean and standard deviation) of our algorithm with baseline meth- ods across various task scenarios and communication constraints.Experineee Experimenta results demonstrate that while MADDPG with communication achieves satisfactory performance in ideal communication environments, it exhibits high sensitivity to communication quality. Any form of communication constraint leads to significant performance degradation. MADDPG shows the lowest performance across all testing environments, with its average rewards substantially inferior to other communication- based algorithms, emphasizing the critical role of effective inter-agent communication for efficient collaboration in MPE tasks. For the MMDDUP algorithm, performance varies with different packet loss rates: the dropout-0.2 configuration generally achieves relatively better results, while higher dropout rates (e.g., 0.8) sometimes cause performance deterioration, as evidenced by its mere 36 average reward in Simple Reference scenarios and consistent underperformance compared to FC-MADDPG in most environments, suggesting that excessive message dropout during training undermines agents' ability to learn effective collaborative policies.
In contrast, CC-MADDPG consistently achieves average rewards comparable or exceeding other algorithms across both idcal and constrained communication environments. Notably, it maintains superior performance even under extreme communication conditions like heavy distance-based constraint (approaching non-communication scenarios), exemplified by its 138.0 performance in Simple_Tes Tag when FC-MADDPG deteriorates 1.5, demonstrating remarkable robustness. Compared with multiple communication-bes -based baselines, MAIC attains the lowest average performance across all four tasks and largely insensitive to the strength of communication constraints, indicating that the incentive messages it produces fail to effectively guide cooperation in these tasks.
An interesting observation emerges from our time-varying packet loss environments: performance variations across different network fluctuation levels (light, medium, and heavy) remain relatively insignificant. It may stem from two factors: (1) The predefined Markov-based packet loss patterns, while simulating bursty and correlated wireless channel characteristics, might generate insufficient average loss rates or inadequate durations of "bad states" to persistently disrupt critical collaboration moments, potentially mitigated by inherent robustness in agent policies; (2) The fixed 25-step episode length MPE tasks may limit full manifestation of aarvvvryingg channel impacts.
5.2.2 Impact of Communication Constraint Priors
This section investigates the effectiveness of incorporating communication constraint priors during the multi-agent reinforcement learning training phase, and analyzes how different prior strategies specifically influence model performance in communication:constiiinn environments.
7

---

Table 1: Performance Comparison of Multi-Agent Algorithms Under Communication Constraints
Algorithms
Task Sceuario
Testing Environment
MAIC
FC-MADDPG
Prepou-MAADDGG
MADDPG CC-MADIPG
Unsestricted
1.5+4.9
75.9=65.3
70.3+65.9
63.9+402.0
72.1=66.1
5.2+12.8
134.7=89.9
Light MBC
1.8+6.0
62.9-65.3
70.9+65.9
63.94462
62.
5.2412.8
1A3.6-89.9
Mediam MBC
1.6.1866
67.2.153.7
71.0165.7
67.1162.3
72.1.66.1
5.51156
134.9 90.9
Simple_Tagg Tag
Heavy MBC (8)
Light DBC (5)
2.3+7.8
1.7426.2
5444:43888
39.5=38.2
69.8166.5
69.0=6588
67.916322
67.3+62.4
72.7466.0
70.8=64.2
23.5+41.5
6.2214.2
136.9=89.7
131.4-66.2
Medium DBC (3)
2.4:6.55
10.9:26.5
70.1165.8
67.816624
70.7163.8
44.81.38.4
835
Heavy DBC (1)
2.7=9.1
1.::5.2
68.7=64.3
66.3+60.5
71446657
71,2470.9
138.0=88.1
Unrestricted
299.0==7..1
2138,7=260
-145
-137,5=21.6
-138.
1422.
-94.9=2667
-129.4220.1
Light MBC
293.3=63.8
-138.8-25.5
-138.14-774
-136.14-22.7
-193.5426.7
19.9
Medium MBC (6)
295.91.884
1138.7125.4
-145.4122.5
138.0.127.6
-136.01..2.5
199.99227.1
22.2.0120.22
Simple_Spreull
Heavy MBC
283.7451.8
-142.02269
-144.34242
-138.1220
-135.84-2255
190.99-27.7
19.2.7419.2
Light DBC (5)
301.2.165.5
2138.7124.4
-138.8124
176.6.127.6
-138.
1.220
19.0.5124.7
22.8.7120.3
Mediam (3C
282.4426388
-156.3=24.9
-139.74-21.2
-1434=27.0
-138.01:22.9
277..4225.6
1122.6119.2
Heavy DBC (1)
-289.5=63.1
-191.92268
-140.9420.8
144.55227.6
-138.0+23.5
169.6=27.1
-128.0+20.6
Unrestricted
0.2+14
61.0=6266
74.4278.3
51.6069.0
38.0=57.9
2.44.11
76,9476.8
Light MBC (3)
0.3+117
47.2=62.66
$4.3278.3
51.6166.4
57.3
2424.1
76,54766.8
Medium MBC
0.2+14
42.3=-3.2
74.4=78.0
81846799
58.
2,144.0
75.5+75..8
Simple_Referene Reference
Lighi DBC
Heavy MBC (8)
0.2+11.3
04.12.0
27.-
49.3.62.3
54.3478.2
53.31774
54.946733
53.0.70033
56.0
M. 58.
23.514.1
2.744.3
76.5
76.24-77.9
Medium (3C
0.34144
39.1556.5
53,3476.4
53.31799
37,24994
32,01-422
73.8:74.5
lleaa DBC (1)
0.1.1.00
4.0.12.9
528116.55
$3.2170.7
34.6.60.3
41.014.1
62.1170.1
Unrestricted
-29.51:26.4
6.711..1
668.500
6..14888
6.7.14.8
75.55148.5
-5.15.00
Light
-33.9426.7
-6745.11
-68450
-6744488
-75.52-18.5
-5.9500
Medium MBC (6)
-35.44:29.5
-7.145000
-7245.1
466-49
-6744.88
-629441.6
-6.155.0
Simple_Advent Advverrdyyryrry
Heavy MBC
-33.1428.7
.7.4..0
-684-..00
668449
-6.744.7
-51,5:3455
4.115.11
Light DBC (5)
-34.9431.0
-26.+115.1
-1084133
-6.-50
-6744.7
-140=80
664588
Medium DRC (3)
-29.34-25.4
-88664213
12.4419..
6.9-522
-6.744.7
888588
.777-7.2
Heavy DBC (1)
.35.01.9.3
45.31228
27.3.1995
-7.7...3
-68.1477
778.5.55
-887744
Table 2: Performance Comparison of CC-MADDPG with Different Priors
Priors Type
Task Scenario
Testing Environment
dropout-0.2
Test-Matched
Light DBC
20..
-119.0++29.4
Simple_Spread
Medium DBC
19,2
39.5
Heavy DBC
-128.0+20.6
-107.0133.3
Light DBC
75.5
84.2184.1
Simple_ Reference
Medium DBC
73.8
107.5+109.1
Heavy DBC
62.1
80.2
As evidenced by comparative results in Table algorithms introducing communication constraint priors during training demonstrate significant advantages. Using FC-MADDPG as baseline refer- ence, although it achieves high performance under ideal testing conditions, its performance suffers catastrophic degradation when encountering any form of communication constraints. For instance, in the Simple Tag scenario under distance-bases constraints, FC-MADDPG's average episode cumula- tive reward plummets from 75.9 in ideal conditions to 19.5, 10.9, and 1.5 respectively. This drastic performance deterioration highlights the vulnerability of models trained under ideal conditions when facing communication constraints.
In stark contrast, algorithms incorporating communication constraint priors during training, such DroDout-MADDPG and CC-MADDPG, exhibit superior robustness. Specifically, Dropout- MADDPG with dropout-0.2 prior achieves an average reward 68.7 the heavy distance-based constraint environment of Simple_ Tag, vastly outperforming FC-MADDPG's 1.5. Remarkably, CC- MADDPG utilizing the same dropout-0.2 prior even surpasses its ideal communication environment performance with score of 138. These results conclusively demonstrate that introducing commu- nication constraint priors during training enables better adaptation to constrained communication environments during testing, yielding enhanced performance stability.
To ensure satisfactory generalization unknown and variable testing environments, CC-MADDPG in this study adopts dropout-0.2 as the standard communication constraint prior by default. This

---

generalized random message dropout prior covers multiple potential communication constraint patterns, endowing the model with fundamental robustness. Building on this foundation, we further investigate whether employing training priors that precisely match the actual test environment constraints could yield additional performance improvements. Verification experiments reveal in Table 2 that models trained with lesstenviromeen--mtccelll priors (where training constraints exactly match testing constraints) consistently outperform those using generic dropout-0.2 priors. For example, in Simple_Spread's medium distance-based constraint environment, the generic prior model achieves 127.66 average reward versus -98 for the matched prior model. These findings indicate that while generic message dropout priors provide baseline robustness, precisely tailored priors reflecting target deployment environments can substantially optimize model performance.
5.2.3 Impact of Dual Mutual Information Optimization Module
This section investigates the effectiveness of the proposed dual mutual information optimization module through ablation studies. To ensure fairness in comparisons, all algorithm variants in this ablation study employ identical communication constraint priors (dropout-0.2). We designed four algorithm variants for systematic comparison:
Baseline Model: Excludes all mutual information optimizations, equivalent to the MADDPG with dropout-0.2 communication constraint prior during training.
Variant 1: Activates only the lossless messages utilization component in Du-MIE module, which employs JSD-based MIE to maximize the lower bound of mutual information between agent decisions and valid messages.
Variant 2: Activates only the lossy messages suppression component in the Du-MIE module. which only utilizes CLUB-based MIE minimize the upper bound of mutual information between agent decisions and invalid messages.
Full Model: Represents our complete proposed algorithm with both optimization directions activated in Du-MIE for messages.
Table summarizes the average episode cumulative rewards of these four variants across vari- ous conmunication-constaaaa
testing environments in Simple_Tag and Simple_ Simple_Spread scenarios. Between the baseline and Eigllecommponet variants, both mutual information optimizations inde- pendently enhance performance. In Simple_ Tag, the baseline achieves 70 average rewards across environments. Variant 1 (maximization-only elevates rewards to 81, while Variant 2 (minimization- only) significantly boosts performance to 120. This confirms the intrinsic value of mutual information optimization beyond prior knowledge utilization.
The full model consistently outperforms all variants, demonstrating synergistic complementarity between the two optimization components. For instance, in Simple_ Taae' 's most constrained short- distance environment, the full model achieves 138.0 rewards versus 68.7 7 (baseline), 81.7 (Variant 1), and 120.4 (Variant 2). Similar patterns emerge in Simple_Spread to -28..0 substantially surpasses the baseline (-143.4 to 145.6). These results validate that dual- directional mutual information optimization synergistically enhances multi-agent collaboration and system robustness under communication constraints.
where the full model (-127.6 ...6
Table 3: Ablation Study on Dual Mutual Information Module
MI Coefficients
Average Episode Cumulative Reward
Scenario
min
max
Unrestricted
Light MBC
Medium MBC Heavy MBC
Light DBC
Medium DBC
Heavy DBC
70.3:15.9
70.9:165.9
71.0265.7
69.8166.5
69.0165.8
70.1
68.71664.3
Simple_ Tag
0.01
81.3175.2
81.3:15.2
81.7477.0
81.6174.0
818874.5
81.725566
81.7173.3
0.001
120.5+82.1
119.8=81.0
120,3:8818
119.3:48.33
121.5+88455
119.5:80.5
120.4=78.7
0.001
0.01
134.74-89.9
133.6+89.9
134.9-190.9
131.4486.2
136.9418.77
135.3+45.11
138.0+88.1
-145.4123.5
14:2.2
-145.4222.5
-144.3424.2
-145.6123.4
-143.4121.2
-144.5420.8
Simple_Spreaa
0.01
-139.1424.8
-139.1124.88
-138.7=24.6
-138.7125.2
-138.8224.4
-139.7123.9
-140.0124.0
0.01
0
-143.0++24.8
-142,542488
-142.0=27.1
-141.92254
-1424=24.8
-141.7+26.8
-139.8422.6
0.01
0.01
-129.4+220.1
-129,2+19.9
.129.0=20.2
.128.7+1922
-128.7 20.3
1127.6+19.2
-1.8.0+20.6

---

Conclusion
Lossy communication remains a critical barrier hindering the practical deployment of MARL in real-world scenarios. To address this challenge, we propose a novel communicationn-contraind MARL framework that first establishes a unified prior over communication constraints via system- atic modeling of lossy communication patterns, enabling agents to adapt strategies across diverse watpuumniaiioooonstriiinn
scenarios. Second, by distinguishing between lossy and lossless mes- sages, we develop the Du-MIE to quantify the impact of messages agent behavior, integrating this into the reward function to enhance the positive influence of reliable messages and mitigate the negative effects of corrupted messages. Finally, when integrated with the MADDPG algorithm, our approach demonstrates superior performance in both overall tasks and ablation studies across bench- mark environments, validating its effectiveness in maintaining robust cooperative decision-makin under varying communication constraints.
In the future, several directions warrant further investigation: (1) Algorithmic Scalability: Whether the framework can be extended Value-based learning frameworks; (2) Adaptability to Dynamic Environments: Whether it can adaptively learn robust policies in highly dynamic communication- constrained environments?
Acknowledgments and Disclosure of Funding
This work is supported in part by National Natural Science Foundation of China (62192783, 62276122 62506157, (22), Jiangsu Natural Science Foundation (BK20243051) Jiangsu Science and Technology Major Project (BG2024031), the Fundamental Research Funds for the Central Uni- versities (14380128, KG202514), Nanjing International/Hong Kong, Macao and Taiwan Science and Technology Cooperation Plan (20230803 1) and the Collaborative Innovation Center of Novel Software Technology and Industrialization.
References
[1] Stefano V Albrecht, Filippos Christianos, and Lukas Schäfer. Multi-agent reinforcement learning: Foundations and modern approaches. MIT Press, 2024.
[2] Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In International conference machine learning, pages 530-539, 2018.
[3] Marwa Chafii, Salmane Naoumi, Reda Alami, Ebtesam Almazrouei, Mehdi Bennis, and Merouane Debbah. Emergent communication in multi-agent reinforcement learning for future wireless networks. IEEE Internet Things Magazine, 6(4):18 24, 2023.
[4] Dong Chen, Kaixiang Zhang, Yongqiang Wang, Xunyuan Yin, Zhaojian Li, and Dimitar Filev. Communication efficient decentralized multi-agent reinforcement learning for cooperative adaptive cruise control. IEEE Transactions Intelligent Vehicles, page 1-14, 2024.
[5] Pengyu Cheng, Weituo Hao, Shuyang Dai, Jiachang Liu, Zhe Gan, and Lawrence Carin. Club:
A contrastive log-ratio upper bound of mutual information. In International conference on machine learning, pages 1779-1788 2020.
[6] Yann Dubois, Douwe Kiela, David J. Schwab, and Ramakrishna Vedantam. Learning optimal representations with the decodable information bottleneck. In Advances in Neural Information Processing Systems 33, pages 18674-18690, 2020.
[7] Benjamin Freed, Guillaume Sartoretti, Jiaheng Hu, and Howie Choset. Communication learning via backpropagation in discrete channels with unknown noise. In The Thirty-Foure AAAI Conference on Artificial Intelligence, pages 7160-7168. 2020.
[8] Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchin Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In 7th International Conference on Learning Representations, 2019.
10

---

[9] Guangzheng Hu, Yuanheng Zhu, Dongbin Zhao, Mengchen Zhao, and Jianye Hao. Event- triggered communication network with limited-bandwidde bandwidth constraint for multi-agent reinforee- ment learning. IEEF Transactions on Neural Networks and Learning Systems, 3468)3396633778 2023.
[10] Shengchao Hu, Shen, Ya Zhang, and Dacheng Tao. Learning multi-agent communication from graph modeling perspective. In The Twelfth International Conference Learning Representations, 2024.
[11] Daewoo Kim, Sangwoo Moon, David Hostallero, Wan Ju Kang, Taeyoung Lee, Kyunghwan Son, and Yung Yi. Learning schedule communication in multi-agent reinforcement learning. In 7th International Conference on Learning Representations, 2019.
[12] Mintae Kim, Hoon Lee, Sangwon Hwang, Mérouane Debbah, and Inkyu Lee. Cooperative multi- agent deep reinforcement learning methods for uav-aided mobile edge computing networks. IEEE Internet of Things Journal, 1123))33040-38053, 2024.
[13] Woojun Kim, Myungsik Cho, and Youngchul Sung. Message-dropont An efficient training method for multi-agent deep reinforcement learning. In The Thirty-Third AAAI Conference Artificial Intelligence, pages 6079 6086,6086, 2019.
[14] Joash Lee, Dusit Niyato, Yong Liang Guan, and Dong Kim. Learning to schedule joint moduricommunication with deep multi-agent reinforcement learning, IEEE Transactions Vehicular Technology, 71(1): 71(1):406-427) 4200064-2,2200 2021.
[15] Pengyi Li, Hongyao Tang, Tianpei Yang, Xiaotian Hao, Tong Sang, Yan Zheng, Jianye Hao, Matthew E. Taylor, Wenyuan Tao, and Zhen Wang. Pmic: Improving multi-agent reinforcement learning with progressive mutual information collaboration. In International Conference on Machine Learning, pages 12979-12997, 2022.
[16] Yong Liu, Weixun Wang, Yujing Hu, Jianye Hao, Xingguo Chen, and Yang Gao. Multi-agent game abstraction via graph attention neural network. In The Thirly-Fourth AAAI Conference on Artificial Intelligence, pages 721 7211-7218, 2020.
[17] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-compeiiiii environments. In Advances in Neural Information Processing Systems 30, pages 6379-6390, 2017.
[18] Hangyu Mao, Zhengchao Zhang, Zhen Xiao, Zhibo Gong, and Yan Ni. Learning agent communication under limited bandwidth by message pruning. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, pages 5142-5149, 2020.
[19] Afshin Orvojlooy and Davood Hajinezhad. A review of cooperative multi-agent deep reinforee- ment learning. Applied Intelligence, 53(11):13677-1322 2023.
[20] Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt. Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Monotonic value function factorisation for deep multi-agent reinforcement learning. Journal of Machine Learning Research, 21(178):1-51, 2020.
[21] Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 ieee information theory workshop (itw), pages 1-5, 2015.
[22] Tonghan Wang, Jianhao Wang, Chongyi Zheng, and Chongjie Zhang. Learning nearly decom- posable value functions via communication minimization. In 8th International Conference on Learning Representations, 2020.
[23] Zifeng Wang, Shao-Lun Huang, Ercan Engin Kuruoglu, Jimeng Sun, Xi Chen, and Yefeng Zheng. Pac-bayes information bottleneck. In The Tenth International Conference on Learning Representations, 2022.
[24] Xiaolong Xu, Xuanhong Zhou, Xiaokang Zhou, Muhammad Bilal, Lianyong Qi, Xiaoyu Xia, and Wanchun Dou. Distributed edge caching for zero trust-enabled connected and automated vehicles: A multi-agent reinforcement learning approach. IEEE Wireless Communications, 31 12:36-41,2022
11

---

[25] Rui Yang, Jie Wang, Qijie Peng, Ruibo Guo, Guoping Wu, and Bin Li. Learning robust representations with long-term information for generalization in visual reinforcement learning. In The Thirteenth International Conference on Learning Representations, 2025.
[26] Lei Yuan, Jianhao Wang, Fuxiang Zhang, Chenghe Wang, Zongzhang Zhang, Yang Yu, and Chongjie Zhang. Multi-agent incentive communication via decentralized teammate modeling. In Proceedings of the AAAI conference artificial intelligence, volume 36, pages 9466-944 2022.
[27] Tingting Yuan, Hwei-Ming Chung. Jie Yuan, and Xiaoming Fu. Dacom: Learning delay-aware communication for multi-agent reinforcement learning. In Thirty-Seventh AAAI Conference on Artificial Intelligence, pages 1763-11771, 2023.
[28] Sai Qian Zhang, Qi Zhang, and Jieyu Lin. Efficient communication in multi-agent reinforcement learning via variance based control. In Advances Neural Information Processing Systems 32, pages 3230-3239 2019.
[29] Sai Qian Zhang, Qi Zhang, and Jieyu Lin. Succinct and robust multi-agent communication with temporal message control. Advances in Neural Information Processing Systems 33, pages 17271-17282, 2020.
[30] Li Zhou, Xinfeng Deng, Zhe Wang, Xiaoying Zhang. Yanjie Dong, Xiping Hu, Zhaolong Ning, and Jibo Wei. Semantic information extraction and multi-agent communication optimization based on generative pre-trained transformer. IEEE Transactions Cognitive Communications and Networking, 11(2):725-737, 2025.
[31] Changxi Zhu, Mchdi Dastani, and Shihan Wang. A survey of multi-agent deep reinforcement learning with communication. Autonomous Agents and Multi-Agent Systems, 38(1):4, 2024.
12

---

A
Appendices
A.1 Details on Experiement
A.1.1
Environmental Description
The experimental evaluation of this study was conducted in a widely used multi-agent particle eMVironment(MPE). Specifically, the following three representative MPE scenarios were selected:
Simple_ SampleTTag: This scenario simulates the classic predator-prey pursuit game. N predators with limited field of view radius (N is set to 3 in this experiment, and the evaluation results of and agents are supplemented in the appendix) need to collaborate to capture a prey. The predators are controlled by independent reinforcement learning strategies, with larger body size, lower movement speed and smaller acceleration. The behavior strategy of the prey is predefined (randomly sampling 100 candidate positions, selecting the optimal movement direction within its perception range through the distance evaluation function). When any predator contacts the prey, all predators will receive positive reward. In addition, 2 immovable obstacles are set in the environment increase the complexity of the pursuit task.
Simple_Spreent This scenario requires N agents (fixed to 3 in this experiment) to cover N pre-set fixed landmarks through collaboration. Similar to Simple_ Tag, each agent has a limited field of view. The reward for the agent is the negative of the sum of the minimum distances of all agents to their nearest uncovered landmarks. In addition, when there is a collision between agents, they will be penalized. Therefore, the agents must move quickly cover their respective target landmarks, avoid collisions with each other, and ensure that all landmarks are uniquely covered.
Simple Referencee This scenario is unique collaborative navigation task, which includes 2 agents and fixed landmarks of different colors. Unlike the previous two scenarios, each agent has global observation capabilities. The challenge is that the target landmark of each agent can only be known through the communication messages of another agent. The collective reward depends on the number of agents that reach the target landmark.
A.1.2 Parameters Setting
In this experiment, an NVIDIA RTXA5000 24GB GPU was used. The actor network of each agent is neural network with two hidden layers, each with 64 neurons, activated with ReLU, and the output layer with tanh activation function to output actions. All agents share a centralized critic network, whose hidden layer structure is similar to the actor network. In the JSD network, communication messages and actions are passed through a single-layer encoder with 32 neurons, respectively, and divergence. In the CLUB
the mutual information lower bound is estimated using Sensen-Shanme network, the middle layer has 32 neurons, activated with ReLU, and the output layer uses tanh activation to model the conditional distribution of lossy messages and actions.
Adam optimizer is used for all networks. The learning rate of actor network, JSD network and CLUB network is 1 x 10-4, the learning rate of critic network is 1 x 10-3, the discount factor is set to 0.95, and the target network update rate is set to 0.01. The replay buffer size 1 x 10 the message buffer size is x 103, and the batch size is usually 1024 (in the Simple_ Tag task, when the number of agents is and 9, the batch size is adjusted to 512 to avoid the problem CC-MADDPG training process exceeding GPU memory). The random seed is set to 1. The time step of each round is fixed to 25 steps. The total time step of training for all models is 4.0 x 10S. When the total time steps exceed 1024, the model parameters are updated every 100 total time steps. For more effective action exploration, Ornstein-Uhled Uhhenbeckk noise is added to the output actions of the actor network at the beginning of training with parameters f 0.15 and 0.2. At the beginning of training, the noise scale decays linearly with the number of training rounds.
Simpl_Tag
During the evaluation, the trained model is loaded and run for 100 episodes in various test environ- ments. The training process of each episode fixed to 25 time steps. The average episode cumulative reward and standard deviation of each algorithm are mainly recorded and compared.
13

---

A.2 More Experiemental Analysis
This study further explored the scalability of each algorithm when the number of agents increases. In the Simple_ __aeee_aaaaa scenario. theoretically, since there is no collision penalty between predators, increasing the number of predators should be able directly increase the probability capture and obtain higher cumulative rewards. However, the experimental results (see table 4 show that the increase in the number of agents also significantly increases the collaborative complexity of the task, which poses challenge to all tested algorithms.
Table 4: Performance of Multi-Agent Algorithms under Varying Number of Agents and Communica- tion Constraints
Algorithms
Tusk Scenario Testing Environment
FC-MADDPG
Dropout-MADDPG
MADDPG CC-MADDPG
0.5
Unrestricted
75.9165.3
70.3
65.9162.0
72.116661
5.24
134.7.189.9
Light MBC (3)
72.9
70.9
65.9162.1 72.5
5.2412.8
133.61189.9
Medium MBC (6)
67.2+:3.77
71.0=65.7
67.1 72.1
5.5+15.6
134.94190.9
Simple_ Tag
Heavy MBC (8)
54.4+:43.8
69.8
67.9+63.2 72.7+66.0
6.0+14.22
131.4+86.22
(3 agents)
Light DBC (5)
19.5:18.2
69.0
67.3 70,8:6642
23.5::41.5
136.9.189.7
Medium DBC
10.91:26.5
70.1
67.8112.4 70.7163.8 44.8.18.4
135.3.185.1
Heavy DBC (1)
1.515.2
68.7166.3
66.3:160.5 71.4165.7
71.2110.9
138.0118..1
Unrestricted
138.5+88.0
84.2-1.0.3
78.5:16811 73.1161.6
6.2412288
131.8:19.99
Light MBC (3)
123.9+47..0
83.5+70.3 78.1 72.9+61.3
6.2+13.1
131.9+-90.3
Simple_ Tag
Medium MBC (6)
78.5+63.1
82.1+70.9
79.6#67.9 72.0+64.3
6.4
133,2484.3
Heavy MBC (8)
69.8+59.5
81.3+69.0
78.6+65.4 75.2+61.8
1.0+15.3
131.8
agents)
Light DBC (5)
14.7431..0
83.0+71.0
80.1-68.6 75.7++62.5
6.6+11.8
135.3+89.4
Medium DBC (3)
5.4=13.4
75.5
79.3164.4 77.21:3.5
8.0
127.4419..1
Heavy DBC (1)
3.8=10.0
79.1
72.2+69.3 76.5+63.7
11.1+2009
122.1+77.8
Unrestricted
83.4+72.5
77.7471.5
68.0+61.0
19.4427777
7.2415.4
78,24488.1
Light MBC (3)
77.9+69.5
77.8+71.9
68.1+61.1
19.5+:304
7.3415.4
77.7472.6
Timple_Tag agents)
Medium MBC (6)
50.3++64.5
77.0+:1.77
66.3+-60.9
19.0+225.
8.3+12.7
78.9+72.5
Heavy MBC (8)
38.9135.3
77.4+:2.5
67.3161.9
16.5+22800
8.1417.0
77.6+73.4
Light DBC (5)
9.7-18.6
42.9:15.2
68.9164.6
19.9136.7
13.9422.5
82.7174.4
Medium DBC (3)
8.0=
27.1141.6
56,3160.5 20.4133.7 18.1132.5
83,717100
Heavy DBC (1)
1.87
21.0:33.4 49.0 21.143499 17.5130.8
83.2-170.6
Specifically, although FC-MADDPG achieved average reward of up to 138.5 in ideal environ- ment with agents, its performance dropped significantly after the introduction of constraints or when the number of agents increased, once again proving its dependence and its limitations under complex coordination. For the Dropout-MANI MADDPG series of algorithms, its robustness advantage weakens when the number of agents increases, and it is difficult to cope with environments with high collaborative complexity and strong communication constraints. For example, in the scenario of agents, the average rewards of dropout-0.2 in silance-bbsedd constraint are reduced to 42.9, 27.1 and 21.0 respectively.
ideal communication
In contrast, CC-MADDPG exhibits stronger robustness and scalability. In the scenarios of 6 and agents, its performance in an ideal environment close to that of FC-MADDPG, and always maintains leading performance level after the introduction of communication constraints. Secondly, when the number of agents increases, its robustness advantage over the simple message dropout method is more obvious in constrained environment. For example, in the scenario of 9 agents, under the heavy distance- based constraint, CC-M CC-MADDPG can achieve an average reward 83.2, while dropout-0.5 -0.5 is only 49.0. This shows that CC-MADDPG is more robust and effective than the simple message dropout method when dealing with more complex communication commuuuicationcconsrrnnned constrained problems.
14

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer:[Yes)
Justification:
The abstract and introduction clearly outline the communication-consiiei problem, the proposed communication-constaiin ceammmiiiaationoonttaiiii
MARL framework, its core innovations model, Du-MIE for messages) and experimental validation. These align with the detailed contributions in the paper, including algorithm design and empirical results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Our paper relies assumptions about real-world communication constraints, as validated in Section 5.2.2, but these assumptions can be simulated in certain realistic simulation scenarios.
Justification:
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specifiiattin, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
15

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: Our paper focuses on the design of algorithms for dealing with communication- constrained problems
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Our paper provides comprehensive details on methods, experimental setup, and parameters in the appendix material, ensuring full reproducibility of the main results. This information sufficiently supports the paper's claims and conclusions, independent of code and data availability.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to hosted model (e.g., in the of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
16

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: Our paper offers thorough methodological and parameter details, enabling reproduction of the main experimental results without the need for code. Comprehensive experimental setup and execution instructions are included facilitate this process.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
7/ //iips.c//pple wttpcccppppppppeeeeeeeeeeeeeeeee guwdes/ haaoll/ieggudeee/eeeeeuuuiisssss
sonooooooccs for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: Our paper offers thorough methodological and parameter details, enabling reproduction of the main experimental results without the need for code. Comprehensive experimental setup and execution instructions are included.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
17

---

Justification:
Our paper appropriately reports error bars or information on statistical signifi- cance, which shown in Section 5.2.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi-
dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Our paper provides sufficient information on the computer resources needed to reproduce each experiment in the appendix material.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: The research conducted in our paper fully complies with the NeurlPS Code of Ethics. We have carefully reviewed the guidelines and ensured that our work meets all the ethical requirements outlined.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
18

---

10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our paper is foundational, focusing on robust learning in communication- constrained scenarios. It does not directly address societal impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper does not involve the release of data or models that pose high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification:
Our paper does not use any existing assets such code, data, models.
19

---

Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new datasets, models, code as primary assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicablle as well as details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve any crowdsoureing experiments or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
20

---

Answer: [NA]
Justification: Our paper does not involve any crowdsourcing experiments or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core methods do not involve large language models. Any LLM use, present, would be for writing/formatting. which is not required to declare per guidelines.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy for what should or should not be described.
21


Reviews:
[{"review_id": "3cnKMPe5eY", "reviewer": "Reviewer_ZcE1", "summary": "This paper proposes a novel communication-constrained multi-agent reinforcement learning (MARL) framework designed to address lossy and unreliable communication in real-world environments. The authors introduce a unified communication prior model to distinguish between lossless and lossy messages across diverse scenarios and quantify their impact on decision-making via a dual mutual information estimator (Du-MIE). By shaping the reward with mutual information signals, the framework enables robust policy learning under varying communication conditions. Experimental results on MPE benchmarks demonstrate the method's effectiveness and generalization compared to baseline MARL algorithms.", "questions": "1. **Is the binary communication prior overly simplified?**  \n   The prior $\\iota_{ij} \\in \\{0,1\\}$ models message reception as either fully successful or completely failed. However, in many practical systems, messages may be partially received, noisy, or distorted. Can the current framework handle such intermediate or stochastic communication states?\n\n2. **Why is MADDPG used as the base algorithm?**  \n   Would the proposed Du-MIE framework still demonstrate consistent advantages if built on stronger backbones like MAPPO? An ablation using a more competitive base algorithm would strengthen the claim of generality.\n\n3. **What is the training overhead of Du-MIE?**  \n   Since both the JSD and CLUB modules are trained jointly with the policy, how much additional computational cost do they introduce? A comparison of training efficiency with and without Du-MIE would be helpful for understanding practical applicability.", "limitations": "The authors have adequately addressed the limitations.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n1. The dual mutual information estimator (Du-MIE) is a novel design that separately optimizes for lossless and lossy messages, enabling more robust policy learning.\n2. Extensive experiments across different constrained environments show consistent improvements over competitive baselines.\n3. The paper is well structured and clearly presented, with detailed ablations and intuitive motivation.\n\n**Weaknesses**\n\nDespite its novel integration of communication modeling and message utility estimation, the paper appears somewhat rushed in presentation, with several notable issues:\n1. **Inconsistent Mathematical Notation**: The notation lacks rigor and clarity. For instance, $r = R(s, a)$ should use bold $\\boldsymbol{a}$ to indicate a joint action, and the policy set $\\pi = \\{ \\pi_1, ..., \\pi_N \\}$ should use bold $\\boldsymbol{\\pi}$ for consistency. Such issues appear repeatedly and detract from the formal precision.\n2. **Limited Related Work and Baseline Coverage**:  The related work section is brief and omits prior efforts on communication under noise or loss. Including these works and comparing against stronger baselines would enhance fairness and context.\n3. **Lack of Experimental Detail and Visualization**:  The paper does not specify the number of seeds or clarify whether $ \\text{mean} \\pm \\text{std} $ refers to standard deviation. The absence of training curves also hinders evaluation of convergence and stability.", "was_revised": true, "final_justification": "The authors have addressed most of my initial concerns in the rebuttal:\n\n- Justified the binary communication prior with real-world examples.\n- Explained the choice of MADDPG and committed to adding MAPPO results.\n- Provided training overhead details and acknowledged notation inconsistencies.\n- Agreed to include training curves and expand related work.\n\nSome concerns remain (e.g., stronger baseline comparisons), but the paper is technically solid and the authors show willingness to improve.", "rebuttal": "We sincerely thank the reviewer for the time and efforts. Below please find the responses to some specific comments.\n\nQ1. Is the binary communication prior overly simplified?\n\nA1. Thank you for pointing out this. We would like to clarify this assumption in two points. First, it aligns well with real-world scenarios that the communication could be broken occasionally, e.g., For example, during network transmission, data are segmented into individual packets. Packet loss occurs when one or more packets are lost in transit and fail to reach the destination, yielding an inherently binary outcome—either the packet arrives or it does not. Likewise, in underwater communication among UUVs, once the distance exceeds a certain threshold, message transmission is naturally dropped, producing the same binary-like outcome. \nSecond, this binary-like abstraction of realistic communication scenarios poses a significant challenge to conventional communication-based MARL. As shown in Table 1, their performance degrades markedly as the proportion of lost messages increases. Moreover, the binary-like abstraction is also partially employed in the validation of the MAIC method. \n\nQ2. Why is MADDPG used as the base algorithm?\n\nA2. Thank you for your concern. We choose MADDPG as the base algorithm for two reasons. First, it embodies both the AC architecture and CTED framework, making it representative of many MARL methods. Second, MADDPG integrates naturally with communication mechanisms without requiring additional modifications, enabling ablation studies to cleanly verify the effectiveness of each module in our approach. By contrast, in standard MAPPO the actor network employs an RNN; when combined with a communication mechanism, this typically could necessitate designing new neural network components, making it difficult to disentangle whether performance changes arise from the neural network architecture or from our algorithmic modules. While our method is orthogonal to existing MARL approaches in principle. We will include more results on the backbone of other MARL algorithms, such as MAPPO in our revised version. \n\nQ3. What is the training overhead of Du-MIE?\n\nA3. We report the training time for each algorithm over 10 runs, with 4 million steps per run. All runs were conducted on the same server equipped with an NVIDIA A5000 GPU.\n\n| Scenarios | fc-maddpg | maddpg-dropout-0.2 | maddpg-dropout-0.5 | maddpg-dropout-0.8 | maddpg | cc-maddpg |\n| --- | --- | --- | --- | --- | --- | --- |\n| simple_tag | 5.71hr | 5.42hr | 6.61hr | 6.56hr | 8.03hr | 14.17hr |\n| simple_reference | 3.03hr | 3.24hr | 4.87hr | 3.03hr | 3.06hr | 5.95hr |\n| simple_spread | 4.13hr | 4.35hr | 8.49hr | 4.97hr | 8.21hr | 13.31hr |\n\nIt is noted that the Du-MIE module doesn’t induce heavy workload of training resources, considering the advantages contributed to the final performance.\n\nW1. Limited Related Work and Baseline Coverage. \n\nR1. Thank you for your comment. We have investigated the degradation of MARL performance under lossy communication. However, there are quite few works in this domain. Approaches that assume noisy (rather than lossy) communication differ from our problem formulation: Approaches with noisy communication assume that noise can be modeled and noisy messages can be repaired, while we assume that messages are directly lost and messages are unusable. Thus, approaches with noisy communication not suitable as baselines.\n\nW2. Lack of Experimental Detail and Visualization. \n\nR2. Thank you for your comment. For training our algorithm, we used random seed 1(no average over at least 5 seeds), which yielded the best performance. During evaluation, we did not fix the random seed. \nMoreover, our goal is to address the performance degradation that arises when deploying models in lossy communication scenarios, and robustness is the core metric we intend to highlight. \nWe believe that, instead of presenting numerous training curves for different tasks, a tabular format can more clearly convey the robustness achieved under various lossy communication scenarios.\nNevertheless, we will adopt your suggestion. We have compiled the training curves and will present them in the appendix.\n\nW3. Inconsistent Mathematical Notation. \n\nR3. Thank you for the comment. We will carefully ensure notation consistency in the final version."}, {"review_id": "PkPsE5Yihi", "reviewer": "Reviewer_fm9D", "summary": "This paper proposes a communication-constrained MARL framework that introduces: (1) a generalized binary link model to distinguish lossy/lossless messages across diverse scenarios; (2) a dual mutual information estimator (Du-MIE) using JSD to amplify useful messages and CLUB to suppress noisy ones; and (3) a reward-shaping mechanism integrating MI estimates into global rewards. Validated as CC-MADDPG in particle environments with Markov/distance-based constraints, it achieves significant performance gains by maintaining robust cooperation under dynamic communication degradation.", "questions": "1.See the weaknesses.\n\n2.During practical implementation, how is the agent-pair-specific state component sij\ndistilled from the global state s? Specifically, what key information dimensions does sij encompass?\n\n3.Would significant imbalance in the ratio between lossless and lossy messages lead to substantial performance degradation?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1.The paper is well written and easy to read, with the method/motivation/story communicated clearly to the reader.\n\n2.The paper employs a Dual Mutual Information Estimator (Du-MIE) to optimize the multi-agent communication mechanism, a framework that demonstrates both theoretical soundness and practical feasibility.\n\n3.This method framework represents a significant improvement over the baseline.\n\nWeaknesses:\n\n1.This study demonstrates a fundamental divergence from PMIC in core research focus: while PMIC primarily concentrates on optimizing multi-agent collaborative behaviors, this work specifically addresses the critical challenge of message transmission reliability in communication-constrained scenarios. Although differing in research objectives, both approaches employ the Dual Mutual Information Estimator (Du-MIE) as the key technical component. It should be noted that the methodology substantially inherits PMIC's foundational framework, and this design choice has, to some extent, constrained the innovative demonstration of this research.\n\n2.While the methodological framework of this paper demonstrates sound logic, the experimental design exhibits several limitations:\n\na)Inadequate baseline comparison: The significant improvement over MADDPG alone is insufficient to substantiate performance superiority. Comparative experiments with other mutual information-based communication optimization methods (e.g., NDQ, MAIC) should be included.\n\nb)Limited test scenarios: The validation using only three cooperative scenarios from MPE lacks adversarial/competitive environment testing, which constrains the generalizability of conclusions.\n\nc)Oversimplified test conditions: The observed \"relatively insignificant performance variations across different network fluctuation levels\" may stem from undemanding test environments. More challenging experimental settings should be employed to verify this finding.", "was_revised": true, "final_justification": "The author's response has addressed most of my concerns. Hence, I improve my rating to 4.", "rebuttal": "We sincerely thank the reviewer for the time and efforts. Below please find the responses to some specific comments.\n\nQ1. During practical implementation, how is the agent-pair-specific state component sij distilled from the global state s? Specifically, what key information dimensions does sij encompass?\n\nA1. Thank you for your concern. In MPE, the environment state is formed by concatenating the local observations of all agents. Simply splitting this vector yields the entries relevant to agents i and j, notably their respective spatial positions and velocities.\n\nQ2. Would significant imbalance in the ratio between lossless and lossy messages lead to substantial performance degradation?\n\nA2. Thank you for your concern. It indeed leads to degraded performance, as demonstrated by our experiments. We configured two lossy communication scenarios, MBC and DBC, correspondingly set two parameters—loss probability and distance threshold—that govern the ratio between lossless and lossy messages. We further stratified the severity of communication loss into three levels: high, medium, and low. As shown in Table 1, FC-MADDPG with full communication performs worse when the proportion of lossy messages exceeds that of lossless messages, whereas the no-communication MADDPG deteriorates when the proportion of lossy messages is lower than that of lossless messages due to interference from redundant messages. \nWe attribute the performance degraded mainly to the mismatch between the message distributions at training and testing. Specifically, FC-MADDPG is trained under full communication yet evaluated under lossy communication, while MADDPG is trained without communication but tested in the same lossy communication. We will add the analysis in the revised version. \n\n\nW1. Inadequate baseline comparison.\n\nR1. Thank you for the comment. We have conducted new experiments to compare with the baselines the reviewer suggested. However, we want to clarify that NDQ is incompatible since it addresses limited communication bandwidth and cannot be applied to lossy communication settings. MAIC is designed for discrete action spaces, accordingly, we discretized the MPE setting and have\n\n| Scenarios| Unrestricted\t| Light MBC(3) | Medium MBC(6) | Heavy MBC(8) | Light DBC(5) | Medium DBC(3) | Heavy DBC(1) |\n| --- | --- | --- | --- | --- | --- | --- | --- |\n| simple\\_tag | 2.7±9.1 | 2.4±6.5 | 2.3±7.8 | 1.7±6.2 | 1.6±8.6 | 1.8±6.0 | 1.5±4.9 |\n| simple\\_reference | 0.2±1.4\t | 0.3±1.7 | 0.2±1.4 | 0.2±1.3 | 0.4±2.0 | 0.3±1.4 | 0.1±1.0|\n| simple\\_spread | -298.0±75.1 | -293.3±63.8 | -295.9±78.4 | -283.7±51.8 | -301.2±65.5 | -282.4±63.8 | -289.5±63.1 |\n| simple\\_adversary |-29.5±26.4 | -33.9±26.7 | -35.4±29.5 | -33.1±28.7 | -34.9±31.0 | -29.3±25.4 | -35.0±29.3 |\n\nExperimental results indicate that MAIC performs worse than our proposed method under these lossy-communication scenarios.\n\nW2. Limited test scenarios.\n\nR2. Thank you for the comment. We add an experiment on a cooperative–competitive scenario (simple_adversary), and the results show that our proposed method likewise exhibits a performance advantage.\n\n| simple_adversary | fc-maddpg | maddpg-dropout-0.2 | maddpg-dropout-0.5 | maddpg-dropout-0.8 | maddpg | cc-maddpg |\n| --- | --- | --- | --- | --- | --- | --- |\n| Unrestricted\t| -6.7±5.1 | -6.8±5.0 | -6.6±4.8 | -6.7±4.8 | -75.5±48.5 | -5.8±5.0 |\n| Light MBC(3) | -6.7±5.1 | -6.8±5.0 | -6.6±4.8 | -6.7±4.8 | -75.5±48.5 | -5.9±5.0 |\n| Medium MBC(6) | -7.1±5.0 | -7.2±5.1 | -6.6±4.9 | -6.7±4.8\t| -62.9±41.6 | -6.1±5.0 |\n| Heavy MBC(8) | -7.5±5.0 | -6.8±5.0 | -6.8±4.9 | -6.7±4.7 | -51.5±34.5 | -6.1±5.0 |\n| Light DBC(5) | -26.6±15.1 | -10.8±13.3 | -6.3±5.0 | -6.7±4.7 | -14.0±8.0 | -6.4±5.8 |\n| Medium DBC(3) | -38.6±21.3 | -22.4±19.0 | -6.9±5.2 | -6.7±4.7 | -8.8±5.8 | -6.7±6.2 |\n| Heavy DBC(1) | -45.3±21.8 | -27.3±19.5 | -7.7±5.3 | -6.8±4.7 | -7.8±5.5 | -5.8±5.4 |\n\n\nW3 (original Q2). Oversimplified test conditions\n\nR3. Thank you for pointing out this. We would like to clarify this assumption in two points. First, this test conditions pose a significant challenge to conventional communication-based MARL. As shown in Table 1, their performance degrades markedly as the proportion of lost messages increases. Moreover, the test conditions (binary-like) is also partially employed in the validation of the MAIC method.\nSecond, it aligns well with real-world scenarios that the communication could be broken occasionally, e.g., For example, during network transmission, data are segmented into individual packets. Packet loss occurs when one or more packets are lost in transit and fail to reach the destination, yielding an inherently binary outcome—either the packet arrives or it does not. Likewise, in underwater communication among UUVs, once the distance exceeds a certain threshold, message transmission is naturally dropped, producing the same binary-like outcome. \n\nW4. It should be noted that the methodology substantially inherits PMIC's foundational framework, and this design choice has, to some extent, constrained the innovative demonstration of this research.\n\nR4. Thank you for the comment. We believe that the novelty of our method is by no means limited to this single aspect. We are the first to formalize lossy communication and to propose an innovative communication-constrained prior model. Leveraging a Dual-MIE scheme, we simultaneously (1) mitigate the adverse effects of lossy messages and (2) amplify the beneficial influence of lossless messages. Both the modeling framework and the solution procedure make a substantial contribution to MARL with lossy communication research and facilitate the rapid transfer of MARL techniques to real-world applications.\n\nMoreover, our work is not a mere duplication of PMIC. Two clarifications are in order:\n\n(1) Problem scope. We address robust policy learning under lossy communication, whereas PMIC focuses on solving overgeneralization issues in MASs.\n\n(2) Methodological framework. Although both approaches employ JSD and CLUB as MIE, their roles differ markedly: we measure the MI between messages and individual behaviors to control the influence of communication on policy training, while PMIC uses it to distinguish the optimal trajectories and sub-optimal ones. Consequently, our method does not require the additional dual buffer used in PMIC."}, {"review_id": "pmwP43kfPS", "reviewer": "Reviewer_UbAw", "summary": "This paper investigates the problem of lossy communication in communicative multi-agent reinforcement learning. To overcome the challenges, a natural goal is to maximize the utilization of lossless messages while minimizing the adverse effects of lossy messages. This paper proposes a novel framework that incorporates communication-constrained priors to model the lossy communication and uses a dual mutual information estimator (Du-MIE) to optimize the impact of messages on agent behaviors. The learned policies can efficiently communicate and collaborative act in multi-agent scenarios with lossy communication. The experimental results demonstrate the effectiveness of the proposed approach in maintaining robust cooperative decision-making under varying communication constraints in MBC and DBC.", "questions": "Q1: Can the proposed method still work well in cooperative scenarios where the agent can defect for self-interest? \n\nQ2: It seems one key related paper on limited bandwidth with information bottleneck is missing: \"Wang et al. 2020. Learning Efficient Multi-agent Communication: An Information Bottleneck Approach.\"", "limitations": "See above comments", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strength:**\n1. This paper investigates an interesting lossy communication problem in MARL and introduces a novel communication-constrained MARL framework that effectively addresses the challenges of lossy communication in multi-agent systems.\n2. The proposed method can be applied to many MARL methods, showing its robustness and adaptability.\n3. The authors provide extensive experimental results on MBC and DBC.\n4. This paper is well-written and clearly organized. \n\n**Weakness:**\n1. It is not clear the scalability capacity of the proposed method in multi-agent scenarios where the number of agents is huge.", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the time and efforts. Below please find the responses to some specific comments.\n\nQ1. Can the proposed method still work well in cooperative scenarios where the agent can defect for self-interest?\n\nA1. Our proposed method does not specifically address this issue. Our approach targets the alleviate the degradation of MARL performance under lossy communication, whereas cooperative scenarios in which agents may defect for self-interest—often termed social dilemmas—require dedicated methods (e.g., incorporating social preferences), which is out of our scope. We are happy to investigate the influence of lossy communication in such scenarios as future work\n\nQ2. It seems one key related paper on limited bandwidth with information bottleneck is missing: \"Wang et al. 2020. Learning Efficient Multi-agent Communication: An Information Bottleneck Approach.\"\n\nA2. Thank you very much for the reminder. We will add the relevant work in the revised version.\n\nW1. It is not clear the scalability capacity of the proposed method in multi-agent scenarios where the number of agents is huge.\n\nR1. Thank you for pointing out this. We have supplemented Appendix A.2 with results for settings with 6 and 9 agents. In addition, for verity the scalability capacity, we also add an experiment with 24 agents, presented below. The results indicate that as the number of agents increases, our method maintains its advantage.\n\n| simple_tag(24 agents) | fc-maddpg | maddpg-dropout-0.2 | maddpg-dropout-0.5 | maddpg-dropout-0.8 | maddpg | cc-maddpg |\n| --- | --- | --- | --- | --- | --- | --- |\n| unrestricted | 77.5±64.4 | 13.4±18.5 | 10.6±15.9 | 13.1±16.3 | 8.1±13.2 | 82.9±69.9|\n| Light MBC(3) | 77.3±64.5 | 13.5±17.2 | 10.6±15.9 | 13.1±16.2 | 8.2±13.2 | 82.6±69.2|\n| Medium MBC(6) | 67.3±51.0 | 14.1±19.5 | 9.5±14.4 | 13.2±16.4 | 7.9±13.1 | 83.6±69.9|\n| Heavy MBC(8) | 53.5±46.4 | 13.8±18.7 | 10.9±18.0 | 13.0±16.5 | 8.0±13.4\t| 85.0±71.5|\n| Light DBC(5) | 14.5±23.4 | 13.5±17.2 | 11.9±19.4 | 12.9±16.0 | 9.7±16.2 | 87.5±71.9|\n| Medium DBC(3) | 12.9±23.8 | 14.7±22.2 | 12.5±18.6 | 13.4±17.0 | 9.3±15.5 | 88.1±75.4|\n| Heavy DBC(1) | 10.5±19.2 | 13.6±20.3 | 12.2±18.1 | 14.4±20.6 | 9.0±14.9\t| 92.0±75.7|"}, {"review_id": "zj98qMPRcV", "reviewer": "Reviewer_c2rM", "summary": "The paper proposes a communication-constrained multi-agent reinforcement learning (MARL) framework to address the challenges of lossy communication in real-world scenarios. It introduces a generalized model to characterize communication conditions and uses this model as a learning prior to distinguish between lossy and lossless messages. The framework employs a dual mutual information estimator (Du-MIE) to quantify the impact of messages on agent behaviors, enhancing the positive effects of reliable messages and mitigating the negative effects of unreliable ones. The proposed approach is validated through experiments in various communication-constrained environments, demonstrating superior performance and robustness compared to existing methods.", "questions": "Please refer to the section \"Weaknesses\" for detailed information.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths\n1. The paper is reasonably well-written, with a clearly defined research problem and a clearly stated motivation. The overall structure is logical and the paper is generally easy to follow.\n2. The paper exhibits a certain level of originality.\n3. The paper demonstrates a certain degree of originality and is also highly readable, particularly Section 4.\n\n# Weaknesses\n1. Regarding the motivation, the main issue the paper aims to address is the problem of communication loss. Therefore, mentioning the issue of limited communication bandwidth (line 23-30) may be off-topic and could dilute the focus of the study. It is recommended to remove this part.\n2. The Introduction mentions that existing methods suffer from poor scalability. To better support this claim, it is suggested to include evaluations on larger-scale scenarios. Additional relevant experiments could be included to strengthen the paper.\n3. Another suggestion is to include a visualization section in Section 5 to further demonstrate the effectiveness of the proposed method, which would make the work more comprehensive.\n4. There are also minor issues, such as missing symbols in Equation (2)(6). The authors are advised to conduct a thorough review for such details.\n\nIf the authors can address this, I would consider upgrading my score.", "was_revised": true, "final_justification": "**Final Decision**: I decide to raise my score to 4.  \n**Justification**: The paper proposes a communication-constrained multi-agent reinforcement learning (MARL) framework to address the challenges of lossy communication in real-world scenarios. Overall, the writing is clear, the motivation is relatively well-defined, and the experiments are reasonably comprehensive. During the rebuttal phase, the issues that have been addressed through active interaction with the authors include the discussion on motivation and the addition of experiments in large-scale scenarios. The issue that remains unresolved is that the experimental scenarios are somewhat simple and need validation in more diverse and complex environments. Taking into account the authors' responses and other reviewers' comments, I decide to raise my score to 4.", "rebuttal": "We sincerely thank the reviewer for efforts and the useful suggestions. We have prioritized and organized the reviewer’s comments, and our responses are as follows: \n\nS1. It is suggested to include evaluations on larger-scale scenarios.\n\nR1. Thank you for the suggestion. We have supplemented Appendix A.2 with results for settings with 6 and 9 agents. In addition, we also add an experiment with 24 agents, presented below. The results indicate that as the number of agents increases, our method maintains its advantage.\n\n| simple_tag(24 agents) | fc-maddpg | maddpg-dropout-0.2 | maddpg-dropout-0.5 | maddpg-dropout-0.8 | maddpg | cc-maddpg |\n| --- | --- | --- | --- | --- | --- | --- |\n| unrestricted | 77.5±64.4 | 13.4±18.5 | 10.6±15.9 | 13.1±16.3 | 8.1±13.2 | 82.9±69.9|\n| Light MBC(3) | 77.3±64.5 | 13.5±17.2 | 10.6±15.9 | 13.1±16.2 | 8.2±13.2 | 82.6±69.2|\n| Medium MBC(6) | 67.3±51.0 | 14.1±19.5 | 9.5±14.4 | 13.2±16.4 | 7.9±13.1 | 83.6±69.9|\n| Heavy MBC(8) | 53.5±46.4 | 13.8±18.7 | 10.9±18.0 | 13.0±16.5 | 8.0±13.4\t| 85.0±71.5|\n| Light DBC(5) | 14.5±23.4 | 13.5±17.2 | 11.9±19.4 | 12.9±16.0 | 9.7±16.2 | 87.5±71.9|\n| Medium DBC(3) | 12.9±23.8 | 14.7±22.2 | 12.5±18.6 | 13.4±17.0 | 9.3±15.5 | 88.1±75.4|\n| Heavy DBC(1) | 10.5±19.2 | 13.6±20.3 | 12.2±18.1 | 14.4±20.6 | 9.0±14.9\t| 92.0±75.7|\n\nS2. Another suggestion is to include a visualization section in Section 5.\n\nR2. Thank you for the suggestion. Our goal is to address the performance degradation that arises when deploying models in lossy communication scenarios, thus robustness is the core metric we intend to highlight. We believe that a tabular format can more clearly convey the robustness achieved under various lossy communication scenarios. Nevertheless, we will adopt your suggestion. We have generated training curves and a bar chart comparing the final rewards based on the log data. These figures will be presented in the revised appendix.\n\nS3. It is recommended to remove the related work mentioning the issue of limited communication bandwidth.\n\nR3. Thank you for the suggestion. We will remove in the revised version.\n\nS4. The authors are advised to conduct a thorough review for minor issues, such as missing symbols in Equation (2)(6).\n\nR4. Thank you for the suggestion. We will carefully ensure equation consistency in the revised version."}]

Author rebuttal:

Meta-review: The paper proposes a systematic training method to cope with communication loss in multi-agent reinforcement learning with communication. With the availability assumption of ideal communication at the training phase, the authors used the communication loss prediction model based on communication link modeling. Then, the authors identified correctly received messages and lost messages and generated a shaped reward for training, boosting the impact of correct message and suppressing the impact of lost messages, based on bounds on mutual information. The idea flow is natural and systematic, and the proper use of lower and upper bounds of mutual information is interesting. The proposed method shows better performance than previous message-dropout. So, AC recommends the acceptance of the paper.
Decision: Accept (poster)