Wonder Wins Ways: Curiosity-Driven Exploration through Multi-Agent Contextual Calibration
Authors: Yiyuan Pan
Zhe Liu
Hesheng Wang
Venue: NeurIPS 2025 poster ()
Keywords: Multi-Agent Reinforcement Learning
Intrinsic Reward
Artificial Curiosity
Primary area: reinforcement_learning

Abstract: Autonomous exploration in complex multi-agent reinforcement learning (MARL) with sparse rewards critically depends on providing agents with effective intrinsic motivation. While artificial curiosity offers a powerful self-supervised signal, it often confuses environmental stochasticity with meaningful novelty. Moreover, existing curiosity mechanisms exhibit a uniform novelty bias, treating all unexpected observations equally. However, peer behavior novelty, which encode latent task dynamics, are often overlooked, resulting in suboptimal exploration in decentralized, communication-free MARL settings. To this end, inspired by how human children adaptively calibrate their own exploratory behaviors via observing peers, we propose a novel approach to enhance multi-agent exploration. We introduce CERMIC, a principled framework that empowers agents to robustly filter noisy surprise signals and guide exploration by dynamically calibrating their intrinsic curiosity with inferred multi-agent context. Additionally, CERMIC generates theoretically-grounded intrinsic rewards, encouraging agents to explore state transitions with high information gain. We evaluate CERMIC on benchmark suites including VMAS, Meltingpot, and SMACv2. Empirical results demonstrate that exploration with CERMIC significantly outperforms SoTA algorithms in sparse-reward environments.

Wonder Wins Ways: Curiosity-Driven Exploration through Multi-Agent Contextual Calibration
Yiyuan Pan Zhe Liu" Hesheng Wang"
Shanghai Jiao Tong University
(pyy030406, liuzhesjtu, wanghesheng) @sjtu. edu. cn
Abstract
Autonomous exploration in complex multi-agent reinforcement learning (MARL) with sparse rewards critically depends on providing agents with effective intrinsic motivation. While artificial curiosity offers powerful self-supervised signal, it often confuses environmental stochasticity with meaningful novelty. Moreover, existing curiosity mechanisms exhibit uniform novelty bias, treating all unex- pected observations equally. However, peer behavior novelty, which encode latent task dynamics, are often overlooked, resulting in suboptimal exploration in de- centralized, communication- free MARL settings. To this end, inspired by how human children adaptively calibrate their own exploratory behaviors via observ- ing peers. we propose novel approach to enhance multi-agen exploration. We introduce CERMIC, a principled framework that empowers agents to robustly fil- ter noisy surprise signals and guide exploration by dynamically calibrating their intrinsic curiosity with inferred multi-agent context. Additionally, CERMIC gener- ates theoretically-groundde transitions with high information gain. We evaluate CERMIC on benchmark suites including VMAS, Melting Meltingpot, and SMACv2. Empirical results demonstrate that exploration with CERMIC significantly outperforms SoTA algorithms in sparse- reward environments.
intrinsic rewards, encouraging agents explore state
Code: https://gitiie http:/hhwwb.ccoo/yyi////////ee
Introduction
Achieving effective exploration in complex Multi-Agent Reinforcement Learning (MARL) settings, particularly those characterized by sparse rewards and partial observability, remains a formidable scientific challenge [7, 43]. Intrinsic motivation, instantiated artificial curiosity, has emerged key ingredient for unlocking autonomous learning by providing self-supervised signals in the absence of immediate extrinsic feedback [29, 38]. This internal drive enables agents to acquire skills and knowledge that support robust, adaptive intelligence.
However, such novelty-seeking algorithms is susceptible to the stochastic environment dynamics or other unlcarnable noises (the "Noisy-TV" problem) [23]. Existing algorithms mitigate this chal- lenge through uncertainty quantification by exploiting global information in multi-agent systems. However, these strategies prove insufficient for intelligent agents, particularly heterogencous ones or those in large-scale systems: Firstly, such agents frequently encounter severe partial observability, rendering inaccurate uncertainty estimates due to insufficient replay experiences [20]; Secondly, in decentralized execution without effective communication, agents struggle to form accurate beliefs
"Corresponding authors. The authors are with the School of Automation and Intelligent Sensing, Shanghai Ji Tong University, the Key Laboratory of System Control and Information Processing, Ministry of Education of China, Shanghai 200240. Zhe Liu is also with the National Key Laboratory of Hlmann-Mchhine Hybrid Augmented Intelligence, Institute Artificial Intelligence and Robotics, Xi'an Jiaotong University.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

about others' latent states, undermining methods that presuppose shared inter- agent information. [13]. Altogether, these limitations highlight critical need for exploration mechanisms that are robust to partial observable and communication-les environments.
Insights from human cognitive development suggest pathway forward: children rapidly adapt to new social games not only through solo trial -rie-or,,,rrro, but also by observing peers, inferring intentions, and selectively imitating successful strategies [42, 16]. Such form of social learning, often driven by an innate curiosity about 'why" others act as they do, allows for swift coordination and an understanding of task dynamics, even without complete information or explicit instruction [21. 30]. Naturally, the success of this human-centra learning process motivates translating its core principles Multi-Agent Systems (MAS). Therefore, this paper seeks answer:
How can robust. context-aware calibration of intrinsic curiosity unlock truly autonomous and effective exploration for MARL agents in challenging real-world settings?
To this end, we propose Curiosity Enhancement via Robust Multi-Agent Intention Calibration (CERMIC), modular, alug-and-play component designed to augment existing MARL exploration algorithms. Based on the Information Bottleneck (IB) principle [41, 40), CERMIC learns a multi-agent contextualized exploratory representation that steers exploration toward semantically meaningful novelty, and filters unpredictable and spurious novelty. Specifically, it incorporates a graph-based module model the inferred intentions of surrounding agents and use the context calibrate raw individual curiosity signal at a given coverage level. At each episode, CERMIC yields a loss for self-training and grooretiaalyy-rrouneedd intrinsic reward for exploration. We empirically validate CERMIC by integrating it with various MARL algorithms and evaluating its performance across challenging benchmark suites, In summary, our contributions are threefold:
We introduce CERMIC, novel framework that empowers MARL agents with socially con- textualized curiosity. Inspired by developmental psychology, this offers a novel perspective the crucial challenges of effective exploration in sparse-reward settings.
We propose robust and controllable multi-agent calibration mechanism in challenging partially observable and communication-line limitee environments. CERMIC allows for adap- tive tuning based on the learned reliability of the intention graph, effectively dampening exploration instability often plaguing vanilla govelty-seeking agents.
We deliver CERMIC as lightweight, readily integrable module and demonstrate consistent gains over strong baselines across standard benchmarks under sparse rewards.
2
Related Work
Naïve Exploration in RL. Early exploration strategies in reinforcement learning (RL), such as e-greedy or Boltzmann exploration, gained popularity due to their simplicity and ease of integration into various algorithms [25, 35, 10]. However, these "naive" exploration methods often perform suboptimally, particularly in challenging scenarios characterized by sparse rewards or deceptive local optima. Effectively, the agent explores by taking largely random sequences of actions, which, especially in continuous state and action spaces, makes comprehensive coverage exceptionally difficult. Even in the foundational setting of multi-armed bandits (MAB) continuous spaces [6]. more theoretically sound yet still model-free exploration strategies like Thompson Sampling (TS), Upper-Confidenne Bound methods, and Information-Dires
Sampling (IDS) have been developed [37, 33, 11]. However, while these methods offer improvements over purely random approaches, their efficacy remains limited when progress requires discovering semantically meaningful novelty far from the agent's initial experience distribution. This motivates the development of intrinsic reward that can provide more targeted and adaptive exploration signals.
Curiosity-Driven Exploration Intrinsic rewards are a powerful tool for driving exploration in RL, especially sparse-reward single-agent tasks [29, 4]. Early methods often quantified novelty via visitation metrics like pseudo-counts or hashing. Subsequent approaches broadened this by measuring surprise through prediction errors (of transitions or features), state marginal matching, uncertainty estimates, even TD errors from random reward predictors [18, 32). Then, multi-agent exploration, however, presents unique inter-agent dynamics requiring tailored intrinsic rewards. Some works have explored inter-agent observational novelty or influence over others' transitions/values
2

---

[44]. Others have investigated shared intrinsic signals (e.g., summed local Q-errors) or information- theoretic objectives like maximizing trajectory- mutual information for behavioral diversity [27]. Techniques such as goal-hased methods with dimension or observation-spen space goal selection also aim to manage state space complexity [12, 22]. However, many multi-agent intrinsic rewards implicitly rely on centralized training signals, privileged communication, or access to global state-assuuan assumptions that break down under decentralized execution. addition, severe partial observability and heterogeneous policies can mis-calibrate uncertainty-basee novelty estimates, while curiosity mechanisms that treat all unexpected events uniformly tend to amplify spurious novelty and overlook socially informative cues encoded in peers' behaviors. To address these shortcomings, propose CERMIC, novel framework for robustly calibrating intrinsic curiosity via learned models of inferred inter-agent intentions.
3 Background
Problem Setup. We consider a MAS operating within a communication-les, partially observ- able Markov Decision Process (POMDP) [5]. Formally, the environment described by tuple (O,A, P, R, N, 7). At each timestep 1, each agent E receives a local observation of E O and select action aj A via decentralized policies, which induces state transition. All the agents receive shared extrinsic rewards rf E R after executing the joint action. In this paper, we focus on the individual agent and omit index throughout; We use uppercase letters denote random variables and lowercase letters to denote their realizations.
Preliminary. The Information Bottleneck (IB) principle [2, 17] guides the learning of a compressed representation of an input X that maximally preserves information about a target Y. This is achieved by optimizing the trade-off:
max T(Z - -x((;
where -(: denotes mutual information. The first term, Y): measures how much information the representation Z contains about the target variable Y, ensuring that Z preserves the relevant predictive information for Y; The second term, I(X; Z), quantifies the amount of information that Z retains about the original input X, compelling Z to be a succinct summary of X. In our work, the IB principle serves as a foundational concept.
Method Overview. CERMIC processes transition experiences (st, at Sc+1, r{-1) to generate intrin- sie rewards and guide exploration, where state embeddings 8t and 8,+1 obtained from observations or and O++1 using encoders with the same structures. To ensure stable representation learning, the parameters of 0c-1 's encoder (6m) is updated via momentum moving of o, 's ones (f). The core of CERMIC is to learn a latent representation Te, parameterized as a Gaussian distribution conditioned on the current state-action pair: 22 go (8t, dg). Following the IB principle, CERMIC seeks maxi- mize mutual information I I(Xx; S++.1 to retain predictive information of novel states and encourage exploration, while minimizing 7 [(x;; [Se, Ar)) exploit information about the current context and gain compressed representation. The compression process min I (X,; [Se, Arl) incorporates robust calibration mechanisms leveraging multi-agen context. The overall objective formulated as Eq. (1):
max 1 (Xx; S,-1) - (X,; A.))
(I)
where a is Lagrange multiplier. In what follows, this work tries address two key questions: (i) How is CERMIC trained (Section 4.3)? (ii) How does CERMIC generate intrinsic rewards to drive exploration (Section 4.4)?
4 Method
4.1
Novelty-Driven Exploration
To drive exploration, we aim to maximize the mutual information I(Se+1; Xt). However, direct optimization is intractable. Instead, we resort to maximizing tractable variational lower bound.
Po (0(
S441) = =Epponaeta)
log p($(41)
+ DKL (
(2)
2 Epter [log (8441/2))) H(S++1)
3

---

Out
Lass
MARL
Qurissity Genoration
Sc-1
Module
S
Pp(K)S
sompling
(y(S++1(X,)
Couptore
Multi-Agent Inference
betectedi
θ
Expechatinn Optirsatioon
Previous Hemory
explact b min (s))Ae))
exploration
Intention
Fecture
f
Module
attn
InfoNCE
componsate
A
Figure 1: Workflow of CERMIC. (Left) Illustrates CERMIC's per- per-timestep objective. (Right) Depicts the network architecture. By projecting the raw observation o, into dower-dimensional embedding 8t, subsequent computations within CERMIC operate on compact representations, contributing to its overall computational efficiency.
where ( (xy) is a variational encoder parameterized by approximate the unknown true conditional distribution 1($++1/2;) Then, by the non-negativity of the KL-divergence, we obtain the second line expression. Since the entropy the true next state distribution H(S is independent of the model parameters ø, it's equivalent discard this term. Thus, the exploration objective simplifies to maximizing a log-likelihood loss under the variational approximation.
Lexplore
E Epla
Po 2x)]
(3)
4.2 Multi-Agent Contextualized Exploitation
Similarly, to minimize the intractable mutual information I(X,; [S,, A,]), optimize its variational upper bound. Following prior work [2], we introduce a Gaussian variational approximation g(x;) ~
N (0, I) for the intractable marginal distribution p(ar.) = p( St, as (p(e,) (St, as ddsdda;:
I([S,, 2x) Ep(er. log p(a,lst, a4) DKLlp((e)) S a Dt-
g(xx)
where the inequality holds due to the non-negativity of KL-divergence. For conciseness, let VA log p(gp 9t, az) (z)/)(z)). The mean piy and variance Eg V, ~ p (No. Myy) can be numerically computed through p(aa|ss, az) and q(x.). However, directly computing the expectation in Eq. (4) remains challenging due to the distribution shift caused by replacing p(z;) and the high-dimensional Monte- Carlo (MC) sampling required [36]. To address this, enforce upper bound č on L to relax the minimization term using a chance constraint:
(V S - P
(5)
Crucially, to imbue this compression process with awareness of other agents in communication- less MAS setting, we condition the chance constraint on an inferred multi-agent contextual feature, fa. This feature f encoded from an underlying intention modeling of other agents and the specific architecture of this intention modeling can be varied. In our primary exposition, we exemplify the intention model as a dynamic graph, G, (additional analyses on the effects of pre-trained intention models and other forms of intention modeling can be found in Section 5.3 and Appendix G). The nodes in G, encapsulate predictive state representations for each agent, while the edges represent their relative spatial relationships. We maintain a memory queue of the intention modeling over time, and the graph G, will advanced to G. only when other agents are detected by a simple MLP module, ensuring efficient computation under partial observability (the trained MIP can identify and match different agents). The generation of G, at each step utilizes an attention mechanism over historical intention modeling and current observations. Subsequently, Graph Neural Network

---

(GNN) [34] processes G, to yield the contextual feature f.. as detailed in Appendix A. Ultimately, regardless of its precise origin, this inferred context fn is integrated into the chance constraint distributional robustness:
infp,epo (V - 2) 2 21 cal
(6)
where
{
(Es, [v] - H.v (f()) E S m
Pu
P. R Es (VV, pv) ) v 22
cal
Here, 71 >> O and 72 > max(n.1 1 are constants, and the mean value now is dependent on the feature fn. The first inequality of P represents the true mean value is within an ellipsoid centered at pi (Ja) with bound 71 while the second one constrains the true variance with a bound 72- The robustness of the set has been proved using McDiarmid's inequality [8], and the optimal 71 72 can be correspondingly derived. Next, this robust chance-constraint formulation can then be tractably converted into the following second-order cone loss Eq. (7) via Cantelli's inequality [26] (see Appendix B for proof), resulting in loss function where 3 is constant regarding 7i 72, In essence, minimizing this loss promotes exploitation within the defined robust calibration set.
C Cexpliit
- ReLU (f+) BVV c).
To ensure the robustness and stability of the curiosity system in the worst case, a lower bound loss also applied as follows, transformed from (V,, Z g) after normalizing Vr,
We can set task-agnostic č and g
CLB
A ReLU(-Hu (f+) + B -c). exploit
(8)
Task-Adaptive Calibration. This paragraph details the implementation of the calibration mechanism for the mean parameter po (fR). We formulate the calibrated mean as network-parametteee function h, regarding the inferred multi-agent context fü, task-adaptive factor 7, and the original mean pag. 7 is the mutual information between the inferred intention fü and the preceding context composed of the previous intention f2-1 and the received extrinsic reward rf
po ( fh = h(TS2 mw)
7 quantifies the consistency of the inferred intention S within the task context provided by the reward signal. It enables adaptation during different stages of learning and task execution: (i) Learning Progress Adaptation. Early in training, when the intention model fü is inaccurate due to partial observability and limited experience, the resulting low mutual information f automatically down-weights the influence of these unreliable inferences on the calibration of (uw; (ii) Task Progress Adaptation. The factor y also captures the alignment between inferred intentions and the underlying task dynamics (cooperative/adversariall even if the intention modeling (e.g., predict state) is accurate, failing to comprehend the task context can lead to actions yielding unexpected rewards rf_1 and result a low 7 as well.
Yet, the mutual information / /i (i) remains intractable. To address this issue, we draw inspiration from contrastive learning, which effectively utilizes negative sampling and acts as regularizer, preventing collapsed solutions and enhancing the stability of the calibration process. Specifically, we employ the InfoNCE loss [39] to establish tractable bounds for the mutual information:
- S
fi) S Doc
(10a)
I 2 Ep( log EREF us: exp(c([rc-1) fi
(10b)
where c is a score function (implemented as boltnaar+softtax in our work) assigning high scores to positive pairs Tf and F denotes the set of negative examples [ff=1,77f1 (TT-::f)))1 / During learning, the positive samples are generated from the memory module, while the negative ones J are computed by adding additional noise to f The derivation of these bounds is detailed in Appendix C. For consistency with the inequality directions in the chance constraints, we substitute the left-hand side of Eq. (10a) into Eq. (7), while substituting the right- hand side into Eq. (8).
5

---

end for Algorithm CERMIC
1: Initialize: CERMIC and the actor-critic network
2: for episode j = 1 to M do
3: for timestep = 0 to T 1 do
4: for each agent n = 0 to N do
5:
Obtain action from the actor o, π (s;), then obtain the state $1411
6:
Add (st, at, 8t+-1) into the on-policy experiences;
7:
Obtain the intrinsic reward rł by Eq. (12) and update rf with r1 rī + rg:
8:
end for
9:
end for
10:
Update the actor and critic with the collected on-policy experiences as the input;
Update CERMIC by gradient descent based on Eq. (11) with the collected on-policy experi-
ences;
4.3
Loss Module
The final loss for training CERMIC is a combination of the upper and lower bounds established in previous sections. CERMIC'S parameters θ encompass the components illustrated in Fig. 1. Owing to its operation a low-dimensional space and reliance on inputs that are readily available standard MARL pipelines CERMIC serves as an efficient, plug-and-play module.
mine Le
CLB Exploit + Cexploit C.Coppore.
(1I)
4.4 Intrinsic Reward Module
In this section, detail how CERMIC generates intrinsic rewards foster exploration. Our aim is to not only encourage novelty seeking but also to be theoretically compatible with extrinsic rewards. To this end, drawing inspiration from Bayesian Surprise [24], we formulate the intrinsic reward rt:
D (p( dz St+1) U D..)) P (O.)..
(12)
where @ represents the parameters of the CERMIC module and the dataset 2m comprises transition tuples (St, ae, St++1) collected over the past m episodes. Intuitively, this intrinsic reward encourages agents prioritize the exploration that maximally informative for optimizing the CERMIC module itself. In summarize, we show the overall MARL algorithm with CERMIC in Algorithm 1.
Theoretical Analysis in Linear MDPs Analyzing dynamics of intrinsic rewards in high- high-dimensional non-linear environments presents significant theoretical challenges. However, we provide a theoretical analysis within the framework of linear Markov Decision Processes (MDPs), where the transition kernel and reward model are assumed to be linear (i.e., F2 = wen(sst at), where w, is the CFRMIC parameters in linear MDP settings and 7) is a feature embedding function). Within this well-studied setting, algorithms like LSVI-UCB [14] are known to achieve near-optimal worst-case regret bounds, largely due to their principled exploration strategy.
The exploration bonus in LSVI-UCB, denoted r JCBB-DB quantifies the uncertainty in the value estimate for the state-action pair (st, ar) and encourages optimistic exploration. We proved that, Our proposed intrinsic reward r is closely related to this UCB-DB bonus, as formalized in Theorem I.
Theorem Consider linear MDP setting where the estimation noises of optimal parameters and transition dynamics are assumed to follow standard Gaussian distributions N (10, I). Then, for any tuning parameter p 0, it holds that
UCB.BD DB
< r
22p T DBB.DB
(13)
By tracking the UCB-DB bonus, which naturally decays as uncertainty about state-action values diminishes, rf also attenuates over time. This prevents persistent, potentially destabilizing intrinsic motivation when exploration is no longer paramount, facilitating convergence towards policies optimized for extrinsic rewards.
While the intrinsic reward may be challenging due to the non- con-Bayesian parameterization of the CFR- MIC model, We empirically approximate it with IB IB-trained representation and Gaussian marginal,

---

yielding a tractable lower-bound bonus that supports robust exploration in noisy environments as follows:
While directly computing r' is challenging, we address this by deriving a computable lower bound using the Data Processing Inequality (DPI), which states that post-processing cannot increase infor- mation. Applying learned representation function Rsx, az) (a part of CERMIC) to the transition (st, des St+1), the DPI yields:
(st, az) = M(((s, ae 8141)lpm) - H((ss, at 84+1) 1/2
(14)
> (R((s) an st+))(pm) H(R(s;, a, $441] P.m)) 1//2 -
apprax
The approximated reward ri Tapprox thus becomes the KL divergence between the conditional represen- tation Rx 8t, at) and its marginal Pmmrgin Pm,)
"ipprox (st, az) = Ee [DEL (R (s,, at) Pmmsrgin (24 (
Dmm))
(15)
The marginal Pmargin (1x Pn)) over model parameters is intractable. Following common practice in information-theoreiic exploration with non- Bayesian models, we approximate Pmmagin (2t (Dm) with a fixed standard Gaussian distribution (O, I). This renders Eq. (15) empirically computable.
5
Experiments
5.1 Task Setup
Benchmarks and Evaluation Metries. We evaluate our approach on a diverse set of MARL bench- marks: VMAS (9 tasks) [3], MeltingPot (4 [asks)[[], and SMACv2 (2 tasks) [9]. All environments were adapted to sparse-reward configurations to rigorously test exploration capabilities; specific modifications and implementation details are provided in Appendix E. Different metrics for each benchmark: (i) mean episodic reward for VMAS, (ii) mean episodic return for MeltingPot, and (iii) mean test win-rate for SMACv2.
Baselines. We integrate CERMIC with two widely-used MARL algorithms, MAPPO[15] and QMIX [31]. Our approach is compared against three categories of algorithms: (i) Standard MARL baselines:
MAPPO, QMIX. (we also include naive exploration method MAPPO-E GREEDY) (ii) State-of-the- art MARL methods: CPM [19] from VMAS and QMIX-SPECTHR [28] from SMACv2. (iii) Other
curiosity-driven exploration algorithms: MAPPO-DE [2]. MACE [13], and ICES [20]. Crucially, all
compared algorithms are required to operate under communication-limitee decentralized execution settings to ensure fair and relevant comparisons.
5.2 Comparison with State-of-the-Art
We present a comparative performance analysis of CERMIC-augmented algorithms against various baselines in Table 1. A general observation from these results is that curiosity-driven approaches, on average, tend outperform traditional MARL. methods. Building upon this, CERMIC consis- tently enhances its base algorithms, achieving new SoTA performance on 12 out of the 16 evaluated scenarios. Furthermore, CERMIC shows its strongest gains MeltingPot, where rewards depend on emergent, dynamic inter-agent interactions rather than fixed rules. In such settings, intention-awart exploration and curiosity calibration give clear advantage over simple novelty seeking.
Table 2: Per-agent contributions in Balance*.
# Agents
2
4
6
8
MAPPO-DB
12.3
13.7
13.6
12.1
MAPPO-CERMIC
14.4
16.7
16.4
17.3
Additionally, Agents driven by single-agent curiosity formulations (e.g., MAPPO-DB), which ignore peers' intentions, are persistently distracted by others' behaviors even in later training (see Table 2). Our contextual calibration explicitly addresses this issue by transforming socially induced randomness into learnable signal. To further validate this, we evaluate how per-agent contributions change with the number of agents in the Balance* task. Results show that MAPPO-DB's per-agent contribution decreases as agent count increases, while CERMIC mitigates this degradation:
7

---

Table I: Performance comparison: against baselines, SoTA,, and other curiosity-driver methods.
An * indicates environments adapted to sparse-reward configurations; others are originally sparse. Task names omitted (see appendix for details)
VMAS
MeltingPot
SMACY2
Disper
Naviga
Sampli"
Passag
Transp"
Balanc
Givella"
Wheel*
Flocki
Staffun
CleaUp
ChiGam
PriDil
pro5v5
2er5u5*
Baseline Methods
QMIX [32]
0.69 0.65 25.4
0.42 48.5 3.40
-2.83 0.55
5.11
71.2
6.83
4.57
0.60
0.39
[1PPO[15]
1.44 1.08
21.8
159 0.23
60.3
3.71
-3.70
-0.37
5.02
74.2
8.44
4.93
0.61
0.44
MAPPO-C GREEDY
1.51 1.22
22.8
161 0.27 63.3
3.74
-3.53
-0.12
5.33
74.5
8.62
5.17
0.61
0.43
Current SoTA
CPM [19]
1.46 1.25
25.7
162 0.44 61.0 3.73
-2.51
0.20
5.11
74.4
8.35
5.21
0.64
0.44
QMIX-SPECTI SPECTRA [28]
1.52
1.14
24.2
154 0.47 52.9
3.82
-2.84
0.02
5.75
70.4
8.51
5.17
0.65
0.45
Curiosity Methods
MAPPO-DB [2]
1.34 0.88
23.0
155 0.51 55.2 3.82
-2.05
-0.04
5.08
71.2
8.47
4.99
0.63 0.43
MACE [13]
1.48 1.22
28.1 166 0.60 60.0 3.62
-1.62
0.24
6.18
75.2
8.94
5.52
0.67 0.44
QPLEX-ICES [20]
1.56 1.36 25.5 164 0.61 63.2 3.96
-1.44
0.63
7,20
76.7
9.07
5.42
0.74 0.48
QMIX-CERMIC (ours)
1.02
0.92
27.2 163 0.84 62.7
3.77
-1331 0.78
8,43
76.2
8.47
5.02
0.73 0.44
MAPPO-CERMIC (ours)
1.57 1.44 25.4 172 0.64 67.3
3.94
-1.47
1.06
7.11 78,3 10.03 6.74
0.70 0.48
+-SNE Visualization of Multi- Agent States
Visualization of Current and Latent States
MAPPO CERMIC
Timestop:
ratreat
Rozz trojectory
approach
Timestop:
MAPPO
MAPPO
CERMIC
Agent-1
2gent-2
Agent-1LLtent
Figure 2: Visualization of agent observation embeddings 8; and latent states dz. (Left) Compara- tive 8, distributions for agents under CFRMIC-augmented vs. baseline algorithms. (Right) Influence of curiosity- latent states 24 task exploration.
5.3 Qualitative and Quantitative Analysis
Qualitative Analysis. To provide intuitive insights of the operational dynamics, visualize the temporal evolution agent states 8, and their corresponding curiosity latent variables xı (Fig. 2).
We define state trajectory temporally contiguous sequence of embeddings.
(i) Curiosity to Peers: The left panel of Fig. which displays these state trajectories, reveals that CERMIC-augmenter agents exhibit more frequent close proximities and intersections in their pathways compared to those under vanilla MAPPO. These convergences, without full trajectory overlap, suggest that agents actively approach others for informed observation to refine their distinct plans, rather than engaging in mere imitation.
(ii) Curiosity to Enviroument: In the right panel, the visualization of latent states offers further compelling evidence. Firstly, H, often displays a more dispersed distribution around the current embedding, reflecting an intrinsic drive towards novel or uncertain aspects of the current state. Secondly, these latent states tend concentrate at the endpoints of the state trajectories, indicat- ing CERMIC's 's heightened activity prior to significant behavioral shifts, aligning with the original motivation behind designing an intrinsic reward that prioritizes critical transitions. Furthermore, inter-agent encounters typically trigger surge in the intrinsic reward, indicating that CERMIC effec- tively captures and values the increased novelty and learning opportunities presented by these crucial multi-agent contextual shifts.

---

Nermalize
Wind Flacking
Whial"
Discovery"
Timestops (le6)
-APPOOGERIITC
MMPO--ERMIC-PeTriinn
Figure 3: The impact of pretrained intention memory module on CERMIC's performances
Quantitative Analysis. Fig. 3 illustrates the mean episodic return curves on the VMAS task, with comprehensive results across additional tasks deferred to Appendix F. While CERMIC-augmented algorithms ultimately achieve superior final performance, their learning curves can exhibit slower initial ramp-up compared some baselines. We attribute this initial lower sample efficiency to the inherent challenge of learning effective multi-agent intention modeling from scratch, a difficulty induced by severe partial observability.
5.4 Ablation Study
We conduct three ablations on CERMIC, summarized in Table 3: (i) Loss module: Ablating the exploration loss Lesplore markedly impedes adaptation in sparse-reward settings, yielding slower learning; removing the exploitation losses - + C prevents effective noise filtering, leading to instability and pronounced fluctuations. (ii) a: increasing the curiosity scale (for our cali- brated intrinsic curiosity) typically lowers immediate returns greater exploration, yet improves learning stability and reduces performance variance, reflecting exploration exploration-exploitaiio exploitation trade-off. (iii) Intention model: More expressive intention models that better extract contextual features I consistently improve task performance.
Table 3: Ablation studies and hyperparameter analysis of CERMIC.
Flocki+
Naviga
Passag*
CleanUp
ChiGam
Loss Ablations (a 0.2)
w/o Lesgioe
0.82 (1:0.1))
1.41 (1:0.44)
166 (133666)
75.4 (#1005
9.22 (1:0.80)
w/o Lesphot
0.72 (#0.48)
1,36 (=00.10)
164 (#:4.41)
71.5 (=1.59)
8.47 (#1.18)
MAPPO-CERMIC
1.06 (+0.11)
1.43 (++0.06)
171 (+3.722
78.1 (+0.82)
10.02 (+0.65)
Coverage Level
1.0
0.80 (10.07)
1.40 (1:0.02)
163 (11.56)
71.2 (::0.77)
9.04 (1:0.46)
0.5
0.88 (+0.10)
1.42 (#0.06)
169 (#3.78)
74.6 (#0.68)
9.50 (+:0667)
0.2
1.06 (+0.12)
1.43 (+0.06)
171 (+3.72)
78.1 (+0.82)
10.02 (+0.65)
Memory Type (a
0.2)
GRU
0.78 (#:0.27)
(1:0.04)
161 (#:3115)
75.2 (#11.03)
(1:097)
Graph
1.06 (10.12)
1.43 (10.06)
171 (#3.72)
78.1 (+0.82)
10.02 (+0.65
5.5
Generalization Across Reward Densities
To validate this and demonstrate CERMIC'S potential with proficient intention modeling, we conducted an auxiliary experiment: We use pre-trained agent detection and intention modeling modules to provide more accurate initial estimates of other agents' states, leading to CERMIC variant that exhibited significantly faster convergence. Given advancements in related areas like motion prediction, this result underscores CERMIC'S strong applicability.
We further investigated the generalization capabilities of agents by evaluating their performance when trained under one reward density (dense or sparse) and tested under sparse-reward setting. The results are presented in Table 4. consistent trend observed across all algorithms is performance degradation when agents trained on dense rewards are deployed in sparse-reward scenarios. However, CERMIC demonstrates notable ability to mitigate this performance drop. We attribute this to CERMIC's capability to enable agents to understand the task by observing other agents, rather than

---

solely relying on trial-and-error learning. These findings underscore the importance of research in sparse-reward reinforcement learning and highlight the potential for CERMIC'S broad applicability.
Table 4: Agent generalization performance VMAS tasks. A denotes the performance drop from
dense-trained to sparse-trained.
Method
Balance'
Give Way" Dense
Passage" * Densef
Sparse
Denset
A 4
Sparset
A
Sparse
|
CPM
61.0
58.2
3.73
3.58
.0.5
162
153
QPLEX-ICES
63.2
60.7
-25
3.96
3.85
-0.11
164
159
MAPPO-CERMIC
67.3
65.6
-1.7
3.94
3.82
.0.12
172
168
6
Conclusion
This paper introduced CERMIC, a novel plug-and-play module significantly enhancing exploration in communication-limite. partially observable MARL under sparse rewards. CERMIC'S core innovation a multi-agent contextualized calibration intrinsic curiosity. Grounded in Bayesian Surprise and supported by theoretical guarantees, we also propose novel intrinsic reward guides this calibrated exploration. Extensive experiments show CHRMIC-augmented algorithms achieve state-of-the-art performance, underscoring the efficacy of context-aware intrinsic motivation. In essence, CERMIC CERMIC offers a crucial step towards enabling more autonomous and socially intelligent agent teams for real-world deployment. No negative societal impact.
Limitations and Future Work. Despite its promising results, CERMIC has limitations that open avenues for future research. First, learning accurately model other agents' intentions solely from local observations in communication-less, partially observable settings remains a challenging endeavor. Future work could explore integrating pre-traiee pre-trained components, such LLMs, to potentially bootstrap or replace aspects of this ad-hoc intention modeling process. Second, our current graph construction and one-step latent transition model may require extensions (e.g., hierarchical latent models) to effectively scale to higher-dimensional or multi-modal MAS tasks.
10

---

Acknowledgment
This paper was supported by the Natural Science Foundation of China under Grant 62303307, 62225309, U24A20278, 62361 166632, U21A20480, and Sponsored by the Oceanic Interdisciplinary Program of Shanghai Jiao Tong University, and in part by National Key Laboratory of Human Machine Hybrid Augmented Intelligence, Xi' an Jiaotong University (No. HMHAII20220S).
References
[1] John Agapiou, Alexander Sasha Vezhnevets, Edgar A Duéfiez-Gzzmán, Jayd Matyas, Yi- Mao, Peter Sunchag, Raphael Köster, Udari Madhushani, Kavya Kopparapu, Ramona Comanescu, et al. Melting pot 2.0. arXiv preprint arXiv:22 133746, 2022.
[2] Chenjia Bai, Lingxiao Wang, Lei Han, Animesh Garg, Jianye Hao, Peng Liu, and Zhaoran Wang. Dynamic bottleneck for robust self-supervised exploration. Advances in Neural Information Processing Systems, 34:17007- 2021. 3, 7,
[3] Matteo Bettini, Ryan Kortvelesy, Jan Blumenkamp, and Amanda Prorok. Vmas: A vectorized multi-agent simulator for collective robot learning. In International Symposium on Distributed Autonomous Robotic Systems, pages 42-56. Springer, 2022. 7
[4] Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv:1810.1289. 2018.
[5] Anthony R Cassandra. survey of pomdp upplications. Working notes of AAAI 1998 fall symposium on planning with partially observable Markov decision processes, volume 1724, 1998.
[6] Sayak Ray Chowdhury and Aditya Gopalan. On kernelized multi-armed bandits. In International Conference on Machine Learning, pages 844-853. PMLR, 2017. 2
[7] Christian Schroeder De Witt, Tarun Gupta, Denys Makoviichuk, Viktor Makoviychuk, Philip HS Torr, Mingfei Sun, and Shimon Whiteson. Is independent learning all you need in the starcraft multi-agent challenge? arXiv preprint arXiv:2011.09533 2020.
[8] Erick Delage and Yinyu Ye. Distributionally robust optimization under moment uncertainty with application to data-driven problems. Operations research, 58(3):595-112, 2010. 5
[9] Benjamin Ellis, Jonathan Cook, Skander Moalla, Mikayel Samvelyan, Mingfei Sun, Anuj Mahajan, Jakob Foerster, and Shimon Whiteson. Smacv2: An improved benchmark for cooperative multi-agent reinforcement learning. Advances Neural Information Processing Systems, 36:37567-37597 2023.
[10] Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.00 0411044 2023. 2
[11] Botao Hao, Yasin Abbasi Yadkori, Zheng Wen, and Guang Cheng. Bootstrapping upper confidence bound. Advances in neural information processing systems, 32, 2019. 2
[12] Jeewon Jeon, Woojun Kim, Whiyoung Jung, and Youngchul Sung. Maser: Multi Multi-agent rein- forcement learning with subgoals generated from experience replay buffer. In International conference on machine learning, pages
PMLR, 2022.
[13] Haobin Jiang, Ziluo Ding, and Zongqing Lu. Settling decentralized multi-agen, coordinated exploration by novelty sharing. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 17444-17452 2024. 2, 7,8
[14] Chi Jin, Zhuoran Yang. Zhaoran Wang, and Michael I Jordan. Provably efficient reinforcement learning with linear function approximation. In Conference on learning theory, pages 2137- 2143. PMLR, 2020. 6
11

---

[15] Hongyue Kang, Xiaolin Chang, Jelena Misic, Vojislav B Misić, Junchao Fan, and Yating Liu. Cooperative uav resouree allocation and task offloading in hierarchical aerial computing systems: A mappo-based approach. IEEE Internet of Things Journal, 10112):0497-10509) 2023. 7,8
[16] Kuno Kim, Megumi Sano, Julian De Freitas, Nick Haber, and Daniel Yamins. Active world model learning with progress curiosity. In International conference on machine learning, pages 5305- PMLR, 2020. 2
[17] Youngjin Kim, Wontae Nam, Hyunwoo Kim, Ji-Hoon Kim, and Gunhee Kim. Curiosity- bottleneck: Exploration by distilling task-specific novelty. In International conference on machine learning. pages 3379-338300 3388. PMLR, 2019.
[18] Chenghao Li, Tonghan Wang, Chengjie Wu, Qianchuan Zhao, Jun Yang. and Chongjie Zhang. Celebrating diversity shared multi-agent reinforcement learning, Advances in Neural Infor- mation Processing Systems, 34:3991 -4002, 2021. 2
[19] Jingchen Li, Yusen Yang, Ziming He, Huarui Wu, Haobin Shi, and Wenbai Chen. Cournot policy model: Rethinking centralized training in multi-agent reinforcement learning. Information Sciences, 677:120983, 2024. 7,8
[20] Xinran Li, Zifan Liu, Shibo Chen, and Jun Zhang. Individual contributions as intrinsic explo- ration scaffolds for multi-agent reinforcement learning. arXiv preprint arXiv:2405. /8/ 10, 2024.
1, 7, 8
[21] Emily G Liquin and Tania Lombrozo. Explanation-seeking curiosity in childhood. Current Opinion in Behavioral Sciences, 35:14-20, 2020.
[22] lou-Jen Liu, Unnat Jain, Raymond A Yeh, and Alexander Schwing. Cooperative exploration for multi-agen deep reinforcement learning. In International conference on machine learning, pages 6826 6836.666.222 PMLR, 2021. 3
[23] Augustine Mavor-Parker Kimberly Young, Caswell Barry, and Lewis Griffin. How to stay curious while avoiding noisy tvs using aleatoric uncertainty estimation. In International Conference on Machine Learning, pages 15220-15240. PMLR, 2022. 1
[24] Pietro Mazzaglia, Ozan Catal, Tim Verbelen, and Bart Dhoedt. Curiosity-driven exploration via latent bayesian surprise. In Proceedings of the AAAI conference on artificial intelligence, volume 36, pages 7752-7760, 2022. 6
[25] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv: 2013. 2
[26] Haruhiko Ogasawara. The multiple cantelli inequalities. Statistical Methods & Applications, 28
(3):495-506, 2019. 5
[27] Georgios Papoudakis, Filippos Christianos, Lukas Schäfer, and Stefano V Albrecht. Benchmark- ing multi-agent deep reinforcement learning algorithms in cooperative tasks. arXiv preprint arXiv:2006.07886 07866, 2020.
[28] Hyunwoo Park, Baekryun Seong, and Sang-Ki Ko. Spectra: Scalable multi-agent reinforcement learning with permutation-frre networks. arXiv preprint 1rXiv:2503.11726, 2025. 7,
[29] Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning, pages 2778- 2787. PMLR, 2017. 1,
[30] Graham Pluck and Helen I. Johnson. Stimulating curiosity to enhance learning. GESJ: Education Sciences and Psychology, 2, 2
[31] Tabish Rashid, Gregory Farquhar, Bei Peng, and Shimon Whiteson. Weighted qmix: Expanding monotonic value function factorisation for deep multi-agent reinforcement learning. Advances in neural information processing systems, 13:10199-11022 2020. 7
12

---

[32] Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt. Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Monotonic value function factorisation for deep multi-agent reinforcement learning. Journal of Machine Learning Research, 21(178):1-51, 2020. 2, 8
[33] Daniel Russo and Benjamin Van Roy. Learning to optimize via information-directed sampling. Advances in neural information processing systems, 27, 2014. 2
[34] Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE transactions neural networks, 20(1):61-80) 2008.
[35] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford. and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.066 2017. 2
[36] Alexander Shapiro. Monte carlo sampling methods. Handbooks in operations research and management science, 10:353-425 2003.
[37] Niranjan Srinivas, Andreas Krause, Sham M Kakade, and Matthias W Seeger. Information- theoretic regret bounds for gaussian process optimization in the bandit setting, IEEE transactions on information theory. 58(5):3250 -2665 2012.
[38] Bhavya Sukhija, Stelian Coros, Andreas Krause, Pieter Abbeel, and Carmelo Sferrazza. Maxin- forl: Boosting exploration in reinforcement learning through information gain maximization. arXiv preprini urXiv:2412.1 2n:::2//2//20000 2098, 2024. 1
[39] Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? Advances in neural information processing systems, 0116827-6839 2020. 5
[40] Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 ieee information theory workshop (itw), pages 1-5. Ieee, 2015.
[41] Naftali Tishby, Fernando C Pereira, and William Bialck. The information bottleneck method. arXiv preprint physics/0004057, 2000. 2
[42] Peter Washington, Catalin Voss, Nick Haber, Serena Tanaka, Jena Daniels, Carl Feinstein, Terry Winograd, and Dennis Wall. A wearable social interaction aid for children with autism. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems, pages 2348-2282 2354, 2016. 2
[43] Kaiqing Zhang, Zhuoran Yang, Han Liu, Tong Zhang, and Tamer Basar. Fully decentralized multi-agent reinforcement learning with networked agents. In International conference on machine learning, pages 5872-5881. PMLR, 2018.
[44] Lulu Zheng. Jiarui Chen, Jianhao Wang. Jiamin He, Yujing Hu, Yingfeng Chen, Changjie Fan, Yang Gao, and Chongjie Zhang, Episodic multi- reinforcement learning with curiosity- driven exploration. Advances in Neural Information Processing Systems, 34:3757-3769, 2021.
3
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: See Abstract and Introduction Section.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: see Conclusion Section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
14

---

Justification:
see Method Section.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Method Section, Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [Yes]
Justification: see Supplementary Material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
wwwiiisssooooioooooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ //nips. wttpccccpppccccceeececccccccccct /ttp:////giieeeeeeeuuuubiiiiiiii /CodeSubmic ssisnnoillll onPolicy))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: see Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: see Experiments Section.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
16

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /ttp///eerrpp.ee/ee//ee cc
Answer: [Yes]
Justification: This work conform with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Conclusion Section.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
17

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: CC-BY 4.0.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
18

---

Answer: [Yes]
Justification:
we include our code/model in the Supplementary Material.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
19

---

Justification: [NA]
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
: Please refer to our LLM policy (https://neurd) htt.arrpps.cccooofeeerrreeee/e/e for what should or should not be described.
20

---

A Memory Update Module
The agent memory update module primarily consists of an agent detection mechanism and memory encoding structure. In our main approach, this involves an MLP-based agent detector and a graph- based memory module. This section elaborates on the graph structure and the pre-training of the detector.
A.1 Graph-based Memory Representation
The graph-based memory module offers a structured approach model inter-agent states and their relationships. For each observing agent, local graph G = (V,, & is dynamically constructed or updated each timestep t.
Nodes (V:): Each node E Va corresponds to an inferred latent state representation of specific agent i (either the self-agent or a detected peer). These node features, denoted Z; E I PPoddd are generated by encoding the raw observation associated with agent i.
Edges (E,): Edges Cij E & link pairs of nodes (v, tý)). representing the inferred relationship between agent and agent j. In our work, these spatial relationships, with edge features fij E Pedge derived from their relative positions.
This dynamic graph serves as input a Graph Neural Network (GNN), which processes the relational information to produce a contextualized memory representation for the observing agent, thereby informing its subsequent actions.
A.2 Pre-training Process
Key components of the agent memory module are pre-trained to establish meaningful initial represen- tations, facilitating more effective subsequent MARL training.
MLP Agent Detector. This module processes an agent's observation to output probabilities [pi..... .PN.....PNN]'' PPN]".
where Pk denotes the likelihood of observing agent of type-k. These probabilities are later used to gate memory updates during MARL training via a threshold Tda. Pre- training is supervised using ground-truth presence labels yk {0, 1}, which set to 1 if agent-k appears within the field of view or one body length of the observer, and 0 otherwise. To facilitate reliable detection, agents in environment visual appearances, and each agent type is represented by a dedicated channel in the observation tensor, enabling effective training of an MLP-based detector.
either designed with inherently exhibit distinet
Ldet - k=1 [uk log(pk) + (1 yle) log(1 pk)]-
(16)
Graph Memory Encoders. The node and edge encoders within the graph memory are pre-trained as follows: (i) Node Encoder: for an observing agent l. its encoder produces feature Zg-zj representing its understanding of another agent j's state. This is trained to predict agent j's true latent state 8,. The MSE loss is weighted by a detection mask, which is derived from the predicted transition probabilities Pk-tj produced by pre-trained detector, using a fixed threshold to determine valid entries; (ii) Edge Encoder: this encoder generates 2D edge features fly from inter-agent cues, trained to directly predict the true relative position vector (Arij, Aw;;) between agents and j using an MSE loss.
c Ledge R. f1. (A r.g. A
Σ Ejet Pk- E 1 -yp-y-ssll' sy
(17)
These pre-training steps provide the memory module with an initial capacity for relevant information extraction, potentially accelerating overall learning.
B Derivation with Cantelli Inequality
In this section, we detail the transformation of the distributionally robust chance constraint, (q. of the main paper. This transformation leverages Cantelli's inequality to arrive tractable loss
21

---

under second-order cone program (SOCP) constraint. In this section, we set ė = for illustration. We begin by defining auxiliary variables based on the quantities in the chance constraint:
V - (P
(18a)
b b= - jvv (f%)
1-1-
(18b)
We also define the set S describing the ambiguity in the first and second moments of s, and the set D3 of distributions consistent with these moments:
{ (11, : 11al S Vn p7
S 12
(19a)
Er,
S V 22
D; Pr E P(R):
(19b)
Ee,
where P(R) is the set of all probability distributions on R. Then, the original chance constraint infp,eP... (V, 1) 1 e can be rewritten using and b. The infimum term is:
inf P. S P.EPeni
inf (5 I b)
P,ED;
(20)
inf inf (r - b). (m..))SSR.EP(u..0)
where Peal refers to the set of distributions under multi-agent contextual intention uncertainty in the main paper. To this end, Eq. (20) formulates the problem as a bi-level optimization. The outer layer finds the worst-case mean μn and variance of within the ambiguity set S. The inner layer finds the worst-case probability (5 = b) for a given mean and variance.
To solve the inner optimization problem, we employ the one-sided Cantelli's s inequality. For random variable X with mean E[X] and variance Var[X] = o², Cantelli's inequality provides bounds on tail probabilities. Specifically, for the lower tail, the bound is defined as:
0²
(X E(X) 2 - 0² X2
Applying this to our inner problem inf PPFM..F)) 13 < b), we get:
(b-1))² (b
if 2 A1
inf P)
Pr (r b)
(21)
0,
otherwise.
In practical multi-agent reinforcement learning scenarios, the desired confidence level positive. For the chance constraint inf (P, < b) 2 1 c to hold with c > O, we must be in the regime where b 2 f4- b < fr, the infimum probability would be O, violating the constraint. Therefore, for any relevant (1), S. the condition 2 μi must be satisfied. This implies b 2 (UP())) M1 = En (assuming Σn > 0). Consequently, Eq. (20) simplifies to:
is
inf (V, 1 inf (b (6-1) 21)²
(22)
E
(I of (b m1)
Solving the optimization problem in Eq. (22) over the set S (defined by moment bounds |p1| S Vn and in + oo S xn) yields the following worst-case probability:
S
S
22 /n
V72 by W//h - (b/ (/VE)/ p-72
+1
P. 1) P.PPon
inf
if
/2,) infeasible,
V2
VT
if < /n1.
The chance constraint requires this infimum probability to be at least e: infe,e P, 11) 1 By substituting the expressions above and performing algebraic manipulations, we arrive the tractable second-order cone constraint. Substituting = - (22),, the condition becomes:
e:
ua (1%) + B Ev 1
(23a)
V
+ V L= (72 m), 1f Se51
4n
(23b)
if <0<E5 <o<e<<m/> n/12.
22

---

This provides the final tractable form for the first inequality in Eq. (6) of the main paper. For this second-order cone constraint, we wrap it with a ReLU function construct the following loss foopploit> which forms upper bound of the exploit loss. We substitute é to recover the Eq. (7)
presented in the main text.
To ensure stability and prevent variables V from shrinking indefinitely, we also impose a lower-bound constraint P,(W U 2 c) 2 - e, which can be converted into loss LLB exploit in similar manner.
C InfoNCE Bounds on Mutual Information
This appendix details how the InfoNCE objective provides lower and upper bounds for the mutual information / /(rr fü- 11; fū). We begin our derivation by establishing the lower bound relationship
(c) & I( f2 1 fR). For notational simplicity, we denote the context 1. as 2n.
In work, the InfoNCE objective is defined using critic function f2- measures the compatibility between the context [%-1] and the current inferred intention so In practice, -) is implemented
which
bilinear layer followed by softmax, yielding scores in the
range (0, 1.. Given a positive pair (fo z.) drawn from the joint distribution p(zn, fh), and a set of N negative samples F (Sr
.. )= drawn from the complement of 5% in its state space F, the
InfoNCE objective is defined as:
I Eg) log exp(c(En; f2))
(24)
Σ exp(c(zns ()))
where the positive pairs (Zn, fa) are obtained from the current prediction pair. Negative samples are generated by independently sampling N times from the state space of fü Let ll be indicator variable, with I 1 indicating a positive sample randomly drawn from the state space, and I = indicating a negative one (i.e., f2 is replaced by fn The corresponding conditional probabilities have the following properties:
p(zn: sull 1) p(zn p(zn: o p(z,)p(ff))
(25)
We aim to maximize the InfoNCE objective I We assume that the optimal critic e* can identify the log-pood pog-posteroor probability of a randomly sampled pair being the positive one. Thus, we have:
I I _
< Eppen. (2) Ex (c" (Znrfa)) få)]
(26)
= Ep(an.tt) BF llogp[I = 2n, fn.
Next, we expand log p(I = 1|2n, R F-) using Bayes' theorem. Considering one positive sample
and N negative samples, the prior probabilities are p(I 1) 1/(N and p(I 0) N/(N+1)
for the collection of negative samples.
p22n f2 [I 1)p(II = 1)
log p(T = 1|2n: S2; 82,
log
p( loll = [I [ = 1) + p(=n, f 0)p(I 0)
(27)
1.)
log
p2n: fR) + N p2 ((.)
This expression for log p(1 = 1 1|zm, f FF) represents the log-probability of the given fü being the true positive, relative N distractors drawn from p(2n)p(fR). Continuing from this expression:
p(zn: f%)
p(zn, f2)
p(zn)p(f:)
log p(2n, fa) Np(zn)
log
((zz))([F
p(zn, ND(Z.)p(SR)
(28)
p(zn, fo)
log
log N.
p(z,) p (f%)
Taking the expectation (E)( Ey- on both sides of the result from Eq. (26) combined with the inequality above:
p(zn. f() log p(2, (1mn)((f3
E Ep) [logp(1= = 1/2ns fu FF < Ep Ep(tn.))
log N.
(29)
23

---

The term Ep))...a)
log pl (f)
is the definition of mutual information [223 fa). Therefore, by back into Eq. (26) and rewriting [znn fR) with the original notation,
pl p(an ((R)
substituting Zn
1i-1::8-1
we have:
2 iice log N.
(30)
This demonstrates that the mutual information lower-bounded by the InfoNCE objective I plus a term log N. This justifies using !ice as a tractable surrogate to maximize a lower bound on the mutual information.
Similarly, the upper bound inequality can be derived using the same method.
{
M(1r--1 In fa-1li Si) S -
exp of (:)) M A log N Bp()n.f)) By log د exp(1-c(3nn.))
(31)
D Intrinsic Reward in Linear MDPs
This appendix provides a theoretical justification for our proposed intrinsic reward by connecting it to the exploration bonus in LSVI-UCB within the context of linear MDPs.
D.1 Preliminary
Least-Squares Value Iteration with Upper Confidence Bounds (LSVI-UCB) is an algorithm designed for efficient exploration and learning in linear MDPs. In linear MDP, the transition kernel and reward function are assumed to be linear with respect to a d-dimensional feature map r7(s, a) of state- action pairs. Consequently, for any policy म, the action-value function Q" (s, a) can be expressed as X" "7(s, a) for some parameter vector X E Rd
LSVI-UCB iteratively collects data and updates the Q-function parameters. In each episode, the agent acts according to the current optimistic Q-function. The parameter Xt is then updated via regularized least-squares:
m
Xx f arg mün XERé
(1) max Lurget ( (1)41) TTxxTTnsss "ss), + AlxxI
where is the number of collected transitions (indexed by i). X is regularization parameter, and Quaree is typically previous estimate slowly updating target. The closed-form solution involves the Gram matrix A, = Σi= n(s at (ms)), at)T T + Al. Crucially, LSVI-UCB employs a UCB-style exploration bonus to construct an optimistic Q-function: Qi a) X² (7s,, a) + + r
a), where the bonus y pub a) = C [n](,, a) A, (ms(ss)))) This bonus quantifies the epistemic uncertainty associated with the value estimate of (s, as.
D.2 Connection to LSVI-UCB Bonus
We establish connection between our intrinsic reward and the LSVI-UCB exploration bonus. In linear MDPs, CERMIC'S parameters is rewritten wt and curiosity representation 2; can be represented as i2 = wx7)(tt, at) E Re. Here, 17(8,, 0.) E IRd the state-action encoding, and wi cxd a parameter matrix. This representation learned by predicting the next state 8:-1 via the regularized least-squares problem:
m
We arg min Σ - afll A!WWIE
(32)
The proof proceeds by vectorizing the matrix wi and defining an expanded feature matrix f). Specif- ically, vec(w,) E adt is the column- tiimnn-wisee vectorization of w,, and fj(s,, 0,) E Rodxe is block- diagonal matrix with v7(s,, a.) repeated times along its diagonal. This construction satisfies veo(wr)" (7s,, az) = (win(s, az)) For clarity, the defimitions are:
veo(wx)) = [wi1, Wid, W21...., Wedd E RRod ?is,,,,)) a.) I. 17{8,, ax) Redxx
(33)
where I(I.) denotes the identity matrix (with dimensions), and is the Kronecker product.

---

Assumption (Gaussian Prior) We consider a linear model for predicting the c-dimensional next state 1141 from d-dimensional state-action features n[3,, (t), formulatd as 81+1 WT,(st, a+) &t- The parameter matrix W Rexd is assumed to follow zero-mean Gaussian prior distribution W ~ - 1),, and the model noise & E IR is assumed to follow a standard multivariate Gaussian distribution & N'(0, Le), independent of W and m[st,a)))
To analyze the intrinsic reward, we adopt a Bayesian linear regression perspective for Eq. (32). Our goal of the analysis obtain the posterior distribution p(we[Dm) to compute the Bayesian Surprise in the intrinsic reward. Firstly, under Assumption 1, we have:
St+1 (5t; az), We N (wom((s;, oz), N (ñ(s), (T)s veo(wu),
(34)
Given the Gaussian prior vec(W) ~ ~O
-000,-110 1), Bayes' rule states:
logp(vec(w;)))mm) = log Mogp(veo(uz))
log logr(s ol vec(w))) + C.
(355
where C constant. Then, substituting the Gaussian PDF for Eq. (34) into Eq. (35) yields the log-posterior:
log ((ve(c((u)))mm = \vec(w)))1 1 m
|| at) a veo((u) - -/ 112 +C
2
im
(36)
(vec(wr) (ee) )c(vec(ur) - +
where C' constant and jı A Σm (s), ( As En is ot M(s d T + AI. Thus, the posterior distribution is veo(ws) D'm ~ NN 1. The covariance matrix of vec(we) is A, The structure of A, is:
A In &
Σ 1=0
mss a(msss)) A In @ Ae,
(37)
Notably, Ar = Ei 1755 of m,)) a)) + AI the Gram matrix from LSVI-UCB. To this end, the intrinsic reward, related Bayesian Surprise, can be simplified to change in differential entropy, where Cov(-) denotes the covariance:
D D (p(wel((s ar, s141) U DD (m)llp(w.(Pmm))
k (p( (er((w)))(( ae 8141) U Dm (l(ppllpp(weww))pp...
(38)
= H(vec(wx) (D..) ((vee[w:))(s>, a,, 1) U Dm)
[log det (Cov(vec(w:) ()..)) log det (vo))ve(()))((s,,, ae, Umm))l.
Substituting the covariance 7 and the updated covariance after observing (St, at se+1):
Dxz(p(wel(sr a s1+1) U D (pppw:
log det (A. + ii at (Ms a:) ) det (A.)
(39)
[log det (I + ii{,,s ax) ti a:)) /2.
where the last equality follows from the Matrix Determinant Lemma. Given the block-diagonal structure of is(s; az) and A, the term (ss ax) A, "i(ss,, a,) is diagonal matrix with e identical scalar entries T7ss,, a T A, "[ss,, az). Thus, Eq. (39) simplifies
D 1 (p(w: at, S(+1) ) Dmn)ll((w:))
= [log (Ie @ (78,, a.) A, 7/8,, a:) ) + I.)] /2
(40)
(c/2) log (1 + n(s,, a.) T A, A ((s,,,))))
By leveraging the inequality / s log(1 + x) $ x, we obtain:
(7,,, az) A, m) az)
< log (1 + n|s,, a.) A, mss ar)) S (()) TT) (r))(s),)))
(41)
25

---

Combining these results, we obtain the bounds for the square root of the mutual information:
[n((s, ax) AAAnnnnsss a:)] srl V [n((,e, ar) ", " "ss,, ac) 12
(42)
Recalling that the LSVI-UCB bonus is yach ( (n)(), Ar An'nn(s, an)] 4/2 we can express the relationship as:
C neb S r <V 2
pdb
(43)
<V
Defining p = غ v J. this can be written as p-ruc < r/s V2p This proof establishes a connection between our proposed intrinsic reward and the UCB bonus, providing theoretical grounding for the stability of our reward mechanism.
E Benchmark Adaptation to Sparse-Reward Settings
Our CERMIC model and all compared algorithms were trained in environments with sparse reward signals to specifically assess exploration capabilities. However, for a comprehensive evaluation against baselines, particularly those not designed for extreme sparsity, final performance was measured using the original dense extrinsic rewards provided by the benchmarks. To facilitate the sparse- reward training, adapted standard MARL benchmarks, VMAS and SMACv2, into sparse-reward configurations as detailed below. This transformation creates challenging scenarios where intrinsic motivation is paramount for effective learning.
VMAS Benchmarks. For VMAS tasks that originally featured dense rewards (e.g., based on distance- to-target or control efforts), we induced sparsity by identifying clear success condition for each task and restructuring the reward mechanism accordingly. Specifically, original dense or intermediate reward components were removed or nullified, except for a significant positive reward granted only upon the achievement of the predefined success condition (e.g.. reaching a target zone, achieving formation). Episode termination conditions remained largely unchanged, but rewards became predominantly contingent on successful terminal states. For instance, in a Bal ance" task, agents receive a large positive reward only upon moving the object to the goal region, with zero minimal rewards during transit. This approach ensures that learning signals are tied to meaningful task completion rather than continuous incremental progress.
SMACv2 Benchmarks. Similarly, for SMACv2 scenarios, while many already possess somewhat sparse rewards, we further amplified sparsity to stringently test exploration. The primary reward signal was strictly tied to the ultimate battle outcome: a large positive reward for winning, a negative zero reward for losing, and neutral slightly negative reward for draw. All intermediate battlefield rewards, such as bonuses for damaging enemy units or penalties for sustaining damage, removed. This concentrates the learning signal at the conclusion of an engagement, compelling agents to learn long-term coordination and strategy based purely on the sparse win/loss feedback, especially in scenarios where incremental damage yields no immediate reward.
These modifications ensure that agents in both benchmarks must explore extensively to discover successful action sequences, as intermediate feedback is largely absent, thus providing robust testbed for curiosity-driven mechanisms like CERMIC.
F
Supplementary Experimental Results
Performance Curve Comparison. We present further comparative experiments showcasing the performance CERMIC against other methods in Fig. 4 below. The results demonstrate that CERMIC consistently surpasses prior SoTA models and other curiosity-driven approaches, achieving superior task performance. Furthermore, is observable that our method exhibits more stable learning curves, characterized by narrower error bands. We attribute this enhanced stability and performance to CERMIC'S 's ability to effectively filter noisy signals while concurrently encouraging robust exploration.
Robustness to Noise. To further validate the moise-filterin capabilities of CERMIC, we conducted comparative experiments in environments augmented with "random box" noise, as depicted in Fig. The results indicate that, in these noisy conditions, CERMIC maintains more stable learning curves
26

---

Normolized Return
Navigation"
Buzz Wire"
Flocking
Reverse Transport*
Sompling
Simple Push"
Timesteps (le6)
QPLEX ICES
MAPPO
Figure 4: Comparative performance on VMAS.
and narrower error bands compared to other algorithms. Notably, this advantage in stability is more pronounced than in the noise-free task counterparts (Fig. 4). Furthermore, we observe that the error bands in the Simple Adversary task are smaller than those in the Ball Trajectory task. This difference can be attributed to the richer multi-agent contextual information available the former, which includes signals from both adversaries and teammates, whereas the latter primarily involves information from a single partner. Consequently, the increased contextual feedback translates to stronger noise-filtering effect, reflected in the learning curves. This underscores CERMIC's ability to leverage multi-agent information to enhance task understanding and mitigate the impact of environmental stochasticity.
Normaizzd Return
Ball Trojectory
Simple Adversary"
Timesteps (le6)
QPLEX-ICES
IPPO-DB DB
CPM
MAPPO-CERMEE
Figure 5: Performance comparison under environmental noise VMAS task.
Role Diversity and Task Adaptation. While CERMIC consistently improves performance, we observe more substantial performance gain over traditional baselines in tasks involving diverse agent roles (e.g., both cooperation and competition, as in Simple Adversary") compared to purely cooperative/competiive settings (e.g., Ball Trajectory'), see Fig. 5. We attribute this to the challenge faced by traditional methods in interpreting heterogeneous agent behaviors under partial observability, where inferring intent and role from observations is inherently difficult. In contrast, CERMIC'S task-ada adaptive weighting mechanism (7) enables agents to better infer and adapt to others' intentions and plans, thereby enhancing performance in complex social interactions.
Visualization of Intrinsic Reward. To illustrate the temporal behavior of CERMIC'S intrinsic reward, we visualized its values throughout a complete episode of the Balance" task, as shown in Fig. 6. A clear trend is the gradual decrease of this intrinsic reward over time, indicating diminishing role of curiosity-drine driven exploration as the agent gains familiarity with the task environment. Furthermore, peaks in the intrinsic reward consistently coincide with an agent encountering novel situations, such as initial environmental entry first-time interactions with other agents. This observation aligns with our design objective for the intrinsic reward, which is to incentivize exploration of unfamiliar states and interactions.
27

---

Encounter
Relativv Reward
Steps
Figure Visualization of CERMIC'S intrinsic reward during one episode of the Bal ancee task.
G Alternative Memory Architectures
While graph-based memory offers structured relational reasoning, alternative architectures provide different trade-offs. Table compares key architectural paramcters with typical example values for these memory types.
MLP-based Memory, An MLP processes the agent's s current (potentially encoded) observation to output a fixed-size memory vector. This is stateless, feed-
GRU-based Memory. Gated Recurrent Unit (GRU) maintains hidden state that is updated each timestep based on the current encoded observation and its previous state, capturing temporal dependencies.
Table 5: Architectural parameters for different memory modules.
Parameter Type
Graph-hased
MI.P-based
GRU-based
Core Encoding
Node Embedding Dim (Dnode)
256
Edge Embedding Dim (Dadge)
16
Obs. Encoder Output Dim (Doba_enc)
64 (for detection/enccoding)
64 256 (Dm)
64
Memory/State Output Dim
512 (DGN N__oNlll out)
256 (DR)
Network Structure
Obs. Encoder (CNN) Layers/Depth
conv layers
3 layers
conv layers
GNN Layers (LGNNN)
GNN Hidden Dim (DGNN_Ma)
256
GNN Heads (Nnends)
MLP Hidden Layers (LMLP)
(Node/Edge Encoders: 2)
2 (post obs-encader)
MLP Hidden Units/Layer (UmGP)
(Node/Edge Encoders: 128)
128
GRU Layers (LGRU)
2
Est. Total Parameters
High
Low
Medium
The Graph-based module is generally the most expressive due to its explicit relational structure but also tends to be the most parameter-heavy. MLP-based memory is the simplest, while GRU- based memory offers a balance for capturing temporal context. The choice depends on the specific requirements of the MARL task, including the complexity of inter-agent interactions and available computational resources.
H Implementation Details
For brevity, we previously use abbreviated task names Table 1. Their full names are as follows: Disper Disperse, Naviga Navigation, Sampli = Sampling, Passag = Passage, Transp Transport, Balanc Balance, GiveWa = GiveWay, Wheel WheelGathering, Flocki Flocking (*: sparse-reward version), StaHun StagHuntRepeated, CleaUp Cleanup, ChiGam PriDil
Chi meiickeaaeeReeeeteddd PriDil ariiooersDiilemmReppeeee pro5v5 Protoss5v5, and zer5v5 Zerg5v5.
28

---

Table 6: Key training hyperparameters.
Parameter Category
Value / Setting
General Optimization
Discount Factor (7)
0.99
Learning Rate (Adam)
le-4
Adam Optimizer
1e-6
Target Network Updates
Update Type
Soft (Polyak averaging)
Polyak Tau (T)
0.005
Exploration Strategy
Initial Epsilon (Ciit)
0.8
Final Epsilon (Cend)
0.01
Training Duration
Max Frames
3000000
On-Policy Collection & Training
Collected Frames per Batch
36000
Environments per Worker
60
Minibatch Iterations
30
Minibatch Size
2400
Off- Policy Collection & Training
Optimizer Steps per Collection
1800
Train Batch Size
1024
Replay Buffer Size
1500000
This paragraph outlines the core hyperparameters used for training models and baselines, unless specified otherwise particular tasks or algorithms. Specific parameters related to intrinsic rewards (e.g., exploitation weight factor a) detailed separately in the experimental setup they vary across tasks and model configurations. Training/evaluation was conducted via BenchMARL suite on NVIDIA Quadro RTX 8000 GPUs.
29


Reviews:
[{"review_id": "6qmi8mgNMr", "reviewer": "Reviewer_ckU6", "summary": "This paper investigates the exploration-exploitation problem in MARL from an information-theoretic perspective. Specifically, the authors design CERMIC, a plug-and-play module that helps multi-agent systems learn curiosity-driven signals, and integrates with existing algorithms in the form of intrinsic rewards. Experiments on several benchmarks demonstrate that the proposed method achieves SOTA performance in sparse reward scenarios.", "questions": "1.Besides intrinsic rewards, what other approaches (of utilizing CERMIC) does the author believe can improve the training effectiveness of MARL? I would like the author to provide an in-depth analysis on this question.\n\n2.How should the ∪ symbol in Equation 12 be interpreted?", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**:\n\n(Major) The sparse reward problem is a commonly challenging issue in RL, while the exploration-exploitation problem represents a more fundamental challenge in RL. The proposed novelty-based exploration and Multi-Agent Contextualized Exploitation effectively address the corresponding problems in the MARL domain.\n\n(Minor) This paper is well-written with clear logic and reasonable structure, making it smooth to read.\n\n(Minor) The experimental evaluation is comprehensive, comparing against SOTA methods and curiosity-based approaches in the field, and achieving new SOTA performance. Additionally, the well-designed visualization experiments enhance the overall clarity of the paper.\n\n---\n\n**Weaknesses**:\n\n(Minor) The individual ablation study (Section 5.4) is conducted only in one scenario of one benchmark, which lacks convincing evidence. I suggest the authors conduct experiments across all scenarios within one benchmark or select representative scenarios from all benchmarks.\n\n(Minor) The proposed method introduces additional computational overhead and complexity. How do the authors analyze this aspect?", "was_revised": true, "final_justification": "Most of my concerns addressed.", "rebuttal": "We sincerely appreciate your thoughtful and detailed feedback. We are encouraged by your recognition of our work’s novelty, clarity, and empirical strength. Below, we respectfully address the identified weaknesses (W1, W2) and respond to the insightful questions (Q1, Q2).\n\n---\n### **W1: Ablation Study**\nWe thank you for this valuable suggestion. To thoroughly address this point, we have conducted extensive new ablation studies across diverse scenarios (8 seeds, 95% confidence interval), which robustly validate our claims. \n|                      |Flocki*     |Naviga     |Passag*     |CleanUp     |ChiGam     |\n|-|-|-|-|-|-|\n|**Loss Ablations ($\\alpha=0.2$, Graph-Memory)**|||||  \n|w/o $L_{explore}$        |0.82 (±0.15)|1.41 (±0.04)|166 (±3.66) |75.4 (±1.05)|9.22 (±0.80)|\n|w/o $L_{exploit}$       |0.72 (±0.48)|1.36 (±0.10)|164 (±4.41) |71.5 (±1.59)|8.47 (±1.18)|\n|MAPPO-CERMIC         |1.06 (±0.12)|1.43 (±0.06)|171 (±3.72) |78.1 (±0.82)|10.02 (±0.65)|\n|**Hyperparameter $\\alpha$ (Graph-Memory)**|||||  \n|1.0                  |0.80 (±0.07)|1.40 (±0.02)|163 (±1.56) |71.2 (±0.77)|9.04 (±0.46)|\n|0.5                  |0.88 (±0.10)|1.42 (±0.06)|169 (±3.78) |74.6 (±0.68)|9.50 (±0.67)|\n|0.2                  |1.06 (±0.12)|1.43 (±0.06)|171 (±3.72) |78.1 (±0.82)|10.02 (±0.65)|\n|**Intention Memory Type ($\\alpha=0.2$)**|||||  \n|GRU                  |0.78 (±0.27)|1.39 (±0.04)|161 (±3.15) |75.2 (±1.03)|9.19 (±0.97)|\n|Graph                |1.06 (±0.12)|1.43 (±0.06)|171 (±3.72) |78.1 (±0.82)|10.02 (±0.65)|\n\n\nThe new results, which we will update in Figure 4 and Table 2 of the revised manuscript, consistently demonstrate that: \n\n1. These results confirm that each loss component is essential to CERMIC's performance. This demonstrates that the exploit loss helps the agent better leverage learned knowledge and filter out noise, while the explore loss improves the agent’s overall performance in sparse-reward environments.\n\n2. Employing more sophisticated memory modules yields better performance in complex scenarios.\n\nThese consistent results across a diverse set of environments provide strong and convincing evidence for the necessity of each component in CERMIC's design.\n\n---\n### **W2: Computational Complexity**\nWe appreciate your attention to computational cost. We have carefully considered this point both from theoretical and empirical perspectives.\n\n***Theoretically***, CERMIC is designed with computational efficiency in mind. As illustrated in our workflow (Fig. 1), the computationally intensive raw observations are first projected into a low-dimensional latent space. All subsequent CERMIC computations, including the multi-agent inference and chance constraint optimization, are performed within this compact latent space, inherently limiting the computational burden. We further analyze the computational complexity. The primary overhead lies in the GNN used for intention modeling, while other components incur only constant-time costs (see Appendix Tables 4 and 5 for parameters). The GNN has a complexity of $O(N_o^2)$, where $N_o$ is the number of agents currently detected. However, due to partial observability, the number $N_o$ is bounded, and the overall computation remains manageable.\n\n***Empirically***, to further quantify this, we have conducted new profiling experiments measuring the runtime (inference time) and memory overhead. We compared CERMIC agents against SoTA method QPLEX-ICES on the VMAS benchmark as below. The results show that CERMIC remains a modest memorial cost (MAPPO-CERMIC: 10.4MB, 83% of QLPEX-ICES's weight under the same visual perception module) while achieving SoTA performance, underscoring CERMIC's effectiveness as an efficient module. CERMIC's lightweight yet promising performance gains highlight its plug-and-play capability.\n| Inference Time (per step: ms) | Balanc* | Disper* | Flocki* |\n|-|-|-|-|\n| QPLEX-ICES                    | 2.6     | 2.1     | 3.7     |\n| MAPPO-CERMIC                 | 2.5     | 1.8     | 3.5     |\n\n---\n### **Q1: Training Effectiveness**\nWe thank you for raising this insightful and forward-looking question. In addition to intrinsic rewards, we believe the core elements of CERMIC—specifically the multi-agent contextual feature $f_n^o$ and the adaptive factor $\\gamma$—can be leveraged to address other key challenges in MARL training. We envision two promising directions:\n\n1. **Hierarchical Reinforcement Learning (HRL)**: In MARL, agents often need to adopt specific roles (e.g., cooperative or adversarial) based on the macroscopic tactical situation. The contextual feature $f_n^o$ generated by CERMIC is key to assessing this situation as it encapsulates inferred peer intentions. This would allow for a principled design of a two-level hierarchy where: (i) a high-level policy $\\pi_{high}$ takes $f_n^o$ and the agent's state $s_t$ as input to select a subgoal or role $g_t$; (ii) a low-level policy $\\pi_{low}$ then executes primitive actions based on $g_t$ and $s_t$ to accomplish the current subgoal. This hierarchical design leveraging multi-agent intention modeling would enable more structured and long-horizon strategic reasoning. \n2. **Reward Credit Assignment**: For reward credit assignment, we propose a novel usage of $f_n^o$ to estimate how an agent is perceived by others. This facilitates fairer credit attribution and quicker role stabilization in cooperative scenarios. For instance, in a soccer game, how does an agent identify the most helpful teammate leading to a goal? The feature $f_n^o$ represents the current agent's judgment of others' intentions. It can help us understand what role other agents expect our agent to play. Based on this, in a cooperative game, if an agent is consistently perceived by others as a valuable collaborator, we can assign it a higher share of the team reward. This approach provides a novel mechanism for agents to quickly identify and settle into positions that maximally contribute to the team's success.\n\nIn both scenarios, to handle the initial inaccuracy of the $f_n^o$ model, we can leverage the task-adaptive factor $γ$ to temper reliance on $f_n^o$ until the intention predicting module “matures”, ensuring stability during the early stages of training.\n\n---\n### **Q2: Symbol Interpretation**\nWe thank you for pointing out the ambiguity of this notation. The symbol is intended to signify the addition of a new data point to the dataset of past experiences. This $\\cup$ symbol denotes the action of incorporating the current transition $(s_t, a_t, s_{t+1})$ into the dataset $D_m$, which contains corresponding transitions from the past m episodes. To better illustrate this, we explain the meaning of the formula in which it appears: the purpose of the intrinsic reward in Eq. 12 is to measure the magnitude of change in the CERMIC module's parameter posterior after being updated with this new data. A larger reward implies a greater impact on the parameter distribution, indicating that the new data is novel and has not been encountered in the historical data from the past m episodes. This design sensitizes the CERMIC module to new situations, aligning with the fundamental goal of curiosity-driven exploration. To eliminate this ambiguity in the revised manuscript, we will first define the updated dataset $D_m^{\\prime} = D_m \\cup (s_t, a_t, s_{t+1})$, and then expressing the formulation of the intrinsic reward."}, {"review_id": "8yGxOBtxCd", "reviewer": "Reviewer_iVNP", "summary": "In this work, the authors propose CERAMIC, a method for multiagent exploration which robustly filters noisy surprise signals and guides exploration by dynamically calibrating intrinsic curiosity with inferred multi-agent contextual awareness. CERAMIC also builds off of results from the single agent exploration community to adapt an approach for computing information gain from state transitions as a curiosity reward signal. CERAMIC achieves state of the art results on the benchmark suits VMAS, Meltingpot, and SMACv2 with particularly high performance in sparse reward settings.", "questions": "I understand the point about the clustering behavior around trajectory endpoints in the qualitative evaluation of the agent dynamics, but I do not understand the other points of significance in the qualitative analysis. Could you explain further what is significant about the agent state dynamics? \n\nAt first, I thought that the method was too similar to the intrinsic curiosity formulation proposed for the single agent case. I found that there were a lot of additional modeling considerations that you have to take into account. Could you explain why you would or would not necessarily expect to see the same relative performance and behavior in the multiagent setting if you use intrinsic rewards built and designed for single agents?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The paper is well written and clear. The related work is thorough but some of the references could be introduced earlier. Specifically, the references to the very similar curiosity objective for the single agent case could have been discussed and contextualized in the related work. Two additional references to prior work with very similar intrinsic reward design in single agents:\n\nRuo Yu Tao, et al. Novelty Search in Representational Space for Sample Efficient Exploration. NeurIPS, 2020.\nBernadette Bucher, et al. Perception-driven Curiosity with Bayesian Surprise. RSS Workshop on Combining Learning and Reasoning Towards Human-Level Robot Intelligence, 2019.\n\n\nThe system has theoretical guarantees about exploration performance which the authors highlight. The intention graph is an interesting and novel methodological approach to modeling agent intention in the multiagent settings. The experiments in Table 1 seem to validate the claims of the paper as well as the model ablations. The training dynamics result was interesting and educational about the method. \n\nI did not fully understand the point about the qualitative results for agent dynamics, and I think a clearer explanation is needed about what is desirable in multiagent system dynamics and how the results in the paper show this behavior. The line plots are very hard to read because the chosen colors are two close together. For example, in Figure 4, several very similar shades of blue are used.\n\nThe authors give full details for reproducing the method including code and all relative hyperparameters. Substantial additional experimentation is included in the appendix as well as in depth theoretical explanations of the system and its behavior. Overall, this paper is novel and interesting, technically deep, and experimentally thorough. It was also well written and educational to read.", "was_revised": true, "final_justification": "I thank the authors very much for their detailed response. I maintain my score and agree with the other reviewers on acceptance for all the points previously discussed.\n\nIn particular, I think that the authors did an excellent job in the rebuttal of describing the specific contributions significant to multiagent system dynamics that were unclear. I highly encourage in the final revision of this paper to include these descriptions as well as improvements to the figures for clarity.", "rebuttal": "We sincerely thank you for the insightful and encouraging feedback. We are pleased that you found our work novel and interesting, technically deep, and experimentally thorough. We address the specific points on clarity (W1) and the questions (Q1, Q2) below.  \n\n---\n### **W1: Clarity**\nWe appreciate you pointing out this point for improvement. To enhance readability and better contextualize our work, and will implement the following concrete changes in the revised manuscript:\n1. **Figure Readability**: We will revise the color schemes in Figures 3, 4, and 6 using distinct and colorblind-friendly palettes to ensure all data series are easily distinguishable.\n2. **Related Work Context**: We thank you for pointing us to the two highly relevant papers (Tao et al., 2020; Bucher et al., 2019). We agree they are important and will integrate them into our Related Work section to better contextualize our work and highlight the transition from single-agent to multi-agent curiosity.\n3. **Notation and Background**: To further improve clarity, we will add a comprehensive notation table in the Appendix and briefly expand on the background of chance constraint optimization in Section 4.2 to make the manuscript more self-contained.\n\n---\n### **Q1: Elaborating on the Significance of State Dynamics (Figure 3)**\nThe qualitative analysis in Figure 3 aims to demonstrate how CERMIC fosters a more sophisticated, human-like curiosity, which manifests in four key phenomena. The first phenomenon illustrates curiosity towards peers (social curiosity), while the subsequent three illustrate curiosity towards the environment (environmental curiosity). \n\n**Observed in the left panels of Figure 3 (Social Curiosity)**\n\n***Phenomenon 1. Intentional Trajectory Crossing***: Unlike conventional MARL agents (e.g., MAPPO agents in the figure) whose state trajectories tend to run in parallel (indicating mutual ignorance), CERMIC agents exhibit converging-then-diverging trajectory paths (forming an 'X' shape). This signifies that agents will first intentionally approach a peer to observe, and then diverge to execute their own plans. Crucially, the trajectories do not fully merge into a Y-shape, which would imply mere imitation. This behavior demonstrates an advanced social curiosity, where agents strategically leverage peer observation to inform their own decision-making. \n\n**Observed in the right panels of Figure 3 (Environmental Curiosity)**\n\n***Phenomenon 2. Endpoint Clustering (as you noted)***: The clustering of latent states $x_t$ at the endpoints of state trajectories indicates that the curiosity module is most active during significant state transitions, which represent novel situations. This aligns with the human tendency to be most curious when facing new and unfamiliar circumstances.\n\n***Phenomenon 3. Latent State Dispersion***: We observe that for any given state $s_t$ (the blue point), the corresponding latent states $x_t $ (green points) do not collapse onto $s_t$ but rather form a dispersed cloud around it. This signifies that the agent's curiosity is directed not at its current, known state, but at exploring the adjacent, unvisited states in its local state manifold. This is a targeted and sample-efficient form of exploration, akin to a human's curiosity about what lies \"just around the corner\".\n\n***Phenomenon 4. Intrinsic Reward Dynamics***: The right panel shows the agent's intrinsic reward value. For context, the two blue spheres in the \"Global Task State\" are the agents, and the red sphere is the object they need to transport. We observe a marked increase in intrinsic reward when the two agents approach each other for cooperative action. This empirically validates that our module successfully identifies and incentivizes agent’s attention towards their peers during critical interactive moments, consistent with our design philosophy.\n\n---\n### **Q2: Distinguishing CERMIC from Single-Agent Curiosity**\nThis is a crucial question that gets to the heart of our contribution. We believe directly applying single-agent intrinsic rewards to MARL settings is suboptimal due to their inability to handle socially-induced stochasticity and leverage social learning opportunities. We explain this based on two key observations from our experiments and analysis:\n\n1. **Susceptibility to Social Distraction**: We observe that agents driven by single-agent curiosity formulations (e.g., MAPPO-DB), which do not explicitly model others, are persistently distracted by their peers' actions, even in later training stages. From a single agent's perspective, a peer's complex behavior, without inferred intent, can be viewed as high-entropy, socially-induced stochasticity. A novelty-based algorithm will thus compel the agent to pay undue attention to this \"noise\", hindering task progress (compared to CERMIC, MAPPO-DB exhibits an almost 10% performance drop across nearly all tasks). Our contextual calibration mechanism is specifically designed to address this issue by transforming this perceived noise into a learnable signal, allowing agents to distinguish between meaningless randomness and intentional peer behavior. \n\n    To further analyze this, we conducted additional experiments to examine how the contribution of each agent to the final task performance changes as the number of agents increases. We selected the **Balance*** task scenario and used the task return weighted by the number of agents ($total$-$return$/$num$-$agent$) to represent the per-agent contribution. We find that in traditional methods, the per-agent contribution actually decreases as the number of agents increases, whereas NeuRO is able to mitigate this trend.\n\n    | Num. Agents | 2    | 4    | 6    | 8    |\n    |-|-|-|-|-|\n    | MAPPO-DB   | 12.3 | 13.7 | 13.6 | 12.1 |\n    | MAPPO-CERMIC | 14.4 | 16.7 | 16.4 | 17.3 |\n\n2. **Inability to Leverage Social Learning**: Beyond mitigating interference, explicitly modeling others’ intentions accelerates learning by enabling agents to infer task rules from their peer behaviors, thus enhancing task performances. This is further validated by our pre-training experiment (Fig. 2), where CERMIC agents initialized with a pre-trained intention model exhibit significantly faster convergence. These results, as highlighted by your insightful questions, underscore the importance of a robust mechanism for peer understanding in accelerating adaptation to task rules This mirrors human learning, where observing teammates is crucial for understanding a task and coordinating effectively. Single-agent methods inherently lack this capability."}, {"review_id": "n0oll4p2Ho", "reviewer": "Reviewer_NuKT", "summary": "This paper introduces CERMIC, a framework for intrinsically-motivated MARL in a decentralized, communication-less, sparse reward setting. CERMIC includes a GNN-based context learning mechanism that, intuitively, models agent intentions with an interesting information bottleneck for representation learning intended to deal with stochasticity in the environment. It presents both a theoretical analysis under linearity assumptions as well as experiments in several MARL environments, including both comparisons to MARL and intrinsically motivated MARL baselines as well as ablations.", "questions": "-Clarify why the loss ablation on Wheel is sufficient to demonstrate robustness to noise?\n\n-Am I missing an eval environment, or were there 15? Do you get SotA on 13 or 12 environments?\n\n-Could you explain how the pre-training process works? Does this make for an unfair comparison with baselines?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n+Quality: the paper includes extensive experimentation on a number of benchmark environments, both in terms of baseline comparisons and ablations. The analyses/visualizations are very helpful.\n\n+Significance: the experimental results demonstrate clear success on several benchmark environments, including challenging melting pot mixed motive environments.\n\n+Originality: a number of the components to the architecture are, to my knowledge, new. I’m in particular not familiar with anything that uses this chance constraint component.\n\nWeaknesses:\n\n-Clarity: The paper is quite hard to read at times, leading to confusion that I think could be cleared up by editing. Some of these are small, e.g. l52 “given coverage level” is hard to understand in that surrounding text, l120 “updated via a momentum moving of o_t’s ones”, l144 “for conciseness, let…” when the term was introduced in an earlier equation, section 5.3 has two \n”qualitative analysis” subsections, l155 “2e”, Eq 7 should I assume have “exploit” not “explore”. Others are more significant, e.g. the paper mentions 16 evaluation scenarios, whereas as far as I can tell, there are 15.\n\n-Significance: Continuing with the headline results, the main text notes SotA performance on 13 out of 16 scenarios, but as I mentioned above, there appear to b 15 scenarios with 12 bolded (ours) results, and a baseline on zer5v5 ties your best on one but is not bolded. The rest appear not to hav errors like that, however, it’d be helpful to understand which differences are significant e.g. Disper on VMAS is very close. Results still look good but appear overstated.\n\n-Significance/Quality: Ablations appear to be run on a small subset of environments, e.g. loss module ablations are run just on Wheel. I wouldn’t necessarily ask for more (you’ve run a lot of experiments) but it’d be helpful to understand why these choices make sense. E.g. the ablation on Wheel appears to be the primary supporting evidence that the explore/exploit loss configuration helps the model deal with noise, and a good deal of exposition was devoted to this property early on.\n\n-Significance/Clarity: you use pre-training, but the particulars of how this is done are barely mentioned in the main text and not super clear in the supplementary, either. One concern is that this makes for an unfair comparison with baselines, as it relies on e.g. pre-training agent identifiers using ground truth signal.", "was_revised": true, "final_justification": "Provided in a separate comment.", "rebuttal": "We sincerely thank you for the thorough and detailed feedback. We appreciate the positive acknowledgment of our work's originality and the extensive nature of our experiments. In response to your valid concerns, we have conducted new experiments and will make substantial revisions to the manuscript to address every point. We address the weaknesses (W1, W2, W3, W4) and questions (Q1, Q2, Q3) below.\n\n---\n### **W1: Clarity**\nWe sincerely apologize for the lack of clarity and the numerous typographical errors; we have performed a thorough revision to address all identified issues and improve overall readability. Your meticulous reading has been invaluable for improving the quality of our manuscript. \n\nWe have carefully corrected all specific errors you pointed out:\n- l155: \"2e\" → \"we\", \n- Eq 7: \"explore\" → \"exploit\", \n- Second “Qualitative Study” corrected to \"Quantitative Analysis\"). \n- l52 \"coverage level\": We respectfully clarify that this is standard terminology in chance-constrained optimization, referring to the probability $1-\\epsilon$ that the constraint must hold.\n- l120 \"momentum moving\": We have revised this to clarify it refers to a standard momentum update for the target encoder, where its weights are a moving average of the online encoder's weights.\n\nFurthermore, to fully enhance readability and better contextualize our work, we will also implement the following concrete changes in the revised manuscript:\n\n1. **Figure Readability**: We will revise the color schemes in all figures using distinct and colorblind-friendly palettes to ensure all data series are easily distinguishable.\n2. **Notation and Terminology**: To ease the reading process, we will add a comprehensive notation table in the Appendix to serve as a quick reference. At the same time, we will present the relationships between modules in a more intuitive manner to clarify the underlying logic.\n3. **Background Context**: To make the text more self-contained for a broader audience, we will expand on the background of key prerequisite concepts, notably chance-constrained optimization, in the relevant sections. \n\n---\n### **W2 & Q2: Experimental Rigor**\nYou are correct, and we sincerely apologize for the typos and the lack of precision in our initial results presentation. We evaluated on 15 total scenarios and our method achieved SoTA performance in 12 of them. We have corrected this in the revised version. To address your valid and crucial concerns about significance and potential overstatement, we now provide a more rigorous, statistically-grounded analysis.\n\n***Firstly***, to directly address whether our improved scores are genuine or merely due to chance, we will report the 95% confidence interval for the mean performance of all methods as follows (due to space limitations, we only show some of the results). The results show that the confidence interval of our method lies entirely above the confidence interval of a baseline, it provides strong statistical evidence that our performance is genuinely higher. \n\nAlso, by examining the size of the confidence intervals in the revised table, one can observe that CERMIC's intervals are consistently tighter than those of the baselines. This demonstrates that CERMIC not only performs better on average but also converges more reliably and robustly.\n|          | Flocki*        | Naviga        | Passag*       | CleanUp       | ChiGam        |\n|-|-|-|-|-|-|\n| MAPPO         | -0.36 (±0.11)  | 1.07 (±0.10)  | 157 (±5.62)   | 74.2 (±0.93)  | 8.43 (±0.84)  |\n| QMIX-SPECTRA  | 0.01 (±0.14)   | 1.13 (±0.07)  | 155 (±4.38)   | 70.3 (±0.84)  | 8.50 (±0.79)  |\n| QPLEX-ICES    | 0.64 (±0.15)   | 1.34 (±0.07)  | 162 (±4.20)   | 76.5 (±1.06)  | 9.07 (±0.80)  |\n| MAPPO-CERMIC  | 1.06 (±0.12)   | 1.43 (±0.06)  | 171 (±3.72)   | 78.1 (±0.82)  | 10.02 (±0.65) |\n\n***Secondly***, while achieving such superior performance, CERMIC maintains a moderate memory overhead (MAPPO-CERMIC: 10.4MB, 83% of QLPEX-ICES's weight under the same visual perception module), indicating that CERMIC can be lightweightly integrated into existing algorithms to enhance agent performance. CERMIC's lightweight yet promising performance gains highlight its practical potential for seamless integration into existing pipelines. Your request for experimental rigor was highly insightful and has greatly improved the presentation and reliability of our results.\n\n---\n### **W3 & Q1: Ablation Study**\nWe thank you for this valuable suggestion. To thoroughly address this point, we have conducted extensive new ablation studies across diverse scenarios (8 seeds with 95% confidence interval), which robustly validate our claims. \n|                      |Flocki*     |Naviga     |Passag*     |CleanUp     |ChiGam     |\n|-|-|-|-|-|-|\n|**Loss Ablations ($\\alpha=0.2$, Graph-Memory)**|||||  \n|w/o $L_{explore}$        |0.82 (±0.15)|1.41 (±0.04)|166 (±3.66) |75.4 (±1.05)|9.22 (±0.80)|\n|w/o $L_{exploit}$       |0.72 (±0.48)|1.36 (±0.10)|164 (±4.41) |71.5 (±1.59)|8.47 (±1.18)|\n|MAPPO-CERMIC         |1.06 (±0.12)|1.43 (±0.06)|171 (±3.72) |78.1 (±0.82)|10.02 (±0.65)|\n|**Hyperparameter $\\alpha$ (Graph-Memory)**|||||  \n|1.0                  |0.80 (±0.07)|1.40 (±0.02)|163 (±1.56) |71.2 (±0.77)|9.04 (±0.46)|\n|0.5                  |0.88 (±0.10)|1.42 (±0.06)|169 (±3.78) |74.6 (±0.68)|9.50 (±0.67)|\n|0.2                  |1.06 (±0.12)|1.43 (±0.06)|171 (±3.72) |78.1 (±0.82)|10.02 (±0.65)|\n|**Intention Memory Type ($\\alpha=0.2$)**|||||  \n|GRU                  |0.78 (±0.27)|1.39 (±0.04)|161 (±3.15) |75.2 (±1.03)|9.19 (±0.97)|\n|Graph                |1.06 (±0.12)|1.43 (±0.06)|171 (±3.72) |78.1 (±0.82)|10.02 (±0.65)|\n\nThe new results, which we will update in Figure 4 and Table 2 of the revised manuscript, consistently demonstrate that: \n1. The results confirm that each loss component is essential to CERMIC's performance. This demonstrates that the exploit loss helps the agent better leverage learned knowledge and filter out noise, while the explore loss improves the agent’s overall performance in sparse-reward environments.\n2. Employing more sophisticated memory modules yields better performance in complex scenarios.\n\nThese consistent results across a diverse set of environments provide strong and convincing evidence for the necessity of each component in CERMIC's design.\n\n---\n### **W4 & Q3: Pre-training Procedure**\nThe pre-training of our intention modeling module is conducted as follows. We deploy $N$ agents in the environment and allow them to move. At each timestep, the intention module for a given agent is tasked with producing two outputs for the other $N-1$ agents: (1) a judgment on whether each peer is within its observation field, and (2) an estimation of the state for any detected peers. These outputs—the presence judgment and the state estimation—are then supervised using the ground-truth information. The network is updated by computing losses via Eq. (14) and Eq. (15) in Appendix A, which correspond to a detection loss and a state prediction loss, respectively.\n\nWe acknowledge your valid concern that introducing pre-training could lead to an unfair comparison with baselines. To ensure fairness, we want to clarify that, unless explicitly stated otherwise, all primary results for CERMIC reported in the main text (e.g., Table 1) are from models trained from scratch, WITHOUT any pre-training. The pre-trained variant is only used for a specific ablation study to analyze the impact of a proficient intention model. However, results show that the CERMIC agent without the pretraining module still achieves SoTA performance in 12 out of 15 scenarios. We provide additional experiments on the role of the pretraining module in Appendix Figure 7.\n\nThe primary purpose of conducting experiments with a pre-trained module is to illustrate the significance of the intention modeling component and to highlight the broad applicability and future potential of CERMIC. The field of motion and behavior prediction is relatively mature, with many lightweight open-source models and APIs available. Our results demonstrate that CERMIC's performance can be further enhanced by leveraging these readily available, off-the-shelf tools. This modularity makes our framework highly adaptable to real-world scenarios where such auxiliary predictive models can be easily integrated to bootstrap performance.\n\n---\n\nOnce again, we express our sincere gratitude for your detailed and constructive review. We hope that our detailed responses and the new experimental results have fully addressed your concerns and have clarified the contributions and significance of our work. We look forward to your further feedback."}, {"review_id": "ofUqT2IJen", "reviewer": "Reviewer_VKXv", "summary": "CERMIC addresses exploration in decentralized, communication-less multi-agent reinforcement learning under sparse rewards. The authors identify the failure modes of standard novelty-driven intrinsic rewards, especially under environmental stochasticity and partial observability; they propose CERMIC: a plug-and-play module that uses a learned multi-agent intention model  to calibrate intrinsic curiosity signals. The method optimizes an Information-Bottleneck objective to maximize mutual information with future states while constraining self-information conditioned on past states and actions. Based on this objective and Bayesian surprise, CERMIC generates theoretical intrinsic rewards and integrates with standard MARL algorithms(MAPPO and QMIX). Empirically, CERMIC achieves state-of-the-art performance on 13 of 16 sparse-reward tasks across VMAS, MeltingPot, and SMACv2 benchmarks.", "questions": "1. Some recent works propose alternative mutual information measures (e.g. sliced mutual information [1,2]) that are more tractable and might work better in optimization and representation learning problems, have you considered incorporating these with your method? Could you provide a few comments on these and how your method (eg intrinsic reward) changes in the light of the new information measure?\n2. How sensitive is CERMIC to the quality of the pre-trained agent detector? Could you report performance when using only on-policy learned detectors without ground-truth supervision?\n\n[1] “Sliced Mutual Information: A Scalable Measure of Statistical Dependence” (Goldfeld & Greenewald, NeurIPS 2021)\n\n[2] “On Slicing Optimality for Mutual Information” (Fayad & Ibrahim, NeurIPS 2023)", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Quality**\n\nThe paper is technically sound and presents an intriguing well-motivated theoretical framework based on the information bottleneck (IB) and robust chance constraints. They show how the intrinsic reward is connected to Bayesian surprise and aligned with UCB bonuses in linear MDPs. There has been an extensive empirical evaluation on diverse benchmark suites under standardized sparse-reward settings where wee see improvements over SoTA baselines. However, the reliance on learning an accurate intention model from scratch slows early training and may be impractical in very large agent populations. The pre-training procedure for the detector and graph memory uses privileged ground-truth labels, thus possibly limiting applicability to domains without such supervision.\n\n\n**Clarity**\n\nThe paper is clearly structured, with intuitive diagrams, however the notation was dense that it might impede reader’s accessibility. \n\n**Originality**\n\nThe combination of robust chance-constrained Information Bottleneck with multi-agent intention modeling is, as far as I know, original. The theoretical link between Bayesian surprise reward and UCB bonuses provides fresh insight.\nHowever, Elements such as InfoNCE-based mutual information bounds and dynamic graphs have appeared in earlier works [eg arXiv:1807.03748]; thus the combination, while novel, builds on many established components.\n\n**Significance**\n\nThis work tackles multi-agent exploration under sparse rewards which is a critical open problem; and as mentioned above: introducing socially contextualized intrinsic motivation represents a novel direction that will likely influence future MARL research.\nThat being said, the requirement for a pre-trained agent detector and graph memory (as described in Appendix A) may limit immediate impact in fully unknown environments. And the method’s complexity may hinder adoption compared to simpler curiosity baselines.\n\nOverall, I found this work to be interesting and potentially impactful with a few limitations.", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the thoughtful and encouraging feedback. We are grateful for the positive assessment of our work's technical soundness, originality, and extensive empirical evaluation. We address the specific weaknesses (W1, W2, W3) and questions (Q1, Q2) below.\n\n---\n### **W1: Reliance on Pre-training Procedure**\nWe acknowledge this as an important practical consideration. However, We would like to clarify that the primary purpose of our pre-training experiments is to illustrate the significance of the intention modeling component and to highlight the broad, practical applicability of the CERMIC framework. We are motivated by the observation that the field of motion and behavior prediction is relatively mature, with many open-source models and APIs available. Our results demonstrate that CERMIC's performance can be further enhanced by leveraging these readily available, off-the-shelf tools, making it highly adaptable to real-world scenarios. Furthermore, we respectfully clarify that, unless explicitly stated otherwise, all primary results for CERMIC reported in the main text are from models trained from scratch, WITHOUT any pre-training.\n\nFundamentally, our core contribution is the principled curiosity calibration mechanism that operates on top of an intention prediction; this mechanism is agnostic to the specific implementation of the intention predictor.\n\n---\n### **W2: Notation Accessibility**\nWe thank you for pointing this out and will make several revisions to improve the paper's clarity and accessibility. To address this, we will make the following concrete revisions in the camera-ready version to improve readability and accessibility for a wider audience:\n1. Add a comprehensive notation table in the Appendix to serve as a quick reference.\n2. Expand on the background of chance constraint optimization in Section 4.2 to make the text more self-contained for a broader audience.\n3. Perform a thorough proofreading of the entire manuscript to correct minor errors and refine phrasing for better readability.\n\n---\n### **W3: On the Manageable and Necessary Complexity of CERMIC**\nWe will demonstrate the manageable nature of our method's complexity from both theoretical and empirical perspectives.\n\n***Theoretically***, CERMIC is designed for efficiency. As illustrated in our workflow (Fig. 1), it first projects high-dimensional raw observations into a low-dimensional latent space. All subsequent, more complex computations are performed within this compact space, which inherently limits the computational burden. We further analyze the computational complexity. The primary overhead lies in the GNN used for intention modeling, while other components incur only constant-time costs (see Appendix Tables 4 and 5 for parameters). The GNN has a complexity of $O(N_o^2)$, where $N_o$ is the number of agents currently detected. However, due to partial observability, the number $N_o$ is bounded, and the overall computation remains manageable.\n\n***Empirically***, to provide concrete evidence, we have conducted new profiling experiments measuring runtime and memory overhead. We compared CERMIC agents against SoTA method QPLEX-ICES on the VMAS benchmark as below. The results show that CERMIC remains a modest memorial cost (MAPPO-CERMIC: 10.4MB, 83% of QLPEX-ICES's weight under the same visual perception module) while achieving SoTA performance, underscoring CERMIC's effectiveness as an efficient, plug-and-play module.\n|Inference Time (per step: ms)|Balanc*|Disper*|Flocki*|\n|-|-|-|-|\n|QPLEX-ICES|2.6|2.1|3.7|\n|MAPPO-CERMIC|2.5|1.8|3.5|\n\nFurthermore, we argue that this manageable complexity is necessary, as our experiments reveal critical two failure modes in simpler curiosity baselines:\n1. **Susceptibility to Social Distraction**: Our empirical results indicate that agents driven by simpler curiosity formulations (without explicit modeling of others) are persistently distracted by peers' actions in later training stages. From a single agent's perspective, a peer's complex behavior without inferred intent is treated as unlearnable noise, which a novelty-seeking algorithm is drawn to, thus hindering task progress. Our calibration mechanism transforms this noise into a learnable signal.\n\n    To further analyze this, we conducted additional experiments to examine how the contribution of each agent to the final performance as the number of agents increases. We selected the Balanc* scenario and used the task return weighted by the number of agents ($total$-$return$/$num$-$agent$) to represent the per-agent contribution. We find that in traditional methods, the per-agent contribution actually decreases as the number of agents increases, whereas NeuRO is able to mitigate this trend.\n\n    |Num. Agents|2|4|6|8|\n    |-|-|-|-|-|\n    |MAPPO-DB|12.3|13.7|13.6 | 12.1 |\n    |MAPPO-CERMIC|14.4|16.7 | 16.4 | 17.3 |\n\n2. **Inability to Leverage Social Learning**: Beyond mitigating interference, we found that explicitly modeling peer intentions accelerates learning by enabling agents to understand implicit task rules and coordinate. This is validated by our pre-training ablation (Fig. 2), where agents with a proficient intention model converge significantly faster, mirroring how humans leverage team observation to learn and cooperate.\n\n---\n### **Q1: Alternative Mutual Information Measures**\nWe sincerely thank you for this insightful question. The methods you've pointed to, Sliced Mutual Information (SMI) and particularly its optimized variant, Optimal SMI (ST*), represent an elegant approach to the challenges of high-dimensional mutual information (MI) estimation. We will focus our discussion on integrating the more advanced ST*\n\nThe central idea of our current method is to learn a multi-agent contextual representation and use it to calibrate an exploration objective based on classic MI. This process is to learning an external auxiliary representation to better optimize a fixed, theoretically-optimal but computationally-intractable MI objective.\n\nThe philosophy behind ST*, however, is fundamentally different. It posits that the uniform random projections of standard SMI are suboptimal, as they waste computational resources on uninformative \"noisy\" directions. The core contribution of ST* is therefore to learn an optimal distribution of projections, concentrating the information measure itself on the most critical subspaces. Therefore, integrating ST* into CERMIC would induce a fundamental shift in the curiosity calibration mechanism:\n\n**Current Framework**: `[Representation Learning Module] → Calibrates → [Fixed MI Objective]`\n\n**New Framework with ST***: `[Representation Learning Module] → Guides → [Adaptive ST* Objective]`\n\nIn this new framework, CERMIC's multi-agent context module would no longer be calibrating a static information objective. Instead, it would be tasked with dynamically guiding the learning process of the \"optimal slices\" within ST*. For instance, the inferred multi-agent context could serve as a prior, helping ST* to more quickly discover projection directions that are not only critical for predicting future states, but also for understanding multi-agent interactions.\n\nAs for the intrinsic reward, the integration of ST* would also lead to a more profound form of the reward signal. Our current intrinsic reward is derived from the Bayesian Surprise on CERMIC parameters. In the new framework, the primary learnable component within the reward would be the slicing policy network of ST*. Therefore, the intrinsic reward could be redefined on the parameters of this slicing policy network.\n\nIn summary the integration would create a powerful synergistic effect: CERMIC provides the high-level social context, and ST*, under this guidance, efficiently identifies the most information-rich low-level subspaces. This combination of \"high-level semantics guiding low-level representation learning\" could be more robust and efficient than our current approach.\n\n---\n### **Q2: Sensitivity to the Pre-Trained Agent Detector**\nWe wish to clarify that, UNLESS explicitly stated otherwise, all main results for CERMIC reported in the text use an on-policy learned detector without any pre-training to ensure a fair comparison.\nOur experiments (Fig. 2 in the main text and Fig. 8 in the Appendix) show that pre-training does lead to moderate performance improvement. However, it also introduces additional training overhead. This suggests that the on-policy learned graph and intention model are sufficient for achieving high performance after a reasonable amount of training.\nTo provide additional support for this observation, we present numerical results. The table below compares the final performance of CERMIC with and without pre-training on several key tasks. The results are consistent with the conclusion above.\n||Flocki*|Naviga|Passag*|CleanUp|ChiGam|\n|-|-|-|-|-|-|\n|CERMIC + pretrain|1.08 (±0.14)|1.46 (±0.07)|173 (±2.29)|83.8 (±0.60)|11.93 (±0.44)|\n|CERMIC|1.06 (±0.12)|1.43 (±0.06)|171 (±3.72)|78.1 (±0.82)|10.02 (±0.65)|"}]

Author rebuttal:

Meta-review: This paper introduces CERMIC, a principled framework for curiosity-driven exploration in multi-agent reinforcement learning that calibrates intrinsic rewards using inferred contextual awareness of peers. The reviewers found the central idea both novel and timely, addressing the longstanding challenge of robust exploration under sparse rewards and partial observability. The method is theoretically motivated via the Information Bottleneck principle, and the authors provide clear algorithmic design along with ablations that highlight the importance of each component. Empirical results across VMAS, MeltingPot, and SMACv2 benchmarks show consistent and often state-of-the-art improvements over strong MARL baselines and prior curiosity-based methods, validating both effectiveness and generality. While limitations remain regarding scalability of intention modeling and reliance on local observations, the authors acknowledge these and suggest promising directions for future work. Overall, the paper makes a significant and well-substantiated contribution, and I recommend acceptance.
Decision: Accept (poster)