GlobeDiff: State Diffusion Process for Partial Observability in Multi-Agent System
Authors: Yiqin Yang
Xu Yang
Yuhua Jiang
Ni Mu
Hao Hu
Runpeng Xie
Ziyou Zhang
Siyuan Li
Yuan-Hua Ni
Qianchuan Zhao
Bo XU
Venue: ICLR 2026 Poster (2026)
Keywords: Multi-Agent Systems
Partial Observability
Diffusion Models
Primary area: reinforcement learning

Abstract: In the realm of multi-agent systems, the challenge of partial observability is a critical barrier to effective coordination and decision-making. Existing approaches, such as belief state estimation and inter-agent communication, often fall short. Belief-based methods are limited by their focus on past experiences without fully leveraging global information, while communication methods often lack a robust model to effectively utilize the auxiliary information they provide.
To solve this issue, we propose Global State Diffusion Algorithm to infer the global state based on the local observations.
By formulating the state inference process as a multi-modal diffusion process, GlobeDiff overcomes ambiguities in state estimation while simultaneously inferring the global state with high fidelity.
We prove that the estimation error of GlobeDiff under both unimodal and multi-modal distributions can be bounded.
Extensive experimental results demonstrate that GlobeDiff achieves superior performance and is capable of accurately inferring the global state.

Published as a conference paper at ICLR 2026
GLOBEDIFF: STATE DIFFUSION PROCESS FOR PARTIAL OBSERVABILITY IN MULTI-AGENT SYSTEMS
Yiqin Yang Xu Yang"; Yuhua Jiang", Ni Mu?, Hao Hu Runpeng Xie', Ziyou Zhang?, Siyuan Li", Yuan-Hua Ni Qianchuan Zhao²; i Bo Xu
'The Key Laboratory of Cognition and Decision Intelligence for Complex Systems,
Institute of Automation, Chinese Academy of Sciences
TTiinghua University MMoonshot AI "Nakaai University
Faculty of Computing, Harbin Institute of Technology
ABSTRACT
In the realm of multi-agent systems, the challenge of partial observability is a critical barrier to effective coordination and decision-making. Existing approaches, such as belief state estimation and inter-agent communication, often fall short. Belief-based methods are limited by their focus on past experiences without fully leveraging global information, while communication methods often lack robust model to effectively utilize the auxiliary information they provide. To solve this issue, we propose Global State Diffusion Algorithm (GlobeDiff)) to infer the global state based on the local observations. By formulating the state inference process multi-modal diffusion process, GlobeDiff overcomes ambiguities in state estimation while simultaneously inferring the global state with high fidelity. We prove that the estimation error of GlobeDiff under both unimodal and multi-modal distributions can be bounded. Extensive experimental results demonstrate that GlobeDiff achieves superior performance and is capable of accurately inferring the global state.
INTRODUCTION
Multi-Agent Reinforcement Learning (MARL) has driven significant progress in complex domains
like robotics (Wang et al., 2022 Lee al. 2022 and autonomous systems (Zhang et al., 20224] Zhou
et al. 2022). enabling agents to learn sophisticated collaborative policies (Feng et al. 2024 Wang
jtt al. 2024b). However, fundamental and persistent barrier to effective multi-agent coordination is
the problem of partial observability (PO), where each agent's view is limited (Amato et al. 2013
Omidshafici et al.) 2017 Srinivasan et al. 2018), and the true global state of the system is unknown.
This challenge, formally captured in the Decentralized Partially Observable Markov Decision Process (Dec-POMDP) framework (Oliehoek et al.] 2016), forces agents to act under uncertainty, often leading to suboptimal conflicting decisions (Spaan 2012).
The core difficulty of partial observability lies in the profound ambiguity it creates: a single agent's s local observation can be consistent with numerous, often dramatically different, global states. This creates a challenging one-to-many mapping problem for state inference. Existing approaches have attempted resolve this ambiguity using discriminative models, such as recurrent networks or Transformers (Hausknecht & Stone 2013 Kapturowski et al. 2018). which learn to predict a single, most likely global state from a history local observations. However, this approach fundamentally flawed. By collapsing rich distribution of possibilities into a single point estimate, these methods suffer from mode collapse. They either average distinct plausible states into a single, nonsensical representation or arbitrarily commit to one possibility while ignoring others, failing to capture the true uncertainty of the environment.
The central thesis of this paper is that the one-to-man many ambiguity inherent in global state inference is best addressed not by discriminative prediction, but by generative modeling (Goodfellow et al.
"These authors contributed equally. Cooresponning author.

---

Published as a conference paper at ICLR 2026
Construct Auxiliary
GlobeDiff Inference Phase
Pa.p (x)
Local Observations
x network
Sampling
Po(A)z) z~ z~N(((().o()))
Observation
Global $
Concat
stepe
Eg
[
[xx,
Action ai
zz((: (8)
Denoising
sK-1
Denoising
5x ~N-((
Multi-Agent Environment
Agent Policy
The overall framework of GlobeDiff. During the execution phase, we first construct auxiliary local observations and then infer the global state using GlobeDiff. Agents make decisions based on the inferred global state 8.
Figure 1:
2014 Ho al. 2020 Song al. 2020). Instead of forcing collapse single mode, generative approach can learn the entire conditional distribution of plausible global states. This allows an agent reason the full spectrum of possibilities, capability that is critical for robust decision-makee under uncertainty. By sampling from this learned distribution, our method can generate high-fidelipy hypotheses about the global state, directly confronting the multi-modalle of the problem.
To realize this vision. we introduce the Global State Diffusion Algorithm (GlobeDiff), novel framework that operationalizes this generative insight using conditional diffusion model. GlobeDiff formulates global state inference as denoising process, learning to reverse a diffusion process that gradually corrupts the global state into noise. Conditioned on an agent's 's local information (such as its own observations communicated messages (Kim et al. 2219 Jiang & Lu 2018)), GlobeDiff can generate a diverse and realistic set of potential global states. This approach not only provides a more principled solution to the one-to-man many mapping problem but also integrates seamlessly into existing MARL frameworks like Centralized Training with Decentralized Execution (CTDE). Our contributions are:
We identify and frame the core challenge of partial observability as a one-to- many mapping problem, highlighting the limitations of existing discriminative approaches.
We propose GifbeDiff, the framework leverage conditional diffusion models for generative global state inference in MARI, offering robust solution to mode collapse.
We empirically demonstrate that GlobeDiff significantly outperforms state-of-the-ae base- lines on challenging multi-agent benchmarks, validating the power of the generative ap- proach.
2 RELATED WORK
Partial Observability To solve the PO problem, particularly in Dec-POMDPs, existing research can be divided into two categories: belief state estimation and explicit communication. First, model uncertainty multi-agent systems, the concept of belief state has been introduced to estimate the state of the environment other agents (MacDermed & Isbell 2013 Muglich al., 2022 Varakantham eet al. 2006). For example, given the effectiveness in handling temporal sequences, RNNs are used to integrate local observation histories over time, providing agents with long-term memory (Hausknechl & Stone 2015 Kapturowski et al.) 2018 WWnn et al., (022). However, estimation errors accumulate over time, leading insufficient information in complex systems and hindering a comprehensive understanding of the global state. In contrast to inferring or estimating the global belief state, inter- agent communication has been introduced to directly acquire information from other agents and expand the receptive field of individual agents (Das et al. 2019 Singh et al.) 2018 Zhang et al. 2019 Kim et al. 2019 Jiang & Lu 2018). However, these approaches suffer from high communication costs and complex protocol design.
Diffusion Model for RL Diffusion models leverage a denoising framework and effectively reverse
multi-step noise processes to generate data (Ho et al. 2020 Song et al.) 2020). These models
2

---

Published as a conference paper at ICLR 2026
Reconstruct Global State via Conditional Diffusion
Larr c ca(s xx,,k)
True Stnte
Forward Procers
Formard Process
Noise e
Noise gRo-k
g0
s
Process
=*,N(0;1)
Condition on (3,
Sample py(xp))
Minimize Prior-Posterior KL Divergence
(*) po(+)))
True State s
9u(alk.s)
Pp(a(k)
Auxiliory Local
Auciliarr Lecal
Posterior Net
Prior Net Pa
Figure 2:
The training process of Globediff is divided into two parts: minimizing the difference between the prior network Ps and the posterior network Gø. and then training the diffusion model based on the forward and backward process.
have increasingly been integrated into sequential decision decision-making tasks to improve performance
and sample efficiency, particularly in single-agent and offline reinforcement learning. For example, diffusion models have been employed as planners, encoding dynamic environmental information and
generating multi-step optimal trajectories (Janner al. 2022 Liang et al.) [023 He et al. [023]
Ajay et al.) Chi et al. 2023). This integration helps mitigate compound errors in auto-regressive
sequence planning and facilitates better long-term pecision-makinng Additionally, diffusion models have recently been applied to address the PO problem in multi-agen systems (Wang et al.) (2024a).
However, they focuses on approximating belief distributions via shared attractors, but not explicitly model the intrinsic one-to-many mapping from local observations to the global state.
3 PRELIMINARIES
Dec-POMDPs We consider Dec-POMDP as standard model consisting of tuple G =< for cooperative multi-agent tasks. Within G, 8 S denotes the global state of the environment. Each agent N := 1,..., n chooses an action a; E A at each time, forming a joint action a E A". The state transition function P(s'|s, a) : A + [0, 1] gives (e'ls,o
transition to the environment. The reward function R(s, a) A + R is shared among all agents and y E [o, 1) is the discount factor.
In partially observable scenario, each agent has individual observations O according to the observation function ((ss : S A + O. Each agent makes decision based on a stochastic policy Tæ, (a: 0z) parameterized by 0g: O A [0, 1]. The joint value function be defined Die 7'rets/9: = 8, at a, T). where π is a joint policy with
as Q (St, ae) = E2 20.,10
parameters =< 2 01,...,
Generative Model for Global State Inference We aim to learn mapping from the auxiliary local observations x to the global state based on the generative model po(s æt), where auxiliary local observations x are composed of local observations Therefore, agents can make decisions based on the global state rather than the local observations o during execution by leveraging the generative model Pe, thereby overcoming the limitations of partial observability. In this work, we consider two scenarios for generating auxiliary states. First, if local observations o are information-richh we can infer the global state based on the individual agent's historical trajectory, in which case the auxiliary local observations 2, is formulated as the integration observations of over the past m steps:
3t
{of-m: oL-m11>
where of denotes the local observation of agent at time step t. On the other hand, if local observations provide limited information, it becomes challenging infer the global state based the individual
3

---

Published as a conference paper at ICLR 2026
agent's historical trajectory. In such scenarios, enable communication between agents, and consequently, the auxiliary local observation 2, is constructed from their joint observations:
21 for of. ,
METHOD
Our methodological approach is designed to tackle the fundamental ambiguity of partial observability; a single local observation x can correspond to many different, yet plausible, global states s. A naive conditional generative model p(s (z) would struggle with this one- one-to-many mapping, likely averaging over the possibilities and producing a blurry global state. To address this, we introduce key architectural choice: latent variable The intuition is
mode selector. Instead of asking the model to solve the ambiguous problem of generating from H, we ask it to solve the well-posed problem of generating from both æ and z. The latent variable provides the specific context needed to select one particular plausible global state from the distribution of possibilities. This transforms the problem into learning a conditional diffusion model 2))...2).
This design, however, introduces new challenge: how do we obtain meaningful 2 during inference when we only have the local observation x? We solve this by bridging the gap between training and inference. During training, we have access to the ground-truth global state which allows us to train an posterior network, (= that learns the ideal required reconstruct s from x. For inference, we train separate prior network, p(z (z), which predicts useful z using only The following sections will detail the mathematical formulation of this diffusion process.
4.1 GLOBAL STATE DIFFUSION PROCESS
4.1.1 TRAINING
Inspired by nonequilibrium thermodynamics (Sohl-Dickstein al. 2013), attempt to formulate pe(s æ) as a diffusion process. However, since the observation function 24 does not assume the unique mapping between elements of the input set (S x A) and output set O different global states may be mapped to the same local observation. Therefore, when inferring global states from local observations, the ambiguity issue caused by the non-unique mapping significantly decreases the accuracy of global state inference. To address this issue, we introduce latent variable z that allows the diffusion process to map single input x to multiple outputs s, that is modeling a one-to-many conditional generative distribution ( r):
-f po(s xr, x),<)p<< [z)d,
(0 x) -
where (=(2 xx) is conditional prior. Then, we introduce an approximate posterior qu (z xr s) and derive the following equation based on Jensen's inequality (Kingma & Welling 2014);
log pa x)) = log / (s 2)po(2 x)d2
x, 2r 2))p(( x)) log / q xx Po
po(s
dz
2o 2 8)
(4)
2 Egg [oog po(s (x x, 8) En, [log
wr, 2)pp( a))
x,, z)] (L x, s))p((( xx)).
in the forward process, we first sequentially introduce Gaussian noise e to the global state s according to the predefined variance:
For (o(s (z, 2).
g(s - V1 (T),
where k E {O..... K) is the diffusion timestep, 8 is the variance parameter, s is the original state and & is the state corrupted with k-step noise. For any sh, we compute from the original state o without intermediate steps:
L - v + V o (s
(6)
4

---

Published as a conference paper at ICLR 2026
where c(s*, k) I), is the k-th step noise of the forward process and ax j with a =1 =1 Br. Then, we represent the global state reference via the reverse process of the diffusion model as
po x) = po x. (s* 0, I)TT
(P 1 ,2,
where the end sample of the reverse chain 50 is the restored global state. Generally, po s x.22 _ could be modeled as a Gaussian distribution .V ; 4sa (s" X, 2, k), a 2, k)). We follow Ho al. (2020) to parameterize po(s 1-1 s x 2) as noise prediction model with the covariance matrx fixed as (s 2, = B²II and mean constructed as
3
po(s 3r 2, Va
co ,,, (V-Vi-a**
8"
2, 2,)
We first sample 5k ~ N(0, I) and then form the reverse diffusion chain parameterized by 0 as
8
s k
s
cols 2, 2,)
VBte, e~ c~N
(9)
Therefore, the global state diffusion process is trained by minimizing the following loss function: C(O, Ø, v)
||e - Eg (va's vI Via'svi--'' "e,1-,,,,,,, [1a]
(10)
SKG. KL(gw(= x, s) s)(lp(( x));
where 24 is a uniform distribution over the discrete set as {1, K]. D denotes the datasets and 3KL. hyperparameter. The overall training process of GlobeDiff is shown in Figure 2
4.1.2 INFERENCE
In the inference phase, each agent first obtains the latent variable via the encoder Po(= a.). Then, each agent initializes s ~ (O,, and performs K iterative denoising sampling steps. Note that 5 is initialized as Gaussian noise, and S obtained after denoising steps is the inferred global state. Crucially, no global information is utilized throughout the inference process. Consistent with the reverse process in Equation 9 the inference at the k-step is performed as follows:
gh
c
x, 222,k)
+ VBEe
(11)
Va
V1 t
where e ~ (O, I) represents standard Gaussian noise. After K inference steps, s0 becomes the inferred global state, where each agent makes decisions by a, no, (0))
The aforementioned global state diffusion process exhibits the following characteristics. First, the diffusion process does not explicitly model the distribution of generated samples but implicitly learns through the denoising network ég. Therefore, the marginal of the reverse diffusion chain provides an expressive distribution that capture complex distribution properties. Second, the proposed global state diffusion process can model the non-unique mapping relationship between local observations and the global state. Finally, the global state inference is conditioned the auxiliary state, enabling sampling those global states relevant to local observations.
4.2 THFORETICAL ANALYSIS
Let å denote the global state generated from GlobeDiff. When the observation function 24 is injective. the mapping between s and x is one-to-one. Assuming that the denoising network and prior network Pa are well trained, we prove that the estimation error of GlobeDiff can be bounded:
Theorem 1 (Single-Sample Expectation Error Bound with Latent Variable). Assume the trained model satisfies the following two assumptions.
(I) Diffussin noise prediction MSE: E. a R.llelll (,, k) <ll?] < 8², (2) Prior alignment: Dxu(po(z | x))lp(z x)) S EXL- x, 2)po(z x)dz and true sample
Then, for any generated sample 8 (o x) I po(s p
s ~u p(s) v), the expected squared error is bounded by:
E [||s =|| < 2W2 (p Ix) x)) 4Var(s x),
(12)
5

---

Published as a conference paper at ICLR 2026
where W2 is the 2- 2-Wasserstein distance between (o | x) and p(s x). Var(s x) Ep(sla) [[ls Mata ||2] is the conditional variance and Mala Bp(s[u)[s] is the conditional mean.
Proof. Please refer to Appendix B.1 for the detailed proof.
However, in practical scenarios, the mapping from is typically one-to-many. We model this situation using multi-modal Gaussian distribution, presented in Theorem [ and prove that the error between the estimated state and the centers of the multi-modal distribution also admits bounded emror.
Theorem 2 (Multi-Modal Error Bound with Latent Variable). Under the following conditions: (I) The true conditional distribution p(s x)) - Ws Ex(x) has N modes with minimum
inter-mode distance D m (z) - (1 2 2/d. (2) Mode separation condition: > C,K2 + C25XL + max, (r) (3) The model satisfies Blllco <ll2] I 8 and DKL (po( 2) [p(( x)) S EKL. Then,for any generated sample ( (s x). there exists a mode 2 (x) such that:
E (05[()11111 < C1K82 + CzEX2 + mmax (x)) + ((e (e-D2/(BoRo)
(13)
where o = max; (r)), and C1, C2 are constants depending on the diffusion scheduler and latent space geometry.
Proof. Please refer Appendix B.3 for the detailed proof.
In essence, Theorem offfer a universal error bound, while Theorem 2 provides a stronger, more specialized guarantee for the multi-modaa settings our method is designed for. These results provides strong theoretical support for our approach. Consistent with our theoretical findings, the empirical results depicted in (igureeaqqannitaaiive verify the high fidelity of our method state reconstruction.
4.3 PRACTICAL IMPLEMENTATION
Architecture We adopt model consisting of repeated convolutional residual blocks to implement the global state diffusion process. The overall architecture resembles the types of U-Nets (Ronneberger al. 20155), but with two-dimensional spatial convolutions replaced by one-dimensional temporal convolute lution.. Because the model is fully convolutional, the horizon of the inference is determined not by the model architecture but by the input dimensionality. This model can change dynamically during inference desired.
Training Mechanism In the practical implementation, first train initial global state diffusion model based on an offline dataset. Subsequently, during online execution, we continuously update the global state diffusion model with the collected data to compensate for the distribution mismatch between offline and online settings. This approach ensures that the global state diffusion model plays a role from the early stages of algorithm training. reducing the instability of MARL algorithms caused by the generative model. In addition, during integration with the CTDE mechanism, we employ the true global state in the policy training phase to reduce computational cost. During the decentralized execution phase, each agent makes decisions based on the inferred global state. The overall process shown Algorithm
EXPERIMENTS
We designed our experiments to answer the following questions: Ql: Can our method accurately infer the global state from the local observations? Q2: Can the global states generated by our method
improve the performance of the MARL algorithm? Q3: Does our method outperform other generative models?

---

Published as a conference paper at ICLR 2026
MMHM2
vS
3552 3562
2he tee-tiller
Berhtterd
0.5
Win Rate
Gaveramer PoRFO
0.4
Drsrictea VINRAFAND
0.3
Conerionna
0.4
0.2
0.2
Win
0.3
10
Stops
Stops
Figure 3: Comparison results with global state inference baselines in SMAC-v1 (PO) tasks with win rate over three random seeds.
protoss Swd
terran
zerg
0.-4
Slowec Sownd
0.25
Ci Ioums
Cloectivt toero
Dyrant:
Dysent Beter Comnarte
Cememic bold Canereree
0.15
0.2
0.05
10
2
10
Steps
Steps
Million
zerg 10v10
terran 10v11
20ng 10v11
0.25
Go0a2iti bust
0.20
Win Rate
0.06
Win Patt
0.10
Commiooree
0.10
0.04
0.02
0.00
0.00
0.00
10
10
Million Steps
Million Steps
Million Steps
Figure 4: Comparison results with global state inference baselines in SMAC-v2 (PO) tasks with win rate over three random seeds.
5.1 SETUP
To answer these questions, we evaluate our method and baselines on SMAC, which is a cooperative MARL environment based on the real-time strategy game StarCraft II (Rashid et al. (020). It includes various unique scenarios where agents can obtain local observations within a certain visual radius. The objective across all scenarios is to command allied units to eliminate enemy units. In addition, SMAC-v2 (Ellis et al.) further reduces the correlation between the local observation and the global state by adding random team compositions and random start positions.
Benchmark for Partial Observability Nevertheless, we are surprised to find that the original SMAC environment is not well-suited for studying partial observability problems. Specifically, adjust the sight range of each agent from 9 to 3 in SMAC-v2 and run the standard MARL algorithm MAPPO. As shown in the Figure 8 the MAPPO's perfor- mance exhibits only a marginal decline (merely 0.03 drop) the observation range narrows. This is because the lo- cal observations retain sufficient environmental information. Therefore, modify the SMAC environment by removing enemy unit types and hit points from local observations to better evaluate partial observability issues, with this adapted environment named as SMAC-v1 (PO) and SMAC-v2 (PO).
0.8
B sight roe sight ranter
6.6
0.4
0.2
Sv5
10v10
Figure Win rate with various sight range in original SMAC tasks.
To ensure fair comparison, all experiments are conducted under identical environmental settings, with three random seeds employed per experiment.
Baselines We compare approach with three representative baselines. Learned Belief Search (LBS) (Hu et al.) 202T) learns an auto-regress nugo-regressive counterfactual belief model to approximate hidden information given the trajectory of agent and uses public-private policy architecture with RNNs to
7

---

Published as a conference paper at ICLR 2026
Agent
Agent
Agent
GlobeDiff
100C
Agent
Agent
Agent
3000
VAE
1000
Agent
3000
Episode
MLP
1000
Figure 5: Visualization of global states generated by GlobeDiff, VAE and MLP. The first plot displays true states and subsequent plots show inferred states per agent. White points denote individual states with polygons highlighting local neighborhoods, Gradient shading (light green purple) indicates training progression. The similarity between the polygon structures of the inferred and true states reflects the predicted quality.
MMM2
1562
0.5
BArro OoT PAPPO
PORD
Winn Rade
Rate
0.3
0.2
Win
0.2
0.1
0.1
0.0
0.0
0.0
2
Million Stops
Million Steps
Million
Stops
Figure 6: Comparison results with generative model baselines in SMAC-v1 (PO) tasks with win rate over three random seeds.
encode cction-observationn histories. Dynamic belief (Zhai et al. 2023) predicts the evolving policies of other agents from recent actionnobservation histories using a variational inference framework. CommFormer (Hu et al. 2024) learns dynamic communication graph via continuous relaxation and attention-based message passing, jointly optimizing the graph and policy parameters. the practical implementation, we combine our method and baselines with the standard MARL algorithm MAPPO. Please refer to Appendix D for the detailed implementation setting.
5.2 MAIN RESULTS
Answer for Question 1: To intuitively demonstrate the GlobeDiff's ability to infer the global state, we visualize both the true global states and the states reconstructed by GlobeDiff' in the SMAC-V2
(PO) Zerg 5v5 scenario. For trajectories sampled during online training, we apply t-SNE (Maaten & Hinton 2008) to project their high nigh-dimensional states into a two-dimensional space. In Figure 5 the first plot shows the ground-truth states, while the subsequent plots depict the global states inferred by each agent. Each white point represents a specific state. To facilitate visual comparison of
the underlying structure, we overlay Voronoi polygons (Balzer et al. 2005). Each polygon defines the region of space closest to single state point, effectively highlighting the local neighborhood. This allows for an intuitive assessment of accuracy: the more the polygon shapes an inferred plot resemble those in the ground-trutt plot, the better the state representation. The background is shaded with gradient from light green to deep purple, where darker regions indicate episodes that occurred later in the online training process.
8

---

Published as a conference paper at ICLR 2026
protoss 5v5
terran 5v5
zerg 5v5
0.4
Gaoeddatt 2wen
0.25
po IHG)
0.20
Win Pato
Earge
Wi Rate
Win Rate
0.15
0.2-
0.10
Disadt tuel
0.1
0.05
0.00
varil PAPP
0.0
vanill MATOO
10
Steps
Steps
Millinn Stops
zera
10v10
terran 10v11
20rg 10v13
0.25
KANO (der
0.29
Win Rato
0.06
MAND Orer
Win Hate:
0.10
0.10
0.04
0.05
0.02
0.00
0.00
0.00
10
10
10
Million Stops
Millinn Stops
Million Steps
Figure 7: Comparison results with generative model baselines in SMAC-v2 (PO) tasks with win rate over three random seeds.
The similarity between the Voronoi polygon structures of the inferred and true states reflects the reconstruction quality. As shown in the Figure the inferred states closely match the real states, indicating that GlobeDiff effectively enables agents to infer the global state from local information. Moreover, as training progresses (i.e., as the background color deepens), the inferred polygons become increasingly similar to the true ones, demonstrating steady improvement in reconstruction performance.
We conduct experiments on SMAC-v1 (PO) and SMAC-v2 (PO) respec- tively. Specifically, for SMAC-v1 (PO), we derive auxiliary information based on the individual agent's historical trajectory, as detailed in Equation For SMAC-v2 (PO), we leverage the communi- cation between agents to construct auxiliary information, as outlined in Equation [ The experimental results in Figure 3 and Figure 4 show that GlobeDiff consistently and significantly outperforms baseline algorithms in most maps. This performance gap can be attributed to the constrained capacity of baseline algorithms in modeling complex multi-modal distributions. For example, LBS tends to gradually accumulate errors when inferring belief states in long-horizon tasks. The inference process of Dynamic Belief remains unimodal, restricting its ability to capture the multi-modal global state distributions. CommFormer requires explicit communication and accurate message aggregation, which can be unreliable under severe partial observability. In contrast, GlobeDiff formulates the state inference process as multi-model denoising procedure, implicitly modeling complex distri- butions within the noise network. This offers a highly expressive model that enhances algorithmic performance through accurate global state inference.
Answer for Question 2:
Answer for Question 3: Same with the scenario de- scribed in Answer for Question 2, we conduct experiments on SMAC-v1 (PO) and SMAC-v2 (PO) respectively. We employ VAE (Kingma& & Welling 2014) and MLP as com-
0.4
GloeOet R'p GieewDll w/
a 0.3
parative baselines for generative models. Specifically, we replace the GlobeDiff with the conditional VAE or an MLP, while keeping everything else unchanged, and name them respectively as MAPPO (VAE) and MAPPO (MLP). Fur- thermore, in the SMAC-v2 (PO) scenario, where agents can obtain information from adjacent agents, we incorporate the agents' joint observations as policy input. This approach named as MAPPO (Joint) and serves an additional base- line to evaluate the role of generative models in state inference. Please refer to Appendix D for detailed description.
zerg Su5
proboss St5
terran $15
zeg 10v10
zeg 10v11
teman 10:11
Scene
Figure 9: Ablation of prior network.

---

Published as a conference paper at ICLR 2026
Diffusion Step
Number of Residual Blacks
Win Rate
Win Rate
0.2
0.2-
0.0
0.0
3
Figure 10: Ablation study for diffusion step and residual blocks on the zerg 5v5 task.
The experimental results in Figure and Figure F that GlobeDiff outperforms all baselines on the super-hard maps. The MLP and VAE show significant performance improvement over vanilla MAPPO in most maps, which is attributed to their limited representation capacity. Moreover, in SMAC-v2 (PO), MAPPO (Joint) performs worse than vanilla MAPPO in some maps. This shows the necessity of the global state inference model, which can extract essential features from such high-dimensional inputs.
Ablation Study for prior network We conduct the ablation study for the prior network Pe by removing the KL constraint in Equation and prior network Ps in training process, which is named GlobeDiff w/o p. We conduct experiments on the various maps. The experimental results in the Figure (egureeinndican indicate that the performance of GlobeDiff can be effectively enhanced by introducing the prior network.
Ablation Study for Hyper-parameters To study the robustness of GlobeDiff across different hyper-parameters, we conduct the following ablation studies. Specifically, we change the diffusion step K from 1 to 8. The experimental results in the left part of Tigure[[0]shov show that the state inference more accurate with the longer denoising steps. addition, we conduct ablation studies for model parameters of the U-Net with various residual blocks. The experimental results in the right right of Figure
that the model's 's capacity has relatively minor impact on the algorithm's performance. We only need small model to achieve accurate global state inference.
CONCLUSION
In this paper, we study the partial observability problem in multi-agent systems. We first propose generative model-based global state inference framework under two scenarios. Then, we propose the Global State Diffusion Algorithm (GloheDifff) which formulates the state inference process as multi-modal diffusion process. The theoretical analysis shows that the estimation error of GlobeDiff can be bounded. Extensive experiments demonstrate that GlobeDiff can not only accurately infer the global state, significantly enhance the algorithm's 's performance, and easily integrated with current MARL algorithms. In the future, we will apply our algorithm to real-world tasks to address the challenges of partial observability in real environments.
REPRODUCIBILITY STATEMENT
We have provided the source code in the supplementary materials, which will be made public after the paper is accepted. We have provided theoretical analysis in the Appendix B We have also provided
implementation details in the Appendix D
ACKNOWLEDGMENTS
This work was supported in part by the National Natural Science Foundation of China under Grant 62192751, in part by the 111 International Collaboration Program of China under Grant B25027, in part by the in part by the InnoHK Initiative, The Government of HKSAR, and in part by the Laboratory for Al-Powered Financial Technologies. This work was also supported by the National Key R&D Program of China (No.2022/D01166A4))
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Anurag Ajay, Yilun Du, Abhi Gupta, Joshua Tenenbaum, Tommi S Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision making? In The Eleventh International Conference on Learning Representations.
Christopher Amato, Girish Chowdhary, Alborz Geramifard, Kemal Ure, and Mykel Kochenderfer. Decentralized control of partially observable markov decision processes. In 52nd IEEE Conference on Decision and Control, pp. 2398-2408 2405. IEEF, 2013.
Michael Balzer, Oliver Deussen, and Claus Lewerentz. Voronoi treemaps for the visualization of software metrics. In Proceedings of the 2005 ACM symposium on Software visualization, pp. 165-172, 2005.
Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, pp. 02783649241273668, 2023.
Abhishek Das, Théophile Gervet, Joshua Romoff, Dhruv Batra, Devi Parikh, Mike Rabbat, and Joelle Pineau. Tarmac: Targeted multi-agent communication. In International Conference on machine learning, pp. 1538-1544 PMLR, 2019.
Benjamin Ellis, Skander Moalla, Mikayel Samvelyan, Mingfei Sun, Anuj Mahajan, Jakob N Fo- erster, and Shimon Whiteson. Smacv2: An improved benchmark for cooperative multi-agent reinforcement learning, 2022. URI https://arxiv. org/abs/2212.07489
Pu Feng, Junkang Liang Size Wang, Xin Yu, Xin Ji, Yiting Chen, Kui Zhang, Rongye Shi, and Wenjun Wu. Hierarchical consensus-basee multi-agent reinforcement learning for multi-robot cooperation tasks. In 2024 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 642-649. IEEE, 2024.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances neural information processing systems, 27, 2014.
Matthew Hausknecht and Peter Stone. Deep recurrent q-learning for partially observable mdps. In 2015 aaai fall symposium series, 2015.
Haoran He, Chenjia Bai, Kang Xu, Zhuoran Yang, Weinan Zhang, Dong Wang, Bin Zhao, and Xue- long Li. Diffusion model is an effective planner and data synthesizer for multi-task reinforcement learning. Advances in neural information processing systems, 36:64896 66644896-69917, 2023.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Hengyuan Hu, Adam Lerer, Noam Brown, and Jakob Foerster. Learned belief search: Efficiently improving policies in partially observable settings. arXiv preprint arXiv:2106.0908 2021.
Shengchao Hu, Li Shen, Ya Zhang, and Dacheng Tao. Learning multi-agent communication from graph modeling perspective. arXiv preprint 240.:22005008500 2024.
Michael Janner, Yilun Du, Joshua Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In International Conference on Machine Learning, pp. 9902-9915. PMLR, 2022.
Jiechuan Jiang and Zongqing Lu. Learning attentional communication for multi-agent cooperation. Advances in neural information processing systems, 31, 2018
Steven Kapturowski, Georg Ostrovski, John Quan, Remi Munos. and Will Dabney. Recurrent experience replay in distributed reinforcement learning. In International conference on learning representations, 2018.
Daewoo Kim, Sangwoo Moon, David Hostallero, Wan Ju Kang, Taeyoung Lee, Kyunghwan Son, and Yung Yi. Learning to schedule communication in multi-agent reinforcement learning, arXiv preprint 102. 2019.
11

---

Published as a conference paper at ICLR 2026
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv.org, 2014.
Hyeoksoo Lee, Jiwoo Hong, and Jongpil Jeong. Marl-based dual reward model on segmented actions for multiple mobile robots in automated warehouse environment. Applied Sciences, 12(9))4703, 2022.
Zhixuan Liang, Yao Mu, Mingyu Ding, Fei Ni, Masayoshi Tomizuka, and Ping Luo. Adaptdiffuser: Diffusion models as adaptive self-evolving planners. arXiv preprint arXiv:2302.01877, 2023.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, ((Nov) 2008.
Liam C MacDermed and Charles Isbell. Point based value iteration with optimal belief compression for dec-pomdps. Advances in neural information processing systems, 26, 2013.
Diganta Mísra. Mish: A self regularized non-monotonic activation function. arXiv preprint 0rXiv:1908.08681 2019.
Darius Muglich, Luisa M Zintgraf, Christian A Schroeder De Witt, Shimon Whiteson, and Jakob Foerster. Generalized beliefs for cooperative ai. In International Conference on Machine Learning, pp. 16062-16082. PMLR, 2022.
Frans A Oliehoek, Christopher Amato, et al. concise introduction to decentralized POMDPs, volume 1. Springer, 2016.
Shayegan Omidshafiei, Jason Pazis, Christopher Amato, Jonathan P How, and John Vian. Deep decen- tralized multi-task multi-agent reinforcement learning under partial observability. In International Conference on Machine Learning, pp. 2681-2690. PMLR, 2017.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Monotonic value function factorisation for deep multi-agent reinforcement learning. Journal of Machine Learning Research, 21 (178):1-51,200)
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer- IiterventionMMICOO 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pp. 234-241, Springer, 2015.
Amanpreet Singh, Tushar Jain, and Sainbayar Sukhbaatar. Learning when to communicate at scale in multiagent cooperative and competitive tasks. arXiv preprint arXiv: 18/2.0975, 2018
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. International conference on machine learning. pp. 2256-2265. PMLR, 2015.
Yang Song. Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011 1, 134466 2020.
Matthijs TJ Spaan. Partially observable markov decision processes. In Reinforcement learning: State-of-the-arr, pp. 387-414. Springer, 2012.
Sriram Srinivasan, Mare Lanctot, Vinicius Zambaldi, Julien Pérolat, Karl Tuyls, Rémi Munos, and Michael Bowling. Actor-critic policy optimization in partially observable multiagent environments. Advances in neural information processing systems, 31, 2018
Pradeep Varakantham, Ranjit Nair, Milind Tambe, and Makoto Yokoo. Winning back the cup for distributed pomdps: planning continuous belief spaces. In Proceedings of the fifth international joint conference on Autonomous agents and multiagent systems, pp. 289-296, 2006.
Tonghan Wang. Heng Dong. Yanchen Jiang, David C Parkes, and Milind Tambe. On diffusion models for multi-agent partial observability: Shared attractors, error bounds, and composite flow. arXiv preprint ar arXiv:2410. 13953, 2024a.
12

---

Published as a conference paper at ICLR 2026
Weizheng Wang, Le Mao, Ruiqi Wang, and Byung- Chee-l Min. Multi-robot cooperative socially- aware navigation using multi-agent aggent reinforcement learning. In 2024 IEEE International Conference Robotics and Automation (ICRA), pp. 12353-12360. IEEE, 2024b.
Yutong Wang, Mehul Damani, Pamela Wang, Yuhong Cao, and Guillaume Sartoretti. Distributed reinforcement learning for robot teams: A review. Current Robotics Reports, 3(4):239-257, 2022.
Muning Wen, Jakub Kuba, Runji Lin, Weinan Zhang, Ying Wen, Jun Wang, and Yaodong Yang. Multi- agent reinforcement learning is a sequence modeling problem. Advances in Neural Information Processing Systems, 15:99-61116,21,, 2022.
Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European conference computer vision (ECCV), pp- 3-19, 2018.
Yunpeng Zhai. Peixi Peng, Chen Su, and Yonghong Tian. Dynamic belief for decentralized multi-
agent cooperative learning. In IJCAI, pp. 344-352, 2023.
Ruiqi Zhang, Jing Hou, Florian Walter, Shangding Gu, Jiayi Guan, Florian Röhrbein, Yali Du, Panpan Cai, Guang Chen, and Alois Knoll. Multi-agent reinforcement learning for autonomous driving: A survey. arXiv preprint arXiv:2408.09677 2024.
Sai Qian Zhang, Qi Zhang, and Jieyu Lin. Efficient communication in multi-agent reinforcement learning via variance based control. Advances in neural information processing systems, 32, 2019.
Wei Zhou, Dong Chen, Jun Yan, Zhaojian Li, Huilin Yin, and Wanchen Ge. Multi-agent reinforcement learning for cooperative lane changing of connected and autonomous vehicles in mixed traffic. Autonomous Intelligent Systems, 2(1):5, 2022.
13

---

Published as a conference paper at ICLR 2026
A ALGORITHM
Algorithm 1 Global State Diffusion Process
1: Initialize the parameters of MARL algorithm and the diffusion model
2: Add offline data to online buffer
3: for each episode do
4: for + 1 to 7"' do
5: Obtain local observation O2
6: Construct auxiliary state 2;
Calculate the encoded latent variables based on Po [2
(x))
8: Infer the global state of based on the Eq. 11
9:
Each agent makes decision based on the inferred state sf Send a to environment and receive 0141, Store transitions in replay buffer DD end for
11:
13: Update MARL algorithm
if Update Global State Diffusion Model then
15:
Update global state diffusion model based on the Eq. 9
16: end if
17:
end for
18: end
14

---

Published as a conference paper at ICLR 2026
B THEORETICAL ANALYSIS
B.1 ERROR BOUND ANALYSIS FOR SINGLE SAMPLES WITH LATENT VARIABLE
Theorem. [ (Single-Sample Expectation Error Bound with Latent Variable) Assume the
trained model satisfies the following two assumptions.
(1) Diffusion noise prediction MSE:
2 ( 2, 2, -cl²] I 8², (2) Prior aligument: Dxu(Pole I +))lp(zz x)) S EKL.
Then, for any generated sample š Po,o(s z) po(s
po(s x, 2)2)) z)dz and true sample
~ p(8] x),, the expected squared error is bounded by:
E [| - 2W (p x), x)) 4Var(s x),
(14)
where W2 is the 2-Wasserstein distance between (e.o(s | x) and p(s x). Var(s I x) Ep(sle) [lls Mala [1=] is the conditional variance and Mala Ep(sla)ss] is the conditional mean.
Proof. Step 1: Error Decomposition via Variance-Bian Tradeoff Let Pals E [s]. For any á and s, we expand:
|| 8 2 = (1 m2//z) - - (Ms121
(15)
By the triangle inequality and Young's 's inequality:
|| 0|| < 2| Ma|xr| 112 + 2|18 Mate 11.
(16)
Taking expectation:
[[|S- =|
S
2E [|| Ma|
+ 2E [||s ar 1=1
(17)
Step 2: Bounding the First Term via Wasserstein Distance
The first term represents the deviation of the generated sample from the true conditional mean. By properties of the Wasserstein distance:
(elz))
[||s - [=]=]]?] <
W2 (pa. x)) 8p < W2 (po (s x). +))+ x), ongz):
(18)
where oyop is the Dirac delta at Mala- The second term equals Var(s x), giving:
[|
|1 (po,0o( x). x)) Var(s ar).
(19)
Step 3: Bounding the Second Term
The second term is exactly the conditional variance:
E [||s |=| Var(s xr).
(20)
Step 4: Final Synthesis Combining all results:
E ||| =||2] < 2 (W3 o xx) x)) + Var(s xx) + 2ar( (x)
(21)
2w3 (p 1x),p(s x)) + 4Var(s x).
(22)
which completes the proof.
B.2 CONNECTING TRAINING LOSS TO WASSERSTEIN BOUND WITH LATENT VARIABLE
Lemma 1. Let the following hold. (1) Noise prediction MSE: E, a...olllco(s' x, 2.2,k)
cll31 I 82,
(2) KL divergence: DKI. (ps(= x)][p(z x)) < exI. Then the Wasserstein-2 distance between (a (8 x) and p(s x) is bounded by:
(2 p). | x)) s C,K8 + C2EKL.
(23)
(a))- C2 is a constant depending on the latent space dimension and geometry, and K is the total number of diffusion steps.
where C1 = maxk
1-a" (a*(I-
15

---

Published as a conference paper at ICLR 2026
Proof. Step 1: Single-Step Error Propagation for Diffusion The reverse process update at step l conditioned on is:
- a
Va - @*) co(s 2,2, + VBLe.
(24)
The deviation caused by noise prediction error Des = ED satisfies:
a
A- (a at)
Dex + A&t.
(25)
Va
Step 2: Error Accumulation Over K Steps
Unrolling the eπror through all K steps:
1 a a Vat(1 ax)
Ack.
(26)
Va
k=
li=k+1
Taking the expectation of the squared norm:
E1 [|]] E
Σ A D
(27)
where A₁ = (I1 vor)
l-at
Vat
Expanding the square:
E A
12]
Σ
|
11 +2 Σ
(28)
IERCIEK
Assuming the noise prediction errors at different steps are uncorrelated, the cross terms vanish:
K || < 82
E|| 1
|
(29)
R=
Now, we need to bound Σ k=
11. Let:
K II
a
2
||Ax|1
(30)
a
Va
aa)
2
Let C₁ max; (I1 =
1-a" (
Then:
K
A < K,K,
(31)
Thus:
E 1 SKK
(32)
Incorporating KL Divergence Error The KI. divergence bound EKL ensures that the prior pa(2 x:) is close to the true posterior p(= x)).
Step 3:
By the data processing inequality for Wasserstein distance:
w? ( po(s x. =))p((> w))z,, / p(s x, 2)p(= wr)dz S &CWW3 x),p(z 1))),
x)), (33)
16

---

Published as a conference paper at ICLR 2026
where C2 depends the Lipschitz constant of the mapping p(s 2, The constant C2 depends on the latent space dimension for the following reasons. (1) Dimension Scaling of Wasserstein Distance: For distributions in R the Wasserstein distance typically scales with vd, due to concentration of measure phenomena. This is known as the "curse of dimensionality" in optimal transport.
(2) Talagrand's Inequality: If p(= x) is log-concave (e.g., Gaussian), then Talagrand's incquality gives:
W2(po(z 12) 2)) s 2CnDxL <x)l[(< x)).
(34)
where CT the Poincaré constant of p(= 1r). isotropic Gaussians in IR this constant scales as O(d2). (3) Lipschitz Constant: The mapping - p(s x, 2) typically has Lipschitz constant that grows with the dimension dg due the increased complexity of the conditional distribution. Thus, we can write:
(pp(2) x), x)) S C2d2SKL.
(35)
where C/2 a cemeension-ineeeendeee constant.
Step 4: Combined Bound
Combining both error sources using the triangle inequality for Wasserstein distance:
(p x), x)) s 2W2 (pa.o(> v)) p(s x, =)p(= | x)) 2W (p(s x, x). p(s x))
(36)
2C, K8 + 20/d..KL
(37)
Absorbing constants into C and C₁ Denote C2 as C2, and noting that C depends on d2, obtain the final bound:
W2 x), x)) s C₁KO² + +ZEKL
(38)
B.3 ONE-TO-MANY MAPPING CASE WITH LATENT VARIABLE
Theorem. ( Modall Error Bound with Latent Variable) Under the following conditions: (1) The true conditional distribution p(s x²) Ee WyN(s; (x)))) Ex(x)) has N modes with minimum
inter-mode distance D min;zj [1p((z)) - (=)] 2 2/d. (2) Mode separation condition: D
C/KK + C2 + max, Tr(E. Tr(E,(z)) (3) The model satisfies Billco ell S 8 and DRL (polz x)) [p(= x)) S EKL. Then, for any generated sample ( x). there exists a mode (1((2) such that:
E - - S C,K82 + C2EKI + 2max T((x())) + + (e-D(118802)))
(39)
where maa max; (x)), and C1, C2 are constants depending on the diffusion scheduler and latent space geometry.
Proof. Step 1: Voronoi Partitioning and Projection Operator
The state space Rd is partitioned into N Voronoi regions [V:j centered the mode centers (u((r)) Define the projection operator:
((s) Σ m((x) [(eV)))
(40)
which maps any point s to the center of its containing Voronoi region.
Step 2: Conditional Distribution Definitions
For each Voronoi region V,, define the conditional distributions:
True state conditional distribution: Pe(s) = p(s ar, E Vi))
Generated state conditional distribution: q:(6) p0 8 E V.)
17

---

Published as a conference paper at ICLR 2026
Under the mode separation condition, the true state conditional distribution pı(s) approximates unimodel Gaussian distribution N Ez) with exponentially small error:
(()(()) Ex))) S So (e D2 -0/(())))))
(41)
Step 3:
Per-Region Projection Error Bound For each Voronoi region Vi, consider the conditional expectation of the projection emror:
E|ll ((3)1 & E vil = Eill (+(x)) & Vi].
(42)
Using the triangle inequality for Wasserstein distance:
B||ls (=) E Vij = W (ge: (x))
(43)
I (W2 (g: pá) + W2(p;, 6 )
(44)
I 2W2 (g: p.) + 2W2(p: (x)()));
(45)
where (ps(x)) is the Dirac delta distribution at (x)).. For the first term, according to Llamal, we have:
(2(g:.p) W2(po p(, w)) +o (e (6-D//(ooun) CiK&2 + C28KI. +0 (e (e-D)/(80mm)
P(s Vi)
P(sEV)
(46)
And for the second term, we have:
Wh(p:: ()(()) = Ep. |||s- (+(ls-) +x) (c-D9/(Bou)))) S S +0 20(( (c-D2/(8omms)
(47)
Step 5: Aggregation Over All Regions
Compute the global expectation:
N
E|||$ 8
(())|
Σ
P(s EV)
E[lis [[8)1= | | Vij
(48)
N P(s E Vi)
C1K82 + +C2ELL + 0 ( (-D2/(8oun))) )) + (o +0 D2
2
P(s E Vi)
i=1
(49)
Σ
2(C + CZExL) 2P Vi)o + 0 (e-D)/
(50)
2N + C2SKI. max (r)E + +0 (E-D2((Bouu))
(51)
Redefining constants Cj = 2NC1 and C²22 = 2NC₂, and absorbing factors:
Ell| (8) < K8 + C2 + max Tr(E,())) + O (e-D//1olmn))
(52)
When D >> Tmax, the exponential term becomes negligible.
18

---

Published as a conference paper at ICLR 2026
C ADDITIONAL EXPERIMENTS
C.I QUANTIFYING THE DIVERSITY
We conducted additional experiments to illustrate the semantic representation learned by 2. Specifi- cally, in the SMAC-v2 Zerg 5v5 environment, we collected observational data across all timesteps and identified both the true global states and the global states generated by GlobeDiff. These state samples were then projected into a two-dimensional plane using t-SNE for visualization, and a 3D density surface was estimated using Kernel Density Estimation (KDE). The experimental results in Figure 11 show that the resulting 3D plots illustrate the normalized probability density (2-axis) over the 2D t-SNE embedding (x- and y-axis).
In the leftmost subplot, we observe that the same observational input often corresponds multiple real global state clusters at distinct locations in the 2D plane, forming several prominent Gaussian modes. The latent variable these Gaussian modes in the embedded space. The three subplots on the right display the global states generated by three selected agents, which faithfully reconstruct the multi-modal characteristics of the true state distribution. Notably, the location, shape, and amplitude of the reconstructed Gaussian modes closely resemble those of the real state distribution, empirically demonstrating z's role in capturing semantic variations.
thus be interpreted as positional encoding for the coordinates of
Roal State
Agent 1 (selected)
Agent 3 ise cellected)
Agant (selected)
Tax Neraateee dessit
Figure 11: Experiments of the qualitative analysis.
C.2 EXPERIMENTS UNDER SAME PARAMETER COUNT
We conducted new experiment where increased the parameter count of the vanilla MAPPO networks to match the total number of parameters GlobeDiff (which includes both the diffusion model and the RL policy). We calculated the total parameters in GlobeDiff to be approximately 12-14M. Accordingly, we expanded both the MAPPÓ actor and critic networks to four-layer MI.Ps with 2048 hidden units per layer, creating a "Vanilla MAPPO (Large)" baseline with comparable parameter budget (approx. 13.5-14M parameters).
The experimental results in Table [ show that despite this significant increase in capacity, the performance gain of Vanilla MAPPO (Large) remains limited. These results suggest that merely increasing the capacity of standard recurrent policy is insufficient for implicitly learn the complex task of multi-modal global state reconstruction, highlighting the necessity of an explicit generative module like GifbeDiff.
19

---

Published as a conference paper at ICLR 2026
Vanilla MAPPO
Vanilla MAPPO (Large)
GlobeDiff
zerg 5v5
0.22++0.01
0.23+-0.00
0.33++0.02
protoss 5v5
0.21
0.24
0.38+10.01
terran 5v5
0.16110.01
0.17+0000
0,2410.01
zerg 10v10
0.13
0.15+00.11
0.25+10.01
zerg 10v11
0.06++0.01
0.07
0.12+0.01
terran 10v11
0.02+0.00
0.03
0.07+00001
MMM2
0.27
0.01+10.01
0.49-10.11
3s5z 3s6z
0.20++0.01
0.01+00.01
0.28+10.04
6h vs 8z
0.1210.01
0.01
0.47+0004
Table 1: Comparison results between Vanilla MAPPO, Vanilla MAPPO (Large), and GlobeDiff
20

---

Published as a conference paper at ICLR 2026
D IMPLEMENTATION DETAILS
In practical implementation, to mitigate the data bias the offline dataset and ensure the effective state inference, we update the GlobeDiff during online learning with update interval of 50 episodes. Under the CTDE framework, we select the historical observation length m 3. Please refer to Table Tables and for the detailed hyper- papae--arameters.
The architecture of the Diffusion model consists of a U-Net structure with two repeated residual blocks, shown Figure [2] Each block consisted of temporal convolutions, followed by group norm (Wu & He. 2018), and final Mish nonlinearity (Misra 2019). The U-Maze dim refers to the multiplicative factor that reduces the output dimension relative the input dimension during the down-sampling process.
RC
ON Miai
Figure 12: U-Net architecture Diffusion model.
Table 2: prrper-parameterss for MAPPO.
Hyper-parameters
Value
Hyper-parameters
Value
Hyper-parameters
Value
Critic Learning Rate
5e-4
Actor Learning Rate
5e-4
Use GAE
True
Gain
0.01
Optim
le-5
Batch Size
Training Threads
Num Mini-Batch
1
Rollout Threads
Entropy Coef
0.01
Max Grad Norm
10
Episode Length
400
Optimizer
Adam
Hidden Layer Dim
64
GAE A
0.95
Activation Function
Relu
PPO Epoch
15
7
0.99
Table 3: Hyper-parameters for prior network pe and posterior network Jø-
Prior Network pa
Value
Posterior Network de
Value
z dim
16
dim
16
Hidden Layer Dim
1024
Hidden Layer Dim
1024
Hidden Layer Num
Hidden Layer Num
3
Activation Function
Relu
Activation Function
Relu
Learning Rate
2e-4
Learning Rate
2e-4
Weight Decay
le-4
Weight Decay
le-4
Batch Size
32
Batch Size
32
21

---

Published as a conference paper at ICLR 2026
Table 4: Slyper-paramett for Generative methods.
GlobeDiff
Value
VAE
Value
MLP
Value
Down-sampling Factor
Latent Dim
256
Hidden Layer Dim
1024
Diffusion Steps
Hidden Layer Dim
1024
Hidden Layer Num
4
Residual Blocks
Activation Function
Relu
Activation Function
Relu
Learning Rate
2e-4
Learning Rate
le-4
Learning Rate
3e-4
Batch Size
32
Batch Size
32
Batch Size
32
22


Reviews:
[{"review_id": "ydENmKabMF", "reviewer": "Reviewer_m4w8", "summary": "The paper presents a novel method “GlobeDiff” for tackling partial observability in multi-agent systems by framing global state inference as a conditional, multi-modal denoising process. The proposed GlobeDiff algorithm leverages diffusion models to generate a belief over the global state from local observations, enhanced by a latent variable to handle ambiguity. The paper is supported by theoretical error bounds and extensive empirical evaluation on modified StarCraft Multi-Agent Challenge (SMAC) environments.", "questions": "1. how to apply the methods to a latency-aware environment?\n2. The authors compare against generative model baselines like VAE and MLP. However, given that you are using a history of observations (Eq. 1) in one scenario, how would a powerful sequence model like a Transformer [1,2], trained to directly predict the global state from the historical trajectory, perform as a baseline? \n\n[1] Sequential Asynchronous Action Coordination in Multi-Agent Systems: A Stackelberg Decision Transformer Approach   \n[2] PTDE: Personalized training with distilled execution for multi-agent reinforcement learning", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Novel and Innovative Technical Approach: The core idea of applying diffusion models to the problem of global state inference in Dec-POMDPs is highly innovative. Instead of traditional belief estimation or communication, the authors formulate the problem as a conditional generative task. The introduction of a latent variable z to model the one-to-many mapping from local observations x to potential global states s is a sophisticated and technically sound solution to a fundamental challenge in this domain.\n\n2. Strong Theoretical Foundation: The paper provides rigorous theoretical analysis, which is often lacking in deep learning-based MARL research. Theorems 1 and 2 establish formal bounds on the estimation error for both unimodal and multi-modal distributions. This theoretical grounding significantly strengthens the claims of the method's efficacy and provides insight into the conditions under which it is expected to perform well.\n\n3. Comprehensive and Convincing Experimental Evaluation: 1) The proactive modification of the SMAC benchmark to create a more challenging and appropriate testbed for partial observability (SMAC-v1/2 PO). 2) Extensive comparisons against a diverse set of strong baselines (LBS, Dynamic Belief, CommFormer) and ablations against other generative models (VAE, MLP).", "weaknesses": "1. Dependence on Offline Data and Potential Distribution Shift: The training mechanism relies on an initial offline dataset to pre-train the GlobeDiff model. In many practical MARL scenarios, such a representative offline dataset may not be available. Furthermore, while the model is updated online, the initial dependence on offline data and the potential for distribution shift between offline and online data phases remain non-trivial challenges that are not fully addressed.\n\n2. High Computational Complexity and Practical Overhead: Training and maintaining three separate components—the diffusion denoising network ϵθ, the prior network p_ϕ, and the posterior network q_ψ—is computationally expensive. The multi-step iterative denoising process during inference also adds latency. This complexity could be a major barrier to real-world deployment, especially in applications requiring low-latency decision-making or with limited computational resources.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "myju0aORFQ", "reviewer": "Reviewer_mKZR", "summary": "This paper proposes GlobeDiff, a method that leverages a conditional diffusion model with a latent variable to infer the global state from local observations in partially observable multi-agent systems. The approach is integrated into the CTDE framework and evaluated on modified SMAC benchmarks. The paper includes theoretical error bounds and extensive experimental results showing superior performance over several baselines.\n\nWhile the core idea of using a generative model for state inference is novel in the MARL/PO context and the results are promising, the paper's presentation significantly undersells its contributions. The writing often obscures the key insights and motivations, making the work appear more incremental than it likely is.", "questions": "The latent variable z is central to the method. Can you provide more intuition or analysis of what z learns to represent? Does it correspond to semantically meaningful aspects of the global state?\n\nThe communication scenario (Eq. 2) is presented but not analyzed. How critical is the quality and scope of the communicated information to the performance of GlobeDiff? Have you tested scenarios with limited or noisy communication?\n\nTheorem 2 requires a \"mode separation condition.\" How often do you expect this condition to hold in practice, and what is the empirical performance of GlobeDiff when the modes of the global state distribution are less well-separated?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Formulating global state inference as a conditional, multi-modal generation problem is a compelling and timely approach. The use of a diffusion model to capture the \"one-to-many\" mapping inherent in PO is a significant conceptual contribution beyond simple belief estimation.\n\nThe method is well-designed. The introduction of the latent variable z to handle multi-modality, coupled with the specific variational training objective (Eq. 10) that aligns the prior and posterior networks, is a non-trivial and thoughtful adaptation of the diffusion model framework for this problem.\n\nThe experimental results are comprehensive. GlobeDemonstrates consistent and significant performance gains across a wide range of challenging SMAC (PO) scenarios against a diverse set of baselines (belief-based, communication-based, and other generative models).\n\nProviding theoretical error bounds (Theorems 1 & 2) for both unimodal and multi-modal cases is a notable strength that adds rigor and credibility to the proposed approach.\n\nThe ablation studies (prior network, diffusion steps) effectively validate key design choices. The state reconstruction visualization using t-SNE and Voronoi polygons (Fig. 5) is excellent and provides clear, intuitive evidence of the model's capability.", "weaknesses": "（1） The introduction successfully outlines the problem but fails to clearly and forcefully state the paper's core intellectual contribution. The description of GlobeDiff is presented as a method, not a solution to a fundamental challenge (the \"one-to-many\" mapping). The key insight—that generative models can sample from the distribution of plausible global states while discriminative methods collapse to a single mode—is buried in the methodology rather than being the central thesis of the paper.\n\n（2） The method section dives too quickly into mathematical derivations without first providing a high-level, intuitive overview of the framework. The reader is left to reverse-engineer the design philosophy from the equations. A paragraph explaining the roles of z as a \"mode selector\" and the loss function as a way to bridge the training-inference gap for z is crucial.\n\n（3） The analysis largely stops at reporting that GlobeDiff performs better. It lacks a deep, diagnostic analysis of why and when it excels.\n\n（4） There is no qualitative analysis of specific episodes or timesteps. For example, showing a situation where a local observation is ambiguous, and illustrating the different global states sampled by GlobeDiff versus the incorrect or averaged state estimated by a baseline (like Dynamic Belief) would be immensely powerful.\n\n（5） The paper's central claim is handling multi-modality, but this is not directly tested or contrasted. Quantifying the diversity of generated states (e.g., using metrics like Average Distance to Nearest Neighbor within a batch of samples for the same x) and comparing it to the diversity of the true underlying states would provide strong, direct evidence.\n\n（6） Attributing VAE/MLP's poorer performance merely to \"limited representation capacity\" is insufficient. A discussion, supported by visualization, showing that the VAE produces blurry \"averaged\" states would directly support the multi-modal argument.\n\n（7） The computational overhead of running a multi-step denoising process during execution is not discussed. This is a critical consideration for real-time applications.\n\n（8） A discussion of the method's limitations (e.g., performance in environments with extremely high-dimensional state spaces, or when the \"one-to-many\" mapping is exceptionally complex) would provide a more balanced view and guide future research.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qjX8hY1X57", "reviewer": "Reviewer_TZGo", "summary": "The paper introduces the idea of using diffusion models to infer full state observations from local (single agent) observations in general partially observable multi-agent reinforcement learning problems. They provide some analysis of how to cast the learning problem in terms of the generative models, they derive theoretically some error bounds in the expected prediction errors between the generative model and the true fully observable state, and include a wide range of empirical results demonstrating how incorporating a diffusion model into a baseline MARL algorithm helps improve significantly the average returns obtained by the agents.", "questions": "1. Can the authors reference [1] above and frame their work in this context? I believe the author's experimental results are more thorough than [1], but I believe it affects the claims that diffusion models have not been used in Dec-POMDPs.\n2. Is theorem 1 a contribution of equal weight as Theorem 2? I am not sure it's a relevant result, I believe more space dedicated to Theorem 2 would probably benefit the paper.\n3. In eq (4) authors derive the ELBO of a generative model, which is a well known expression and a common approach to training variational approximations to posterior densities in general Bayesian generative problems. A reference to this and some nuance would make the paper clearer.\n4. I have a fundamental question about the baselines used to compare. In my view, GlobeDiff+MAPPO is in some way adding a large amount of model complexity to vanilla MAPPO. While there is no good way of measuring this and ensuring it's constant among all baselines, I don't believe it is fair to compare baselines against each-other without making any reference to the size of the policy/value networks. Were the architectures of the policies kept constant across all baselines? I would then argue that it is trivial that GlobeDiff+MAPPO provides a performance increase, since at the very least this is adding model complexity to the representation the agents learn. What I would be interested in knowing is, if for example, one can train GlobeDiff+MAPPO where the number of parameters in the networks (of the diffusion model + the RL policy) is **the same** as the number of parameters in a vanilla MAPPO policy. If even under comparable model complexity the authors observe such improvements, I will be more convinced that the approach adds any value to the MARL field. If under a similar (total) number of parameters the performance is similar, I would argue perhaps deep RL recurrent policies with enough complexity generate some form of internal representation of the global state reconstruction based on agent observations (which is, in fact, the implicit argument behind using recurrent policies in Dec-POMDPs).", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The problem is highly relevant. Partial observability in dec-POMDPs is a prevalent bottleneck in terms of general MARL algorithms succeeding at producing optimal distributed policies in complex MARL environments.\n- The solution the authors propose is reasonable and very straightforward: use powerful generative models to aid agents form posterior densities over true fully observable states based on local observations.\n- The empirical results seem thorough, and the authors compare against a reasonable amount of baselines.", "weaknesses": "- Using diffusion models to approximate posterior distributions of fully observable states in Dec-POMDPs is not really a novel idea. See [1]; authors even test MAPPO implementations with diffusion generative models in the SMAC benchmarks.\n- Theorem 1 does not really add any insight or seems to be particularly useful; it states that the expected approximation error between true samples and generated ones is bounded by a linear combination of some distance between distributions and the variance of the generative model (this is not a very insightful result). The proof also follows almost immediately from the triangle inequality.\n- I am slightly concerned of the validity of the baseline comparisons. I am not convinced it is fair to compare GlobeDiff+MAPPO against a vanilla MAPPO with the same number of trainable parameters. See below for a question on this.\n- Some sentences in the introduction are not very clear. First, I would appreciate some nuance in the assumption that a MARL problem is necessarily partially observable (this is not the case in general, there are game theoretic problems that are fully observable like eg bimatrix games...). Also, the following sentence (lines 49-51) is not clear: \"Unfortunately, the focus on interpreting past experiences without significantly enhancing the agent's ability to leverage global information\". Isn't interpreting past experiences a way of inferring global information?\n- (*Minor but I had to raise it*) In the paper title, as well as in the first line of the introduction, 'multi-agent system' should be plural, 'multi-agent systems', and the first line should be 'Multi-agent systems have...'.\n \n\n[1] Wang, Tonghan, et al. \"On Diffusion Models for Multi-Agent Partial Observability: Shared Attractors, Error Bounds, and Composite Flow.\" Proceedings of the 24th International Conference on Autonomous Agents and Multiagent Systems. 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)