Reconstruction-Guided Policy: Enhancing Decision-Making through Agent-Wise State Consistency
Authors: Liang Qifan
Yixiang Shan
Haipeng Liu
Zhengbang Zhu
Ting Long
Weinan Zhang
Yuan Tian
Venue: ICLR 2025 Poster (2025)
Keywords: multi-agent reinforcement learning
partial observability
cooperation
centralized training distributed execution
global state
Primary area: reinforcement learning

Abstract: An important challenge in multi-agent reinforcement learning is partial observability, where agents cannot access the global state of the environment during execution and can only receive observations within their field of view. To address this issue, previous works typically use the dimensional-wise state, which is obtained by applying MLP or dimensional-based attention on the global state, for decision-making during training and relying on a reconstructed dimensional-wise state during execution. However, dimensional-wise states tend to divert agent attention to specific features, neglecting potential dependencies between agents, making it difficult to make optimal decisions. Moreover, the inconsistency between the states used in training and execution further increases additional errors. To resolve these issues, we propose a method called Reconstruction-Guided Policy (RGP) to reconstruct the agent-wise state, which represents the information of inter-agent relationships, as input for decision-making during both training and execution. This not only preserves the potential dependencies between agents but also ensures consistency between the states used in training and execution. We conducted extensive experiments on both discrete and continuous action environments to evaluate RGP, and the results demonstrates its superior effectiveness. Our code is public in https://anonymous.4open.science/r/RGP-9F79

Published as a conference paper at ICLR 2025
RECONSTRUCTION-GGUDDD POLICY: ENHANCING DECISION-MAKING THROUGH AGENT- WISE STATE CONSISTENCY
Qifan Liang', Yixiang Shan', Haipeng Liu', Zhengbang Zhu", Ting Long' Weinan Zhang", Yuan Tian'
1 Jilin University, Shanghai Jiao Tong University
[liangqf23, shanyx22, 1i 1iuhp24 @mails...........l...aalmllllmsl jlu. edu. cn flongting, yuantian] jlu.lllu.u. edu.
ABSTRACT
An important challenge in multi-agent reinforcement learning is partial observ- ability, where agents cannot access the global state of the environment during execution and can only receive observations within their field of view. To ad- dress this issue, previous works typically use the dimension-wise state. This state is obtained by applying MLP or dimension- basood attention on the global state for decision-makee making during training and relying on a reconstructed dimension- wise state during execution. However, dimension states tend divert agent attention to specific features, neglecting potential dependencies between agents, making optimal decisions more difficult. Moreover, the inconsistency between the states used in training and execution further increases additional errors. To Policy resolve these issues, we propose a method called Reconstruction-GGide (RGP) reconstruct the agent-wise state, which represents information of inter- agent relationships, as input for decision-making cution. This not only preserves the potential dependencies between agents but also ensures consistency between the states used in training and execution. We conducted extensive experiments on both discrete and continuous action environ- ments to evaluate RGP. and the results demonstrate its superior effectiveness. Our code is public in https:://// // //gi thub. cwm/Mut wwt.o//Musee//GPee http////////////eee//e///e//
during both training and exe-
INTRODUCTION
Cooperative multi-agent reinforcement learning (Cooperative MARL) Netoo 2005 Bukharin et al.,
2024] refers the scenario where multiple agents work together within an environment towards common goal through maximizing global reward. Due to its practical applications in fields like
robot collaboration (Orr & Dutta 2023), intelligent traffic systems (Mushtaq el al.) 2023). distributed
energy management (Zhu al. 2202g. network resource allocation (Allahham al. 2022 and
virtual simulations (Liang et al. 2022), this area has gained significant attention in recent years.
A typical challenge in multi-agent reinforcement learning is partial observability (Tuyls & Nowé 2005 Panait & Luke 2005), where each agent can only access observations within its field of view instead of the global state, which represents the complete information about all agents and the environment. One direct approach to addressing this issue is to reconstruct the global state (Chen) al. 2022. Xu al. 2024) within a centralized training and decentralized execution framework to enhance decision-making
As shown in Figure (c), these methods input the dimension-wise attention on the global state, into the policy during training to obtain actions for interacting with the environment, and then optimize the policy based on the rewards from the environment. Simultaneously, the global state used to train reconstructor, which is designed to reconstruct the global state (or the dimension-wise) when it becomes unavailable. During the execution phase, since the global state is inaccessible, the reconstructed state is used as substitute input for the policy network to generate actions and interact
state, which obtained by applying an MLP or dimension-wind
Corresponding author.

---

Published as a conference paper at ICLR 2025
Time step:
IIn STEEL
100
Gramal PTDE
Cental STDNT
5,1
Damtte Win Merr (P
Decenral PIDE
Decentall SiDet
1
60
polley
Ippesdracke
proteee_ S_M9.
10_
3
6 10
(a)
(b)
(c)
(d)
Figure I: (a) and (b) illustrate a toy example of UAV exploration and rescue. The global state is an 8-dimensional vector: the first two dimensions represent the the state of environment, and the remaining six correspond the start and end coordinates of the coverage areas for Hy. I1, and 22- (c) represents the training and execution process of previous state reconstruction methods, which have inconsistent state input for decision-making (d) represents the training and execution process of our RGP, which has consistent state input for decision-makinng (e) shows the mean win rates of previous state reconstruction methods PTDE (Chen al. 2022) and SIDiff (Xu al. 2024) both centralized and distributed execution.
with the environment. However, these methods have two limitations: (1)State inconsistency. Using a reconstructor replace the state with reconstructed state during execution can introduce additional errors. The policy is trained on the true state, but during execution, it operates on the reconstructed state, which may differ due to reconstruction inaccuracies. This discrepancy reduces the policy's 's robustness, this error, it is crucial to ensure consistency between the states used during training and execution. (2) Inadequacy of dimension- wise state. The dimension-wise state is simplified low-dimensional representation of the global state, which may cause the agents to overly focus on specific dimensions rather than concrete interactive objects. In toy scenario of UAV search and rescue in Figure I[a, Iy may overly focus on (0, 1) in the global information instead of the information of 3o (1, 3), leading overlap with zo during exploration, as illustrated in the figure. To enable agents focus on interaction targets rather than specific dimensions, a better approach is to split the global state by agent (as shown in Figure (b)). creating an agent-wise state that accounts for inter-agent relationships to improve decision-making.
was never trained handle imperfect states. Therefore, to minimize
With that consideration, we propose method called Reconstruction Guided Policy (RGP), which consists of two modules: the decision module and the guidance module. The decision module is active during both training and execution, it reconstructs the agent-wise state and feeds the agent- wise state to make decisions. The guidance module is only active during training and is used to guide agent-wisg wise state reconstruction. As shown in Figure the guidance module shifts the focus to the agents rather than the dimensional feature. By dynamically adjusting attention for each agent, potential relationships between agents are explicitly captured, avoiding overlap and improving the decision-making Since both training and execution rely the reconstructed agent-wiss agent-wise state for decision-making as shown in Figure [ (d), the inter-agent relationships are incorporated into the agent-wise state, and the additional error is reduced through state consistency, enhancing decision- making effectiveness. To evaluate the performance of RGP, we conduct extensive experiments on discrete and continuous environments, experimental results demonstrate the effectiveness.
In summary, our contributions are: i) We propose a novel method that maintains consistency be- tween training and execution, reducing error accumulation. ii) Our method reconstructs a more comprehensive agent-wise state, which is extracted based on agents rather than the dimension-win state, capturing inter-agent relationships and enhancing their performance. iii) Experimental results show that our method outperforms existing knowledge distillation based approaches, especially in dynamic, complex environments.
PRELIMINARY
Dec-POMDP: A fully cooperative multi-agent reinforcement learning task comprising n agents be represented by Decentralized Partially Observable Markov Decision Process (Dec-
2

---

Published as a conference paper at ICLR 2025
where N POMDP) OOliehock et al., 2016). which is defined as (N..AS. (1...., n) is a set of agents, A [A')eeN is the set of joint actions a; = {at) ieN at time step S is set of global states sc, which computed by state transition function P(s a.): S x A x S + |0, 1]. Global state 8t is a high-dimensional vector which contains the information of all agents and environment time L. For agent i. we denote the vector that indicates its inter- relationships with other units as the agent-wise state 87- O is a set of local observations of, which generated by the joint observation function (0+++ at, Sr+1) S X A + O. All agents share the [), is the discount factor. Each agent (Ω) X A)". Based on this history, the agent conditions of) J x A + [0, I]. The joint action-observation history T E J defined similarly. Based on the observed history 7", we introduced a trajectory state h for the agent, which is -igenndieniional vector represents the action-observation history of agent i. The joint policy π has a joint action-value function: Qior (5,, Q4) = 350+1:0,,044410 [G2 (G](ss [tt arl, where G Ek=>k=o>* The ultimate goal is to obtain an jotimal.join joint policy T with the optimal joint action-value function Q = Q
same reward function r(st az) S x A + (R) and
has an ooboovntoserviionnn history E a stochastic policy x' (
Diffusion Probabilistic Models: Diffusion models (Ho et al. 2020 Rombach et al. 2022 generate samples matching the target data distribution by denoising Gaussian noise. The models consist of a forward process and denoising process. The diffusion process introduces noise into the original dis- tribution p(ær) until it becomes a pure Gaussian noise distribution, and can be described as a Markov process: g g (
(x /I - BxI), where To..... 2K are latent variables, and is an identity matrix. 3% is the noise schedule measuring the proportion of noise added at cach step.
Given ag2 1 33g and Tig. = II (1 3:). then a (za) = Naxx; Vikg.ro, (1 (1-Tx) J)). The de- noising process starts with a Gaussian noise distribution p(xx) = (O, I) and restores the original data step by step, which can be described as: pe (xx-1 l = (xx-1; po (2x, k),, Eg (æx, k)), where pi (a*; k) (ox(1-au)
(2t,, k). and g presents model used to reconstruct I-ax-1 2. + - 1-6e (2
a. can be optimized by:
[(6) = =1--
- (2g, -
(1)
3 RELATED WORK
Multi-agent reinforcement learning faces the partial observability issue. To address this, Centralized Training with Decentralized Execution (CTDE) uses the global state to optimize the network during centralized training (Claus & Boutilier 1998) while relying only on local observations for decen- tralized execution Tan 1993). To further mitigate partial observability, some methods (Guan al., 2022 Wang et al. 2021) adopt agent communication, which allows agents to exchange informa- tion with each other. However, such methods typically require expensive communication channels. Information-sharii methods provide additional information for training and gradually reduce the additional information for distributed execution, such as SUPER (Gerstgrasser et al., 2024), which shares prioritized experience; MACPF (Wang et al. 2002). which shares the actions of other agents; and CADP (Zhou et al., 2023). which shares suggestions between agents. But this can reduce the performance of the agent during the execution phase. Other methods adopt a modeling approach. Opponent modeling methods (Papoudakis et al., 2021] [un et al., 2024) overlook the potential en- vironmental information. World model methods (Liu et al. (024) are employed to model the entire environment directly. However, modeling errors can be progressively amplified through interac- tions and feedback loops between agents, thereby affecting the overall learning performance and the convergence of strategies. State reconstruction methods aim to learn a model to reconstruct the global state during execution. For instance, PTDE (Chen et al., 2022) uses knowledge distillation to train a student network that reconstructs linear mapping of the global state. SIDiff (Xu et al. 2024) reconstructs the global state through diffusion. They have incorporated the inconsistent dimension- wise state for decision maasing in training and execution, which introduces additional errors and suboptimal decisions.
4 METHOD
To capture the inter-agent relationships and decrease the additional errors, we propose a method called ReconstructionnGiuide Policy (RGP), which consists of decision module and a guidance
3

---

Published as a conference paper at ICLR 2025
Agent
Qut Que
x +-
M-
Q.
Q:
Mixinn Network
8
replay
MLP
reconstructed agent-wise state
GRU
GRU
Mi reconstructed trajectory state
Q
Qu
concel
Q: reconstructed action value
nu reconstructed joint action value
Agent
Agentn
Dot Product
8 agent- wise state
Scalled Del
M J: trajectory
Product Sollmax
(of (naffrr
action value
Query
Kay
Value
oonoot
ruc' joint action value
81
(a) Decision Module
(b) Guidance Module
Figure 2: The framework of RGP. Figure (a) illustrates the Decision Module, which is active during both training and execution, reconstructing agent states 3 to assist in decision-making and action selection. Figure (b) shows the Guidance Module, which only active during training, generating agent states st as ground truth guide the training of the generative model. Both modules optimized through the same mixing network.
module. illustrated in Figure 2 the decision module is designed to reconstruct the agent-wise state and make decisions, which active in both training and execution. The guidance module is designed to guide the learning of the decision module, which is only active in the training. In the following, first describe the decision module, followed by the guidance module.
4.1 THE DECISION MODULE
As we discussed previously, the dimension-wise state lacks inter-agent dependencies, which is un- favorable for decision decision-making, Therefore, the decision module takes the observation and trajectory to reconstruct the agent- agent-wisc state, and makes decisions based on the agent-wise state.
Specifically, at arbitrary step & given the agent i's observation of and latest reconstructed trajectory state h our decision module takes the following steps to make the decision at.
First, it takes the observation of and latest reconstructed trajectory state hi-1 condition to reconstruct the agent-wise state agent i, which represents inter- relationships with other units discussed previously. Considering the generative models (Kelkar & Anastasio 202 Qi al., 2023 Chen et al., 201 8hare effective for reconstruction, we apply a generative model to reconstruct the agent-wise state of the agent:
5 9 M(-1)).
(2)
where | denotes the concatenation. of denotes the reconstructed agent-wise state of agent i. 9(-) denotes the generative model and we implement the g(.) with diffusion model, which reconstructs the agent-wise state by iteratively conducting Eq. K times to obtain si and use s as s/:
3e
So (s lot, M M B&C. ~ N'O, I), for A = K,... 1. (3)
V OKk
V a4
3e is the noise schedule measuring the proportion of noise added each step, and 0 is worth noting that although we chose diffusion models as the generative model in this work, other generative models are also applicable (we compared the results of reconstructing the agent-wise state using different generative models, and the detailed results can be found in Appendix C.2
Bc. It
Then, the reconstructed agent-wise state concatenated with the observation and fed a function f to predict the optimal action and the corresponding individual Q-value of the optimal action:
hi, aj Q; = f([s], of]. Aj-1),
where function f is composed of a recurrent neural network (RNN) (Zaremba, 2014 Graves & Graves 2012 Chung et al. 2014) and individual Q-network (Mnih 2013). Particularly, the
4

---

Published as a conference paper at ICLR 2025
concatenation of reconstructed agent-wise state sj and observation of is first fed to a RNN to obtain the reconstructed trajectory state of the current step:
hi RNN( oil, M111)
where RNN(-) denotes the RNN, and we implemented with Gated Recurrent Unit (GRU) et al. 2014). hi. 1 denote the reconstructed trajectory state of the current step and previous step, and hgg is initialized with a zero vector, that h O. Then, the reconstructed trajectory state of the current step is fed to the individual Q-network, which is implemented with multiple-layer perceptron pereepetto (MLP), to predict the individual Q-value of each action and obtain the action.
af c-Greedy (arg max MLP(h),a))
(6)
Q, = MLP (h1, af)
(7)
where of is the action taken by agent i to interact with the environment. is the hyperparameter in training, and 0 in execution, Qi denotes the individual Q-value.
is obvious that the reconstruction of the agent-wise state is uncontrollable within the decision module, which might lead to bad decisions in Eq. Therefore, introduce guidance module to guide the reconstruction.
4.2 THE GUIDANCE MODULE
The guidance module is simi-es network paired with the decision module, guiding the recon- struction and the learning of the decision module. As shown Figure 2 (b), the guidance module guides the decision module by incorporating the global state of the environment to estimate the agent-wise state 8 and trajectory state h4. They are derived from the true global state and can there- fore act constraints that guide the decision module to reconstruct the high- figh-iidelity reconstructed agent-wise state and reconstructed trajectory state when the global state is unavailable. It is impor- tant that the primary function of the guidance module is to guide the learning of the decision module, therefore, it is active only during training. In the following, we will first introduce the pipeline of the guidance module and then explain how it guides the decision module.
The Pipeline of the Guidance Module. At arbitrary step given the agent i's observation of and the global state st. our guidance module works with the following steps and produces the guidance agent-wise state 3; and guidance trajectory state h (Here, we prefix the variables produced by the guidance module with "guidance" to avoid confusion).
Since 8, is vector that contains the state of all the agents in the environment (if environmental information is present, we treat it as an independent agent), we first decompose the global state by agent, mapping each agent's state to the same dimension d, stacking them into matrix S.. S, E Raxd each row of the matrix S, denotes the state of one agent. Then we encode the ID of the agent, and concatenate them with global state matrix and observation. That
1 p). of = po
where P is the embedding matrix of agents' ID, which is a learnable matrix and each row vector of which corresponds to one agent. p¹ is the i-th row vector of P, which denotes the embedding of agent i's IDD. Then, we apply a multi-head attention on 5. and of to compute the guidance agent-wise state, which represents the inter- relationships with other units:
5 = Attn(f (o)),,
where f4, fo are implemented with MLP. Att(-) denotes the multi-head attention, in which f..) act as the query, key and value correspondingly. s denotes the guidance agent-wise state of agent i. Next, we feed the guidance agent-wise state 8 to (Eq 4 to obtain guidance trajectory state, guidance action and guidance Qavluee
ht at Q2 ((5t.,0))
(10)
5

---

Published as a conference paper at ICLR 2025
The Guidance Decision Module. To guide the reconstruction of the agent- wise state, we treat 8; as the ground truth data that diffusion model ς is supposed to reconstruct given the latest trajectory state and observation. Therefore, apply the learning target of the diffusion model to guide the agent-wise state reconstruction in decision module.
L B.e-(1_K)1,--0dd3...--..0.
< (e + vI Vi-aVa, Lo,,
11 (1I)
Recall that one of the factors influencing the decision made by the decision module is the trajectory state (hj-1 in Eq.44), which is computed from the agent-wise states of previous steps s;-1. As decisions are made sequentially over multiple time steps, any slight inaccuracy in the reconstructed agent-wise state at each step can accumulate in the trajectory state, affecting subsequent decisions. To address this, we utilize the guidance trajectory state generated by the guidance module to refine the reconstruction of the agent-wise state in an indirect manner. Specifically, the guidance trajectory state is employed to constrain the trajectory state derived from the reconstructed agent-wise state:
C. EDi(k ht)
(12)
4.3 MODEL LEARNING
Although the guidance module guides the agent-wise state reconstruction of the decision module, the parameters both the decision and guidance modules are randomly initialized. The constraints from Eq. [T] and Eq.(12) can only make the local and trajectory states generated by the two modules closely approximate each other, but cannot guide them toward convergence in a direction favorable for making optimal decisions. Following previous work, we also adopt Mixing network (Rashid) cet al. 2020) to further constrain the agent's decisions. Specifically, we apply two TD losses (Peng & Williams 1993 Mnih al. 2015) constrain the actions produced by the decision module and the guidance module.
La Ep[(r mmaxQtod (Te+1, ar+1, 0 ) - -Qrat (Te, ar 81; +))
(13)
ail1
C = Epl(r monaxQtoe (T++1, a,+1. 8:+1; 0 .7 Qroi (e, ae. sc; 0, n))²1, ax+1
(14)
where vi are the parameters of the generative model, v are the parameters of the attention and 0 are the parameters of the other networks. , . and 0 represent the parameters of the target network, which are periodically copied the parameters and kept constant for a definite number of iterations. It is important to note that although the actions generated by the guidance module are not used for interaction with the environment constraining them helps guide the agent-wise state and trajectory states to converge toward optimal actions.
Therefore, the learning target of our method is:
L Li La + La L.g. The details of training are illustrated in Appendix A.I algorithm
(15)
EXPERIMENTS
We conduct extensive experiments to evaluate the performance of RGP, and we particularly focus on the research questions: How does RGP perform compared with other methods (RQ1)? ii) Can RGP reduce the gap between training and execution (RQ2)? iii) Why does RGP can achieve better performance than other methods(RQ3)) iv) Can RGP explore the potential relationships between agents? (RQ4)? v) Can RGP adapt to continuous action environments? (RQ5)? vi) How does the RGP perform under more challenge partially observable conditions? (RQ6)?
5.1 EXPERIMENT SETTING
Environments. We primarily evaluated RGP on SMAC (Samvelyan et al. 2019) and SMACv2 EI- lis et al. 2024). SMAC is the most widely used discrete multi-agent environment, while SMACv2 introduces stochasticity based on SAMC. We set up the SMACv2 maps with ally agents against enemies. Additionally, to further demonstrate the portability of RGP, conducted experiments in
continuous predator- eeddator-rryyaaooo and continuous cooperative navigation scenarios (Lowe et al. 2017). For
detail of continuous predator-prey and continuous cooperative navigatio, Appendix B

---

Published as a conference paper at ICLR 2025
MAP
VDN
QMIX
QPLEX
CADP
PIDE
SIDiM
RGP
HPN-
RGP+
Q
HPN
27mn_vs..00 3.m
92.5a6.0
96.423.5
96.743.0
92.6a4.5
96.6a1.9
93.243.9
98.1=2.1
99.5a0.7
100.0:0.0
MMM2
88.345.4
92.745.4
94.8+2.9
92.0++4.
94.7+344
87.248.4
98.1
99.4+0.3
365z_v._ 26,62
65.945.4
70.8+6.4
30.6+37.3
90.2+3.1
67.4+6.9
10.0+8.7
76.9+2.8
94.7+2.2
96.6+2.3
corrider
86.916.9
92.144.6
30.5+39.9
87,8+4.3
91.422.7
70.7=17.0
93.3:3.0
96.4+3.4
96.7+2.9
6b 8z
23.3417.9
38.1428.3
7.642.3
51.8&16.3
52.4a8.5
23.8
73.1
94,423.5
95.612.3
Protoss. 360
58.845.2
69 69..6
70.0+5.3
64.6+-6.8
64.4+3.4
25.3
76.2+3.6
79.424.8
84.5+4.3
Terran 360
65.16
72.8+5.9
74.6+6.5
70,2+5.9
63 63.4
47.8+10.9
75.0=5.9
78.0s5.2
82.8a4.44
Zorg- 360
51.6+7.1
55.645.3
40.9=28.7
56.7+7.7
55.3+5.6
46.6+8.9
61.427.1
67.47700
70.1=7.0
Table I: The mean win rate (%) of different methods on SMAC and SMACv2, with + denoting the standard deviation. We applied RGP to QMIX, denoted RGP. Protoss_360 indicates that in the protoss map, agents have a 360 field of view. Similarly, zerg 360 and terran 360 represent analogous settings for the zerg and terran maps. We further applied RGP to HPN-QMIX, denoted as GRP+HPN. The corresponding training plots are provided in Appendix C.1 for reference.
Baselines. In discrete action environments, we used traditional CTDE methods as baselines, in- cluding VDN (Sunehag et al., 2017). QMIX (Rashid et al. (020), QPLEX (Wang et al.] 2020) and HPN-QMIX (Jianye et al. 2022). Additionally, we compared RGP with CADP (Zhou et al.)
2023 which enhances CTDE by providing agents with supplementary information during execu-
tion.
We further evaluated our method against PTDE (Chen et al., 2022) and SIDiff (Xu et al.) which reconstruct the dimension-win state through local observations during decentralized execution. In continuous action environments, we compared RGP with policy-based methods, such MADDPG (Lowe et al. 2017) and FACMAC (Peng et al. 2021).
2024
Implementation Details. Our model was trained on a setup with 4 NVIDIA A40 GPUs, an Intel
Gold 5220 CPU, and 504GB of memory, optimized using the Adam optimizer (Kingma & Ba 2014).
Due to limited computational resources, we replaced the Unet used in the original paper DDPM (Ho) al. 2020, Rombach et al. 2022) with an MLP. We set the timestep of diffusion to 10, and the
heads of attention to 4. The details of other hyperparameters can be found in Appendix [A. table
We also referred to PyMARL2 (Hu et al. (202]) for extensive tuning and proper training of all
baselines to ensure they achieved optimal performance.
5.2 BENCIMARK RESULTS
We compare RGP to baseline methods with respect to the normalized battle mean win rate obtained during online evaluation. We conducted trials with different seeds, reporting the average results and standard deviation. The results of RGP and baseline methods summarized in Table
From Table [ we can observe that (RQ1): (1) Compared with the methods without reconstructing the states, such VDN, QMIX, QPLEX, our RGP achieves the best performance, which demon- strates that relying solely on local observations is insufficient in distributed execution. Additional information is needed to assist the agent in decision-male making achieve better performance. (2) We noticed that RGP outperformed QMIX in most cases, which has exactly the same backbone with QMIX. Similarly, RGP+HPN also outperformed HPN-QMIX. That demonstrates that the recon- structed agent-wise state provided by the decision module beneficial for the decision of agent. (3) Compared with the methods that reconstruct the states, such as PTDE and SIDiff, our RGP achieves the best performance in most cases, which shows that our method provides agents with more refined additional information, specifically the agent-wise state, through the decision and guidance modules, which can effectively enhance the agents' eecciiion-mainng performance.
5.3 PERFORMANCE RETENTION
To investigate whether RGP reduces the performance gap between training and execution (RQ2), we compare the retention ratio of RGP with the PTDE and SIDiff, which also reconstruct the state for decision making, on the environment of SMAC and SMACv2, and illustrate the results in Table 2 From Table 2 we can observe that: PTDE and SIDiff exhibit significant performance discrepancies between centralized training and distributed execution, with both methods having lower perfor- mance retention rate than our proposed RGP. Since both PTDE and SIDiff aim to reconstruct the dimension- wise state and have an inconsistent state in decision-making of training and execution,

---

Published as a conference paper at ICLR 2025
PTDE
SIDiff
RGP
Map
CF
DE
PRR
CE
DE
PRR
CF
DE
PRR
3ahz.
77.646.2
67.4a6.9
86.9&11.3
10.5+2.8
95.248.7
79.342.1
76.9
97.0:444
6b_v8_82
71.245.3
52.428.5
73.6+13.1
29.7=22.8
23.8+15.9
80.12816
76.546.1
73.124.6
95,629.7
protoss_360
73.245.5
64.42344
88.048.1
59.845.7
25.3+1144
42.3=19.5
76.944.1
76.243.6
99.1=7.1
360
71.644.8
63.4a6.3
88.5+10.6
68.1
47.8+10.9
70.2+17.4
73.143.8
75.0=5.9
102.649.7
Table 2: The mean win rates (%) and performance retention ratios (%) (Chen al. 2022) of different
methods on SMAC. We tested their results under both centralized and decentralized execution. CE stands for centralized execution, DE stands for decentralized execution. PRR stands for Performance Retention Rate, which can be calculated by dividing the win rates of DE by it of CE. The uncertainty of PRR is measured by the standard deviation propagation technique.
-LLoss
-TTooss
-QLoss
Cottate
Istaee
RGP
80
Bartle Win Mean
40
770
50
20
20
10
10
profoss 360
Z0Tg 360
protoss 90
zerg 90
Figure 3: The ablation study. We tested the mean win rate under different variants of RGP. Pro- toss. 90 indicates that the agents have a 90 field of view in the protoss map, and zerg_90 represents similar setting for the zerg map.
this result suggests that learning the reconstruction of the agent-wiss state and adopting a consistent state in decision-nig decision-making is beneficial.
5.4 ABLATION STUDY
To further investigate the factors that support the performance of RGP (RQ3), conduct the abla- tion study. specifically, we compare the performance RGP with the following variants:
-LLoss removes the constraint (Li, Eq. in the guidance module. That means the reconstruc- tion of the agent-wise state is only guided by the constraint on trajectory state (C2).
-Lsss removes the constraint on the trajectory state (C,, Eq 12 in the guidance module. That means the reconstruction of the agent-wise state only guided by the constraint on agent-wise state (Ci).
-Loss removes the constraint ( Eq on the Q-value.
Gstate applies the global state reconstruction in decision module and make decision based on the the reconstructed global state.
Istate applies the inconsistent state in training and execution on RGP.
The results are illustrated in Figure 3, from which we have following findings: (1) Removing the constraint the Q-value of guidance module (-QLoss) significantly decreased the performance of RGP and introduced greater variance. This is because without the constraint of Lg. the guidance module struggles guide the reconstructed agent-wise state and trajectory state favoallef states, hindering the agent's s ability to make optimal decisions. (2) Removing L₁ (-LLoss) and L, (-TLoss) decrease the performance. This because both Ci and C, is used guide the agent- wise state reconstruction. Removing one of them affects the reconstruction of the agent-wise state, thereby impacting overall performance. Also, we observed that the impact of both settings on per- formance is less significant than that of -LLoss. This is because removing one constraint still leaves the other constraint in place to properly guide the reconstruction of the agent-wise state. (3) Making decisions based on the reconstructed global state (GState) instead of the agent-wise state decreases the performance. We believe this is due to two facts. On one hand, making decisions directly based on the global state without considering the agent-wise states of the agents results in decisions that do not align with the actual circumstances of the agents. On the other hand, the global state it- self is discrete, and the diffusion models implemented may not be adept generating discrete
8

---

Published as a conference paper at ICLR 2025
Agent 1
0.6
Agents
Agent
0.4
Agent 3
Agent
0.2
Agent 5
10
15
20
25
30
Time Steps
Figure 4: The attention weight of each agent.
MADDPG
FACMAC
RGP+MADDPG
RGP+FACMAC
2008
Epissdde Returns
Episode Returns
Episode Returss
100
1010
200
0.0
20
0.3
2.
25
20
0.9
29
(a) Predator-Proy agents, prey)
(b) Predstor-Prec (6 agents, pray)
(c) Cooperative Navigation (3 agents)
Figure 5: The average reward of different methods. RGP+MADDPG denotes using MADDPG as the subsequent information processing and critic network, and RGP+FACMAC refers to using FACMAC as the critic network. The experiment conducted using three random seeds. The thick, dark-colored lines represent the mean, while the shaded area indicates the standard deviation.
data, leading inherent biases in the reconstructed global state. 4) Applying inconsistent states to make decisions in training and execution (IState) decreases the performance. The reason as we discussed previously, the policy trained on the true state, never trained handle imperfect states (the reconstructed one), which might be slightly different from the actual one due to reconstruction inaccuracies. Feeding the imperfect states policy increase the suboptimal action.
5.5 AGENT-WISE STATE FOR AGENT
To further verify whether our RGP indeed accounts for the potential relationships between agents (RQ4), we randomly select an arbitrary agent and visualize the weight of attention assigned to other agents when computing the agent-wise state for decision-making. The results are illustrated in Figure 4 which the agent is denoted in Agent 1, and other agents are denoted with Agent 2 5. assigns to other agents when calculating the agent-wise state. The redder the color, the higher the weight of the corresponding agent's s state in Agent 1 agent-winn wise state.
The horizontal axis represents time steps. Each cell indicates the attention weight that Agent
From Figure we observed that: (1) Before step 15, Agent 1 focuses on its own state, but after step 15, its reliance its state decreases. This is likely because, in the early stages, it needs its own state to posítion itself in the environment, but once positioning is complete, it relies only on local observations and longer needs its state. (2) Before step 15, the weight of Agent is relatively high, while after step 15, the weight of Agent 3 increases more than that of Agent 2. We assume that likely because, Agent dynamically adjusts its cooperation strategy based on the evolving importance of other agents, first relying more on Agent 2 and later shifting its focus to Agent 3 as the context changes. This suggests that the system is able flexibly respond varying circumstances, optimizing collaboration as needed throughout the task. These results demonstrate that RGP effectively captures inter-agent relationships and selects information of important agents.
5.6 PORTABILITY ON CONTINUOUS ACTION ENVIRONMENT
To further investigate the portability of RGP on continuous action environment (RQ5), we transport the decision module and the guidance module of RGP to the MADDPG (Lowe et al. 2017) and FACMAC (Peng et al. 2021), which tackle continuous action scenarios, and compare their perfor- mance in the continuous action environment: Cooperative Navigation and Predator-Prey (Lowe) et al.) 2017). The results illustrated in Figure 5 in which RGP+MADDPG denotes the setting we transport the RGP to MADDPG, and RGP+FACMAC denotes the setting we transport RGP to FACMAC (for the details of implementation please refer Appendix A. Algorithm

---

Published as a conference paper at ICLR 2025
Metric
protoss_ 366
protoss, 90
protoss
terran_3ol
terran.
terran.30
zerg.360
zerg.90
zerg..0
QMIX
69.444.6
27.6+8.2
7.2+1.6
72.845.9
41.9+5.3
5.2+2.0
55.645.3
23.2+5.6
2.1
RGP
76.233.6
37.2+6.5
10.1+5.0
75.0&5.9
46.3s4.1
9.4x1
61.427.1
30.0a4.7
4.2+1.1
9.8+5.2
34.8+25.7
40.3+70.0
3.068.1
80.8+38.7
29.3421.5
100.0+59.7
HPN.QMIX
79.4&4.8
47.545.8
8.843.6
78.045.2
49.547.4
4.442.8
67.447.0
17.1=5.3
2.5.2.3
RGP+HPN
84.544.3
53.946.5
10.243.6
82,8+4.4
60.0+5.6
12.8+3.4
70.1+7.0
26.3+8.0
4.1+1.6
PTR 6.445.4
13.5+13.8
15.9a41.4
6.245.7
21.2
190.9:144.0
4.0=10.4
53.8+49.7
64.0=87.0
Table 3: The winning rates (%) and performance improving ratios (PIR)(%) of different methods on SMACv2. We tested their results under decentralized execution and calculated the performance improving ratio during decentralized execution. Protoss 30 indicates that the agents have a 30 field of view. Similarly, zerg-30 and terran.30 represent analogous settings for the zerg and terran maps, respectively. The uncertainty of PIR is measured by the standard deviation propagation technique.
We can observe from Figure that: (1) RGP+MADDPG achieves better performance than MAD- DPG, and RGP+FACMAC achieves better performance than FACMAC in all the settings. That shows introducing our RGP benefits the performance, and our RGP is not only applicable to discrete action environments but also continuous action environments. (2) In the Predator-Prey (Figure (a) and (b)), RGP+FACMAC achieves the average return which several times than FACMAC. Espe- cially the setting with 6 agents and prey, the average reward of FACMAC is around 1200, while the average reward of RGP+FACMAC is around 1600! We assume that is because the introduction of agent-wise states helps agents effectively cooperate with surrounding agents, facilitating their collaboration to better corner the prey.
5.7 FURTHER INVESTIGATION
In multi-agent decision-making the field of view an agent is key factor affecting its decisions; a smaller field of view makes it more likely for the agent to make suboptimal decisions. To investigate the performance of RGP under more challenging partially observable conditions (RQ6), we tested RGP's performance across different fields of view (360", 90", and 30%) and compared it with QMIX and HPN-QMIX under the same settings.
The results are illustrated in Table 3 in which both RGP+HPN and RGP incorporate the decision module and the guidance module, whereas UPN-QMIX and QMIX lack these modules. Moreover, QMIX is the backbone of RGP, and HPN-QMIX is the backbone of RGP+HPN. From Table 3 can observe that: (1) Regardless of the field of view, RGP outperforms the QMIX and RGP+HPN outperforms the HPN-QMIX, which demonstrates the effectiveness of method again. (2) In all settings, as the field of view decreases, the difference between the setting of introducing the deci- sion module and guidance module and the setting without these two modules becomes increasingly pronounced. It suggests that our method can reconstruct state that better characterizes the actual state of the agent for decision-maker
6 CONCLUSION
In this paper, we propose a method called Reconstruction-Guuieed Policy (RGP), which consists of a decision module and guidance module. The decision module responsible for reconstructing the agent-wise state and making decisions, while the guidance module helps guide the agent-wise state reconstruction. We conduct extensive experiments to evaluate the performance of RGP, and the results demonstrate the effectiveness of the proposed method. We found that ensuring consis- tency between training and execution states effectively prevents error amplification and improves performance retention ratios during execution, particularly in dynamic and complex environments. Additionally, considering the potential relationships between agents helps them identify interaction targets, which is beneficial for promoting cooperation among agents.
However, our method still has limitations. In environments where complex information such as images is used for state representation, it becomes difficult to decompose agent-specifie information. In the future, techniques like object detection or image segmentation could be introduced to address this issue.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
The Jilin University team is partially supported by the Ministry of Science and Technology of China (No. (0002YFFF9005000 and the National Natural Science Foundation of China NN.662070200 and No.U2341229). The Shanghai Jiao Tong University team is partially supported by National Natural Science Foundation of China (62322603) and Shanghai Municipal Science and Technology Major Project (2021SHZDZZ00102),
REFERENCES
Mhd Saria Allahham, Alaa Awad Abdellatif, Naram Mhaisen, Amr Mohamed, Aiman Erbad, and Mohsen Guizani. Multi-agent reinforcement learning for network selection and resource alloca- tion in heterogeneous multi-rat networks. IEEE Transactions Cognitive Communications and Networking. 8(2):1287-1300, 2022.
Alexander Bukharin, Yan Li, Yue Yu, Qingru Zhang, Zhehui Chen, Simiao Zuo, Chao Zhang, Songan Zhang. and Tuo Zhao Robust multi-agent reinforcement learning via adversarial reg- ularization: Theoretical foundation and stable algorithms. Advances in Neural Information Processing Systems, 36, 2024.
Nutan Chen, Alexej Klushyn, Richard Kurle, Xueyan Jiang, Justin Bayer, and Patrick Smagt. Metries for deep generative models. International Conference on Artificial Intelligence and Statistics, pp. 1540-1550. PMLR, 2018.
Yiqun Chen, Hangyu Mao, Tianle Zhang, Shiguang Wu, Bin Zhang, Jianye Hao, Dong Li, Bin Wang, and Hongxing Chang. Ptde: Personalized training with distillated execution for multi- agent reinforcement learning. arXiv preprint arXiv:2210.0887) 2022.
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555. 2014.
Caroline Claus and Craig Boutilier. The dynamics of reinforcement learning in cooperative multia- gent systems. AAAI/AА 1998(746-752)22, 1998.
Benjamin Ellis, Jonathan Cook, Skander Moalla, Mikayel Samvelyan, Mingfei Sun, Anuj Mahajan, Jakob Foerster, and Shimon Whiteson. Smacv2: An improved benchmark for cooperative multi- agent reinforcement learning. Advances Neural Information Processing Systems, 36, 2024.
Matthias Gerstgrasser, Tom Danino, and Sarah Keren. Selectively sharing experiences improves multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Alex Graves and Alex Graves. Long short-term memory. Supervised sequence labelling with recurrent neural networks, pp. 37-45, 2012.
Cong Guan, Feng Chen, Lei Yuan, Chenghe Wang. Hao Yin, Zongzhang Zhang, and Yang Yu. Efficient multi-agent communication via self-supervised information aggregation. Advances in Neural Information Processing Systems, 15:1000-1033, 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Jian Hu, Siyang Jiang, Seth Austin Harding, Haibin Wu, and Shih wei Liao. Rethinking the imple- mentation tricks and monotonicity constraint in cooperative multi-agent reinforcement learning. 2021.
HAO Jianye, Xiaotian Hao, Hangyu Mao, Weixun Wang, Yaodong Yang, Dong Li, Yan Zheng and Zhen Wang. Boosting multiagent reinforcement learning via permutation invariant and permuta- tion equivariant networks. In The Eleventh International Conference on Learning Representations. 2022.
Varun A Kelkar and Mark Anastasio. Prior constraind reconstruction using style-based generative models. International Conference on Machine Learning, pp. 5367-5377. PMLR, 2021.
11

---

Published as a conference paper at ICLR 2025
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Zhiuxan Liang, Jiannong Cao, Shan Jiang, Divya Saxena, Jinlin Chen, and Huafeng Xu. From multi- agent to multi-robot: scalable training and evaluation platform for multi-robot reinforcement learning. arXiv preprint arXiv:2206.0953 2022.
Qihan Liu, Jianing Ye, Xiaoteng Ma, Jun Yang. Bin Liang, and Chongjie Zhang. Efficient multi- agent reinforcement learning by planning. arXiv preprint arXiv:2405. 1778, 2024.
Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cupprative-competiiie environments. Advances in neural information processing systems, 30, 2017.
Volodymyr Mnih. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602. 2013.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529 533, 2015.
Anum Mushtaq, Irfan UI Haq, Muhammad Azeem Sarwar, Asifullah Khan, Wajeeha Khalil, and Muhammad Abid Mughal. Multi-agent reinforcement learning for traffic flow management of autonomous vehicles. Sensors, 23(5):2373, 2023.
Gonçalo Neto. From single-agent multi-agent reinforcement learning: Foundational concepts and methods. Learning theory course, 2, 2005.
Frans A Oliehoek, Christopher Amato, et al. A concise introduction to decentralized POMDPs, volume Springer, 2016.
James Orr and Ayan Dutta. Multi-agent deep reinforcement learning for multi-robot applications: A survey. Sensors, 23(7))3625, 2023.
Liviu Panait and Sean Luke. Cooperative multi-agent learning: The state of the art. Autonomous agents and multi-agent systems, 11:387-434, 2005.
Georgios Papoudakis, Filippos Christianos, and Stefano Albrecht. Agent modelling under partial ob- servability for deep reinforcement learning. Advances in Neural Information Processing Systems, 34:19210-19222, 2021.
Bei Peng, Tabish Rashid, Christian Schroeder de Witt, Pierre-Alexandre Kamienny, Philip Torr, Wendelin Böhmer, and Shimon Whiteson. Facmac: Factored multi- aalt-- centralised policy gra- 2021.
dients. Advances in Neural Information Processing Systems,
J Peng and RJ Williams. On the convergence of stochastic iterative dynamic programming algo- rithms. Adaptive Behavior, 1:437-454, 1993.
Zekun Qi, Runpei Dong, Guofan Fan, Zheng Ge. Xiangyu Zhang, Kaisheng Ma, and Li Yi. Contrast with reconstruct: Contrastive 3d representation learning guided by generative pretraining. In International Conference on Machine Learning, pp. 18223-28244 PMLR, 2023.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Monotonic value function factorisation for deep multi- reinforcement learning. Journal of Machine Learning Research, 21 (178)): 2020.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 10684-10695, 2022.
Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt. Gregory Farquhar, Nantas Nardelli, Tim G. J. Rudner, Chia-Man Hung, Philiph H. S. Torr, Jakob Foerster, and Shimon Whiteson. The CtafCraft Multi-Agent Challenge. CoRR, abs/1 902.04443, 2019.
12

---

Published as a conference paper at ICLR 2025
Jing Sun, Shuo Chen, Cong Zhang, Yining Ma, and Jie Zhang. Socision-making with speculative opponent models. IEEE Transactions on Neural Networks and Learning Systems, 2024.
Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Mare Lanctot, Nicolas Sonnerat, Joel Leibo, Karl Tuyls, et al. Value-decomposition networks for cooperative multi-agent learning. arXiv preprint Si..iv:1706.02299090 2017.
Ming Tan. Multi-agent reinforcement learning: Independent vs. cooperative agents, Proceedings of the tenth international conference machine learning, pp. 330-337, 1993.
Karl Tuyls and Ann Nowé. Evolutionary game theory and multi multi-agent reinforcement learning. The Knowledge Engineering Review, (009), 2005.
Jiangxing Wang, Deheng Ye, and Zongqing Lu. More centralized training, still decentralized exe-
cution: Multi-agent conditional policy factorization. arXiv preprint arXiv:2209.1220 12681, 2022.
Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. Qplex: Duplex dueling
multi-agent q-learning, arXiv preprint arXiv:2008.01 01062. 2020.
Yuanfei Wang, Fangwei Zhong, Jing Xu, and Yizhou Wang. Tom2c: Target-oriented multi-agent communication and cooperation with theory of mind. arXiv preprint arXiv:2111.00 11N::2211..9199 2021.
Zhiwei Xu, Hangyu Mao. Nianmin Zhang, Xin Xin, Pengjie Ren, Dapeng Li, Bin Zhang, Guo- liang Fan, Zhumin Chen, Changwei Wang, et al. Beyond local views: Global state infer- ence with diffusion models for cooperative multi-agent reinforcement learning. arXiv preprint DXXi::2408.09501 2024.
Wojciech Zaremba. Recurrent neural network regularization. arXiv preprint wrXiv:1409.2329. 2014.
Yihe Zhou, Shunyu Liu, Yunpeng Qing. Kaixuan Chen, Tongya Zheng, Yanhao Huang, Jie Song, and Mingli Song. Is centralized training with decentralized execution framework centralized enough for marl? arXiv preprint 1rXi::2305.17352. 2023.
Dafeng Zhu, Bo Yang. Yuxiang Liu, Zhaojian Wang, Kai Ma, and Xinping Guan. Energy man- agement based on multi-agent deep reinforcement learning for a multi-energy industrial park. Applied Energy, 311:1 18636, 2022.
A ALGORITHMS AND HYPERPARAMETERS
A.1 RGP WITH VALUE DECOMPOSITION METHOD AND POLICY GRADIENT METHODS
Our method can effectively integrated with both value decomposition and policy gradient ap- proaches, which we have summarized Algorithm Tandd and Algorithm respectively.
A.2 HYPERPARAMETERS DETAIL
Details of RGP's hyperparameters provided in Table 4 For bascline VDN, QMIX. QPLEX, they were implemented with the hyperparameters of PyMARL2 (Hu et al.) 2021). For HPN-QMIX, CADP, PTDE, and SIDiff, they were implemented with their optimal hyperparameters, specified
in their respective papers (Jianye et al. 2022 Zhou et al. 2023 Chen et al. 2022 Xu et al. 2024)
B ENVIRONMENT DETAILS
B. CONTINUOUS PREDATOR- PREY
We followed FACMAC Peng et al. 2021) and improved the simple tag scenario in MPE (Lowe al. 2017), which is variant of the classic predator-prey trrt problem. Each agent operates in a tinuous wio-dimensional action space and must capture mistingmoving prey a randomly generated 2D environment, where obstacles hinder movement. To create fully cooperative environment, the
13

---

Published as a conference paper at ICLR 2025
Algorithm Training of RGP with Value decomposition methods
1: Initialize Generative model and attention
2: Initialize individual Q networks and target networks
3: Initialize mixing network and target mixing network
4: Initialize replay buffer D
5: for each episode do
6: Initialize global state 80. guide hidden states and reconstruction hidden states for each timestep 0 to 7 do
7:
8:
for each agent = 1 to do Obtain the global state so and local observation of
9:
10:
Calculate S; and 0; according Eq 8
11:
Calculate agent-wise state 8 according Eq
12:
Calculate reconstructed agent-wise state s according Eq 2
13:
Select action at according to s with c-greedy policy w.r.t Qi(o s h!-1) end for
15:
Take joint action a2
Obtain the global reward re+1, the next observation o, and the next global state 8
17:
Store the episode in replay buffer D
Sample batch episodes from replay buffer D
19:
Update the parameters of the decision module, guidance module and the mixing network
according Eq. [5]
20:
Replace target parameters every M episodes
21:
end for
22: end for
Algorithm 2 Training of RGP with policy gradient methods
1: Initialize Generative model and attention
2: Initialize individual actor and the critic
3: Initialize replay buffer D>
4: for each episode do
5: Initialize global state su, guide hidden states and reconstruction hidden states
6: for each timestep = 0 to T do
7:
for each agent 1 to N do
8:
Obtain the global state so and local observation of
9:
Calculate S, and of according Eq
10:
Calculate agent-wise state 5 according Eq
Calculate reconstructed agent-wing wise state s; according Eq 2
Select action at according to sf with distribution w.r.t m. (o). s h 11
13:
end for
Take joint action a;
15:
Obtain the global reward 1++1, the next observation o, and the next global state 8;
16:
Store the episode in replay buffer
17:
for each agent = 1 to N do
18:
Sample batch of episodes from replay buffer D
19:
Fit value function by regression hean-squared error and update critic Update actor according to policy gradient methods end for
20:
21:
22: Update target parameters by polyak averaging
end for
24:
end for
14

---

Published as a conference paper at ICLR 2025
Module
Hyperparameter
Value
Diffusion process timestep
10
Type of optimizer
Adam
Learning rate
0.001
Decision Module
Batch size
128, (64 for 3s52_vs 336zz)
TD lambda
0.6, (0.3 for 6h, 6h_vs_82)
Training epochs
10M, (8M for SMACv2)
Buffer size
5000, (2500 3s5z vs. 386z)
Target update interval
200
Attention heads
Guidance Module
Attention embedding dim
Agent information mapping dim
32
Table 4: Hyperparameter settings for the RGP training.
prey's strategy was designed as hard-coded heuristic, where it always moves to the farthest posi- tion from the nearest predator at any given time. Agents can only receive rewards when they are close enough to the prey.
To introduce partial observability, a vision radius was added for each agent. Agents can only receive information within their vision radius (including other agents, prey, and obstacles), limiting their ability to observe other agents to approximately 60% of the time.
B.2 CONTINUOUS COOPERATIVE NAVIGATION
The continuous cooperative navigation is a fully cooperative environment based on MPE (Lowe al. 2017). In a two-dimensional space, n agents must collaborate cover landmarks scattered in the environment. Agents receive rewards from the environment when they get close enough a landmark. They can only access information within their field of view, including their own position and velocity, the position and velocity of other agents, and the location of the landmarks. We implement the continuous action space by setting cont inwuosuaacttt.oooo iaaaiiiooTruu in the code. For more details, Further details available in the official PettingZoo documentation
C MORE EXPERIMENT
C.I TRAINING CURVES
To clearly illustrate the training procedure of the model, we present our curves in Figure 6 in which the x-axis represents the training episodes, while the y-axis represents the battle win rate in executing. be found that RGP+HPN outperforms HPN-QMIX and performs optimally in most cases. RGP also outperformed the rest of the baselines in most cases.
C.2 THE IMPACT OF GENERATIVE MODELS
Our original intent of applying diffusion is generate the agent- state. Therefore, diffusion serves as one module in our method. Theoretically speaking, any generative model, including VAE, GAN,
simple MLP, can replace the diffusion in our method. To further validate the impact of generative models, we replaced the diffusion model in our RGP with MIP, VAF, and GAN respectively and conducted experiments in the SMACv2 environment. Experimental results are shown in Figure 7 We found that (1) Even though the diffusion model is replaced with VAE and MLP, we can observe that RGP outperforms the baseline QMIX in both settings, demonstrating the effectiveness of RGP. (2) Replacing the diffusion model with GAN, the performance slightly decreases. We assume that because GAN requires simultaneous training of both the generator and discriminator, and this stepwise training negatively affects the overall quality of generation. (3) We can observe that replacing the diffusion model with VAE and ML.P results performance much
https::/peettinggzoo. farama www.://awww......./oeeeeooorooaa //mm.//r/ooeeerrronnntt///eeee
15

---

Published as a conference paper at ICLR 2025
RGP RGP+HPN
VDN QMIX
QPLEX
CADP PTDE
SIDiff
HPN-QMIX
ATM
27m, vs 30m
MMM2
3552
25_3962
carridor
1.0
battle win rate
battle win rste
10.6
0.5
battle
0.2
0.2
02
0.2
epesides
eposides
1n6
epasides
Ind
6h vs Bz
10gen protoss 360
10gen terran, 360
10gen_zerg zerg_360 260
1.0
1.0
1.0
1.0
battle win rate
rate
bettle win rate
bettle Win rate
0.6
0.6
066
966
0.2
0.
82
eposides
Iet
eposides
166
eposides
1nd
eposides
let
Figure 6: Training curves in SMACv2.
QMIX
Diffusion
GAN
VAE
MLP
Battle Win Mean
29
66
43
34
74-
69
61
38
29
64
56
33
%
51
28
24
54
46
23
protoss
zerg_
protoss 90
z012
Figure 7: Performance comparison across various generative models. Protoss 360 and Protoss 90 represent agents having 360" and 90º fields of view, respectively, in the protoss map. Similarly, zerg_360 and zerg_90 represent analogous settings, respectively, in the zerg map.
closer to that of the diffusion setting. We assume the underlying reason is the substitution of the U- Net used in the original paper DDPM (Ho et al. 20200 Rombach et al., (2022) with MLP (to save the cost of computational resources). Nevertheless, the setting of diffusion still outperforms the setting of VAE and MLP in most cases.
C.3 THE BIAS OF GENERATIVE MODELS
To explore the potential biases introduced by the generative model in state reconstruction, we con- ducted experiments on zerg-90. Specifically, we sampled 1,000 trajectories from PTDE and RGP, respectively, to calculate the cosine similarity between the true ground truth and the reconstruction. The results are visualized in Figure which each grid represents the cosine similarity for the cor- responding agent at specific time step. We can observe that (1) the reconstruction of our method significantly closer to the true ground truth compared to the baseline. Furthermore, the similar- ity between the reconstruction of our method and the true ground truth consistently exceeds 0.9, demonstrating that our method introduces minimal bias during reconstruction. (2) The reconstruc- tion performance of RGP in the first step is relatively poor compared the subsequent steps. It is likely because of the cold state, in which the length interaction history is zero, interferes with the reconstruction. However,even under this condition, the cosine similarity of RGP still remains above 0.9, indicating that this interference does not significantly impact the overall performance.
C.4 MORE COMPLEX ENVIRONMENTS
C.4.1 MORE AGENTS
To further explore RGP's performance in scenarios with a larger number of agents, we conducted experiments in the zerg 90 20 20 environment. This setting involves 20 agents battling against
16

---

Published as a conference paper at ICLR 2025
Cosine similarity of Z and 2p in PTDE
Cosine similarity of sy and 5, in RGP
agents
2
0.40
0.30
0.00
25
30
20
25
30
time steps
time steps
Figure 8: Cosine Similarity of PTDE and RGP. The heatmap's horizontal axis represents the timesteps, while the vertical axis corresponds to the agents. The low-dimensional representation of the true global information constructed by PTDE is denoted as 2, and its reconstructed global information is denoted as Zp. To illustrate the detailed variations more clearly, we enhanced the color band by adding more vibrant colors to the key intervals require emphasis.
20 enemies under the zerg-90 configuration, presenting significantly higher challenge for the al- gorithm. The experimental results are presented in Figure 9
(a). We observe that RGP outperforms all baselines, demonstrating its scalability in environments with larger number of agents.
RGP
QMIX
PTDE
RGP
QMIX
Battle Win Mean
15
22
zerg protoss
7.4+4.5
2.7+1.9
17
14.2+4.2
1.6+0.1
10-
12
Table 5: The winning rates (%) of differ- ent methods on environment with changes in the underlying states. Zerg indicates that we trained on zerg. 360 and executed on zerg 360 epo, and protoss indicates that we trained on protoss_360 and exe- cuted on protoss_360.cpo.
zerg 90 20 vs 20
(b) zorg 360 epo
Figure 9: The winning rates (%) of different methods on more complex environments.
C.4.2 MORE HIGHLY DYNAMIC ENVIRONMENTS
To further explore RGP's performance in environments with higher randomness, conducted periments on the zerg zerg_360 1e..3600ppppp _epo map. This map builds upon SMACv2 by introducing random mask- ing of enemy observations for each agent. Specifically, in an episode, when enemy is observed by the first agent, the first agent is guaranteed to observe it as normal. Other agents have a 50% chance of being unable observe the enemy for the remainder of the episode. This significantly limits the agents' observation capabilities, further increasing environmental dynamics (for more details, refer the SMACv?7. The experimental results are presented in Figure (b). We observe that: (1) RGP outperforms other methods in environments with higher dynamics. This demonstrates RGP's scala- bility to highly dynamic environments. (2) PTDE performs poorly, likely because the inconsistency between states used during training and execution is further amplified in more complex settings. This magnification of inconsistency errors causes its performance to fall below RGP and QMIX.
C.5 ROBUSTNESS
To explore robustness RGP under changes in the underlying states of the environment, we trained zerg_360 and executed perg_360_epo. Similarly. we trained protoss 360 and exe- cuted on protoss 360 epo. The results are shown in Table We can observe that under such strict environmental constraints, QMIX is almost entirely unable to achieve any victories. In contrast, RGP retain a certain level of mecisi capabílity. This demonstrates that our approach main-
17

---

Published as a conference paper at ICLR 2025
tains robustness in environments with latent state changes. Even when unexpected variations in state dynamics occur, our method continues to maintains a clear advantage over the baseline.
18


Reviews:
[{"review_id": "Amg2KDTwHt", "reviewer": "Reviewer_aujm", "summary": "This paper addresses two issues with multi-agent reinforcement learning: the errors from reconstructing the global state of the environment during execution, and respecting dependencies between agents which are necessary for optimal decision making. The paper shows how the model addresses these two issues and empirically dissect it to show its better performance than current MARL solutions, as well as reasons for what causes this.", "questions": "* What is the uncertainty measure in Table 1? Is it a standard error?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "* The writing flows well and is well-motivated. It was easy to understand the MARL background and the problems of state reconstruction error and inter-agent coordination that this paper tackles.\n* Important assumptions and details (critical to the solution method and results) were made clear often to the reader: such as the guidance network only being used during the training, and the implementation details for the experiments.\n* Enumerating the research questions tremendously helped with the paper's clarity and structure. This practice should be adopted into more papers in the field!", "weaknesses": "* Figures need a bit more clarity: What do the shaded regions in Fig 5 represent? How many runs were used for Figure 5? Depending on the number of seeds and the claim being made, this has to be specified as context to the reader. For example, tolerance intervals or bootstrapped confidence intervals are great for showing the variability in performance when there, even when there are a small number of runs. 95% confidence intervals would be appropriate if there are more runs and it's safe to assume the y-axis variable is normally distributed.\n* The paper lacks a limitations section. Besides the one limitation regarding smaller fields of view, it  would be nice to see what other challenges to do with partial observability in MARL would need to be addressed next, and if there are any other shortcomings of RPG. Certain parts of the work speculated reasons about why results were seen (such as the impressive score of RGP+FACMAC in Predator-Prey). This could be summarized at the end of the conclusions section.\n* [Very Minor] Line 483: typo “continuous”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7hs1cWT9Xq", "reviewer": "Reviewer_gRGP", "summary": "Partial observability in multi-agent reinforcement learning (MARL) limits agents to local observations, often leading to suboptimal decisions and inconsistencies between training and execution. To solve this, the paper proposes the Reconstruction-Guided Policy (RGP), which reconstructs an agent-wise state capturing inter-agent relationships, ensuring consistent input for both training and execution. Experiments across discrete and continuous action environments show RGP’s good performance.", "questions": "- How did the authors make the MPE' Navigation task (Spread) continuous ?\n- What was the time horizon in SMAC tasks? Can the authors provide the plots from these experiments?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The paper studies an important problem in cooperative MARL, that is dealing with partial observability and non-stationarity\n- The paper proposes a novel method, which is based on state-of-the-art techniques such diffusion models and attention mechanism.\n- The paper includes strong ablation study, which examines and justifies the selection of the method's components.\n- The proposed method achieves good results across different benchmarks.", "weaknesses": "- The comparison with the related work can be improved: The paper ignores significant works based on agent (opponent) / state modelling, world models, transformer-based methods which also aim to deal with partial observability. For instance, the authors argue that one main challenge is state inconsistency. In particular, many works (e.g. [1]) leverage state embeddings (which can be much more compressed and informative) instead of using the reconstructed state (which can be less informative for each agent's training, also due to redundant information, see [2]). Therefore, although I believe that the proposed method is interesting, the state inconsistency challenge has partially been addressed by related work, and thus the authors should better highlight their contributions in the context of related work.\n- One concern is that the baselines regarding partial observability are not very established in MARL community. Some of them have not even been published at a top-tier AI/ML venue. To further improve the paper, I would suggest that the authors also include well-known methods (as baselines) based on alternative ways to deal with partial observability (e.g. [3], [4], [5]).\n- The attached repo does not contain any files.\n- Minor weakness: The authors do not include information about hyperparameter details of the baselines.\n- Minor weakness: There are many typos and grammatical errors (e.g. lines 240-244). I suggest that the authors proofread carefully the whole manuscript.\n\n[1] Papoudakis, Georgios, Filippos Christianos, and Stefano Albrecht. \"Agent modelling under partial observability for deep reinforcement learning.\" Advances in Neural Information Processing Systems 34 (2021): 19210-19222.\n\n[2] Guan, Cong, et al. \"Efficient multi-agent communication via self-supervised information aggregation.\" Advances in Neural Information Processing Systems 35 (2022): 1020-1033.\n\n[3] Yang, Yaodong, et al. \"Transformer-based working memory for multiagent reinforcement learning with action parsing.\" Advances in Neural Information Processing Systems 35 (2022): 34874-34886.\n\n[4] Wen, Muning, et al. \"Multi-agent reinforcement learning is a sequence modeling problem.\" Advances in Neural Information Processing Systems 35 (2022): 16509-16521.\n\n[5] Sun, Jing, et al. \"Decision-Making With Speculative Opponent Models.\" IEEE Transactions on Neural Networks and Learning Systems (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aWFDNnhclH", "reviewer": "Reviewer_qsBB", "summary": "The paper introduces Reconstruction-Guided Policy (RGP), a method in cooperative multi-agent reinforcement learning (MARL) to address partial observability, where agents rely on limited, localized information instead of global state access. Traditional MARL approaches train with global state access but deploy with reconstructed state representations, often reducing robustness and performance. RGP is evaluated on both discrete (SMAC, SMACv2) and continuous (predator-prey, cooperative navigation) environments, demonstrating superior performance and higher retention rates compared to methods like PTDE and SIDiff during centralized training and decentralized execution. Ablation studies and retention analysis further validate RGP’s effectiveness in preserving agent dependencies and decision consistency, showcasing its versatility in various MARL scenarios with observability challenges.", "questions": "1. The paper uses $\\epsilon$ repeatedly across equations, specifically in Eq (3) and Eq (6). Could you clarify why the same $\\epsilon$ notation is applied in both contexts? Given that Eq (3) refers to a noise variable for the diffusion process while Eq (6) applies to the exploration parameter, distinguishing between these variables would improve clarity and reduce confusion.\n2. Figure 2 is challenging to interpret with respect to the network design for RGP. Could you provide a more detailed explanation of how each component in Figure 2 maps to the decision and guidance modules described in Sections 4.1 and 4.2? Specifically, illustrating the data flow within and between these modules and explaining how key operations like the generative model, RNN, and attention mechanisms are integrated would be very helpful.\n3. Could you elaborate on the robustness of the learned policies when deployed in environments with potential state changes? How well does RGP handle discrepancies between the training and execution environments, especially if the state dynamics change unexpectedly? Additionally, have you considered incorporating techniques like domain randomization or online adaptation to improve robustness in dynamic deployment settings?\n4. The large standard deviations observed in baseline methods like SIDiff (Table 2) and QPLEX (Table 1, 3s5z vs 3s6z) suggest significant variability in these baselines’ performance. Could you clarify the potential sources of this variability? Additionally, could you explain the rationale behind selecting baselines with such high variance, as they may impact the reliability of comparative assessments?\n5. While the paper uses a diffusion model for reconstructing agent-wise states, the ablation study in Figure 6 suggests that alternative models (e.g., VAE, MLP) yield similar performance. Could you provide further justification for the choice of the diffusion model over simpler architectures, especially given the increased computational complexity?\n6. The paper evaluates RGP in multi-agent environments with relatively small agent counts. How scalable is the proposed approach in scenarios with a significantly larger number of agents, where maintaining agent-wise state consistency could become computationally intensive?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Proposes an innovative approach for state consistency in MARL, which mitigates errors from training-execution state inconsistencies.\n\n2. Effectively captures inter-agent relationships, improving collaborative performance.\n\n3. Demonstrates versatility and strong performance in both discrete and continuous action environments through extensive experiments.", "weaknesses": "1. Limited exploration of potential biases introduced by the generative model in state reconstruction, particularly in complex environments.\n2. The approach’s scalability to more agents and highly dynamic environments is not thoroughly evaluated.\n3. The theoretical foundation for using diffusion models over other generative models could be further clarified, as comparable methods like VAE and MLP produced similar results.\n4. Misleading use of letters/symbols, unclear annotations on some images, poor writing and reading experience", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vGubwQiXnm", "reviewer": "Reviewer_hsoz", "summary": "The authors introduce a method, Reconstruction-Guided Policy (RGP), which consists of a decision module and a guidance module. The decision module uses a diffusion model to reconstruct a per-agent agent-wise state from an agent’s partial observation of the full environment state which captures the agent’s relationship to other agents. This reconstructed agent-wise state is then used to condition a policy for agent action selection. The guidance module uses an attention-based model to guide the reconstructed agent-wise states. This ensures that the information agents condition on during acting and training remain aligned which reduces error accumulation and improves learning.", "questions": "* There is no code available, the link the authors have provided seems to only have a readme containing no text except the work readme. Could the authors please fix this?\n* Could the authors elaborate on how the guidance module shifts focus to agents instead of to observation features?\n* Could the authors please elaborate on how the decentralised execution differs from centralised training with decentralised execution in Section 5.3? \n* The term dimensional-wise state is used but not clearly defined. What do the authors mean by this and can the authors clearly contrast this with the agent-wise state which is a key concept in the work?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "* The approach to guide the reconstruction of the individual agent-wise states is interesting and using a diffusion model in reinforcement learning is also interesting.\n* The authors conduct detailed ablation studies to disentangle which aspects of their method have the greatest impact on algorithm performance. \n* The authors test their method in continuous and discrete action space environments.", "weaknesses": "* The paper has multiple grammatical errors and spelling mistakes. \n* The illustrative example in Lines 71 - 80 is difficult to follow. I think the authors are trying to say that agent $x_1$ incorrectly assumes that agent $x_0$’s coverage is represented by the 2nd and 3rd elements in the state vector instead of the correct 3rd and 4th elements. Is this what the authors mean here? Since this is an example used to illustrate the necessity of the method, I think it can be explained more clearly. \n* While it is beneficial that the authors benchmark on SMACv2 instead of only on SMAC it is still quite a limitation in the work that most of the conclusions regarding the method are drawn from results on SMAC. This together with only 3 continuous action space benchmarking tasks is a rather limited evaluation set and not very extensive as the authors mention in the paper. \n* When there is improvement over a baseline it is slight and oftentimes within one measure of error of other methods. For example, in Table 1 RGP+HPN overlaps with the results of HPN-QMIX on all scenarios and RGP overlaps with QMIX, VDN or QPLEX on all scenarios. \n* In Table 3, the error is not taken into account when computing the Performance Improving Ratio of one method over another. This sketches an incomplete picture since it only uses the mean. For a more statistically sound comparison, error propagation should be incorporated into these calculations using standard error propagation techniques. Similarly, for the Performance Retention Ratio of one method over another in Table 2. \n* In Appendix C, the authors show that using an MLP to generate the per-agent state leads to similar performance as using a diffusion model. The MLP is less computationally expensive, why do the authors not use this?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)