Multi-Agent Guided Policy Optimization
Authors: Yueheng Li
Guangming Xie
Zongqing Lu
Venue: ICLR 2026 Poster (2026)
Keywords: multi-agent reinforcement learning
teacher-student learning
centralized training with decentralized execution
Primary area: reinforcement learning

Abstract: Due to practical constraints such as partial observability and limited communication, Centralized Training with Decentralized Execution (CTDE) has become the dominant paradigm in cooperative Multi-Agent Reinforcement Learning (MARL). 
However, existing CTDE methods often underutilize centralized training or lack theoretical guarantees. 
We propose Multi-Agent Guided Policy Optimization (MAGPO), a novel framework that better leverages centralized training by integrating centralized guidance with decentralized execution. 
MAGPO uses an autoregressive joint policy for scalable, coordinated exploration and explicitly aligns it with decentralized policies to ensure deployability under partial observability. 
We provide theoretical guarantees of monotonic policy improvement and empirically evaluate MAGPO on 43 tasks across 6 diverse environments. 
Results show that MAGPO consistently outperforms strong CTDE baselines and matches or surpasses fully centralized approaches, offering a principled and practical solution for decentralized multi-agent learning.

Published as a conference paper at ICLR 2026
MULTI-AGENT GUIDED POLICY OPTIMIZATION
Yueheng Li,, Guangming Xie**, Zongqing Lu
School of Advanced Manufacturing and Robotics, Peking University "School of Computer Science, Peking University iliyueheng, xiegming, zongqing. 1utepku.... edu. cn
ABSTRACT
Due practical constraints such as partial observability and limited communica- tion, Centralized Training with Decentralized Execution (CTDE) has become the dominant paradigm in cooperative Multi-Agent Reinforcement Learning (MARL). However, existing CTDE methods often underutilize centralized training lack theoretical guarantees. We propose Multi-Agent Guided Policy Optimization (MAGPO), novel framework that better leverages centralized training by in- tegrating centralized guidance with decentralized execution. MAGPO uses an autoregressive joint policy for scalable, coordinated exploration and explicitly aligns it with decentralized policies to ensure deployability under partial observ- ability. We provide theoretical guarantees of monotonic policy improvement and empirically evaluate MAGPO on 43 tasks across 6 diverse environments. Results show that MAGPO consistently outperforms strong CTDE baselines and matches surpasses fully centralized approaches, offering a principled and practical solution for decentralized multi-agent learning.
INTRODUCTION
Cooperative Multi-Agent Reinforcement Learning (MARL) provides a powerful framework for solv- ing complex real-world problems such as autonomous driving (Zhou et al.) 2020). traffic management (Singh et al.) 2020), and robot swarm coordination (Hüttenrauch et al. 2019 Zhang et al. 102 However, MARL faces two fundamental challenges: the exponential growth of the joint action space with the number of agents, which hinders scalability, and the requirement for decentralized execution under partial observability, which complicates policy learning.
A widely adopted solution is Centralized Training with Decentralized Execution (CTDE) (Olichock al. 2008 Kraemer& & Banerjee 2016), where agents are trained using privileged global information but execute independently based on local observations. CTDE forms the foundation of many state-of-the-art MARL algorithms and typically incorporates centralized value function to guide decentralized policies or utility functions during training. This setup allows algorithms to benefit from global context without violating the constraints of decentralized deployment.
Parallel developments in single-agent RL have explored analogous ideas under the Partially Observ- able Markov Decision Process (POMDP) setting (Olichoek & Amato 2016). Two representative paradigms have emerged. The first is asymmetric actor-critic (Pinto et al. (2118), where the critic has access full state information during training while the actor is restricted to partial observations. Con- ceptually, this setup can be viewed as the single-agent counterpart of CTDE: centralized information is exploited in training but not used at execution time. The second paradigm is teacher-student learning, where teacher policy trained with privileged information provides direct behavioral supervision to student policy that must operate under partial observability. Unlike asymmetric actor actor-critic, which transfers value information, ceacher-studen methods transfer action-level knowledge, potentially enabling stronger guidance.
While asymmetric designs have long been embedded in CTDE-style MARL algorithms through centralized critics, the teacher-student paradigm has only recently been extended to multi-agent settings. This has led to the Centralized Teacher with Decentralized Student (CTDS) framework (Zhao et al. 2024),, which augments CTDE with an explicit centralized teacher policy. In CTDS, a
"Corresponding authors.

---

Published as a conference paper at ICLR 2026
teacher outputs joint actions based on the global state, and decentralized student policies trained to imitate these actions.
Although CTDS holds the promise of leveraging centralized coordination more effeecivvly than value-based CTDE methods, introduces structural challenges that are particularly pronounced in MARL. First, learning a centralized teacher over the joint action space suffers from poor scalability, as the space grows exponentially with the number agents. Second, even if a strong teacher is obtained, decentralized students may suffer from fundamental imitation gap (Weihs et al. (022]) In multi-agent settings, this gap is exacerbated by policy asymmetry: the centralized teacher conditions on global state and joint context, whereas decentralized students must act based solely on local observations. As a result, the space of realizable decentralized joint behaviors may not contain the teacher's 's strategy, leading to unavoidable performance degradation.
To overcome these limitations, we propose Multi-Agent Guided Policy Optimization (MAGPO), novel framework that bridges centralized training and decentralized execution through a principled and MARL-specific design. MAGPO addresses the scalability and policy asymmetry problem by constraining a centralized, autoregressive guider policy to remain closely aligned with decentralized learners throughout training. The guider policy allows agents to act sequentially conditioned on previous actions, utilizing global information and coordinated data collection (Wen et al., 2022] Mahjoub et al., (025). The alignment ensures that the coordination strategies developed under centralized supervision remain realizable by decentralized policies, thus mitigating the imitation gap that undermines prior CTDS approaches. Unlike direct extension of single-agent GPO (Li) iet al. 2025), MAGPO introduces structural mechanisms tailored to multi-agent learning, including sequential joint action modeling and decentraaizzaioon-alinee updates, while preserving scalability and parallelism. We provide theoretical guarantees of monotonic policy improvement and empirically evaluate MAGPO across 43 tasks in 6 diverse environments. Results show that MAGPO consistently outperforms strong CTDE baselines and even matches or exceeds fully centralized methods, estab- lishing it as a theoretically grounded and practically deployable solution for MARL under partial observability.
2 BACKGROUND
2.1 FORMULATION
We consider Decentralized Partially Observable Markov Decision Process (OccPPOMPP) (Oliehoek & Amato 2016) in modeling cooperative multi-agent tasks. The Dec-POMDP is characterized by the tuple (,S,AAA A,A, r, P, P,O,3 7.).)) where A is the set of agents, S is the set states, A is the set of actions, r is the reward function, P is the transition probability function, 2 is the individual partial observation generated by the observation function O, and y is the discount factor. At each timestep, each agent i N receives partial observation o; Z according to O(s; i) state 8 E S. Then, each agent selects an action a, E A according to its action-observation history т; E (Z x A)", collectively forming joint action denoted a. The state 8 undergoes transition the next state s' in accordance with P(s'|s ls, a). and agents receive a shared reward r. Assuming an initial state distribution p A(S), the goal is find a decentralized policy π { that maximizes the expected cumulative return:
V=
E
r'relso ~ p).
(1)
This work follows the Centralized Training with Decentralized Execution (CTDE) paradigm (Olichoek)
al. 2008 Kraemer & Banerjee 2016). During training, CTDE allows access global state to stabilize learning. However, during execution, each agent operates independently, relying solely on its local action-observation history.
In centralized training, we can therefore optimize a joint policy μuaas) that coordinates all agents while enabling joint exploration. To update this joint policy, we will consider the policy mirror descent (PMD) objective (Shani et al. 2019):
1 I- D Daptuisy (He, (un2
n (k+1) arg max
Tie (VV(VVV
2

---

Published as a conference paper at ICLR 2026
where 77 is the step size, P E A(S) is an arbitrary state distribution, dp(x(*) is the discounted state-visitation distribution under (((),, and (d,() denotes the corresponding weighted Bregman divergence. Conceptually, PMD can be viewed as class of preconditioned policy gradient methods. (KL)
Choosing the Bregman divergence to be the Euclidean distance or the Kullback-Leibe divergence recovers projected policy gradient and natural policy gradient (NPG) (Kakade (200I) respectively. Thus, PMD provides a unifying framework that encompasses many modern policy-based RI algorithms, including TRPO (Schulman al., (015a) and PPO (Schulman al., 2017). We build upon this formulation to develop our theoretical results in Section 4.1
2.2 RELATED WORKS
CTDE. CTDE methods can be broadly categorized into value-based and policy-based approaches. Value-based methods typically employ joint value function conditioned on the global state and joint action, alongside individual utility functions based local observations and actions. These functions often satisfy the Individull-Global-e
(IGM) principle (Son et al. 2019). ensuring that the optimal joint policy decomposes into locally optimal policies. This line of work is known as value factorization, and includes methods such as VDN (Sunehag et 2017), QMIX (Rashid et al.) 2020 QTRAN (Son et al. 2019), QPLEX (Wang et | [202aa), and QATTEN (Yang et al. (0220). Policy-based methods, in contrast, typically use centralized value functions to guide decentralized policies, allowing for direct extensions of single-agent policy gradient methods multi-agent settings. Notable examples include COMA (Foerster et al. 2018), MADDPG (Lowe et al., 2017). MAA2C (Papoudakis et al.) 2020) and MAPPO (Yu et al.] 2022). Additionally, hybrid methods that combine value factorization with policy-hased training have been proposed, such as DOP (Wang et al. 202 FOP (Zhang et al., 20216), and FACMAC (Peng et al., 202T). While CTDE has achieved strong empirical performance, most existing methods leverage global information only through the value function. We refer to these as vanilla CTDE methods, as they do not fully exploit the potential of centralized training.
CTDS. More recently, researchers have explored extending the teacher-student framework from single-agent settings to multi-agent systems, leading to the Centralized Teacher with Decentralized Students (CTDS) paradigm (Zho et al. [024]] Chen et al. 2024 Zhou et al., (025). In this framework, a centralized teacher policy-accesse
global state and acting jointly-collecte collects high-quality trajec- tories and facilitates more coordinated exploration. CTDS methods offer stronger supervision than vanilla CTDE methods. However, due to observation asymmetry (Weihs et al. (202I) and policy space mismatch, the learned decentralized policies may still suffer from suboptimal performance- that we explore further in the next section.
In contrast vanilla CTDE and CTDS methods-many of which lack theoretical guaran- wres-another line of research focuses on Heterogeneous Agent Reinforcement Learning (HARL), where agents are updated sequentially during training (Zhong et al. (023). This formulation under- pins algorithms such as HATRPO and HAPPO (Kuba et al. 2022) and HASAC (Liu et al., 2025). While HARL provides better theoretical guarantees and stability, it requires agents to be heteroge- neous and updated one at a time. As result, these methods lack parallelism which is important in large-scale MARL tasks and cannot exploit parameter sharing, which has proven effective in many MARL application GGupa et al. 2017 Terry al. 2020 Christianos et al. 2021a).
HARL.
СТСE. Centralized Training with Centralized Execution (CTCE) approaches treat the multi-agent system as a single-agent problem with a combinatorially large action space. Beyond directly applying single-agent RL algorithms to MARL, promising direction in CTCE has been to use transformers
(Vaswani et al. (017)) to frame multi-agent trajectories as sequences (Chen et al. 2021). This has
led to the development of powerful transformer- baasdd methods such as Updet (Hu et al. 2021).
Transfqmix (Gallici et al./ 2023), and other offline methods (Meng et al., 2023 Tseng et al. 2022
Zhang et al. 2022). Two representative online methods are Multi-Agent Transformer (MAT) (Wnn
et al. 2022) and Sable (Mahjoub et al. 2025), ), which currently achieve state-of-the-as performance in
cooperative MARL tasks. CTCE methods offer strong theoretical guarantees (Wen et al, 2022) and
impressive empirical results. However, they fall short in practical settings that demand decentralized
execution, where each agent must act based solely its local observation and policy.
3

---

Published as a conference paper at ICLR 2026
Update
(A) CTDE
10
10
3
3
Update
(B) CТСE
10
10
7-X
Distill
(C) CTDS
10
10
Figure 1: Illustrative example showing three different MARL settings.
3 PROBLEMS OF CURRENT CTDS
In this section, we examine the limitations of using a centralized teacher to supervise decentralized student policies. Two key challenges arise in this setting: asymmetric observation spaces and asymmetrie policy spaces.
The first challenge-
laallengge-smmmmrrii observation spaces-is is shared with single-agent POMDPs involving
privileged information and has been extensively studied in prior work (Warrington et al. 2020 Weihs et al. 2021 Shenfeld et al. 2023 Li et al. 2025). When the teacher relies on privileged informatinn that unavailable (and not inferable) to the student, the student cannot faithfully reproduce the teacher's behavior. Instead, it learns to approximate the conditional expectation of the teacher's 's action given the observable input (Weihs al. 2021 Warrington et al. 2020). This typically results in an "averaged" behavior that can be significantly suboptimal.
The second challenge-asynnnne
policy spaces-is unique the multi-agent setting. arises from the structural mismatch between the teacher's policy (typically joint and expressive) and the students' policies (factorized and decentralized). We illustrate this challenge through a simple example shown in Figure [] Consider cooperative task where three agents must each output an integer such that their sum equals a target value of 10. Each agent acts once, and the system succeeds only if the total sum is exactly 10. We compare three MARL frameworks:
(A) Vanilla CTDE, Agents share a centralized value function but act independently via decentralized policies. Suppose all three agents use the same deterministic policy (00- = 3. producing a total of which fails the task. Because each agent observes the same global state and optimizes the same objective, they may all simultaneously increase their action to 4 in the next update, producing 12, still failing. Lacking inter-ages agent coordination signals, the agents struggle to determine which one should adjust its action. This leads to classic miscoordination. requiring random trial-and-eroor exploration and memorization of rare successful configurations to eventually coordinate.
(B) CTCE. Agents act sequentially, observing previous agents' actions before choosing their own. Suppose the first agent updates action to 4 and the second still picks 3. The third agent, having observed both previous actions, selects and achieves the correct total. Sequential execution effectively transforms the multi-agent coordination problem into a single-agent decision-making process over joint policy. This makes coordination straightforward and stable, without repeated random exploration. However, this setting assumes centralized execution, which is often infeasible in real-world applications requiring decentralization.
(C) CTDS. Now consider distilling successful CTCE policy from (B) into decentralized student policies. If the teacher's policy deterministic and factorizable (e.g., always producing [4, 3, 3), CTDS can recover an optimal decentralized solution. However, a CTCE teacher may exploit stochastic strategies. For instance, the first agent samples {3, 4] at random, the second agent always outputs 3, and the third agent-having seen the first two actions-outputs 7 ensuring the total is always
10. While this is optimal under CTCE, cannot be factored into independent policies: CTDS would learn two independent stochastic policies for the first and third agents, leading to failures such as
[4, 3, 4]. If the first CTCE agent selects 3 or 4 with equal probability, the distilled decentralized policy succeeds only 50% of the time.
4

---

Published as a conference paper at ICLR 2026
This example highlights the core failure mode: coordination patterns encoded in the teacher's joint policy are lost when forced into a decentralized representation. To address these limitations, we propose new approach that constrains the teacher's 's policy during training, preventing it from exploiting unrepresentable coordination strategies while still allowing to guide decentralized leamners effectively.
4 METHOD
We introduce Multi-Agent Guided Policy Optimization (MAGPO), framework that leverages centralized, sequentially executed guider policy to supervise decentralized learners while keeping them closely aligned. MAGPO is designed to combine the coordination benefits of centralized training with the deployment constraints of decentralized execution.
We begin by presenting the theoretical formulation and guarantee of monotonic policy improvement in the tabular setting. For simplicity, we initially assume full observability-ia. all agents observe the global state reducing the setting cooperative Markov game (Littman 1994). We will return to the partially observable case in the subsequent implementation section.
4.1 MUL.TI-GGENT GUIDED POLICY OPTIMIZATION
Our algorithm maintains centralized guider policy with an autoregressive structure agent actions: u{ass) = μe1 (a" (s)x²² (a'a a²t) n (a'n [s, |s afim-)), where 81.m (with m S n) denotes an ordered subset {ū1, ... } of the agent set N, specifying the execution order. The decentralized learner policy is defined as: π (als) = (a*)s) for any ordering ii:., implying that all agents act independently.
Building on this structure, MAGPO optimizes the centralized guider and decentralized learner policies through an iterative four-step procedure inspired by the GPO framework (Li et al. 2025):
Data Collection: Roll out the current guider policy to collect trajectories.
Guider Training: Update the guider μ4x to fag by maximizing RL objective.
Learner Training: Update the learner ाB to T& by minimizing the KL distance DRI. fts))
Guider Backtracking: Set 14++1 = T for all states 8.
The first step allows MAGPO to perform coordinated exploration using joint policy. the second
step, the guider is updated using the Policy Mirror Descent (PMD) framework (Xiao 2022). which
solves the following optimization:
f arg max (m ((()))) DKL (xss
(3)
where Que is the Q-function of guider and 7 is the learning rate. As discussed in Section 2. PMD a general policy gradient framework that subsumes popular algorithms such as PPO and TRPO. Here, we adopt PMD for theoretical clarity and instantiate it using PPO-style updates in practical implementation. Additional details are provided in Appendix A In the final step, we perform guider backtracking, where the guider is reset to the current learner policy. Theoretically, this is always feasible since any decentralized policy π defines a valid autoregressive joint policy fe by simply ignoring the conditioning on past actions.
Based on the framework introduced above, we have the following theorem for MAGPO.
Theorem 4.1 (Monotonic Improvement of MAGPO). Let (T=) be the sequence of joint learner policies obtained by iteratively applying the four steps of MAGPO. Then,
Vo(T++1) 2 V)(T)), Ak,
(4)
where Vp is the expected return under initial state distribution p.
Proof. See Appendix A
5

---

Published as a conference paper at ICLR 2026
In contrast to CTDS and standard CTDE methods like MAPPO, MAGPO provides provable guarantee of policy improvement. This result can be understood intuitively: the guider identifies a policy that improves return in the full joint space using PMD. The learner then projects this policy into the decentralized policy space via KI. minimization. Since the target was chosen via projected gradient, the resulting learner policy also improves return.
To further clarify the structure of MAGPO, we show that its learner updates can be interpreted sequential advantage-based updates-a procedure known ensure monotonic improvement in multi-agent settings (Kuba et al. 2022). We begin with the following lemma:
Lemma (Multi-Agent Advantage Decomposition (Kuba et al. 2022 In any cooperative Markov game. given a joint policy æ. for any state s, and any agent subset 1imm the following equations hold:
A (8, a ftom
A'd (s. (1s.a"t a ,aat),
where
A (s, hi a ammm) A Quickifio (s,, a abtim) - Qhiskep (s, aluik)
(6)
for disjoint sets Jisk and him The state-action value function for subset defined as
Quamn (s, a ammm)
E Ba-mmm nu "1im [Q(s, a from a a=hmm)
Using this, we derive the following:
Corollary 4.2 (Sequential Update of MAGPO), The update for any individual policy whb with ordered subset i1 can be written as:
T "E+1
arg max E
My b41
Add (s, at10-,
(8)
n
7K
Proof. See Appendix A
This shows that MAGPO's learner updates are equivalent to performing sequential advantage- weighted policy updates. Importantly, unlike methods such as HARL which update agents one at time, MAGPO allows for simultaneous updates of all agent policies. This enables parallel training and improves scalability to large agent populations. Moreover, HARL requires heterogencous agents to guarantee policy improvement, while MAGPO works with either homogeneous heterogeneous agents, allowing to benefit from parameter sharing-a widely adopted practice that significantly improves efficiency and generalization in MARL (Gupta et al. 2017 Terry et al.) 2020 Christianos
et al. 2021a).
4.2 PRACTICAL IMPLEMENTATION
In this subsection, we describe the practical implementation of MAGPO. Our implementation is based on the original GPO-clip framework (Li al.) 2025), extended to the multi-agent setting. The key difference is that the guider in MAGPO is sequential execution policy. Since MAGPO is compatible with any autoregressive CTCE method, we do not specify the exact encoder, decoder, or attention mechanisms used. Instead, we present general training objectives for both the guider and learner components.
Guider Update. As introduced in the previous section, the guider policy (parameterized by Φ) is first optimized to maximize the RL objective, and then aligned with the learner policy. This is achieved via an RL update augmented with a KL. constraint:
((Ø)
min (72 clip, (0),
Tn
(9)
m (8) (5) (1 (.1se, (1s-1)) v/ (1101))]

---

Published as a conference paper at ICLR 2026
where
Ma to "o |sc, as )
Mo [a Scc a
$ (200))
r3 (0)
my m2((6) (6) =
M M (a) |1,, at 1
is
To
(a, 100
and
P (a To (a) (o,)) 8
(1:-11 ser at
1
# 10 Po (a [a/l|s a, )-:--) 1-,, -
clip(r) (0), ()) = clip clip
c,
This objective has two modifications compared to the standard one: double clipping function :iip(;,e,d 6) and a mask function m² (6), both controlled by a new hyperparameter 8 > 1. which bounds the ratio between guider and learner policies within (3 5). The inner clip in the double clipping function stops the gradient when the advantage signal encourages the guider to drift too far from the learner. The mask function ensures the KI. loss is only applied when this ratio constraint violated. The advantage estimate A, is computed via generalized advantage estimation (GAE) (Schulman et al. 2015b) with value functions.
Learner Update. The learner policy æ, parameterized by 0, updated with two objectives: (i) behavior cloning toward the guider policy, and (ii) an RL auxiliary term to directly improve return from the collected trajectories.
T
(()) = Tn
DKL (TTz (-lo,' Du (-1se, fu: 1))-))))-
j=1
(10)
Amin
where
(0) Tg (0
(11)
M (a (1c, a. 1
The auxiliary RL objective helps maximize the utility of collected trajectories. Since the behavior policy (guider) is kept close the learner, this term approximates an on-policy objective. In principle. we could apply sequential updates to each individual pplicy--analoguu to HAPPO to preserve the theoretical guarantees of monotonic improvement. However, this makes the learner updates non- non-parall aan-parallelizable and incompatible with parameter sharing. Since no performance benefits are observed from using HAPPO, we instead adopt a MAPPO-style update: all learners share parameters and are updated jointly and in parallel. The auxiliary RL term can be treated as optional and controlled by X.
EXPERIMENTS
We evaluate MAGPO by comparing against several SOTA baseline algorithms from the literature. Specifically, we consider two CTCE methods- -abil (Mahjoub al. 2025) and MAT (Wen et al.) 2022) two CTDE Buselincs-MAPE vanilla implementation of on-policy CTDS, which be viewed as MAGPO without double clipping, masking, and the RL auxiliary loss. For the joint policy in both MAGPO and CTDS, we use Sable as the default backbone. All algorithms are implemented using the JAX-based MARL library Mava (de Kock et al. 2023).
(Yu et al. 2022) and HAPPO (Kuba et al. 2022) and
Evaluation Hyperparameters. We follow the evaluation protocol from Mahjoub al. (025)) Each algorithm is trained with 10 independent seeds per task. Training is conducted for 20 million environment steps, with 122 evenly spaced evaluation checkpoints. At each checkpoint, we record the task-specific metrics (e.g., mean episode return and win rate) over 32 evaluation episodes. For task-level results, we report the mean and 95% confidence intervals. For aggregate performance across entire environment suites, report the min-max normalized interquartile mean (IQM) with 95% stratified bootstrap confidence intervals. The hyperparameters are tuned on each task for each algorithm, which are detailed in Appendix

---

Published as a conference paper at ICLR 2026
MAGPO
CTDS
MAPPO
HАPPO
MAT
SABLE
1.0
episode return
1.0
1.0
0.8
0.8
0.8
Mean episode
0.6
0.6
0.6
0.4
0.4
0.44
0.2
0.2
0.2
0.0
0.0
0.5
1.0
2:0
0.0
0.0
0.5
1.0
1.5
2:0
0.0
0.0
0.5
1.0
1:5
2.0
Timesteps
Timesteps
Timesteps
(a) CoordSum
(b) LevelBasedForaging
(c) MaConnector
Mean agisode retur
2.0
1.0
1.0
0.8
0.6
0.4
0.2
0.0
1.0 Timesteps
0.0
1.0 Timesteps
1:5
Timesteps
(d) MPE
(c) Robootaachouse
(f) Smax
Figure 2: The sample efficiency curves aggregated per environment suite, where dashed lines represent the CTCE methods. For each environment, results are aggregated over all tasks and the min-max normalized inter-quartile mean with 95% stratified bootstrap confidence intervals shown.
SABLE MAT MAPPO
CTDS
0.30
0.5
0.75
Figure 3: The overall aggregated probability of improvement for MAGPO compared to other baselines for that specific environment. A score of more than 0.5 where confidence intervals are also greater than 0.5 indicates statistically significant improvement baseline for given environment (Agarwal et al. 2021
Environments. We evaluate MAGPO on diverse suite of JAX -bxsed multi-agent benchmarks, including 4 tasks in CoordSum (introduced in this paper). 7 tasks in Level-based foraging (LBF) (Christianos al. 2021bb, tasks in Connector (Bonnet al. 2024 3 tasks in the Multi-agent Parti- cle Environment (MPE) Lowe et al., 2017). 15 tasks in Robotic Warehouse (RWARE) (Papoudakis) al. 2020), and 10 tasks The StarCraft Multi-Agent Challenge in JAX (SMAX) (Rutherford et al. 2024.. The CoordSum environment, introduced in this paper, reflects the didactic examples discussed Section 3 where agents must coordinate to output integers that sum to a given target without using fixed strategies. A detailed description is provided in Appendix B
5.1
MAIN RESULTS
Figure preeents the per-environment aggregated sample-efficiency curves. Our results show that MAGPO achieves state-of-the-ee performance across all CTDE methods and even outperforms CTCE methods on a subset of tasks. Specifically, MAGPO surpasses all CTDE baselines on 32 out of 43 tasks, and outperforms all baselines on 20 out of 43 tasks. Figure 3 reports the probability of improvement of MAGPO over other baselines. MAGPO emerges as the most competitive CTDE method and performs comparably to the SOTA CТСE method Sable in three benchmark environments. Comparing MAGPO to CTDS reveals a significant performance gap in the CoordSum and RWARE domains, suggesting that in these environments, the CTCE teacher may learn policies that not decentralizable- www.riiiaabeeeeeeeinnnnnn direct policy distillation ineffective, Additional tabular results and environment/hakk-leee aggregation plots provided in Appendix C
8

---

Published as a conference paper at ICLR 2026
Mon OM0-1 Roam 5 C
-30
Treettpp
Timeiaaoe
Timeitape
(a) MAGPO with different CTCE methods as guider
(b) MAGPO with different 8
Figure 4: MAGPO performance varies with the choice of guider and the regularization ratio 5.
Duatitn OvetSum-3130-55 2030-50
Lagu-4a--2 4ag-tard
CoordSur--ae 20-30-000000
Ury-2ag
22
03
02
03
23
03
1.0
77
03
(a) MAGPO with different
(b) CTDS with different auxiliary loss weighted by À
Figure 5: The effect of RL auxiliary loss.
5.2 ABLATIONS AND DISCUSSIONS
In this subsection, we discuss several key aspects and design choices of MAGPO.
Bridging CTCE and CTDE. MAGPO's performance intuitively depends heavily on the capa- bility of the guider, which corresponds to the performance of the underlying CTCE method. In Figure a)), we evaluate MAGPO on two tasks where Sable and MAT exhibit different performance. In simple_spread_O0 10ag. MAT performs significantly worse, resulting in poor performance of MAGPO when using MAT as the guider. In contrast, on large-Sag, MAT outperforms Sable, leading to better performance of MAGPO with MAT. This dependency could be seen as a limitation, but it actuall serves as a core feature: MAGPO effectively bridges CTCE and CTDE. In many practical applications that require decentralized policies, MAGPO enables advances in CTCE methods to directly benefit CTDE methods as well-facilitating the co-development of both paradigms.
Effect of the Ratio d. MAGPO introduces a hyperparameter ó to regulate the guider's deviation from the learner, which most strongly influences its performance. A smaller ó enforces a stricter constraint, keeping the guider closer to the learner policy; larger ō allows the guider more freedom, potentially enabling it to explore regions of the policy space that are difficult or even unreachable under decentralized constraints. In Figure 4b). we assess MAGPO's s performance under varying 5 values on two tasks. In (hordSum-5x20-40 centralized guider tends to learn a policy that is not decentralizable, which must be restricted to improve imitability. Conversely, in medium-4ag-hun 4ag-hard, the guider policy is more directly imitable, and restricting it too tightly hinders lcarning. These observations show the importance tuning based on the task's structure and imitation feasibility.
smaller 8 yields better performance because the
Effect of RL Auxiliary Loss. MAGPO incorporates an RL auxiliary loss in the learner update to better utilize collected data and stabilize learning. This component is not as important as the but a properly tuned À can also improve performance, as shown in Figure a)) To understand this, consider the guider's RL objective is towards an undecentralizable direction and the learner pulls backward (due to the imitation constraint), then this back-and-forth may repeatedly stall progress. By incorporating RL updates, the learner can crounter-supervvne the guider, helping it discover more decentralizable update directions. Furthermore, in Figure 5]b,, we test applying the same RL auxiliary loss to a CTDS method. The results show limited benefit. This is because in CTDS, the behavioral policy is the teacher, which is not enforced align with the student. If the teacher-student

---

Published as a conference paper at ICLR 2026
gap is too large, the collected data is off-policy, thus on-policy RL loss the student provides little benefit.
Observation asymmetry. While most of our analysis has focused on asymmetries the policy and action spaces, observation asymmetry is equally critical. In the current framework, CTCE methods like MAT and Sable condition on the union of agents' partial observations, whereas individual policies are limited to their own local views. This mismatch creates an imitation gap, making direct imitation methods like CTDS fail, even when the underlying joint policy is decentralizable. MAGPO addresses this issue similar to the single-agent setting (Li et al. (025), by controlling the divergence between the guider and the learner through the parameter ö. In addition, privileged information- the union of partial observations- is often available during centralized training (e.g., the true global state), although we do not explore it in this paper. Providing such privileged signals to the guider could further enhance its ability supervise decentralized policies under partial observability.
Model capacity. In addition to pol- icy and observation asymmetry, asym- metry can also arise from mismatched model capacity between training and deployment. In many practical sys- tems, a high-capacity centralized model teacher is used during train- ing, while compact decentralized policy must be deployed due to com- pute or latency constraints (e.g., large LLMs distilled for real-time infer- ence, iision-angguage-action mod- els paired with lightweight controllers for high-frequency control). MAGPO naturally accommodates this setting by explicitly constraining the central- ized guider to remain imitable by de- centralized actors throughout training. To evaluate this property, we conduct additional experiment on Amacv2_5_umits where all training-time networks (e.g., critics and centralized policies) are kept at full capacity, while only the hidden dimension of the deployable decentralized actors is reduced at evaluation time. This setup simulates a realistic distillation scenario in which a large teacher must be compressed for deployment. As shown in Figure 6 MAGPO consistently outperforms CTDS across all evaluated actor capacities, and the performance gap widens as the deployed actor becomes smaller. Compared MAPPO, both MAGPO and CTDS CTDS-being seacher-student Tramewors-ttranse knowledge from large models more effectively than value-based supervision alone. Importantly, MAGPO degrades more gracefully under severe capacity constraints, indicating that constraining the teacher to remain aligned with decentralized rcalizability improves robustness to deployment compression.
smacv2 5 units
MAGPO
0.8
CTDS
MAPPO
0.5
Win
0.4
0.
0.1
Hidden Size
Figure 6: Effect of model capacity on performance.
6 CONCLUSION
We presented MAGPO, a novel framework that bridges the gap between CTCE and CTDE in cooperative MARL. MAGPO leverages sequentially executed guider for coordinated exploration while constraining it to remain close to the decentralized learner policies. This design enables stable and effective guidance without sacrificing deployability. Our approach builds upon principles of GPO and introduces practical training algorithm with provable monotonic improvement. Empirical results across 43 tasks in 6 diverse environments demonstrate that MAGPO consistently outperforms state-of-the-art CTDE methods and is competitive with CTCE methods, despite relying on decentralized execution.
ACKNOWLEDGMENTTS
This work was supported in part by the National Natural Science Foundation of China [grant numbers U22A2063 2062,, U23B2037, 12272008, 62450001, 62476008).
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Mare Bellemare. Deep reinforcement learning the edge of the statistical precipice. Advances in neural information processing systems, 34:29304-29322 2021.
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next-generation hyperparameter optimization framework. In The 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. pp. 2623-2631, 2019.
Clément Bonnet, Daniel Luo, Donal Byrne, Shikha Surana, Sasha Abramowitz, Paul Duckworth, Vincent Coyette, Laurence I. Midgley, Elshadaí Tegegn, Tristan Kalloniatis, Omayma Mahjoub, Matthew Macfarlane, Andries P. Smit, Nathan Grinsztajn, Raphael Boige, Cemlyn N. Waters, Mohamed Mimouni, Ulrich A. Mbou Sob, Ruan de Kock, Siddarth Singh, Daniel Furelos- Blanco, Victor Le, Arnu Pretorius, and Alexandre Laterre. Jumanji: diverse suite of scalable reinforcement learning environments in jax, 2024. URL hothttss:/// /ttp://aaavv../..//.a//e.....aia wwwpziivvorrgaeooo abs /206 09884
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:15084-15555 2021.
Yiqun Chen, Hangyu Mao, Jiaxin Mao, Shiguang Wu, Tianle Zhang, Bin Zhang, Wei Yang, and Hongxing Chang. Ptde: Personalized training with distilled execution for multi-agent reinforcement learning, 2024. URL ht.tp::::: /ttp:::///avvvvvov
abs 221 088772
Filippos Christianos, Georgios Papoudakis, Arrasy Rahman, and Stefano V. Albrecht. Scaling multi- agent reinforcement learning with selective parameter sharing, 2021a. URL https:///xxxi
brg/abs/2102. 07475
Filippos Christianos, Lukas Schäfer, and Stefano V. Albrecht. Shared experience actor-critic for multi-agent reinforcement learning, 2021b.
07166
Cédric Colas, Olivier Sigaud, and Pierre-Yvel Oudeyer. Gep-pg: Decoupling exploration and exploitation in deep reinforcement learning algorithms, 2018. URL htttttss///axiv..oo
abs Abs/1802.05054 05054
Ruan de Kock, Omayma Mahjoub, Sasha Abramowitz, Wiem Khlifi, Callum Rhys Tilbury, Claude Formanek, Andries P. Smit, and Arnu Pretorius. Mava: research library for distributed multi- agent reinforcement learning in jax. arXiv preprint arXiv:2107.01460, 2023. URL https
//axxiv.orr/pdf/22000 071.60000..11.........
Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Proceedings of the AAAI conference artificial intelligence, volume 32, 2018.
Matteo Gallici, Mario Martin, and Ivan Masmitja. Transfqmix: Transformers for leveraging the graph structure of multi-agent reinforcement learning problems. arXiv preprint arXiv:2301.0555 2023.
Rihab Gorsane, Omayma Mahjoub, Ruan de Kock, Roland Dubb, Siddarth Singh, and Arnu Pretorius.
Towards a standardised performance evaluation protocol for cooperative marl, 2022. URL https
7/arxiv. www.v/orggeee. /a.pa////22..mm 10485
Jayesh K Gupta, Maxim Egorov, and Mykel Kochenderfer. Cooperative multi-agent control using deep reinforcement lcarning. In Autonomous Agents and Multiagent Systems: AAMAS 2017 Workshops, Best Papers, São Paulo, Brazil, May 8-12, 2017, Revised Selected Papers 16, pp. 66-83. Springer, 2017.
Siyi Hu, Fengda Zhu, Xiaojun Chang, and Xiaodan Liang. Updet: Universal multi-agent rein- forcement learning via policy decoupling with transformers. arXiv preprint arXiv:2101. 1220110.0000 1000000 2021.
Maximilian Hüttenrauch, Adrian Šošić, and Gerhard Neumann. Deep reinforcement learning for swarm systems. J. Mach. Learn. Res,, 11116-19999, January 2019. ISSN 1532-4435.
11

---

Published as a conference paper at ICLR 2026
Sham Kakade. A natural policy gradient. In Proceedings of the 15th International Conference on Neural Information Processing Systems: Natural and Synthetic, NIPS' 01, pp. 1531-1552 Cambridge, MA, USA, 2001. MIT Press.
Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning, In Proceedings of the Nineteenth International Conference on Machine Learning, ICML '02, pp. 267-274, San Francisco, CA, USA, 2002. Morgan Kaufmann Publishers Inc. ISBN 1558608737.
Landon Kraemer and Bikramjit Banerjee. Multi-agent reinforcement learning as a rehearsal for decentralized planning. Neurocomputingg 190:82-95 94, 2016.
Jakub Grudzien Kuba, Ruiqing Chen, Muning Wen, Ying Wen, Fanglei Sun, Jun Wang, and Yaodong Yang. Trust region policy optimisation multi-agent reinforcement learning, 2022. URL https
//w.xiv.cggaabs//// 2tt.///bb/221991... 11251
Yueheng Li, Guangming Xie, and Zongging Lu. Guided policy optimization under partial observabil- ity, 2025. URL httpt:::://aar http:/awwwv....raaaaaaa wrg. abs 250.
Michael Littman. Markov games as a framework for multi-agent reinforcement learning. In Machine learning proceedings 1994, pp. 157-163. Elsevier, 1994.
Jiarong Liu, Yifan Zhong, Siyi Hu, Haobo Fu, Qiang Fu, Xiaojun Chang, and Yaodong Yang. Maximum entropy heterogeneous-agemi reinforcement learning, 2025. /ttpstsss: //arxiv org/abs 2306.10711 10715
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-compettiv environments. Advances in neural information processing systems, 30, 2017.
Omayma Mahjoub, Sasha Abramowitz, Ruan John Kock, Wiem Khlifi, Simon Verster Du Toit, Jemma Daniel, Louay Ben Nessir, Louise Beyers, Juan Claude Formanek, Liam Clark, al. Sable: a performant, efficient and scalable sequence model for marl. In Forty-second International Conference Machine Learning, 2025.
Linghui Meng, Muning Wen, Chenyang Le, Xiyun Li, Dengpeng Xing, Weinan Zhang, Ying Wen, Haifeng Zhang, Jun Wang, Yaodong Yang, et al. Offline pre-trained multi-agent decision transformer. Machine Intelligence Research, 20(2):233-244 2023.
Frans A Oliehoek and Christopher Amato. A concise introduction to decentralized POMDPs. POMeM Springer, 2016.
Frans A Oliehoek, Matthijs TJ Spaan, and Nikos Vlassis. Optimal and approximate q-value functions for decentralized pomdps. Journal of Artificial Intelligence Research, 32:289-353, 2008
Georgios Papoudakis, Filippos Christianos, Lukas Schäfer, and Stefano V Albrecht. Benchmark- ing multi-agent deep reinforcement learning algorithms in cooperative tasks. arXiv preprint arXiv:2006.07869 2020.
Bei Peng, Tabish Rashid, Christian Schroeder de Witt, Pierre-Alexanrre Kamienny, Philip Torr, Wendelin Böhmer, and Shimon Whiteson. Faemac: Factored multi-agent centralised policy gradients. Advances in Neural Information Processing Systems, 34:1 1208-1 2021.
Lerrel Pinto, Marcin Andrychowicz, Peter Welinder, Wojeiech Zaremba, and Picter Abbeel. Asym- metric actor critic for image-based robot learning. RSS, 2018.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Monotonic value function factorisation for deep multi-agent reinforcement learning. The Journal of Machine Learning Research, 21 21(1):7234-724, 2020.
Alexander Rutherford, Benjamin Ellis, Matteo Gallici, Jonathan Cook, Andrei Lupu, Gardar In- gvarsson, Timon Willi, Ravi Hammond, Akbir Khan, Christian Schroeder de Witt, Alexandra Souly, Saptarashmi Bandyopadhyay, Mikayel Samvelyan, Minqi Jiang, Robert Tjarko Lange, Shimon Whiteson, Bruno Lacerda, Nick Hawes, Tim Rocktaschel, Chris Lu, and Jakob Nico- laus Foerster. Jaxmarl: Multi-agent rl environments and algorithms in jax, 2024, URL https://arra http:////exvv..rg/////////// 10090
12

---

Published as a conference paper at ICLR 2026
John Schulman, Sergey Levine, Philipp Moritz, Michael I. Jordan, and Pieter Abbeel. Trust region policy optimization. CoRR, abs/1502.05477, 2015a. URL [RLl:ttt://axxii/ wwtp
05477
John Schulman, Philipp Moritz, Sergey Levine, Michael I. Jordan, and P. Abbeel. high-dimensional continuous control using generalized advantage estimation. CoRR, abs/1506.02438, 2015b. URL
https: /tpps///app semanticschoooo..o wrtg/orrpsstDCC 3075444
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, 01011707706447 2017. URL httt::/arxiv. www.:vi.orrgaa...
1707. 06347
Lior Shani, Yonathan Efroni, and Shie Mannor. Adaptive trust region policy optimization: Global convergence and faster rates for regularized mdps. CoRR, 0abs/1999.02665 2019. URL http
//arxiv.org/ abs///b///// 909. 02769
Idan Shenfeld, Zhang- Wei Hong, Aviv Tamar, and Pulkit Agrawal. TGRL:
An algorithm for teacher guided reinforcement learning. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 31077-31093, PMLR, 23-29 Jul 2023. URL httt::://ppocceeinng www.mpprre//v222222.. ishenfeld23ahtml
Arambam James Singh, Akshat Kumar, and Hoong Chuin Lau. Hierarchical multiagent reinforcement learning for maritime traffic management. In Proceedings of the 19th International Conference on Autonomous Agents and Multiagent Systems, AAMAS '20 Auckland, New Zealand, May 9-13, 2020, pp. 1278-1286. International Foundation for Autonomous Agents and Multiagent Systems, 2020. doi: 10.5555/3398761.3 13399999.
Kyunghwan Son, Daewoo Kim, Wan Kang, David Earl Hostallero, and Yung Yi. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In International conference on machine learning, pp. 5887-5896. PMLR, 2019.
Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Mare Lanctot, Nicolas Sonnerat, Joel Z. Leibo, Karl Tuyls, et al. Value-decommosition networks for cooperative multi-agent learning. arXiv preprint arXiv: 0ee./70660.9999 2017.
Justin K Terry, Nathaniel Grammel, Sanghyun Son, Benjamin Black, and Aakriti Agrawal. Revisiting parameter sharing in multi-agent deep reinforcement learning. arXiv preprint arXiv: 1362. 2020.
Wei-Cheng Tseng,
Johnson Wang, Yen-Chen Lin, and Phillip Isola. Offline multi-agent reinforcement learning with knowledge distillation. Advances in Neural Information Processing Systems, 35:226 237, 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang QPLEX: duplex dueling multi-agent g-lcarning. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021a.
Yihan Wang, Beining Han, Tonghan Wang. Heng Dong, and Chongjie Zhang. DOP: off-policy multi- agent decomposed policy gradients. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021b
Andrew Warrington, Jonathan Wilder Lavington, A. Scibior, Mark W. Schmidt, and Frank D. Wood. Robust asymmetric learning in pomdps. In International Conference on Machine Learning, 2020. URL RRlLhttps: //api. peapi.semmannee semanticschi a http:hhoo....o.oooooooooottt///2
13

---

Published as a conference paper at ICLR 2026
Luca Weihs, Unnat Jain, lou-Jen Liu, Jordi Salvador, Svetlana Lazebnik, Aniruddha Kembhavi, and Alexander Schwing. Bridging the imitation gap by adaptive insubordination. In Proceedings of the 35th International Conference Neural Information Processing Systems, NIPS '21, Red Hook, NY, USA, 2021. Curran Associates Inc. ISBN 13843993..
Muning Wen, Jakub Grudzien Kuba, Runji Lin, Weinan Zhang, Ying Wen, Jun Wang, and Yaodong Yang. Multi-agent reinforcement learning is a sequence modeling problem. Proceedings of the 36th International Conference Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 978171387 1088.
Lin Xiao. On the convergence rates of policy gradient methods. Journal of Machine Learning Research, 23(282):1 2022. URL
html
Yaodong Yang, Jianye Hao, Ben Liao, Kun Shao, Guangyong Chen, Wulong Liu, and Hongyao Tang. Qatten: A general framework for cooperative multiagent reinforcement learning. arXiv preprint arXiv:2002.03939, 2020.
Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang. Alexandre Bayen, and Yi Wu. The surprising effectiveness of PPO in cooperative multi-agent games. Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
Fuxiang Zhang, Chengxing Jia, Yi-Chen Li, Lei Yuan, Yang Yu, and Zongzhang Zhang. Discover- ing generalizable multi-agen agent coordination skills from multi-task offline data. In The Eleventh International Conference on Learning Representations, 2022.
Tianhao Zhang, Yueheng Li, Shuai Li, Qiwei Ye, Chen Wang, and Guangming Xie. Decentralized circle formation control for fish-like robots in the real-world via reinforcement learning. In 202/ IEEE International Conference on Robotics and Automation (ICRA), pp. 8814-8820. IEEE, 2021a.
Tianhao Zhang, Yueheng Li, Chen Wang, Guangming Xie, and Zongqing Lu. Fop: Factorizing optimal joint policy of maxximum-entrope multi-agent reinforcement learning. In International Conference Machine Learning, pp. 12491-12500. PMLR, 2021b.
Jian Zhao, Xunhan Hu, Mingyu Yang. Wengang Zhou, Jiangcheng Zhu, and Houqiang Li. Ctds: Centralized teacher with decentralized student for multiagent reinforcement learning. IEEE Transactions on Games, 16(1):140-150, 2024. doi: 10.1100997G222223223339.
Yifan Zhong, Jakub Grudzien Kuba, Xidong Feng, Siyi Hu, Jiaming Ji, and Yaodong Yang. Heterogeneous-ageen reinforcement learning, 2023. https: wwg.//iww.ww//os.so.o///reb /2304
09870
Ming Zhou, Jun Luo, Julian Villela, Yaodong Yang, David Rusu, Jiayu Miao, Weinan Zhang, Montgomery Alban, Iman Fadakar, Zheng Chen, Aurora Chongxi Huang, Ying Wen, Kimia Hassanzadeh, Daniel Graves, Dong Chen, Zhengbang Zhu, Nhat M. Nguyen, Mohamed Elsayed, Kun Shao, Sanjeevan Ahilan, Baokuan Zhang, Jiannan Wu, Zhengang Fu, Kasra Rezace, Peyman Yadmellat, Mohsen Rohani, Nicolas Perez. Nieves, Yihan Ni, Seyedershad Banijamali, Alexan- der Imani Nowen-Rivers. Zheng Tian, Daniel Palenicek, Haitham Bou-Ame Ammar, Hongbo Zhang, Wulong Liu, Jianye Hao, and Jun Wang. Smarts: Scalable multi-agent reinforcement learning training school for autonomous driving. In Conference on Robot Learning, 2020.
Yihe Zhou, Shunyu Liu, Yumpeng Qing. Kaixuan Chen, Tongya Zheng, Jie Song, and Mingli Song. Is centralized training with decentralized execution framework centralized enough for marl?, 2025. URL http::ttp//..... /tattps///axiv..o
17352
THE USE OF LARGE LANGUAGE MODELS (LLMs)
L.MMs used to polish the paper writing.
14

---

Published as a conference paper at ICLR 2026
A
PROOFS
We will consider policy mirror descent (PMD) objective
(Shani et al. 2019):
arg max 1 (k+
Thk (VV(
11) 1
De Daplutss (Le, u (1
(12)
1
where 7/ is the step size, E ASS) is an arbitrary state distribution and dp (p(())) is the discounted state-viste visitation distribution under the policy pl(),, Daplacor is the weighted Bregman divergence. Considering that
(VV)(u ), = Σ (V.Vo(ul (,((-1))
(13)
sES
obtain
(k+2) arg max
(ol (Qu (8,)),u((16)) ( D(u(-1b), (100))))
k
#ES
(14)
(m (Quil) (s))))((1s)) - D(u(-Is), (1))
arg max
#ES
In this paper, we use KL divergence as special case of Bregman divergence.
Theorem A.1 (Monotonic Improvement of MAGPO). A sequence (Th ===0 of joint policies updated by the four step of MAGPO has the monotonic properly:
Vo(T+++1 (T(T),, Vk.
(15)
Proof. Following the derivation from Xiao 2022), the PMD objective is
L = arg max {T (Q** (s, ), ((-1s))) - DKL (M(-1s))) (x(H))))
(16)
which admits the closed-form solution
ü = (als) exp (n Qu (s, a))
(x)
(17)
exp (1)1 Q a))
(als)
(x)
where we replace 14g with næ due to the backtracking step.
Next, the learner update is defined as
t (-1s) = arg min (T(-()) (i(1s)).
(18)
*
which guarantees the KL divergence decreases:
DKL( (-|s), (+ (((+s)) 2 DKL ( (a(-(s) (11s))
(19)
Exp log t - -log m2 (s, a) 2 Ext-1 log k-1 * - log n. m=Q == (s,a)
(20)
mi B ( (p (s, m) Eyp 1Qmh as) DRI.(T k+1 (+1x), - (+1s))
(21)
Then, by the performance difference lemma (Kkkade & Langford 2002), we obtain:
(o(TT+1)) VT((TT)
1 Band,(ma E Exp++ [or (s,
En [Q=
(s, (s,a)
0. Exat1 (T k- (-1s), न
(-1s))] *++))))))+))]]
(22)
7 T/k
15

---

Published as a conference paper at ICLR 2026
Corollary A.2 (Sequential Update of MAGPO). The update of any individual policy ahy with any ordered subset 11:5 can be written as:
Ale (8, a2:0-1,,,,000000 ,a.))a)))]]) DKL m²)) (23)
kk+1 arg max d 1 9t-1
a'
Proof. We first decompose the guider policy in equation 17
exp (m (s. a))
ps = === (als)
(s)
exp (NA V (s))
(a(a)ss exp (Th Qm (s, a) - m/ (m
=x(s)
TTlaass exp A7 (s, (s, (sx(s)
(24)
is (ab1s)
exp 1/k Σ
A (s. (1t0, /Ea(s)
T
exp
m/ Ait (8,
a an 1, ati)
(a>18)
( al))-1)
/=1
This implies that the marginal guider policy for agent iz is:
exp MaA (s, an1--,
p
(a (a' |s, a dd n. (a+
(25)
2j(8, ahi--1)
Next, we decompose the KL divergence:
(xL(F(-1s), (c(-s)) = Eart (als)(a)s) - log (((ass)
Eaar
log
πI j=1
क (ati1s) - log
II a'' (abb| ls,
n
n Σ log pots (a (
(26)
-Ear
log " (a>1(s)
E 1 115-1 a
[log (a')13(a'')))) log fi' (ats (, abv
Although the objective is directly decoupled, observe that each policy き is conditionally independent of the subsequent agents given the prior ones. Therefore, we can sequentially optimize:
T arg min Ea'ro [log म (af) [s) log ja ( (a*1s))
7/1
L2 arg min n 1 i 2 [log n (a'2 - log fi (a (s
E
arg min E a 4y wn [log nl (a')(a)))) log f (atils, afutt))))
Substituting the expression for A yields:
is = arg min - f1g-2 wet 4ig- at with
[log 4 (a'lss) log p (a' a.
x
log 2 (a's(s) log कंं (s)) A. (s, as
arg min uy fmm
x²
1 (,'',
arg max E
A'd at (+)-,,))
x² an - - wab
7
(27)
which completes the proof.
16

---

Published as a conference paper at ICLR 2026
B COORDSUM DETAILS
We introduce the CoordSum environment, cooperative multi-agent benchmark designed to demon- strate the flaw of CTDS and evaluate the performance of existing paradigm. In this environment, team of agents is tasked with selecting individual integers such that their sum matches a shared target, while avoiding repeated patterns that can be exploited by an adversarial guesser.
Naming Convention
Each task in the CoordSum environment is denoted as:
CoordSum-cnum.agen
x coumacctioss> - <max.target>
where <num agents> is the number of agents in the team, <num act i ions> specifies the size of each agent's discrete action space, and <max_target>> is the maximum possible target sum.
Observation Space At each timestep / E [1, 100), all agents receive the same observation: the current target value lurgel[1 U0, <max.target>) The observation also includes the current step count.
Action Space Each agent selects an integer action from a discrete set:
A = {0, 1, I.... <num_actions - -1}
for ,,... <numagents>> The joint action is the vector of all agents' selected integers.
To encourage agents to coordinate without relying on fixed or easily predictable strategies, the environment incorporates an opponent that attempts to guess the first agent's 's action using majority vote over historical data. Specifically, for each target value, the environment records the first agent's s past actions and uses the most frequent one as its guess. The reward at each timestep defined as follows:
Reward Function
If the sum of all agents' actions equals the current target:
- reward of 2.0 is given if the opponent's guess does not match the first agent's action. A reward of 1.0 given if the opponent's guess does match the first agent's 's action.
If the sum does not match the target, reward of 0.0 is given.
The same reward is distributed uniformly to all agents.
C FURTHER EXPERIMENTAL RESULTS
C.I PER-TASK SAMPLE EFFICIENCY RESULTS
In Figure 7 we give all task-level aggregated results. In all cases, we report the mean with 95% bootstrap confidence intervals over 10 independent runs.
C.2 PER-TASK TABULAR RESULTS
In Table we provide absolute episode metric (Colas al. 2018 Gorsane al. 2022 over training averaged across 10 seeds with std reported. The bolded value means the best performance across all methods, while highlighted value represents the best among CTDE methods.
C.3 HYPERPARAMETERS
All algorithms were tuned on each task with a tuning budget of 40 trials using the Tree-structured
Parzen Estimator (TPE) implemented in the Optuna library (Akiba et al.) 2011).. )). Since some of the tuned hyperparameters
provided in Mava (de Kock al. 2023), we directly adopt them and only tune the additional algorithms and tasks. Specifically, MAGPO and HAPPO in all tasks, all algorithms in the newly introduced CoordSum tasks.
The default hyperparameters for all methods are listed in Table and Table The full hyperparameter
search spaces provided in Table 4 Table 5 Table 6 and Table
17

---

Published as a conference paper at ICLR 2026
Table 1: Performance comparison across tasks. Best overall value is bolded. Best among CTDE methods are underlined.
Task
MAGPO
CTDS
HАРPO
MAPPO
CoordSum
3x-0-30
153. = 1.41
111.98
11.15
153.32 = 1.89 156.37
3.56
68.29 # 16,80 80
153.70 + 3.77
3x30-50
156.62
1.09
76.27 = 14.10
129.35
5.22 158.05 05 4.40
87.79
8.79
125.04
7.18
5x20-80
157.61
3.89
19.62
1. 11.79
119.19 1: 10.05
142.51
4.87
86.86
8.01
48.35
15.76
8x15-100
8.47
1: 127.32
12.76
57.22
18.75
15x1553p-5f
0.99 0.00
0.96 0.02
0.91 0.02
0.97 1: 0.02
0.91 0.02
0.96 0.01
15x1554--3f
0.00
+ 0.00
0.99
MaConneetor LevelBosddooo
15-155-4-55f
0.99 +: 0.00
0.99
0.89 0.02
0.98 + 0.01
0.97 + 0.01
0.99 0.00
23-8x8-2p-2f-coop
1.00 + 0.00
1.00 # 0.00
1.00 0.00
1.00 + 0.00
1.00 + 0.00
1.00 0.00
2s-10x10-3p-3f
0.97 + 0.01
0.87 + 0.02
0.99 0.00
1.00 + 0.00
0.97 + 0.01
0.99 0.00
8x8-2p-2f-coop
1.00
1.00
10x10-3p-3f
1.00 + 0.00
1.00 + 0.00
1.00 + 0.00
1.00 + 0.00
0.99 + 0.00
1.00 0.00
con-5x5x3a
0.94
0.87 0.02
0.85
con-7x7x5a
0.76 :: 0.03
0.71 1: 0.05
0.67 0.02
0.63 1 0.02
0.62 1: 0.04
0.74 0.03
con- con-10x1000 10x10x10
0.42 0.03
0.37 0
0.21 + 0.03
0.30 + 0.01
0.22 # 0.05
0.39 0.03
con- 5xx33a
0.02 0.01
0.02 = 0.01
0.00 + 0.00
0.02 + 0.01
0.00 + 0.00
0.08 0.01
spread 3ag
-6.10 1: 0.21
6.34 1 0.29
66.63 1: 0.49
- 6.72 1. 0.22
6.59 1: 0.23
4.92 0.30
spread 5ag
18.66 + 0.41
20.38
0.39
23. + 0.53
222.81 0.23
25.30
1.74
12.75 + 0.91
spread_10ag
40.51
10.09
0.35
-50.07
36.936.33
large-4ag
7.63 +: 0.98
5.00 0.54
0.61 0.54
3.02 + 2.26
4.61 + 0.25
6.22 1.73
large-4ag-hard
4.56
2:25 1.50
+
2.28
large-Sag
10.40 + 0.52
7.68 7.68 0.69
0.00 0.00
8.33 + 0.66
14.72 0.79
11.01 +: 0.51
large-8ag-hard
8.66 + 0.61
4.32 2.88
0.00 4 0.00
3.38 + 3.10
9.07 + 0.71
9.22 0.48
medium-4ag
11.46 + 1.16
0.22
4.04 0.63
7.82 3.24
3.82
12.74 = 1.44
medium-4ag-hare
8.49 0.54
4.79
0.66
2.80
2.64
6.79 1.34
RobooWarehous
medium-6ag
14.78 3.28
8.49
0.82
12.13 0.53
13.32 0.63
12.97 1.03
small-4ag
15.09 0.71
11.07 1: 0.81
9.64 1: 2.43
10.52 1 0.75
18.27 0.53
0.53
16.47 8.26
smaall-4g-hard
11.48 1 0.41
7.56 1: 1.02
6.50 1: 1.09
9.44 1 0.35
9.68 1:
12.02 1.20
tiny-2ag
19.70 1 1.16
13.70 1.96
10.93 1 2.30
12.28 5.86
17.06 1.61
21.17 1.24
tiny-2ag-her
16.61 0.99
9.23
9.61 3.09
15.93 13.11
tiny-4ag
31.02 + 1.95
17.84 1.14
26.29 2.88
28.19 1.02
43.56 2.69
tiny-4ag-hate
20.19 2.61
1.881
16.66 + 2.50
19.01 + 1.31
20.54 11.99
30.97 = 1.65
xlarge-4ag
5.74 + 0.71
3.02 1.25
0.00 - 0.00
3.73 : 1.19
4.71 + 0.43
3.76 2.30
xlarge-4ag-hard
0.31 + 0.64
0.12 0.34
0.00
0.00 + 0.00
0.39 + 1.01
0.70 1.39
2s3z
1.00 0.00
0.99
0.90
1.00 + 0.00
0.99 0.00
1.00 0.00
3s5z
0.99 + 0.01
0.00
0.98
1.00 + 0.00
1.00 0.00
1.00 0.01
3s5z.vs. v3_66z
+ 0.02
0.89
0.11
0.91 + 0.05
0.93
0.94
3s_vs_5z
0.09 + 0.01
0.90
0.001
0.99 + 0.01
0.90
0.99 0.01
Smax
Bh_ys_88 8z
1.00 0.00
0.99
0.90
1.00 1 0.00
0.99 0.00
1.00 0.00
10m_vs.1 11m
0.98 0.002
0.20
0.00
0.39 1 0.07
0.88 0.19
0.83 0.24
5m.vs.60 6m
0.35
0.23
0.37
1 0.36
smacv2.5 units
0.02
0.80
0.68
0.02
0.82 - 0.02
0.78 0.03
Smacv2_10_unin
0.76 + 0.05
0.066
0.47 +.7
0.76 - 0.02
0.71 + 0.04
0.65 0.04
27m_vs.30m
0.99 0.01
0.99 0.01
0.70
0.83 0.10
0.88 + 0.09
1.00 0.00
Table 2: Default hyperparameters for MAT and Sable.
Parameter
Value
Activation function
GeLU True
Normalize Advantage
Value function coefficient
0.1
Discount
0.99
GAE
0.9
Rollout length
128
Add one-hot agent ID
True
18

---

Published as a conference paper at ICLR 2026
MAGPO
CTDS
MAPPO
HAPPO
MAT
SABLE
CLanhar A4233
PalTy Zicwe
33
anoieepoedd Stg
10g
Ixpedag
optood retu 003000 eptoose
15. Ae
slaeeday
stop Selvel
299
Bir Par
Hoan estagee
St
10t
Timestopp
Timesteps
Time2ops
Figure 7: Mean episode return with 95% bootstrap confidence intervals on all tasks.
19

---

Published as a conference paper at ICLR 2026
Table 3: Default hyperparameters for HAPPO and MAPPO.
Parameter
Value
Value network layer sizes
[128,128]
Policy network layer sizes Number of recurrent layers
[128,123
1
Size recurrent layer Activation function
128
ReLU
Normalize Advantage
True
Value function coefficient
0.1
Discount
0.99
GAE
0.9
Rollout length
128
Add one-hot agent ID
True
Table 4: Hyperparameter Search Space for MAT.
Parameter
Value
PPO epochs
{2, 5, 10, 15)
Number of minibatches
[1,2,4,8]
Entropy coefficient
{0.1, 0.01, 0.001, 1)
PPO clip
(0.05, 0.1,
Maximum gradient norm
50.5, 10}
Learning rate
{le-3, 5e-4, 2.5e-4. le-4, 1e-5}
Model embedding dimension
{32, 64, 128] 1. 2, 4] 1. 2. 3}
Number of transformer heads
Number of transformer blocks
Table 5: Hyperparameter Search Space for Sable.
Parameter
Value
PPO epochs
(2 10, 15}
Number of minibatches
{1,2, 8}
Entropy coefficient
(0.1, 0.01, 0.001,
PPO clip
{0.05, 0.1,0.2)
Maximum gradient norm
(0.5, 5, 10)
Learning rate
(1e-3, 5e-4, 2.5e-4, le-4, 1e-5)
Model embedding dimension
{32, 64, 128}
Number retention heads
{1, 4}
Number retention blocks
{1,2,3 3]
Retention heads scaling parameter
{0.3, 0.5, 0.8, 1}
Table 6: Hyperparameter Search Space for HAPPO and MAPPO.
Parameter
Value
PPO epochs
{2, 4, 4,83
Number of minibatches
{2, 4, 8)
Entropy coefficient
(0, 0.01, 0.00001}
PPO clip
{0.05, 0.1, 0.2)
Maximum gradient norm
(0.5,5, 10)
Value Learning rate
[1e-4, 2.5e-4, 5e-4)
Policy Learning rate
le-4, 2.5e-4, 5e-4
Recurrent chunk size
{8, 16, 32, 64, 128)
20

---

Published as a conference paper at ICLR 2026
Table 7: Hyperparameter Search Space for MAGPO.
Parameter
Value
Double clip RL auxiliary loss À
{1.1, 1,5, {0, 1,,1,,,4,8) 4,8)
21


Reviews:
[{"review_id": "5DS9tYX2KO", "reviewer": "Reviewer_ifsr", "summary": "This paper introduces Multi-Agent Guided Policy Optimization (MAGPO), which follows the Centralized Training with Decentralized Execution (CTDE) paradigm. MAGPO alternates between improving a centralized guider policy and distilling it into a decentralized learner policy. After each distillation step, the guider is reset to match the learner so that the guider remains factorizable and the imitation gap does not grow. Empirically, MAGPO consistently outperforms strong CTDE baselines and matches or surpasses fully centralized training approaches.", "questions": "How is the guider actually reset to the learner in implementation? Is this performed by zeroing out the action-conditioning parameters? Please clarify the exact procedure.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Proposes a simple and conceptually elegant algorithmic framework for multi-agent RL.\n\nDemonstrates strong empirical performance across benchmark tasks.", "weaknesses": "It is not clearly explained how the guider (teacher) is reset to the learner (student) in practice. While in theory a product of decentralized policies defines an autoregressive joint policy, the mechanism of implementing this reset may be nontrivial depending on the network architecture.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gOUN353p60", "reviewer": "Reviewer_MMCD", "summary": "This work proposes MAGPO, a Centralized Teacher with Decentralized Students (CTDS) approach for multi agent RL, that leverages a sequentially executed guider for joint coordinated exploration while constraining it to be close to the decentralized learner policies. The authors identify and aim to address two problems with previously proposed CTDS approaches: (i) training a centralized agent raises scalability issues; and (ii) under partial observability, the students may fail to replicate the teacher's behavior. MAGPO addresses both the scalability and policy asymmetry issues by learning a centralized, autoregressive guider policy that is constrained to remain aligned with decentralized learners policies. The authors prove monotonic improvement of MAGPO under tabular settings. The authors provide experimental results comparing MAGPO against CTCE and CTDE methods, as well as a vanilla implementation of on-policy CTDS, across several domains.", "questions": "- I think the authors should better emphasize since the beginning that they are focused on *autoregressive* policies. The authors should also note that autoregressive policies do not entirely cover the space of all possible joint policies.\n- 176: \"Agents act sequentially, observing previous agents’ actions before choosing their own.\" - I guess the authors wanted to say that the actions of the agents are jointly selected, right? Why are the actions *sequentially* selected? In centralized execution what matters is that they are *jointly* selected, even tough in practice some methods may sequentially select them. And, if the actions are jointly selected, then why the agents cannot solve this task in the first place? Even if the actions are sequentially selected, why is the third agent selecting 3 instead of 4 prior to the update?\n- line 222: How is this KL distance minimized? I guess we minimize it under a given state distribution, right? Under which state distribution? I think this step needs to be further explained in the context of this article.\n- Eq.2: I believe this equation needs to be better explained in the context of this article. Does this hold for any state $s$? I think $Q^{\\mu_k}$ is the Q-function for policy $\\mu_k$, but this is undefined in the article. D_KL is the KL-divergence? Maybe the authors could talk a bit more about Policy Mirror Descent in the background section of the work. I can see that later the authors specify the updates, but I believe the discussion should be clearer in the text close to eq. 2.\n- I suggest the authors to find a way to group the different classes of methods in the plots in some way (so that it is easier to understand which class of methods, e.g., CTDE, CTDS, etc., a given line corresponds to). For example, CTCE can be plotted with a dashed line, CTDE with a solid line, etc. Because I think this could make it easier to interpret the plots.\n- My key question regarding the experimental results is: as far as I understood, this work in not the first proposing a CTDS approach to MARL. Thus, I would expect the authors to also compare their method (MAGPO) against other CTDS approaches. I can see the authors compare against a CTDS approach, but does this baseline represents well the previously proposed approach for CTDS? Why not including other previously proposed CTDS methods as baselines?\n\nMinor comments/typos:\n- line 56: \"The guider policy allows agent to act sequentially conditioned on (...)\" - agent -> the agents?\n- line 155: \"We compare three MARL frameworks:\" is duplicated.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper features a good discussion of related work in Sec. 2.2. The paper seems to do a good job motivating the research gap it aims to address. The experimental protocol seems solid, with the authors performing multiple runs and reporting confidence intervals. The experimental results seem reproducible. The experiments are also extensive, considering several different environments. The authors provide an ablation study. The work appears to feature a sufficient degree of novelty, even though I am not very familiar with previously proposed CTDS approaches.", "weaknesses": "I think the clarity of the paper needs to be further improved in some places. While the experimental results seem to support that the proposed method outperforms pervious baselines, I'm not sure whether the included baselines are well-representative of the previously proposed works; namely, I wonder if the authors are properly comparing their method against previously proposed *CTDS* methods for MARL (see my questions for additional information regarding this matter). This is my main point of concern regarding the article.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6UU0HM4lzM", "reviewer": "Reviewer_vuQz", "summary": "The paper proposes Multi-Agent Guided Policy Optimization (MAGPO), a framework for cooperative multi-agent reinforcement learning that unites centralized training with decentralized execution. MAGPO uses a centralized autoregressive guider for coordinated exploration while keeping it aligned with decentralized policies, ensuring deployability. It offers theoretical guarantees of monotonic improvement and shows state-of-the-art results across 43 tasks, outperforming CTDE baselines and matching fully centralized methods.", "questions": "How well does the proposed autoregressive guider policy actually scale to large multi-agent systems, given that it still requires sequential conditioning over agents’ actions and has not been empirically validated beyond small- or medium-scale settings?\n\nSince the monotonic improvement proof assumes full observability and tabular settings, how meaningful are the theoretical guarantees for realistic partially observable environments, and can the authors extend or justify the theory’s applicability to Dec-POMDPs and function-approximation cases?\n\nTo what extent are the key algorithmic components, including guider backtracking, the δ constraint, and the RL auxiliary loss (λ), essential for achieving stable learning, and how sensitive is MAGPO’s performance to these design choices and to the selection of the guider backbone (e.g., Sable vs. MAT)?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper’s main strength lies in its conceptually coherent integration of centralized and decentralized learning in multi-agent reinforcement learning. By introducing the Multi-Agent Guided Policy Optimization (MAGPO) framework, it provides a principled way to leverage centralized guidance during training while maintaining decentralized deployability.", "weaknesses": "I have several concerns regarding this study, as summarized below.\n\n1. The paper argues that the autoregressive guider policy alleviates the exponential growth of the joint action space. However, this sequential modeling still requires conditioning on all preceding agents’ actions and therefore may not scale well in the number of agents. The computational and communication costs of this approach under large agent populations are not analyzed, and all experiments are limited to small- or medium-scale settings (mostly under ten agents). Consequently, the claim of effective scalability remains largely qualitative and lacks empirical validation on truly large-scale MARL problems.\n\n2. MAGPO’s approach to addressing policy asymmetry by regularizing the centralized guider to stay close to decentralized learners via a KL constraint (parameterized by δ). This seems to be an intuitive mechanism but not a principled solution. The theoretical guarantee of monotonic improvement holds only in the tabular and fully observable setting, leaving its applicability under partial observability and function approximation uncertain. In practice, the δ constraint appears to serve mainly as a tunable heuristic, and the paper provides limited analysis of how sensitive performance is to its choice. Thus, while MAGPO may empirically reduce the imitation gap, it does not theoretically resolve the structural mismatch between centralized and decentralized policy spaces.\n\n3. The theoretical analysis of MAGPO assumes full observability, effectively reducing the problem to a centralized Markov game. While this simplification allows for a clean monotonic improvement proof, it undermines the practical relevance of the result. Realistic multi-agent systems operate under partial observability, where theoretical guarantees often break down due to non-Markovian local histories. Without an extension of the theory to Dec-POMDPs, the presented result is largely of limited practical utility and does not substantiate the paper’s claims of theoretical robustness under decentralized execution.\n\n4. The role of the “guider backtracking” step (resetting the guider to the learner policy each iteration) is not well justified empirically. While it appears necessary for the theoretical monotonic improvement proof, this requirement arises only under restrictive assumptions (tabular, fully observable setting). In practice, backtracking may slow learning by erasing the guider’s accumulated coordination knowledge. The paper would benefit from an in-depth ablation study or clarification on whether strict backtracking is essential for stability, or if a softer synchronization mechanism could yield faster convergence (and better performance) without violating practical stability.\n\n5. The addition of the RL auxiliary loss in the learner update is a pragmatic design choice rather than a theoretically required one. It helps the learner recover from the imitation gap when the guider’s centralized behavior is not perfectly realizable under decentralized observations. However, the motivation is somewhat heuristic. The paper includes a limited ablation (Fig. 5a) demonstrating that λ influences performance, but offers no systematic sensitivity or interaction analysis. In particular, the relationship between λ and the guider–learner KL constraint (δ) remains unexplored. The effect of λ is treated empirically rather than analytically, leaving uncertainty about the stability and generality of this design choice.\n\n6. The experimental results are broad and methodologically sound, showing that MAGPO is a competitive CTDE framework. However, the improvements over strong baselines such as MAPPO, HAPPO, and Sable are often moderate rather than decisive. While MAGPO demonstrates robustness and conceptual coherence, the evidence does not clearly establish a substantial empirical leap over other SOTA methods. Particularly, MAGPO’s performance sees to heavily depend on the guider’s backbone (Sable vs. MAT) and the δ hyperparameter. This makes the results appear fragile, strong when tuned, weaker otherwise. This paper doesn’t present qualitative examples of improved coordination, communication efficiency, or emergent behaviors. So I don’t really see how MAGPO achieves its advantage.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)