Heterogeneous Agent Q-weighted Policy Optimization
Authors: Bor-Jiun Lin
Chun-Yi Lee
Venue: ICLR 2026 Poster (2026)
Keywords: Multi-agent Reinforcement Learning
Heterogeneous Agent Reinforcement Learning
Diffusion Policy
Policy Optimization
Primary area: reinforcement learning

Abstract: Multi-agent reinforcement learning (MARL) confronts a fundamental tension between stability and expressiveness. Stability requires avoiding divergence under non-stationary updates, while expressiveness demands capturing multimodal strategies for heterogeneous coordination. Existing methods sacrifice one for the other: value-decomposition and trust-region approaches ensure stability but assume restrictive unimodal policies, while expressive generative models lack optimization guarantees. To address this challenge, we introduce **H**eterogeneous **A**gent **Q**-weighted Policy **O**ptimization (**HAQO**), a framework unifying sequential advantage-aware updates, Q-weighted variational surrogates, and entropy regularization. Our analysis establishes monotone improvement guarantees under bounded critic bias, extending trust-region theory to diffusion-based policies with intractable log-likelihoods. HAQO achieves superior returns and reduced variance compared to policy-gradient baselines across diverse benchmarks. The ablation studies confirm sequential updates ensure stability, expressive policies enable multimodality, and entropy regularization prevents collapse. HAQO reconciles stability and expressiveness in MARL with theoretical rigor and practical effectiveness.

Published conference paper at ICLR 2026
HETEROGENEOUS AGENT Q-WEIGHTED POLICY OPTIMIZATION
Bor Jiun, Lin Department of Computer Science and Information Engincering, National Taiwan University crlc112358@gail.co com
Chun- Yi Lee Department of Computer Science and Information Engincering & Artificial Intelligence Center of Research Excellence, National Taiwan University cylee@csie. ntu. edu. tw
ABSTRACT
Multi-agent reinforcement learning (MARL) confronts a fundamental lension be- tween stability and expressiveness. Stability requires avoiding divergence un- der non-stationary updates, while expressiveness demands capturing multimodal strategies for heterogeneous coordination. Existing methods sacrifice one for the other: value-decomposition and trust-region approaches ensure stability but as- sume restrictive unimodal policies, while expressive generative models lack op- timization guarantoes. To address this challenge, we introduce Heterogencous Agent Q-weighted Policy Optimization (HAQO), framework unifying sequen- tial advantage-aware updates, Q-weighted variational surrogates, and entropy reg- ularization. Our analysis establishes monotone improvement guarantees under bounded critic bias, extending trust-region theory to diffusion-based policies with intractable likeliiosooss. HAQO achieves superior returns and reduced variance compared policy-gradiens basclines across diverse benchmarks. The ablation studies confirm sequential updates ensure stability, expressive policies enable mul- timodality, and entropy regularization prevents collapse. HAQO reconciles stability and expressiveness in MARL with theoretical rigor and practical effectiveness.
1 INTRODUCTION
Multi-Agent Reinforcement Learning (MARL) has emerged central paradigm for enabling in- telligent agents to cooperate effectively in complex, dynamic environments. Its importance spans across domains such as robotics (Chen et al., 2025; Qin et al., 2025; Cai et al., 2024; Feng et al., 2025), autonomous driving (Shai et al., 2016; Wu et al., 2025; Taghavifar et al., 2025), and smart factory applications (Wascem & Chang, 2025; Leet et al., 2025; Movahed al., 2025). Despite advances in these realms, designing MARL algorithms that achieve both stability and expressiveness remains fundamental challenge. The complexity of MARL arises from four main factors: (i) the joint action space grows exponentially with the number of agents, which renders exhaustive exploration and op- timization computationally infeasible, (ii) the learning process is inherently non-stationary, as each agent's 's policy update alters the environment dynamics observed by other agents, (fii) agents are of- ten heterogeneous, possessing distinct dynamics, observation modalities, or action repertoires, and (iv) decentralized coordination presents additional obstacles where communication constraints and

---

Published conference paper at ICLR 2026
Unimodal Policy
Explored Area
Muttinodal
Poliy
Unexplored
Multineda we
Bnttagy Requlastzaion
Thrgee Objective Function
Training
Figure 1: Policy exploration comparison across different algorithmic components. This multi-modal coordination task requires agents to discover and maintain coverage across four distinct target regions (circular contours) rather than collapsing to suboptimal single-mode strategies. We visualize agent policies through dimensionality reduction to demonstrate learned representations. Top: Gaussian policies exhibit rapid mode collapse, failing to capture optimal multimodal dis- tributions. Middle: Multimodal policies improve coverage but suffer from exploration instability and mode imbalance. Bottom: HAQO combining multimodal policies with entropy regularization achieves stable, balanced exploration that effectively tracks all target modes throughout training.
partial observability prevent centralized control in practical scenarios. Heterogeneity particularly exacerbates all previous challenges, complicates learning when agents operate with different ob- servation spaces, action spaces, functional roles, which usually necessitate specialized approaches that frequently produce brittle policies (Zhong et al., 2024), unlike homogeneons settings where rameler sharing mechanisms function elfectively. These challenges collectively render coordination and convergence substantially more challenging than in homogeneous MARL scenarios.
Among these challenges, heterogeneous multi-agent systems confront additional expressiveness lim- itations that compound the coordination difficulties. Expressiveness in MARL refers to the ability of policy representations to capture the full spectrum of behaviors and coordination patterns that agents may need exhibit complex multi-agent environments. When agents possess different ca- pabilities, observation modalities, or roles, the coordination problem becomes even more complex as cach agent must learn strategies that account for the diverse behavioral repertoires of its heteroge- neous partners. Conventional unimodal policy classes, such Gaussian policies (Tan, 1997), impose severe constraints on the types of strategies agents can learn and execute. These limitations become particularly problematic in heterogeneous scenarios where optimal coordination requires agents to switch between distinct behavioral modes or employ complex, multimodal action distributions that adapt to their partners' diverse capabilities. As result, unimodal policy representations cannot. adequately represent the rich strategic repertoire necessary for effective multi-agent coordination in heterogencous systems, and often result in mode collapse where agents converge to suboptimal behaviors that represent only a limited subset of the available strategy space. Furthermore, simul- tancous policy updates across all heterogeneous agents create destabilizing non-stationarity that impedes convergence to meaningful coordination solutions (Zhong et al., 2024). The limitation of unimodal policies is further illustrated in Fig. 1, where Gaussian policies collapse to single strat-
2

---

Published a conference paper at ICLR 2026
egy and fail to represent the multimodal behaviors needed for coordination. However, multimodal policies alone prove insufficient, demonstrated by their unstable exploration patterns. The inclu- sion of entropy regularization in the proposed HAQO framework enhances the multimodal policy's exploration capabilities, effectively enabling more balanced coverage across all strategic modes.
Beyond these fundamental challenges, three critical issues emerge as particularly significant bar- riers to practical MARL deployment in heterogeneous environments. First, coordinating learning updates presents substantial difficulties, as simultaneous policy updates across multiple heteroge- neous agents create conflicting gradients that cause training instability and hinder convergence to meaningful solutions. This challenge becomes exponentially more complex when agents possess dif- ferent learning capabilities, update frequencies, and policy representations, making traditional ho- mogeneous coordination mechanisms inadequate. Second, policy expressiveness limitations severely constrain the strategic diversity that heterogeneous agents require for effective coordination. Con- ventional unimodal policy representations cannot capture the rich multimodal behaviors necessary for sophisticated coordination strategies, particularly when heterogeneous agents must adapt their decision-makiin processes to accommodate diverse partner capabilities, observation modalities, and action repertoires. Although multimodal RT objectives can be cast as variational lower bounds, extending these guarantees to heterogeneous MARL is non-trivial. The key obstacle is ensuring that expressive policies still yield improvement when agents differ in roles and update sequentially. Third, the integration of expressive policy representations with stable learning algorithms remains a fundamental challenge, as more sophisticated policy classes introduce additional variance and computational complexity that can destabilize the already fragile multi-agent learning dynamies in heterogeneous settings. These interconnected challenges highlight the critical need for algorithmic frameworks that can simultaneously address coordination stability, policy expressiveness, and het- erogeneous agent capabilities while maintaining theoretical guarantees and practical applicability.
To overcome these challenges, we propose Heterogeneoou-AAgent Q-weighted Optimization (HAQO), a framework that addresses the demanding requirements of stability and expressiveness in heterogeneous MARL. HAQO employs sequential advanlage-aware updates to mitigale non- stationarity, a Q-weighted variational surrogate to align expressive policies with return maximiza- tion, and an entropy surrogate to sustain exploration. These components address the limitations of prior methods, and enable heterogeneous agents to maintain diverse behaviors while ensuring sta- bility, providing theoretical guarantees and practical effectiveness. Our contributions are threefold:
1. Introduce MARL framework that couples sequential advantage-bass updates with ex- pressive policy models, enabling coordination in heterogeneous agents with flexibility.
2. Generalize Q-weighted policy optimization to the multi-agent setting, extending advantage- weighted objectives with explicit importance-sampling corrections for sequential updates.
3. Derive an exact bound on return differences under sequential multi-agent updates, yielding tight constants that account. for heterogeneous dynamics and non-stationary environments.
2 RELATED WORKS
Early MARL research adopted independent learning frameworks (Tan, 1997) where agents treated others as environmental components and applied single-agent algorithms. This approach demon- strated fundamental limitations due to non-stationarity from each agent's 's perspective, creating un- stable learning dynamics and poor scalability. Centralized MARL methods (Wen et al., 2022; Pedro
3

---

Published a conference paper at ICLR 2026
al., 2024) addressed these limitations through information sharing and joint optimization, miti- gating non-stationarity while improving coordination. However, these methods required full observ- ability and centralized execution, proving impractical for real-world systems with quadratic compu- tational scaling, The centralized training and decentralized execution (CTDE) paradigm (Sunehag et al., 2018) emerged as a breakthrough, exploiting global state information during training while relying on decentralized local policies for execution. CTDE enabled two dominant algorithmic fam- ilies: balue-based and policy-based approaches. Value decomposition methods (Rashid al., 2018; Yang et al., 2020a; Rashid et al., 2020; Yang et al., 2020b; lqbal et al., 2020; Zhou et al., 2020; Li et al., 2022; Shen et al., 2022; Sun et al., 2021; Liu et al., 2019; Hu et al., 2021; Wang et al., 2020; Hao et al., 2023; Lin & Lee, 2024) factorize joint value functions into per-agent components to address credit assignment. Policy-based approaches (Chao et al., 2022; Papoudakis et al., 2021; Ackermann et al., 2019; Son al., 2019; Wang et al., 2021) extend singlo-agent policy gradient theorems to multi-agent domains, often stabilized through trust-region or proximal objectives.
Recent research has addressed coordination challenges through sequential update schemes that update agents individually while conditioning on others' recent policies, providing formal guar- antees for monotone joint return improvement and Nash equilibrium convergence (Zhong et al., 2024). Stabilization strategies include trust-region methods (Kuba et al., 2022c) that bound policy changes and coordination graphs (Kok & Vlassis, 2006) that approximate inter-agent dependencies. For policy expressiveness, normalizing flows (Ma et al., 2024) enable flexible distributions through invertible transformations, while diffusion models (Janner et al., 2022) demonstrate exceptional capability for generating diverse behaviors in sequential decision- decision-making
addantage-weighhen jectives have emerged as promising approaches for biasing updates toward high-value actions while reducing variance, with methods like RWR (Strupl al., 2022) and AWR (Peng et al., 2021) casting optimization weighted supervised learning. Recent extensions (Ding et al., 2024) connect these concepts with diffusion models through Q-weighted variational bounds approx approxinating policy gradi- ents. However, these approaches remain limited to homogeneous settings and perform poorly with differing observation or action spaces (Liu et al., 2024). Coordination mechanisms either rely on homogeneity assumptions or impose structural constraints limiting applicability in heterogencous environments, while expressive policy representations introduce computational costs and variance, with stable integration into cooperative online multi-agent settings remaining largely unexplored.
ob-
3 PRELIMINARIES
3.1 HETEROGENROUS-AGHH MIRROR LEARNING (HAMT.)
Zhong et al. (2024) generalizes the mirror-eeariigg methodology (Kuba et al., 2022b) to multi-agent settings with heterogeneons agent configurations. In contrast to homogeneous MARL algorithms that rely on parameter sharing, HAML explicitly allows each agent to maintain a distinct policy tai- lored its specific capabilities and constraints. HAML operates within the CTDE paradigm (Sunc- hag et al., 2018), and its primary innovation lies in extending trust-region style convergence guar- antees to heterogencous multi-agent settings through the integration of mirror-descent update rules for each individual agent. Specifically, for each agent i, the policy update takes the following form:
π Thew arg min ( - nB [A' a)]]
(1)
r'Ell'
where nO E II' represents the policy for agent within its designated policy class II' (T (A)) denotes the Kullback-Leiba divergence between the new and old policies serving as a regularization term, 77 is scaling parameter, and A a) is the multi-agent advantage function that evaluates
4

---

Published conference paper at ICLR 2026
the relative value of actions taken by agent i in the context of the joint state-action space. This formulation ensures that each agent's policy update balances exploitation of advantagcous actions with regularization to prevent destabilizing changes that could disrupt coordination with others.
3.2 SEQUENTIAL UPDATE DECOMPOSITION
central challenge in MARL that simultaneous updates can creale instability due to conflicting gradient directions. HAML and related methods (Zhong et al. (2024)) address this by updating agents sequentially in randomized order 11:n- Let wold and Nnew denote the joint policies before and after one round of updates. The return difference can be decomposed into per-agent contributions:
[(Tnew) J (TToll))
Σ m=1
[a Told
(mmm 1, F, (mmt))) [T
fmon FFlld )
This decomposition expresses the global return improvement as a summation of marginal improve- ments contributed by each agent, conditioned on the latest updates of its predecessors. As shown (Zhong et al., 2024), this decomposition principle ensures monotonic improvement guarantees for the overall joint policy, provided that each individual agent's update yields local improvement.
3.3 DIFFUSION POLICY
Diffusion policies represent a paradigm shift in reinforcement learning. They move beyond restric- tive unimodal parametrizations to capture complex, multimodal action distributions essential for high-dimensional control. While foundational works by Zhong et al. (2024) and subsequent ap- plications in single-agent domains have demonstrated their efficacy robotic manipulation, their application heterogeneous multi-agent systems remains underexplored. Our work builds upon the theoretical formalisms established by Chi et al. (2023) and Kuba et al. (2022a), and extends them by utilizing CTDE paradigm to address the specific challenges of heterogeneous coordination. Unlike prior works that rely on parameter sharing, our approach operates in a standard, purely cooperative MARL setting where agents optimize a shared objective ((0) = IET ~ pe mp1E,>5m 7'rr]. The integration of diffusion models serves not merely to enhance expressiveness, but to resolve the babblity--xpreesivenss tension by enabling agents to model complex joint. distributions without. sacrificing the monotonic improvement guarantees provided by our sequential update framework.
3.4 Q-WEIGHTED VARIATIONAL POLICY OPTIMIZTIION (QVPO)
While sequential updates provide stability, policy expressiveness is equally critical for capturing multimodal strategics. Diffusion models Song al. (2021) offer high-capacity generative policies, however, their variational lower bound (VLB) training objective lacks direct alignment with rein- forcement learning. To address this, Ding et al. (2024) introduces value-weighted training objective connecting VLB to policy gradient theory through the Q-weighted variational loss, formulated as:
(Q) -.,a Be [Q(s,a) a) ||e fle--n Eg (Vara
ate, s, -I-a 1]]]
(3)
where eg is the denoising network parameterized by 0, ãe is the schedule at I, and T is the total timesteps. This objective forms a tight lower bound of the policy gradient objective under appropri- ate weighting transformations. This result bridges generative modeling and reinforcement learning, allowing diffusion policies to be trained online while prioritizing actions with higher Q-values.
5

---

Published conference paper at ICLR 2026
4 MOTIVATIONAL EXPERIMENT
Unimodal Agent
EIAQO Molti-geem System
To motivate our approach, design cupation game that agents must cover all largets to maximize the visitation density across multiple targets, which constitutes a multimodal objective that requires co- ordinated coverage. Fig. 2 compares Gaussian policies with HAQO. Baseline methods exhibit mode collapse and con- centrate on target subsets while neglect- ing others. In contrast, HAQO achieves diverse coverage across most targets with consistent performance. This improvement stems from two key designs: expressive multimodal policies that represent diverse strategic behaviors, and sequential advanlage-aware updates that mitigate the instability of simultaneous updates, confirming sequential coordination's 's importance.
SVENO 2IN
High Desity
Low Deniey
Figure 2: Comparison between conventional MARL baseline employing unimodal Gaussian policies (left) and the proposed HAQO framework (right). The con- tours represent the state visitation density distribution.
5 METHODOLOGGY
In this section, we present the Heterogencous-Ageet Q-weighted Optimization (HAQO) framework, which addresses the three critical challenges identified heterogeneous MARL. It. inte- grates: (i) sequential advantage-aware updates that mitigate non-stationarity and provide monotone improvement guarantees in Section 5.1; (ii) Q- Q-weighted lower bound for multi-agent setting in Section 5.2; and (iii) an entropy regularization term which bridges MARL with generative modeling and ensure robust exploration in Section 5.3. We synthesize these theoretical foundations and derive the monotone improvement bound for HAQO, together with the practical algorithm in Section 5.4.
5.1 TRUST-REGION MULTI-AGENT POLICY IMPROVEMENT VIA SEQUENTIAL UPDATES
In MARL, simultaneous policy updates can precipitate performance collapse since advantage func- tions are defined relative to joint policies, which causes concurrent policy changes to invalidate advantage estimates and eliminate improvement guarantees. To address this challenge, sequential update mechanisms to mitigate this non-stationarity by permuting the update order iim and define per-agent surrogate objectives ('(T') = E p a-iwn, Egt~ri[A' a')] that condition on current policies of other agents a' neww while retaining the baseline state visitation distri- bution nold for analytical tractability. ('(x') measures agent i's advantage under updated peer policics while averaging over the bascline state distribution, enabling clean telescoping arguments in performance difference analysis as in Lemma A.8. Since ('(²)) relies on ~ pTala, we must constrain policy update magnitudes to control the distributional mismatch between (pTold, F p ).
adopt
We impose per-agent trust-region constraints E -og (T. (+Is)]] S 6 where 8; is small update radius. This heterogeneous-agent analogue of the TRPO KL-divergence constraint ensures that distributional drift across sequential updates remains quadratically small, which leads to the tighter bound in Proposition 5.1 following Zhong et al. (2024). This proposition establishes that sequential updates guarantee monotone joint return improvement up to quadratic KL drift penalties, with bounds scaling lincarly rather than exponentially with agent count because since sequential execution avoids simultancous update instability. We provide structural explanation for why sequential updates deliver stable practical improvement, with proof derivation in Appendix A.6.
6

---

Published conference paper at ICLR 2026
Proposition 5.1 (Heterogeneous Sequential Improvement Bound). Let agents i E (1, update in random permutation order inn Suppose rewards are bounded (Assumption A.1); policies are regular (Assumplion A.3), and trust-region construints hold with radius ò, for each agent i (A sssumplion A.5). Then there erist constants C₁- - O A for each agent i such that:
(1-7)
n
J (TTww) - (Tooaa)
Σ ((((eee))) L' L'' (T(TTI)) Σ
C C, (proof at Appendix A.6) (4)
1=1
5.2 HETEROGENBOUS AGENT Q-WEIGHTED VARIATIONAL. (QV) DIFFUSION OBJECTIVE
Standard policy-gradient methods rely on gradients of log However, in diffusion-based actors, the likelihood of action is intractable: the policy is defined implicitly through reverse SDE, and log r' cannot be evaluated in closed form. This limitation precludes the use of classical likelihood-ratio surrogates. To resolve this, following Ding et al. (2024), we leverage a Q-weighted denoising objective, which is both (i) implementable for diffusion models via score matching, and (ii) theoretically aligned with return maximization through a VLB. For each agent i, conditioning on the already-updatee policies of other agents, we define the multi-agent QV surrogate as follows:
JP (8') = Bgnppooaa Ba-trari Bati ot (' a" [|e- + v.. (5)
where Egs is the denoiser, and w'(s, a², a-)) is a non-negative weight function derived from the conditional advantage. Intuitively, JQV biases denoising updates toward actions that exhibit high advantage under the current critic. In principle, one could set w'(s ( = A'ss directly. However, since A be negative, this would yield sign-indefinite objective that no longer lower- bounds the improvement surrogate. To preserve positivity and hence variational tightness, we adopt transformations that retain advantage aligmment while discarding negative contributions as followssransformatioos that retain advantage aligmment while discarding negative contributions:
qadv: w'ss,a', a-) = max (0, A'ss,a') Ea A'(,,a)).
(6)
qcut: asss,'' max(0, (s (111)))1(444 ((A' 2 2
(7)
The qadv transformation centers advantages before rectification to maintain weight calibration around the bascline policy, while qcut imposes thresholds to focus on high-advantage samples and filter out noise. These transformations create a practical objective that remains non-negative while capturing relative ad advantage scaling and ensuring the denoising loss serves as a valid policy-gradient surrogate. Proposition 5.2 demonstrates that J,0((0) (0i) serves as a robust policy improvement surro- gate in heterogencous scttings, maintaining tight bounds with exact critics and degrading gracefully under approximation. The positivity constraint preserves monotonicity by preventing negative- advantage interference, while gadv/qcut balances smooth calibration with selective precision, abling expressive diffusion policies to achieve the same thcoretical guarantees as classical policy gradients without restrictive unimodal assumptions. The detailed proof is offered in Appendix A.7.
Proposition 5.2 (QV Lower Bound). Let wt be constructed via qadv qcut such that w 2 0 and is monotone in the conditional advantage. For any feasible diffusson policy xO with denoiser co.,
('(+)) 2 J. - Eir &: O(E)), (proof at Appendia A.7)
(8)
where eQ is the crilic bias from Assumption A.4. The implicil constants in O(eo) scale with the diffusion forward-notk schedule. Equality holds in the bero-bias, well-specified reverse- reverse-process process limit.
7

---

Published conference paper at ICLR 2026
5.3 ENTROPY REGULARIZATION FOR DIPFUSION-BASED POLICIES
Entropy regularization is essential in modern RL (Liu et al., 2024) by preventing premature conver- gence and promoting exploration, typically computed via log m' (a'ls) for tractable densities. How- ever, diffusion-based actors define policies implicitly, rendering log computationally intractable. To resolve this, we introduce an entropy surrogate that injects uniformly sampled actions into the denoising objective, broadening policy support and preventing collapse. Proposition 5.3 shows this surrogate maintains positive spectral floors in action covariance, providing entropy lower bounds that enable diffusion policies to retain diverse stralegies without explicit likelihood computation. Proposition 5.3 shows that diffusson policies can achieve entropy regularization without explicit log- likelihood computation. The surrogate employs uniform reconstruction as a state-agnostic baseline, which guarantees positive variance floors that prevent mode collapse while remaining compatible with denoising formulations. This surrogate balances with the Q-weighted objective, where JQV drives probability mass toward high-advantage actions, while Jont prevents pathological concen- tration around a narrow set of actions. ions. This effectively extends maximum-entropy RT principles to intractable settings. Although one-sided in preventing collapse rather than upper-boundnng entropy, this surrogate enables expressive diffusion policies to maintain exploration guarantees equivalent to those of classical maximum-entropp methods, while preserving HAQO's sequential update stability.
Proposition 5.3 (Eutropy Surrogate for Diffusion Actors). Let x² be a diffusion- based policy whose density (a s) is implicit. Define the surrogate objective:
(0') Eg e( [a: l|ee--oo (Veo-VVa at +VI- s, {. (proof at Appendir A.8) (9)
where a; > 0 is a scaling coefficient and cos is the diffusion denoiser. Then:
1. (0*) 2 0 for all 0', ensuring that il provides only non-negalive exploration pressure.
2. Minimizing Jren enforces a spectral floor Amin (2:) 2 o > 0, guaranteeing the bound
di
H(A') ()
log (2re omin), where Σ Var(A" |) and d, dim
(10)
2
5.4 MONOTONE IMPROVEMENT GUARANTEES FOR HAQO
HAQO integrates three essential components: (i) drift penalty via KL. divergence PPO-style clipping that constrains updates locally around the baseline policy, (ii) the Q-weighted variational surrogate, and (iii) the entropy surrogate. These define a tractable optimization problem whose solution guarantees theoretical improvement and practical stability under sequential heterogeneous updates. For agent the random permutation order, the HAQO objective is defined as follows:
max TiET' (1r)
D'(T'IITToa
(11)
QV lower bound surrogate entropy surrogate drift penalty (KL or PPO-elip)
where (( the feasible trust region around the baseline policy, and D denotes either an expected KL divergence or a PPO clipping that enforces bounded deviation from the current policy.
This per-agent objective synthesizes three interlocking components: J. (00) drives return- improving updates, 1²"*((') injects controlled exploration without tractable deg-desss,,s,, and the drift penalty constrains updates within safe bascline policy neighborhoods. Theorem 5.4 establishes monotone return improvement under quadratic drift penalties and bounded critic error. Algorithm 1 integrates these theoretical components with K-candidate variance reduction into a single iterative procedure with centralized critic updates followed by randomized per-agent policy optimization, translating thcoretical guarantees into scalable, stable algorithm for practical implementation.
8

---

Published conference paper at ICLR 2026
Table 1. Performance Comparison on Multi-Agent MnJoCo Environments
Environment
HAA2C
MAPPO
HATRPO
HАPPO
HAQO
Ant-v2 4x2
5637 (86)
5874 (32)
5013 (432)
5793 (59)
6014 (201)
HallCheetah-v2 2x3
4231 (1069)
6984 (132)
5369 (217)
7024 (103)
6873 (137)
Hopper-v2 v2 3x1
1832 (923)
3612 (57)
3733 (102)
3481 (173)
3884 (81)
Walker2d-v2 2x3
1124 (94)
5013 (483)
3744 (373)
5523 (214)
5681 (301)
Walker2d-v2 6x1
1923 (234)
4693 (247)
2109 (223)
4317 (401)
4789 (293)
Humanoid-v2 17x1
732 (13)
6739 (201)
7013 (311)
Values represent
with deviations. indicates algorithmss best performance, for HAQQ's
Theorem 5.4 (Monotone Improvement of HAQO). Assume bounded rewards (Assumption A.1),
regular policies (Assumplion A.3), crilic bias eQ (Assumplion A.4), and sequential updales with feasible neighborhoods constrained by D'. Then the HAQQ update satisfies (proof at Appendix A.9.):
J
Σ
n JAPY(Bnow)) (fhow) J. (Bhow) (((o(())w)))
Σ C C -(cc), C2 = 0 3Runx (i (12)
n
6 EXPERIMENTAL RESULTS
6.1 EXPERIMENTAL SETUPS
Environments. We evaluate HAQO on establishod benchmarks encompassing key coordination challenges: Multi-Agent Particle Environment (MPE) (Mordatch & Abbeel, 2017) for discrete and continuous control, StarCraft Multi-Agent Challenge (SMAC/SMAC<2) (Vinyals et al., 2017; El- lis et al., 2022) for discrete coordination under partial observability, Google Research bootball (GRF) (Kurach et al., 2020) for higl-dimensional strategic coordination requiring multimodal policies, Multi-MuJoCo (Kuba et. al., 2022a) for continuous control, and Bi-DexterrossHands (Bi- D) (Zhong et al,, 2024) for bimanual manipulation coordination. This suite systematically varies action dimensionality, observability constraints, and inter-agent coupling to provide rigorous eval- uation across diverse coordination paradigms. The complete specifications are in Appendix A.11.1.
Baselines. We compare against statc-of-thh-art (SoTA) value-based and policy-based coopera- live MARL methods. From value decomposition, we include QMIX (Rashid et al., 2018), which demonstrates strong cooperative performance but relies on monotonic factorization that limits het- erogencity handling. Among policy-gradient approaches, we evaluate MAPPO (Chao et al., 2022) with centralized training and parameler sharing, MADDPG (Lowe et al., 2017) with individual actor-critic pairs that faces scalability challenges, and recent heterogeneous methods НАРPO and HATRPO (Zhong et al., 2024) with sequential updates under trust-region constraints. While these approaches improve stability over simultancous updates, they remain limited by unimodal Gaussian policies. All results use three random seeds with consistent hyperparameters for fair comparison.
6.2 COMPARISON TO HOLICY-BASED APPROACHES
In this subsection, we compare HAQO against MAPPO, MADDPG, HAPPO, and HATRPO (i.e., the SoTA policy-based MARL method that combines trust-region updates with advantage- weighted objectives). In Multi-MuJoCCo, where agents jointly control continuous body parts, HAQO consis- tently outperforms policy-gradient basclines. Table 1 shows HAQO achieves higher asymptotic returns and more stable convergence in continuous control tasks, demonstrating effective handling of heterogeneous action spaces. These results confirm that HAQO inherits trust-region stability while leveraging expressive diffusion policies to capture strategies that unimodal baselines miss.
Specifically, the Humanoid task illustrates this advantage: successful locomotion requires multi- modal behaviors including alternating leg phases, torso stabilization, and arm coordination. Gaus- sian policies in MАPPO and HAPPO collapse to homogeneous behaviors causing unstable loco- motion, while HAQO's diffusion policies capture multimodal gait cycles for robust walking. Our additional evaluation results for MPE, SMAC/SMAGV2, GRF, and Bi-D Appendix 1.11.2 further confirm HAQO's superior stability, diverse strategics, and robust coordination across all domains.
9

---

Published conference paper at ICLR 2026
6.3 ABLATION STUDIES
6.3.1 SEQUENTIAL UPDATE
Ablation studies in Bi-D reveal the importance of sequen- tial updates. Without sequential updates, conflicting ac- tions cause dropped objects and unstable grasps, with Fig. 3 showing concurrent updales yield poor performance and frequent failures. Sequential updates allow agents to adapt to partners' recent policies, stabilizing coordination and establishing HAQO's sequential update foundation. 6.3.2 EXPRESSIVE POLICY
Figure 3: Average step rewards with sequential updates versus simultane- ous updates. Sequential updates yield faster learning and higher performance.
We compare diffusion-basel policies with uni- modal Gaussian policies using average returns in Table 4. Diffusion policies consistently achieve higher returns with lower devialions, exemplified in 3 vs 1 with keeper where Gaus- sian agents obtain 66.82 + 9.12 while diffusion agents achieve 97.27 = 1.07. complex tasks like Pass and shoot with keeper (PS) and Run pass and shoot with keeper (RPS), Gaussian agents converge to brittle single-tactic behaviors while diffusion policies sustain diverse tactical modes. The reduced deviation confirms that diffusion policics provide both higher and more stable performance, demonstrating that expressive policies are essential in GRF's multimodal coordination scenarios.
Figure 4: Gaussian Diffusion Policy on GRF.
Environment
Agent w/
Agent w/ Diffusion Pollcy
Gaussian Policy
PS
76.24 (8.23)
82.14 (2.13)
RPS 3 va with keeper
(10.31)
(3.87) 97.27 (1.07)
66.82 (9.12)
6.3.3 ENTROPY REGULARIZATION
MPE Sisple willons Euaragy
NFE Stugle wall Energyy
We evaluate entropy regularization a customized MPE environment with two agents covering three landmarks requir- ing distributed coverage maximization. Without entropy regularization (Fig. 5 lcft), agents collapse to covering only two landmarks, reflecting value-driven objectives' exploixooo.. In contrast, with entropy regularization (Fig. right), agents maintain balanced coverage across all landmarks. These results confirm Proposition 5.3: the entropy surrogate prevents collapse even without explicit log-likelihoods, and can result in robust and generalizable coordination policies.
Figure The effect of entropy regularization for MPE. State visitation density of two agents across three land- marks without entropy (left) and with entropy (right).
7 CONCLUSIONS
We proposed HAQO, a unified framework for heterogeneous-aggen RL integrating sequential up- dates, Q-weighted surrogates, and entropy regularization. Our theoretical analysis established monotone improvement guarantees for expressive policies, and empirical results showed that HAQO consistently outperforms posicy-based baselines. Ablation studies confirmed each component's 's cessity: sequential updates mitigate non-stationarity. diffusion policies capture multimodality, and entropy regularization prevents collapse. These contributions show that stability and expressiveness can be reconciled within a framework, enabling scalable coordination complex MARL systems.
10

---

Published conference paper at ICLR 2026
8 ACKNOWLEDGE
The authors gratefully acknowledge the support from the National Science and Technology Council (NSTC) in Taiwan under grant numbers NSTC 114-2221-F-0002 --22211---00--009-MMY3 NSTC 113- 113-222--F-002 212-MY3, and NSTC 14-2218-E-A49-00 Award (ASSA) under grant number PEAAA---11-02 The authors would also like to express their appreciation for the donation the GPUs from NVIDIA Corporation and NVIDIA AT Technology Center (NVAITC) used in this work. Furthermore, the authors extend their gratitude to the National Center for High-Performance Computing (NCHC) for providing computational and storage resources. The authors also thank the NVIDIA Taipei-l supercomputer for providing essential computing resources.
as well as the support from the Academia Sinica Scholar
REFERENCES
J. Ackermann, V. Gabler, T. Osa, and M. Sugiyama. Reducing overestimation bias in multi-agent domains using double centralized criticss. arxiv, arXiv:1910.0144 465, 2019.
Yuxin Cai, Xiangkun H., Hongliang G., Wei- Yun Y., and Chen L. Transformer-based mulli-agent reinforcement learning for generalization of heterogeneous multi-robot cooperation. In Proc. Int. Conf. on Intelligent Robots and Systems (IROS), 2024.
Yu Chao, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, and et al. The surprising effective- ness of ppo in cooperative multi-agent games. In Proc. Conf. on Neural Information Processing Systems (NewrIPS), 2022.
Chen, Z. Yang, H. Xu, D. Zhang, and G. Mylonas. Multi-agent systems for robotic autonomy with llms, Xiv, 056622500....66, 2025.
C. Chi, S. Feng, Y. Du, Z. Xu, E. Cousineau, and et al. Diffusion policy: Visuomotor policy learning via action diffusion. In Proc. Robotics: Science and Systems (RSS), 2023.
T. Dietterich, Becker, and Z. Ghahramani. A natural policy gradient. In Proc. Conf. on Neural Information Processing Systems (NeurIPS), 2001.
S. Ding, H. Ke, Z. Zhenhao, R. Kan, Z. Weinan, and et al. Diffusion-basse reinforcement learning via q-weighted variational policy optimization. In Proc. Conf. on Neural Information Processing Systems (NewrIPS), 2024.
Benjamin Ellis, Skander M., Mikayel S., Anuj M., Shimon W., and et al. Smacv2: An improved benchmark for cooperative multi-agent reinforcement learning. arXiv, arXiv:2212.07489, 2022.
Z. Feng, R. Xue, L. Yuan, Y. Yu, N. Ding, and al. Multi-agent embodied ai: Advances and future directions. Xiv, arXiv:2505.05100 2025.
J. Hao, X. Hao, H. Mao, W. Wang. Y. Yang, and et al. Boosting multiagent reinforcement learning via permutation invariant and permulation equivariant networks. In Inl. Conf. on Learning Representations (ICLR), 2023.
S. Hu, Zhu, X. Chang, and X. Liang. UPDeT: Universal multi-agent reinforcement learning via policy decoupling with transformers. In Inf. Conf. Learning Representations (ICLR), 2021.
11

---

Published conference paper at ICLR 2026
Shariq Iqbal, Christian A. Schröder de Witt, Bei Peng, Wendelin Böhmer, Shimon Whiteson, and Fei Sha. Ai-qmix: Attention and imagination for dynamic multi-agent reinforcement learning. CoRR, abs/2006.00 2006.04222, 2020.
M. Janner, Y. Du, J. Tenenbaum, and S. Levine. Planning with diffusion for flexible behavior synthesis. In Proc. Int. Conf. on Machine Learning (ICML), 2022.
J. R. Kok and N. Vlassis. Collaborative multiagent reinforcement learning by payoff propagation. Journal of Machine Learning Research (JMLR), 2006.
J. G. Kuba, R. Chen, M. Wen, Y. Wen, Sun, and et al. Trust region policy optimisation in multi-agent reinforcement learning. In Int. Conf. Learning Representations (ICLR), 2022a.
J. G. Kuba, C. S. de Witt, and J. Focrster. Mirror learning: A unifying framework policy optimisation. In Proc. Int. Conf. on Machine Learning (ICML), 2022b.
Jakub Grudzien Kuba, Ruiqing Chen, Muning Wen, Ying Wen, Fanglei Sun, and et al. Trust region policy optimisation in multi-agent reinforcement learning. In Int. Conf. on Learning Represendations (ICLR), 2022c.
K. Kurach, A. Raichuk, P. Stańczyk, M. Zajac, O. Bachem, and et al. Google research football: A novel reinforcement learning environment. In Proc. Conf. Artificial Intelligence (AAAI), 2020.
C. Leet, A. Sciortino, and S. Koenig. Jointly assigning processes to machines and generating plans for autonomous mobile robols in smart factory. arx, arXiv:2502.21001, 2025.
J. Li, K. Knang, B. Wang, F. Liu, L. Chen, and et al. Deconfounded value decomposition for multi-agent reinforcement learning. In Proc. Tnt. Conf. on Machine Learning (ICMT.), 2022.
Bor-Jiun Lin and Chun-Yi Lee. HGAP: Boosting permutation invariant and permutation equiv- ariant in multi-agent reinforcement learning via graph attention network. In Proc. Int. Conf. on Machine Learning (ICML), 2024.
I. J. Liu, R. A. Ych, and A. G. Schwing. PIC: Pormutation invariant critic for multi-agent deep reinforcement learning. In Proc. Ind. Conf. on Machine Learning (ICML), 2019.
Jiarong Liu, Yifan Zhong, Siyi Hu, Haobo Fu, QIANG FU, and et al. Maximum entropy heterogeneous- reinforcement learning. In Int. Conf. on Learning Representations (ICLR), 2024.
Ryan Lowe, Y. WU, A. Tamar, J. Harb, OpenAT Abbeel, P., and et al. Multi-agent actor-critic for mixed cooperatiee-competttive environments. In Proc. Conf. on Neural Information Processing Systems (NewrIPS), 2017.
X. Ma, J. K. Gupta, and M J. Kochenderfer. Normalizing flow model for policy representation in continuous action multi-agent systems. In Proc. Tnl. Conf. on Aulonomous Agents and MulliAgent Systems (AAMA5), 2024.
Igor Mordatch and Pieter Abboel. Emergence of grounded compositional language in multi-agent populations. Xiv, arXiv:1703.0490: 2017.
12

---

Published conference paper at ICLR 2026
A. B. Movahed, H. Nozari, and A. B. Movahed. Systematic review of smart factories production in industry 5.0. arXiv, arXiv:2504.21625, 2025.
G. Papoudakis, F. Christianos, L. Schäfer, and S. V Albrecht. Benchmarking multi-agent deep reinforcement learning algorithms in cooperative tasks. In Proc. Conf. on Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2021.
Pedro P. Pedro, Diogo S. C., Miguel V., Alberto S., Pedro A. S., and et al. Centralizod training with hybrid execution in multi-agent reinforcement learning. In Proc. Inl. Conf. on Aulonomous Agents and MultiAgent Systems (AAMAS), 2024.
X. B. Peng, A. Kumar, G. Zhang, and S. Levine. Advintage- regression: Simple and scalable off-policy reinforcement learning. In Int. Conf. on Learning Representations (ICLR), 2021.
Y. Qin, L. Kang, X. Song, Z. Yin, X. Liu, and et al. Robofactory: Exploring embodied agent collaboration with compositional constraints. arviv, arXiv:2503.64008, 2025.
T. Rashid, M. Samvelyan, C. Schroeder, G. Farquhar, J. Foerster, and S. Whiteson. QMIX: Monotonic value function factorisation for deep multi-agent reinforcement learning. In Proc. Int. Conf. on Machine Learning (ICML), 2018.
T. Rashid, G. Farquhar, B. Peng, and S. Whiteson. Weighted QMIX: Expanding monotonic value function factorisation for deep multi-agent reinforcement learning. In Proc. Conf. on Newral Information Processing Systems (NeurIPS), 2020.
J. Schulman, Sergey L,, Pieter A., Michael J., and Philipp M. Trust region policy optimization. In Proc. Int. Conf. on Machine Learning (ICML), 2015.
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv, arXiv:1707 007477777777 2017.
S. S. Shai, S. Shammah, and A. Shashua. Safe, multi-agent, reinforcement learning for autonomous driving. arx, arXiv:1610.03295, 2016.
S. Shen, M. Qiu, J. Liu, W. Liu, Y. Fu, and et al. Resq: A residual q function-based approach for multi-agent reinforcement learning value factorization. In Proc. Conf. on Neural Information Processing Systems (NeurIPS), 2022.
K. Son, D. Kim, W. J. Kang, D. E. Hostallero, and Y. Yi. QTRAN: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In Proc. Inl. Conf. on Machine Learning (ICML), 2019.
Song, C. Meng, and S. Ermon. Denoising diffusion implicit models. arxi abs 20000.0000,,, 2021.
W. F. Sun, C. K. Lee, and C. Y. Lec. DFAC framework: Factorizing the value function via quan- tile mixture for multi-agent distributional q-learning. In Proc. Inl. Conf. on Machine Learning (ICML), 2021.
Sunehag, G. Lever, A. Gruslys, W. Marian Czarnecki, V. Zambaldi, and et al. Value- decomposition networks for cooperative multi-agent. learning based on team reward. In Proc. Int. Conf. Autonomous Agents and MultiAgent Systems (AAMAS), 2018.
13

---

Published conference paper at ICLR 2026
Hamid Taghavifar, Chuan H., Chongfeng W., Ardashir M., and Chunwei Z. Behaviorally-aware multi-agent rl with dynamic optimization for autonomous driving. In IEEE Transactions on Aulomalion Science and Engineering, 2025.
M. Tan. Multi-agent reinforcement learning: Independent versus cooperative agents. In Proc. Int. Conf. on Machine Learning (TCML), 1997.
O. Vinyals, T. Ewalds, Bartunov, P. Georgiev, A. Vezhnevets, and et al. Starcraft II: A new challenge for reinforcement learning. arXiv, arXiv:1708.04782, 2017.
J. Wang, Z. Ren, T. Liu, Y. Yu, and C. Zhang. QPLEX: duplex dueling multi agent q-learning. In Int. Conf. on Learning Representations (ICLR), 2021.
W. Wang, T. Yang, Y. Liu, J. Hao, X. Hao, et al. Action semantics network: Considering the effects of actions in multiagent systems. In Inf. Conf. on Learning Representations (ICLR), 2020.
M. Waseem and Q. Chang. Enhancing flexibility in smart manufacturing: System property enabled multiagent approach for mobile robot scheduling in multiproduct flexible manufacturing systems. arXiv, ar ArXiv:2504.2000
2025.
M. Wen, J. G. Kuba, R. Lin, W. Zhang, Y. Wen, and et al. Multi-agent reinforcement learning is sequence modeling problem. In Proc. Conf. Neural Information Processing Systems (NeurIPS), 2022.
Yaozu Wu, Dongyuan Li, Yankai Chen, Renhe Jiang, Henry Peng Zou, and al. Multi-agent autonomous driving systems with large language models: A survey of recent advances. arXiv, arXiv:2502116004 2025.
Y. Yang, Hao, Liao, K. Shao, G. Chen, and al. Qatten: A general framework for cooperative multiagent reinforcement learning. arviv, arXiv:2002.03939 2020a.
Y. Yang, Ying Wen, Y. Chon, H. Wang, K. Shao, and et al. Multi-agent determinantal q-lcarning. In Proc. Inl. Conf. on Machine Learning (ICML), 2020b.
Yifan Zhong, Jakub Grudzien Kuba, Xidong Feng, Siyi Hu, Jiaming Ji, and et al. Heterogencous- agent reinforcement learning. In Journal of Machine Learning Research (JMLR), 2024.
M. Zhou, Z. Liu, P. Sui, Y. Li, and Y. Y. Chung. Learning implicit credit assignment for cooperative multi-agent reinforcement learning. In Proc. Conf. on Neural Information Processing Systems (NeurIPS), 2020.
M. Strupl, F. Faccio, D. R. Ashley, R. Kumar Srivastava, and J. Schmidhuber. Reward- regression converges to global optimum. arx, mxXiv:2107.0008: 2022.
14

---

Published conference paper at ICLR 2026
A APPENDIX
A.1 NOTATIONS
We summarize the notations used throughout this paper. These notations span policy representa- tions, value functions, diffusion-based objectives, and auxiliary constants introduced in our theo- retical analysis. Table 2 provides comprehensive overview.
Table 2. Summary of notations used in the HAQO framework.
Notation
Description
Ti, old mew
Policy of agent i; old and updated versions in sequent ial optimization.
II;
Policy class for agent i.
J(x)
Expected return under joint policy x.
('(s,a)
Action-value function (critic) for agent State-value function for agent i.
((()
as)
Advantage function for agent i: A'(s, = Q'(s, Vi().. State visitation distribution under policy m.
Pn
DRr. (1{
Bullback-Leible divergence between two policies of agent i. Trust-region radius (KT. constraint) for agent i.
6,
N
The number of agents in the environment. PPO clipping parameter for ratio constraints. Upper bound on per-step reward. Discount factor.
Rmax
Y
Constant in sequential improvement bound, C. C== = O Rimax
C
(1-7)²
0,
Parameters of agent i's diffusion policy (denoising network).
e, tn
Gaussian noise and denoising network output in diffusion objectives. Diffusion noise schedule parameters timestep f. Advantage-based weight function (e.g., Gadv, (eut).
as, at
wi(s, Oz. a.))
Transformation: max(0, A'ss,, aa) - Eag-rpla A'(s, ā.)
Gadv
Transformation: max A',,,, a)) 1[A'(s, a.) E). Scaling coefficient for entropy regularization.
Jcut
Covariance of action distribution for agent at state s.
d:
Action dimension of agent i. Slack term due critic bias: & O(Eq). Maximum bias in critic approximation.
&
EQ
Am (8, aimm)
Incremental contribution in sequential advantage decomposition. Random permutation order of agents for sequential updates.
i1:
(mgold)
Feasible trust-region neighborhood around baseline policy.
( Ko, Ke
Drift penalty term (KL divergence or PPO-style clipping). Candidate counts: Kù for behavior sampling, K, for target critic up- dates.
15

---

Published conference paper at ICLR 2026
A.2 HAQO ALGORITHM
This section provides the complete HAQO training procedure in Algorithm 1. While the manuscript highlights key theoretical components, i.e., sequential updates, Q-weighted surrogates, and entropy regularization, this algorithm consolidates them into a practical pipeline specifying critic updates, sequential agent permutation, and variance reduction through candidate sampling. This allows readers to directly connect thcorctical analysis with implementation, and ensures reproducibility.
Algorithm 1 HAQO (Sequential CTDE with Diffusion Actors)
1: Inputs: Replay buffer D; Actor policys {To Yi=ı Critic policy {Qu); Trust radii {6.) / PPO clip c; Entropy weights {az]; Candidate counts K2 (bchavior), K, (target).
2: for iteration 0, l,... do
Data Collection
3: Sample candidates: For each state and agent i, draw K₁ candidates af m (-1s).
4: Score actions: Evaluate each candidate using critic: Qu (s, (s,aaj.a))
5: Execute best: Select joint action a = (a".*...,, "..."","""""" aa=*)) with a E arg max, Vu
Store transition: Append (s, as. s') replay buffer D.
Critic Update
7: Targets: Train Qú by minimizing TD loss with K, < Kb:
v = re + E Na (1444) ( a', [
Actor Update
8: Permutation: Sample a random order 11:-
9:
for m m= n do
10:
Sample batch: Draw mini-batch B= a, from D.
11:
Advantage est.:: Compute A (s, a²m) (e.g.; GAF or TD(A)).
12:
Weight computation: Set weights with (s, atm a fm) using Eq. (6):
qadv: W + max{0, A'm
Egt 1 AA (s. a }]
qcut: w'th + max(0, A'm (s, (ima"")) 1(A (s,a'm) 2 o)
13:
Diffusion target: Add noise at. = Vai a VI as e with (O,
14:
QV surrogate: Construct the Q-weighted variational objective as in Eq. (5):
(0'²) = Es, t, [w'm (imm((x amm fm). 116-1-e60 cgim egim (at 4,1)
15:
Entropy regularization: Add entropy surrogate from Eq. (9):
T.en (0') = asim E., (Atm), [--lā²m porm (āt. t, 8) }1]]
Policy gradient: gmm V Vetn (JOV + Jent
16:
from Eq. (11).
Fuim (atm (s)
17:
PPO: Define importance ratio pi (0)
Kyrm (atm and the clipped surrogate:
old
Colip (0) [ss,a)= min (r (0) A aim). clip(r tmx (0), 1 1- 1 + Atm (s, a a²m))
3KL E, (m 6tm (11 mm (115)).
18:
Drift Penality: Take gradient step (O). (0). Parameter update: Hmm +- fio n
19:
16

---

Published conference paper at ICLR 2026
A.3 STANDING ASSUMPTIONS
Our analysis builds upon collection of assumptions that are fundamental in the reinforcement learning literature. We cmphasize their particular significance in the heterogcncous--geen diffusion- based selling of HAQO. Each assumption represents both technical and modeling choice that identifies the precise theoretical regime in which our monotonicity and variance guarantees hold. Assumption A.1 (Bounded rewards). There erists a finăte constant Raax > such that
|r(s,a)] & Rmaxx ((s,a)
(13)
As a consequence:
Rmax 1-7" ViEW,
[Q'( -
(14)
Rmax 1-7' [s(ss & 1-7
2 2maax
[s)) S
(15)
Discussion. This is the classical bounded-reward assumption used in performance difference bounds (e.g., Schulman et al. (2015)). It ensures that all value functions are uniformly bounded by geometric scries argument:
8
2
Rmax 1-9
Riax
(16)
Several important consequences follow:
Finite surrogates. Every surrogate objective we optimize, namely ddbantagg--oightted Q-weighted, entropy-regularivee objectives, remains finite. In particular, (T [E,a a) log (a|s)] in Proposition 5.2 would become ill-defined without bounded Q'.
Concentration control. When analyzing stochastic gradients (Section A.10), the bound- edness translates into a uniform envelope Ig(a)) - = 0 { ), which enables the appli- cation of Hoeffding-tyyy type concentration inequalities for our variance reduction proofs.
Improvement guarantees. The performance difference lemma (Lemma A.8) and se- quential improvement bound (Proposition 5.1) both require bounded rewards to control the magnitude of the constants C; 0 ( appearing in the trust-region penalty terms.
Assumption A.1 is essential beyond theoretical convenience: it establishes a fundamental depen- dency structure wherein bounded rewards induce bounded value functions, which in turn guarantee bounded surrogates and gradients. This property indispensable for both HAQO's thcoretical guarantees and the statistical concentration arguments that support our analytical framework.
Assumption A.2 (Discount factor). The environment admits a discount factor y E (0. This ensures that infimile-horizon returns are well-defined and finile, which enables converyent value function updates. The discount factor also provides exponential decay of future rewards, which essential for the concentration inequalities and finite-sample analysis our theoretical framework.
Discussion. A subunit discount factor ensures well-defined returns and induces the resolvent oper- ator (T - -P. This property is critical for establishing bounds on state-distribution shifts (sce Lemma A.10). At y 1, monotonicity proofs generally fail without ergodicity assumptions.
17

---

Published conference paper at ICLR 2026
Assumption A.3 (Regular policies). Each per-agent policy x² s) satisfies the following:
Absolute continuity: T'(- 1 s) is absolutely continuous with respect to a fized reference mensure (e.g., Lebesgue mensure continuous action spaces or counting measure on dis- crete ones). This ensures that densities r' s) crist and KL divergences are well-definer
Continuous parameterization: The mapping from parumelers 0' to distribulion T 8) is continuous, such that small parameter changes induce small distributional changes.
Bounded gradients: On any compact parameter set, the score function Vo log Ty (a' s) is uniformly bounded. This condition prevents exploding gradients during optimization.
Discussion. This assumption represents a standard smoothness requirement in modern RL theo- ries (Schulman et al., 2015; 2017; Ding al., 2024). Tt. provides the following theoretical guarantees:
Well-defined trust regions: KL divergences and total variation distances between successive policies exist, ensuring that constraints such those in Assumption A.5 meaningful.
Stable optimization: Since policy updates induce only small, gradual changes, mathematical tools such as Neumann series expansions can be systematically employed to bound the magnitude of state visitation distribution shifts, which ensures overall training stability.
Compatibility with diffusion models: In the setting assumed in this study, the denoiser underlying the diffusion actor must propagale gradients stably through multiple timesteps. Boundedness ensures that error signals do not amplify across the reverse diffusion process.
Analogy classicul smoothness: Just as TRPO and PPO require smooth log-likelihoods of Gaussian policies to establish monotonic improvement, HAQQ requires similar regularity such that the Q-weighted diffusion surrogate behaves in stable and predictable manner.
This assumption prevents problematic policies such those with sudden discontinuities or extreme spikes and aligns with standard practice of employing modern neural networks with smooth activa- tion functions, which naturally satisfy these smoothness requirements in typical implementations. Assumption A.4 (Critic bias). There erists a finite constant eQ 2 such that for every agent i N and all
pairs (s,
(((s,a))Q(ss,aa) S EQ.
(17)
Discussion. This assumption explicitly bounds the approximation error of the learned critic Q* relative to the true action-value function Q', Several insights are worth highlighting:
Uniform bound vs. unbiasedness. We do not require Q' to be unbiased (which is generally unrealistic with function approximation); it suffices that the worst-case error is bounded by cq. This aligns with standard practice in approximate dynamic programming and fitted Q-iteration, where crities are treated as approximate oracles.
Propagation into surrogates. The bias directly affects how tight our Q-weighted varia- tional lower bound becomes (Proposition 5.2). Specifically, the Q-function error Eq creates an additive slack term & = O(Eq) in the inequality
(* 2 J.Q((8))
(0') - E.,
(18)
18

---

Published conference paper at ICLR 2026
which then builds up across all agents in our monotonic improvement guarantee (Theo- rem 5.4). Therefore, critic bias is the sole source through which approximation crrors can weaken our improvement bound.
Interpretability of (Q. Since (Q' & Rm ~) under Assumption A.1, the relative error ratio (Q/ 7)) provides a normalized measure of critic quality, In practice, smaller EQ implies tighter lower bounds and faster convergence; larger EQ manifests as looser guarantoes but does not invalidate the overall framework.
Robustness. The assumption covers worst-case deviations due to (i) function class mis- specification, (ii) stochastic training noise, and (iii) finite-sample estimation error. Even when critics are biased, as long as their error is controlled, the HAQO analysis remains valid. This robustness is critical for practical MARL where perfect value estimation is infeasible.
In short, Assumption A.4 formalizes the fact that our guarantees degrade gracefully with critic quality: the algorithm inherits the approximation bias only as an additive slack, preserving monotonicity guarantees up to (leq)-
Assumption A.5 (Trust-region fcasibility). Each per-agent policy update is restricted to local neighborhood around ils previous policy. Specifically, for each agent i E N we require either:
(i) Expected-KT. KI. constraint: Ex- KI(TT'd 18) x 7. (-1s)) s 8,,
(19)
(0) PPO-sbyle ratão clipping: x (a 8) (a 8) E [1 c, c)
which implies KL bound of order 6; = (
Discussion. This assumption provides the structural safeguard needed to ensure provable mono- tonic improvement in the multi-agent setting, Several points are worth noting:
Sequential vs. simultaneous updates. In heterogeneoous-agent optimization, each agent updates in sequence while treating others as fixed. Without a per-agent trust region, devi- ations could compound multiplicatively across agents, causing the surrogate decomposition in Proposition 5.1 to break down. The trust-region ensures each step is "local" enough for the performance difference expansion to remain valid.
Analogy to TRPO. This is the direct multi-agent analogue of the TRPO monotonicity condition (Schulman et al. (2015); Kuba et al. (2022a)). In single-agent TRPO, bounding the KL divergence ensures the new policy does not deviate too far from the old one, thereby controlling dittrdiitti shift. In HAQO, we enforce the same principle per-agent, so that the sequential improvement terms telescope without uncontrolled cross-agent error.
Choice of implementation. Either the explicit expected-KL constraint (solved via con- jugate gradient or line search) or the simpler PPO clipping mechanism sulfices. Both are compatible with diffusion actors, since they only require computing state-wise KL or ratios, not lig-likel gradients. In practice, PPO-style clipping often provides a cheaper and more stable approximation.
Effecc of 5, The radius 6, (or equivalently c) governs the varianee tradeoff: smaller 5; ensures tighter thcoretical guarantecs but slower policy change; larger 6, accelerates
19

---

Published conference paper at ICLR 2026
learning but risks violating the monotonicity bound. In our theory, the effect appears as quadratic penalty term C,07, making the choice of ò; critical for balancing safety and elliciency.
In summary, Assumption A.5 is the cornerstone that enables the sequential breakdown of perfor- mance improvement. Without it, individual agent surrogates could deviate arbitrarily from the true joint advantage, breaking our guarantee of steady progress.
A.4 ADVANTAGE DECOMPOSITION
The global advantage function in multi-agent reinforcement learning be unclear due to its dependence on all agents' actions. To analyze sequential updates, we need decomposition that altributes the joint advantage to incremental contributions from each agent during ordered updates. Lemma A.6 provides this by expressing the overall advantage as a telescoping sum of conditional increments, with each increment associated with one agent's decision given its predecessors. This decomposition serves two purposes: theoretically, it enables tractable analysis by aligning local surrogates with global improvement, and practically, it clarifies how each agent's 's update is evaluated relative to others' most recent policies, reflecting HAQO's sequential optimization structure. This lomma formalizes the concept that global coordination can be understood as sequence of locally zero-meea,, advantage-driven updates.
Lemma A.6 (Sequential advantage decomposition). Fiz permutation iim Define the conditional
value chain
((m)
E 4m mmm mm4 (s,aa")))))
(20)
with ()) (s) and (s, (" a). Set the per-step increment
Am ( = ((m)) (s, (s, amum) - (mm-1)(ss (8, amm-)))
(21)
Then for any (8,a) holds that
(s, = Σ ast and Eg Am =0 O.
(22)
m=l
Proof. We proceed in two steps. First, establish telescoping decomposition of the conditional value chain, which allows to express the joint advantage as sequence of incremental differ- ences along the agent permutation. This ensures that each agent's contribution can be isolated without losing global consistency. Second, we verify that each increment has zero expectation un- der the corresponding agent's 's policy, a property that guarantees local unbiasedness and aligns the decomposition with the standard performance difference arguments in reinforcement learning.
(1) Telescoping decomposition. By definition,
A (s, aftm))
(23)
20

---

Published conference paper at ICLR 2026
If we sum these increments over m I,...,, n, the intermediate terms cancel in a telescoping fashion:
A (s, a')m) (!") -000m)))))0)))) (s).
(24)
By construction (s, = (= a), since all agents' actions are fixed, and ()) VT(s),
since no action has been chosen yet. Therefore,
(s =
= (s,
(25)
77
which establishes the first identity.
(2) Zero-mean property. Fix m. By definition,
Q(m) (s, afum)
y fo tin mmm [Q" (s
(26)
Now take expectation with respect to amm ~ m/ (-18):
Eg Eaimarii (Q(m) (Q adum)) Eggimniii 18 qim fi worfrrtt [Q" afum)].
(27)
But this exactly recovers Q(m-1) ahim-1), because in Q(m-1) the actions of agents im...., in drawn from their policies. Hence,
Egtm ni (Q(m) (s, (s) (((==))((s af.m 1).
(),
(28)
Subtracting gives
Batm Am (8, afum)) = 0.
(29)
Two-agent example.
Fix the permulation 11:2 (1,2). Define the conditional value chain
(s) (*((),
(s,aa" Bea (17 (sT
(s,a' a²) = Q as
The increments are
An(s,at) - -
((ss,a',a)) (1)) - (s,(a,)))
Zero-mean for m= = Take expectation over the local policy of agent 1:
E Bat (s,,, a²)] = E.y (I 1 (s)
Eit Eu, Q (0 ,a)) - (s)
(0= (s) (=(s)-V* V*(s) 0.
21

---

Published conference paper at ICLR 2026
Zero-mean for m 2. Now hold a' fixed and take expectation over the local policy of agent
Rolms A268, a², ,d²]] E2 Q(2) , a²) - (s.
En² Q* ((s,a',o ,0)) Ena ( ,a) =
Telescoping. Finally,
As(s,a)) + A2(ssa a²)== a") (Q)= (Q -00) -00 - (*))) as,
This concrele case shows exactly what "subtracting gives" means: Ea ~xm (Q(m)) = =Q(m-1), 80 subtracting ((m-1) makes the expectation of Am equal to zero.
Conclusion. We have shown (i) the telescoping identity yielding A a) - Emn Am Am( and (ii) that each increment Am has zero expectation under its local policy.
Remark A.7 (Interpretation). This lemma gives an ezact decomposition of the global advantage into a sum of conditional advantage increments, each attached a specific stage of the sequential factorization ii.. Several insights follow:
The decomposition is not unique; each ordering in. yields different chain Am. This márrors the sequential update in HAQO where agents are opti- mized in random order, ensuring fairness and preventing bias from fixed roles.
Permutationn-cwarrnsss
Zero-mean increments. Each Am integrates to zero under the local policy n'm Thuss, Am behaves like a true advantage lerm: only dislinguishes actions relative the baseline of that agent's 's own policy, conditioned on predecessors' actions.
Bridge to surrogates. In practice, the per-agent surrogale Lim. in HAQO uses exaclly Am its advantage signal, together with Q-weighting (qadv/qcut) and diffusion denoising. This makes the learning signal algebraically exact, in contrast to heuristic decompositions (e.g. marginal advanlages) that may introduce bias.
Am resembles the incremental contribution of agent im in a coalition game, analogous to a Shapley- marginal. Hence, the sequential decomposition can be seen as tractable, policy- dependent Shapley approzimation.
Relation to cooperative game theory.
Implementation. When combined with QVPO-style weighting, Am defines the "con- ditional advantage" input to the diffusion policy objective. Sampling multiple candidate actions per agent and evaluating them under Q(m)) directly implements this decomposilion in practice.
A.5 PERFORMANCE DIPFERENCE FOR MULTI-AGENT POLICIES
To establish monotone improvement guarantees, must link changes in joint policy to changes in return. Lemma A.8 generalizes the classical performance-differeee identity to the multi-agent case, showing that the return gap between two joint policies depends on the new policy's occupancy distribution weighted by the old policy's advantage. This result is central to HAQO: it explains why distributional drift matters and why trust-region constraints are essential control it. Intuitively, the lemma tells that if new joint policy consistently chooses actions that look favorable under
22

---

Published conference paper at ICLR 2026
the old advantage, overall return must improve up to corrections from state-distribution mismatch. By grounding our later trust-region bounds and sequential surrogates in this identity, Lemma A.8 provides the foundation upon which HAQO's monotonicity theorem is built.
Lemma A.8 (Multi-agent performance difference). For any joint policies n, n',
J(x) 1-y E Eand' B AA (s,
(30)
Proof. We assume E 10 and bounded rewards (so |V <oo), which ensures all series converge.
(1) Advontage identity. Using (=,ssa) r(s,a) + (( and (s, (7,
4 (s), have for any (s)
(s, r((,,) + (., (s).
(31)
(2) Telescoping along a x' -trajecctory. Let (st, at)e20 be generated by x' from state distribution do
For finite horizon T,
T Σ 0
T
T 7' (s))) V (by (31))
r A (st, az) =
Σ
r (x) +
1=
T
(r) - (s0)s) + 7 2
(telescoping).
t=0
Taking expectation under n~' and letting 00, the last term vanishes since 7 T+IVa T+
a.s. Hence
B
Σ
A' (Bc.
= E Σ
>rr(se as) Eepnnd (su)).
(32)
120
120
But E,,.. 1120 7/1(s,
(r)ss))))) a:)] J(x') and Exp V (so)) [(TT,, so (32) gives
J(z') - J(T) -E Σ A'A (sc: at)
(33)
(3) Discounted visilation form. To rewrite the RHS in terms of a stationary distribution, define the discounted state-visitee visitation distribution (Dietterich et al. (2001); Schulman et al. (2015)) of m':
d' (8) = (1 - 7)
Pr(st = 8 8| , do).
(34)
1=0
23

---

Published conference paper at ICLR 2026
This distribution reflects the long-run frequency with which states are visited under HT, but dis- counted that carlier visits contribute more heavily than later ones. The factor (1 -1 normalizes d' to be valid probability measure.
Now, for any measurable f: A-
BYvT
Σ (()( ar)
Σ
7 E[f((s az)]
7¹ Es Ba, B
B Band R (ial)
The last step follows directly from the definition of d= which absorbs the wwwiighted froquencies of states into single distribution. Thus expectations over discounted trajectories can be equivalently written as expectations over d"
Finally, applying this identity with f(s, a) (s, in (33) yields
1 E Fond ("" a)]
J
(35)
1 I-
which completes the proof.
Remark A.9 (Interpretation). Lemma A.8 tells us that the gap J(x') - J(r) depends only on
how the new policy x' selects actions that look favorable under the old advantage A", Importantly, the state distribution d= oppears on the RHS, which makes the identily exact but introduces a issue in practical policy optimization. The remainder of this section develops rigorous way to control this mismatch.
ceneriittioo-mmmmatede control. To replace expectations under d'' by d=, we need a stability bound. Recall that d' denotes the discounted state visitation distribution induced by policy m. When comparing two policies and n', the difficulty arises because their induced distributions d" and dm may differ, potentially invalidating performance-differenee arguments if left uncontrolled.
Let |f sup, [f() be the uniform norm on test functions, and define the maximal per-state deviation between the action distributions:
a( T) sup (V *-(s)).
(36)
Intuitively, a(T',T) л measures the worst-case total variation (TV) distance between the two policies across all states. This quantity captures how much n' may deviate from a in its action selection at any state, and thus bounds how quickly the trajectory distribution d= can drift away from d", small a<<'', m) implies that the two policies are locally similar at every state, which ensures their long-term discounted visitation distributions remain close.
24

---

Published conference paper at ICLR 2026
Formally, standard coupling and resolvent arguments show that
27 1 S
||d d=
a(T' T).
This inequality quantifies distributionn-misaaach the shift in state visitation frequencies is con- trolled by both the per-state policy deviation and the geometric factor 12 In practice, this bound allows us to safely replace expeclations under d" with those under d' up a controlled error term that scales with a(x', .T.
Hence, a(T', π serves as the stability coefficient. bridging policy divergence and distributional mis- match, playing the same role in multi-agent sequential updates as the KL trust-region radius in single-agent TRPO.
The stationary distribution changes smoothly with n, at a Lipschitz rate proportional to 7/(1 ?
Furthermore, by Pinsker's inequality, (V,, S v (plll)) we obtain
axx',x) x) $ V KTmax (T
(37)
with (=)= sup, ([(((1)))[[=(( This justifies KL-based trust regions principled way to stabilize distribution shift.
Lemma A.10 (TRPO-type lower bound). Let sups,a (s, Then
47 (1 7) e A(T'TT),
J(W') 2 J(T) Band' a (AT(
(38)
where a(T', sup, TV(T'(1ls), ((11)) denoles the mazimum total varialion dislance between n¹ and across states. Proof. From Lemma A.8, we have the perforraace--ifferencce identity
( I
E Eandy' ( (s,
(39)
Insert and sublraci baseline. The difficulty in directly analyzing this expression lies in the fact that the expectation is taken under the new distribution d= whereas the advantage function A is defined relative to the old policy æ. To control this distribution mismatch, we add and subtract the same term evaluated under d=:
E E [A7] = Rat (A*] + (Egr [A7] - Ear (A7)).
(40)
The first term, Ex- [A"], is well-behaved by definition of the advantage function, it vanishes when actions are sampled from π, but it remains informative when actions are sampled from n'. The second term, Ear[A Ez- [A*], isolates the error due distributional shift between d=' and d=. This decomposition cleanly separates policy improvement (captured by the bascline expectation
25

---

Published conference paper at ICLR 2026
under d=) from distribution mismatch (captured by the difference term), allowing bound the latter using the stability cocfficient a(m', TT) introduced above.
Bound the distribution shift. For any bounded test function f f S + I2 we want to control the difference between its expectation under the new discounted visitation distribution d= and the old one d=. By definition,
g [f] E. [f]
Σ -(((s)) SES
d= f(s).
(41)
Taking absolute values and applying Hölder's inequality (duality of 6 norms),
E [/] Ex- [S] & ||d - d= sup [s()]
(42)
This inequality states that the deviation in expectations is at most the product of (i) the total variation distance ||d= -d d= between the two distributions, and (ii) the worst-case magnitude of f. In particular, when f(s, a) is chosen as an advantage function or increment (which is bounded by Assumption A.1), this provides a direct way to quantify how much policy evaluation error arises from state-distribution drift.
Applying this with (s, (r a) yields
g [A"] Eg- [A7]
s ||dd - d'
(43)
Step 3: Control ||dd' - -d d . It remains to bound the deviation of the discounted visitation distri- butions. A key fact (Schulman et al. (2015)) is that
27
||d d= s 1-7
sup (
(44)
This inequality follows from unrolling the recursive definition of d= and applying contraction prop- erties of Markov kernels.
Step 4: Combine bounds. Putting steps (1) (3) together, we obtain
- 1 Egp [A*] + 1- (Rarr(AT E.g (A7)).
1
(45)
The sccond term is bounded absolute value by
1 1|d dd'l 5
27
2 2a( m))
(46)
1 1-
1-7
26

---

Published conference paper at ICLR 2026
where we used 1pp-9p 1/1 2p.V(p, q). This gives the penalty term
4y
(1 7) ca(T', .).
(47)
Step 5: Final inequality. Thus we conclude
4y
J(x)) 2 J(z) + , ( (1 (1-
c ca(T'
(48)
claimed.
Remark A.11 (Connection to HAQO). This lower bound is the malli-agent onalogue of the classic TRPO improvement guarantee. In HAQO, each agent's 's update is restricted by drift term (KI.-ball or PPO clipping), which directly controls a(x', π). Lemma A.10 therefore prowides the theoretical justification for including ('( (la) in the HAQO objective: it ensures that despite diffusion- based actors (which lack traciable log nT, we can still guarantee monotone improvement up to second- order KL and critic-bias errors. This bridges the gop between diffusion surrogates and classical trust-region policy optimization. Two subtleties are worth emphasizing:
The bound is uniform across agents, not tied to a specific decomposition. only requires bounding the largest deviation a( R), which makes it robust to heterogeneous policies.
The penally term scales with Tー showing that high discount factors 7=1 make policy optimization much more sensitine. to distribution mismatch. In mulli-ogent training, this highlights why entropy regularization (to prevent collapse) and careful trust-region schedul- ing are critical in practice.
A.6 PROOF OF PROPOSITION 5.1 HETEROGENEOUS SEQUENTIAL IMPROVEMENT BOUND
This proposition formalizes why sequential updates yield a stable improvement guarantee the multi-agent setting- The key challenge that simultancous updates of all agents break the performanco-diffrrncc analysis, since each agent's advantage is defined relative to the joint basc- line policy. By contrast, sequential updates allow to telescope the return difference into per- agent contributions conditioned on the most recent predecessors, aligning local surrogates with the global improvement objective. The proof combines (i) the sequential advantage decomposi- tion from Lemma A.6, which expresses the joint advantage as a sum of incremental terms, and (ii) cntiiibutioo-mismmate control, which bounds deviations between the baseline visitation distribution and the updated one via per-agent trust-region constraints. Together, these tools yield monotone improvement bound with quadratic KL penalties that scale only linearly in the number of agents.
Proof. Setup and notation. Let T()) Vold and for m = 1,...,7 define the partially-updated joint policy
(m))
neww
Fold
fmm
(49)
80 that (n) = Nnew- Denote d(m); (m)) Recall the conditional value chain Q and increments Am from Lemma A.6, and the per-agent surrogate (with bascline state distribution and already-
27

---

Published conference paper at ICLR 2026
updated opponents)
L (mT = Beruprodd Ba oos (on Ba Sat (m [m
atmm)]
(50)
Step 1: Per-step TRPO bound. Apply the multi-agent performance difference Jemma
(Lemma A. to the pair (T(m-11) T(m)):
(T()) (m(m-)))
1-7 E ennd))), a= (1
(51)
Equ. (51) using the TRPO-type incquality (Lemma A.10) with fm-1:=su1p (m
47
(T(m)) J (T(m-)1) 2 E sm) a [A (s,
6m al ((m) 1)). (52)
(1 7)2
By bounded rewards (Assumption A.1), [V S Rmax and (Q S 1-7
Mmax hence 1-7
2
6m-1 sup LA (m- (s,a))
< 1
(53)
Step 2: Identify the per-step surrogate gain. At step m only agent im changes between T(m-1) and T(m)- Using the conditional advantage decomposition (Lenma A.6), we have for each
E Ea (A (s,a)) E i' wes by nustew
Am atsmm)
(54)
(m)
That is, when all other agents are held fixed, the expected advantage under the new policy reduces exactly to the m-th increment Am: which isolates the contribution of agent im Therefore the first term in (52) equals the stale-averaged slep-gain
G = Egnaim -m
Im Am (8, aft""))]
(55)
Here d(m-1) the state distribution induced by the partially-updated policy after m steps, so Gm represents the true improvement achieved when agent im updates under the actual trajectory distribution. Our surrogate L however, is defined with respect to the baseline distribution pToua, which avoids dependence on the dristing d(m--1 and makes analysis tractable. To connect the two, we insert and subtract naid:
G = Egnipota E, asm [mm] + (Eadd(--) - BonpTona ) Eg im aeton [Am]
state-mismatch correction
= =(C'm (n/tee) L. (TT
mm)
(56)
mismatch
28

---

Published conference paper at ICLR 2026
where the surrogate difference appears because Bai old
[m]] by the zero-mean property of
conditional increments (Lemma A.6). The additional term A (m)
mismatth, quantifies the distribution shift ertor: the discrepancy between evaluating under the drifting d versus the fixed baseline pTole, Later, we will bound this mismatch using trust-region constraints, showing that it scales no faster than (v88) and hence can be controllod.
Step 3: Bounding the state-mismatch correction. Recall that A mismatch mcasures the dis- crepancy between evaluating conditional increments under the drifting state distribution dm-1)) and the fixed baseline pTola, Formally,
(m)
(m) mismatch
(E 1) Reseptol R Ba-ii aam atm (m atmm))
(57)
value, hence by Assumption A.1: Bound increments. Each Am (s, attm)) ) is the difference between conditional Q-value and basclinc
I sup (mm (s, + sup (*(m)))s) < 2Bmax
sup 1mm(s, (m)) 4,aflim
(58)
8,0
Relating mismatch to L distance. Therefore, applying the general inequality [|E - IS, s |pp 9ll2 sup lf] yields
A
(m) mismatch
[m(mm-1) prald ||I
2Rmax
(59)
Bounding the state distribution shift. The I difference arises from the composition of the m--1 pre- vious updates. By the perturbation bound on discounted visitation distributions (cf. Lemma A.8),
we have
[|(mm-1) - -ottall ||
2y
Σ
(60)
7)2
Control via trust region. For each step j m. the single-agent policy change is controlled by the trust region:
a(FG): S sup TV (-18), vidal (-1s))
(61)
By Pinsker's inequality,
sup TV( W (11s), n (-1s) s
v KT Tia |l mieww
(62)
Finally, Assumption A.3 and the finite state-support constant ej tighten the expected-KL trust. region into max-KL. KL bound:
KL KLmaxxTT ll wloww eg 6zg.
(63)
29

---

Published conference paper at ICLR 2026
Consolidated bound. Combining the above, we conclude
||d(m-1) - pMosid 1|1 = 0 Σ
(64)
Jmm
and hence from (59), we get:
mismmtth =
(m)
(F J.m
d Rmax
Σ vos))
(65)
Thus the mismatch term grows at most sublincarly with accumulated trust radii, and will later be absorbed into the quadratic penalty via line search.
Step 4: Assemble the per-step inequality. We now combine the pieces. surrogate decomposition (56) into the TRPO bound (52):
Substitute the
(T(()) J (T((m=1)) 2 (L (Thew) L. (ola))
(m) mismatch
(1 7)2 6m-1 al Thew Told! )
47
(66)
Bounding the penalty term.
From (53), cm-1 S
2Raax I-7
Therefore
4y (1 7)2 cm-1 a(Tie Tora) S
8*y
Rmax a(The. Tola).
(67)
(1 (1-9) 7)3
By Pinsker's inequality,
Am al
Sm new oldd
KLmax (wom ||
S /
Cmn 8pm.
(68)
2
Consolidating constants. Define K A 87 (1-a)3 Rmax. so that
4y 7)2 6m-1 au (n ग bra)) S
K
V
Cun 2
v
(69)
Final per-step bound. Plugging this back into (66), and combining with the mismatch bound
[(((m)) (T(m-))) 2 (Cmm (mrrer L (T'ra)
C. v frmm
B
(70)
jkm
for Ö.,, B B= 02 that incorporate the KL-TVV relaxations and the mismatch scaling.
30

---

Published conference paper at ICLR 2026
Step 5: Trust-region line search quadratic remainder. performance bound of the form
The inequality (70) established
((T()))) - J (T)(--1)) 2 AC Cim 6i.. - B
B jmm
6ig
(71)
where AL L (Táww) -fm (ma The difficulty is the presence of root-ô penalties, which are non-quadratic and thus unsuitable for guarantecing monotonic improvement in sequential update scheme.
Ensuring second-order local model volidity. In TRPO, one controls the KL divergence between consocutive policies such that
(L
T new
8:mm
(72)
This ensures that the local quadratic approximation to the surrogate Lim remains accurate. By convexity and smoothness assumptions (Assumption A.3), the surrogate gain grows least linearly in d,m:
Lim (T Lim (Todd))
Riim dim.
(73)
for some curvature constant Ki depending on the natural gradient direction.
Controlling rool-5 penalties. The penalties in (70) involve terms proportional to V and Ejem vo, To make these manageable, recall the inequality:
V8
8, for any 1 >
(74)
2n
This allows each root-ō term to be upper-bounded by a constant plus a quadratic term in 6. The constants can then be absorbed by a backtracking line-search procedure, which scales down the step size until the improvement term A dominates.
Line-search elimination of constants. Line-search ensures that we never accept step where the constant slack terms overwhelm the surrogate improvement. Thus, in practice, the non-quadratic root-ô penalties are converted into purcly quadratic penalties, while maintaining improvement guar- antees.
Final quadratic remainder. Consequently, there exists a constant
C.. =0 o Rimax
R (
(75)
such that the per-step inequality simplifies into clean surrogateeminu-qquuaaatt form:
[T)m)) J (m 2 (Cim (r/ L (m
C.m
82
(76)
31

---

Published conference paper at ICLR 2026
This is the desired TRPO-style guarantee: each agent's update improves performance up to quadratic penalty in the trust-region size, ensuring stability even in the multi-agent sequential setting.
Step 6: Sum over m (teloscoping). Summing (76) over m 1, ,,.., telescopes the LHS into J( (wwww) - (maaa) and the RHS into the sum of surrogale gains minus quadratic penalties:
[neww - J (Tola)
2
- L (Tola)
-
Cin Sm
(77)
m=
m=1
Relabeling im + yields the proposition.
Remark A.12. Several subtleties are worth emphasizing:
Source of constants. The constants C; come from three layers of bounding: (i) the TRPO-lype lower bound introduces a factor ~Tp (ii) advantage magnitudes are bounded by e S 2Bua> under Assumplion A.1, and (6i)) conversion of total varialion dislances inlo KL divergences requires Pinsker-type inequalities, introducing v. slack Together these yield C, = 0l (7
I-7
From root-ô to quadratic. A naive analysis leads to vo, penalties, which would be too weak to guarantee improvement under small but frequent updates. The line-search or adaptive slep-size mechanism essential: ensures that the local surrogate improvement dominates the first-order term, leaving only a quadratic penalty. This mirrors the role of backtracking in TRPO, where shrinks the step until the sccond-order approzimation is valid. Intuitively, the quadratic dependence reflects the fact that two sources of drift (advantage approzimation and distribution shift) must interact for performance to decrease; if either were absent, the slep would be safe.
Sharpness of the bound. The conslant C₁ is generally conservative: il reflects worst- case perturbations across all states, while in practice the effective drift smaller because policies are only updoted significantly on states frequently visited under pr As result, the empirical improvement observed during training often much stronger than the pessimistic theoretical guarantee. Still, the quadratic penalty term is essential to rule out adversarial counteresamples and to establish monotonicity theorem.
Connection to practice. In praclical MARL inplementalions, one often replaces the KL constraint with either PPO-style clipping or adaptive trust radit. Both mechanisms im- plicitly enforce the same small-step property, keeping 8, sufficiently small for the quadratic remainder lo be negligible relative to the surrogale guin. This explains why monolonic im- provement is observed in practice despite the complerity of multi-ogent updates. Moreover, sequential updates ensure that the constants C; do not scale exponentially in the number of agents, avoiding the curse of dimensionality that would arise under simultaneous updates.
In summory, the proposition provides theoretically conservative but structurally sharp guarantee: sequential updates achieve monotone improvement up to controlled quadratic penalties. The role of conslants is mainly lechnical, ensuring robustmess of the proof, while in practice the trust-region mechanisms make the bound tight enough to stabilize training.
32

---

Published conference paper at ICLR 2026
A.7 PROOF OF PROPOSITION 5.2 HETEROGENEOUS QV LOWER BOUND
This proposition establishes that the Q-weighted variational (QV) objective provides a valid sur- rogate for policy improvement in heterogencous multi-agent settings. The central difficulty that diffusion-based actors do not admit tractable likelihoods, preventing the use of classical log-ratio policy gradients. Instead, rely a denoising formulation weighted by conditional advantages. The proof procceds by (i) showing that, under nonnegative weighting schemes (gadv or qeut), the QV surrogale lower-bounds the per-agent improvement surrogate, and (ii) quantifying the slack introduced by approximate critics as an additive error term order ((eq). These arguments demonstrate that the QV objective retains the monotonicity propertics of advantago- weighted pol- icy gradients, while remaining implementable for expressive diffusion policies without requiring explicit likelihoods.
Proof. We proceed in five steps, moving from conditional advantages to a weighted denoising re- gression and then to a Fisher-type quantity, and finally comparing it to the per-agent surrogate.
Step L is a conditional-avvanage objective. By the sequential decomposition (Lemma A.6) and the per-agent surrogate definition, for the update position with in =i,
('(T') = EompTola E Bi- En'na [ (s, (ftmm) }]
(78)
where Am(s, a Am(s)ahmm) = ((m) (x, afrm))))m))))))m)) ((m afm--1) is the conditional increment. Interprela- tion: L is the old-state expectation of the (new-opponents) advantage signal that the i-th agent realize by changing its marginal x¹.
Step 2: QV surrogate as a weighted denoising regression (law of total variance). The QV surrogate (5) reads
(0') = Eanatata Eg inmi E Ba
ona oft [u'(s, a' a ||e. c (z, (2,) == +V1 ai
(79)
Fix (s,a and abbreviate expectations by E|-] Eageal.. Let mlz) = Ele s, (the Bayes estimator).
To analyze the risk, recall the Pythagorean identity in L2; for any iquare-integrable random variable X and any measurable function f(2),
[[| [(=)] E[IX- - E[X =]|]]] B[| | =] - [=))]]]]
(80)
This is a direct consequence of orthogonality of conditional expectations in Hilbert spaces, since
E [X - E[X x)) [E[ 2] - f())) = O.
(81)
Applying this identity to X = e with weight wt gives
J,O((8) s.a E[w" ||c m + Blw [1m(2) ear (z, (z,
(82)
33

---

Published conference paper at ICLR 2026
Hence the minimal (best achievable) weighted risk is the weighted conditional variance:
int J, (0' 188a Blw" Var(e | [2,8,
(83)
Averaging over (s, (s then yiclds
inf J.
(0)) -E,,a-1 B(w Var(c 2,
s,)) ]
(84)
Further derivation.
Note that the conditional variance can be written as
Var(c 8, [[[[]] s, Ilx(2)]1?
(85)
that
inf JiO s,, = B[wlElllel? s, EEW" [[)))))
El
(86)
This decomposition makes clear that the irreducible part of the risk comes from the intrinsic noise level of e (the variance term), while the reduction due to eg: depends only on approximating μ(=). Consequently, the Bayes estimator μ(z) uniquely achieves the minimal risk, and any parametric approxination egi incurs an excess risk
E(0') = B[w])l((2) - c (2,
(87)
so that
J. 18,a inf J.Q (, + E(O').
(88)
Thus, the optimization of the surrogate is equivalent to minimizing the excess risk E(0") over the hypothesis class for Epc.
Step 3: From denoising risk to a (weighted) Fisher divergence. We now make precise the claim that the QV denoising risk (up schedule constant) a weighted Fisher divergence between the true and model reverse conditionals.
Fix (s,a=1,() and recall the Gaussian forward corruption
Vaca + VI- E~ 10,
(89)
Let pc(x s) denote the t-marginal under the old policy (integrating out a' given and a²). Define the true conditional score and the score model by
(e(2 (s) Vo 1s),
$0.0r 8) is the score induced by the model at time
(90)
34

---

Published conference paper at ICLR 2026
The QV loss with weights w' 20 is, from (82),
J.aY (0' (s = Es fe (2, = B|w
(l)(2, 1)) (91)
where 811, ((2,,,t)-Eee Ele 8, is the Bayes estimator.
(i) Tweedie/scon identity for Gaussian corruption. Write the joint density Pr(z, a' s)) p( a²) Tola (a' s) with p(z aV)= vara", (1 - T)) Diffcrentiating the marginal pc (s)
J p(> Tla (a' () da under the integral sign,
/
V. (oggp(==s))
V v=p(z a²) aaaaaa
8) da' E
V logp(z (2 aa') 2
pe(z 8)
E
2
2,8
2
E aa 2,5)
(92)
O2
Since (z / 1/11 Cit; laking the conditional expectation of given s, yields
M(2,8, 8, == -Elc| 2, 8, 2 v 0g E[a' s. VI-
δt V log 8) = VI - ā. se(2 s).
Oz
(93)
Thus, the Bayes-optimal denoiser linearly related to the true score.
(ii) Relating excess denoising error to score mismatch. For any (measurable) denoiser Egs, decompose
Egi (E-H) u) + (# -Eg),
(94)
and note that Efe - / 2, s, = 0. Hence, for any weight w' that is measurable w...t (z, a', s,t),
B[wi||c co. |13] = [B['|lc ||c - 14|33] + B[w"|l co. [13] + 2 BL n. cor>]
[[[w"|c 10|l3] + Biw²|| co. [1²], (cross term 0 by conditional mean zero).
(95)
Using (93), the second term satisfies
|a Eg: 122=1
=1 -I as se(2 5) Egi (2, s, s)]l
(96)
If we parameterise the model denoiser via score network as co; vI ai s0,0 s) (the
standard DDPM/DDIM correspondence), then
|| co: 212 (1 a.) [1 s) ( ss][
(97)
35

---

Published conference paper at ICLR 2026
Plugging (97) into (95) and taking expectations over (s, "9,'"',a', yields
(0')= F("(((')= Blw ml1
Blw (1 (xs) || 81,0+ [1
weighted irreducible variance
= Var(c 2,8,2) t)]
+
Ki Essuptold B, inaain Fa', w (0, a², a- " ||st 8 (98)
where k; (1 ā.) in the denoising parameterization above (more generally, Hi > 0 is a schedule- dependent proportionality constant when mapping denoiser parameterizations score parameter- izations).
(iii) Taking the infimum and identifying Fisher divergence. The first term in (98) does not depend on 0'. Therefore,
inf (0') const Σ Kr inf VEw* ||s: 01,0011 const +
Σ Kt Drisher (P: Pr.o:) (99)
E Elw" || - $c,0
5 10 11
which is exactly weighted Fisher divergence between the true and model reverse conditionals (the weighting being w²). At the optimum 0** within the model class,
inf J.OV((0)))) (0') Σni No Banuprota B -aan Eais as, 11: - 12] const. (100)
The schedule factor Ke scales the contribution of each noise level, such that deeper noising (smaller āt) contributes proportionally more to the overall objective. The weight w'(s, a², aa") 2 0 focuses the score-matching loss on as regions with high (rectified or centered) advantage, ensuring that reducing the QV loss improves precisely the parts of the action space that matter most for return. Finally, if the model class contains the true reverse conditionals (8t, = 8t), the Fisher terms vanish and the infimum reduces the (weighted) irreducible variance, consistent with Eq. (83).
Step 4: Nonnegative, advantage-aligned weights yield a lower bound. We make the informal statements precise in three micro-steps: (i) show that the QV loss minorizes its own best- achievable value (a weighted conditional variance) whenever w 2 0; (ii) relate that best-achievable value to (positive) conditional-advantagg mass via bounded-variant and alignment arguments; (iii) normalize away schedule constants and collect terms.
(i) Weighted orthogonal decomposition. Recall the weighted L from (82):
Jo (0' |s,a =[ww' 1eellle-(a) [(=) + E[w" [|x(()) (2,
(101)
with m(z) Eie s). The cross-term vanishes because TE (=(=) | 2, 2, = and wf independent of given (s, a² at). Since wt 2 0 by construction (qadv/qcut) the "excess-risk" term
36

---

Published conference paper at ICLR 2026
is nonnegative, hence for every 0¹
J.a a 2 inf (.0(0 = E[w Var(e 2, s,s,
(102)
8
Averaging over (s, a t) yields:
J,PY(0") (0') 2 inf J. (0)= =Bga :Egie (wr Var(e 2,,,
(103)
(ii) From weighted conditional variance to (positive) conditional-advantage mass. Let max Ssup,,., Var(e s,), (finite under standard Gaussian schedules). Then
E [wr Var(e 2, s.) < o Batway [w a a-))
(104)
since the inner conditional variance is uniformly bounded and wt does not depend We now bound the RHS for each weighting transform.
Case A (qadv): we (A' - Ba A 2 O. Conditioned on (8, (s,a the conditional incre- ment/ade cont/advantage identity (Lemma A.6) gives a'ss,a²;a-i) = (m afum) and EgiNay Am = 0 (zero mean). Then
Baiway [w'] Engia ((Am I= < Batna [+]]
(since EAm 0). (105)
When agent updates to increase the surrogale L (ef. (78)), the new policy n² places (weakly) more probability mass on improving regions {Am 0}, hence! Eaenath A I Eninoi [A Therefore,
Eag [w' Varle 2,88 0) S of B A (8, attm))]
(106)
Cose B (qcut): w' = A4. 11A' 2E. E). As A 1(A 2 $) S A4r
the same alignment argument gives
Es Earazi
[w*)
S E Bainei [ Ey a Nay
A S Eainn
A
(107)
slightly sharper bound uses the boundedness 0 Am S M with M= 2Bmax we have A s M and thus A 1(A' 2 e) S (AL) <
1-7 (cf. (53)): on (A 2c)
M A giving
s á min 1, M E Exrr A
Ea [ Var(e 2, s,,
(108)
A fully rigorous statement replaces this monotonicity by F_g ( R.. mba) Imll- and then the per-agent trust region to bound the TV term; this only tightens the final constants and does not affect the direction of the inequality.
37

---

Published conference paper at ICLR 2026
Combining (104) with (106) (108) and averaging over (s,a-) yields, in both cases, inf J,0((0) (0) S Cvaz Beuuptota E B-wws Eat Am (8, afrm)))
(109)
with Cvar = max for qadv and Cvar = max min[1, M/E} for qeut.
(iii) Normalization across and identification with L'. Per-step schedule factors (the Ke 's's of (100)) can be absorbed into the definition of the normalized QV loss (see the "Normalization and unit-matching" bullet in the main text), so we treat J.QV as already rescaled into a unitless weighted Fisher-type quantity. Using A. A Am with B Am 0 and the fact that w' is chosen to increase Egtnas Am under (s, a a), one has
Exinat A ml S
Batwo [Am) + Baiwnt A
S
Ba'N
(110)
Therefore, by the definition of the per-agent surrogate (ef. (78)),
E., Buwwar
[ S L' (TT).
(111)
Plugging this into (109) and recalling the minorization in (i) gives
J. (0') 2 inf J,
(0) S Cvar (x(").
(112)
Absorbing the finite constant Ovar into the normalized definition of J. yields the ideal (zero-bias, well-spocifid)) variational relation
JO s C'T'',, equivalently ('(=²) 2 (O')
(113)
Where the constants come from. A canonical Gaussian schedule ensures that o S 1, while the bound M 2Raux arises from the supremum norms of Q and V (ef. (53)). The threshold e in qcut provides trade-off between bias and robustness, effectively tightening Cvar through the factor min M/e).
Support/TV refinements. Tf one prefers to avoid the heuristic step of "placing more mass on improving sets," " can be replaced by the inequality |Ezig Erg gl S (T'',, ( sup [gl, with g = A The total variation distance can then be bounded via Pinsker's inequality and the per- agent trust-region constraint. This refinement introduces an O(vō,) slack, which is subsequently absorbed by the line-scarch procedure into the quadratic remainder (scc Step 5).
Bias terms. When using an approximate critic Q' under Assumption A.4, each of the steps above accumulates an additive O(rq) slack-and arising in weight construction, boundedness, and alignment. We capture this residual error by the term in Proposition 5.2.
38

---

Published conference paper at ICLR 2026
Step 5: Bias control via EQ. The previous steps established that, under an exact critic Q' the QV loss J. (0*) forms a valid lower bound to the policy improvement functional (* In practice, however, the critic learned and inevitably biased. We now quantify how this bias propagates and show that the lower-bound guarantee degrades gracefully.
Bias model. Assume the approximate critic Q' satisfies the uniform error bound
('(s, Q'(ss & EQ: V(s,a),
(114)
as Assumption A.4. This error enters both through the weights w', a) (constructed from A') and through the conditional increments Am in (78).
Perturbation of objectives. Because all objects appear linearly in expectations and are uniformly bounded (by Assumption A.1), a standard stability argument yields
(T'( Cirue(TT) S C1 EQ,
J (0') - (0') 02 EQ:
(115)
for constants C1 <2 > depending only on Rmax and the variance schedule. Here, a captures perturbations of advantage increments, while 02 accounts for the effect on the Fisher-divergence terms.
Final inequality with slack. Substituting (115) into the lower-bound relation of Step gives
('(T') ( (01 + 02) eq.
(116)
Define the slack variable & = (a1 + 02) cq = O(cq). Then
('(m²) 2 J.o (0' - E7-
(117)
Interpretation.
If EQ = and the reverse diffusion family well-specifiel, the slack vanishes and the incquality is tight.
If eq > 0, the lower bound remains valid up to an additive slack linear in critic error. Thus, critic bias cannot break the surrogate property but merely shifts it downward by O(cq).
Hence the QV loss remains a reliable surrogate for policy improvement even with approximate critics, cnsuring the robustness of QVPO practice.
Remark A.13 (What the bound is (and is not) saying). The inequality asserts that optimizing the iiffusiion-copptttie surrogate J.QV cannot overstate the per-agent improvement signal L': small QV loss guarantees at least as much (conditional) advantage mass, up to critic bias. The positivity of w is the linchpin without it, negative-adountage regions could let the denoiser "explain away" errors and violate monotonicity. Practically, qadv is smoother/star calibrated while qcut more selective wwwhhhhhhhhiiieee both yield a safe lower-bound surrogate for diffusion actors.
39

---

Published conference paper at ICLR 2026
Remark A.14 (Constants and modeling choices). The hidden constants absorb (i) the forword- noise schedule via faj (ti) strchitt/smmootnesss of the landscape in local trust regions (implicit in Assumplion A.3); and (ii) the centering scale in qadv. Tuning the schedule or and thresholds e controls where the denoiser concentrates learning capacity, trading tightness of the bound against sample efficiency.
A.8 PROOF OF PROPOSITION 5.3 ENTROPY SURROGATE FOR DIFFUSION ACTOR
This proposition establishes that entropy regularization be enforced for diffusion-based policies without requiring explicil log-likelhhoods. The main idea is to introduce surrogale objective that perturbs the denoising process with uniformly sampled actions, thereby ensuring a positive variance floor in the induced action distribution. The proof procoeds by showing (i) non-nogativity of the surrogate term, which guaranlees it can only encourage exploration, and (ii) a lower bound on the action covariance spectrum, which translates directly into an entropy floor. Together, these arguments confirm that diffusion actors inherit the exploration guarantees of maximum-entropy RL, even when likelihoods are implicit and intractable.
The surrogate (9) is motivated by the Gaussian (and more generally, covariance) entropy lower bound:
1 H(A 8) 2 2 (2 det Ei) 2 di
log((2r) 2
logdet E
(118)
where d, is the action dimension and Σ = Var(A' s). Thus, maintaining nontrivial covariance (in particular, nonvanishing eigenvalues) guaranlees that entropy remains bounded away from zero. Although n' implicit, uniform-action denoising provides a tractable way to increase the effective dispersion of the induced policy, and hence to raise the entropy lower bound.
Proof. We refine each step with explicit inequalities and mappings.
Entropy variance connection with spectral floors. Let A' E R4 be absolutely continuous with mean pt and covariance Σ = Var(A' 8). The differential conditional entropy is
Step 1:
H(A' 8) = -f
-fpto p(a' logp(a' s) da'.
(119)
Among all distributions with fixed covariance Σg, the Gaussian N Σ{) marimizes entropy, yield- ing the classical bound
H(A
log ((2Te)a det E.), 2
(120)
with equality iff A I Gaussian. While (120) is an upper bound for fixed covariance, it also implies monotone surrogate: increasing det Ei (or any spectral lower bound Σ,) necessarily raises the right-hand side, hence raises any entropy lower bound of the form
H(A' (s) > H(E)),
(121)
40

---

Published conference paper at ICLR 2026
that depends monotonically on 2 Two convenient spectral relaxations are:
d. II (b 2 (Amin (E:))
det E,
(122)
k=1
de log Aa (xx) 2
logdet Σ
d; log Amin (E²).
(123)
R=1
Combining (120) and (123) gives the spectral prozy
1 log (2xe)4 det Da) 2
di
di log Amin (Ea).
log((
+ 2
(124)
2
2
Therefore, if maintain spectral floor Amin (En) o > 0, then
di
H(A 8) 2 H floor
loog (iin) min) C2
(125)
2
for some constant C'2 0 that depends on shape assumptions of p(a" 8) (e.g., = 0 for Gaussian; more generally C can be bounded for log-concave or sub- Gaussian families).
While the Gaussian expression provides an upper envelope at fixed E, any mechanism that increases dispersion (e.g,, pushes eigenvalues upward) monotonically increases natural entropy prozies such as (124),
mild regularity Nog-ccocavittysiin Gaussian hails)-yeelds a quantitative lower Sub--aasianan bound of the form (125). In HAQO, J. is designed precisely to enforce such spectral floor by preventing variance collapse in every action direction.
Step 2: Exact equivalence up to a known scale between e-prediction loss and uniform reconstruction error. The forward diffusion for actions uses
Von To + v1 - Cr. e, To ā' ~ U(A',, e~ N00
(126)
Two equivalent parameterizations of the reverse model are standard:
c-prediction:
c.. s,t),
(127)
To preerciittion:
u0,0- (8,
(128)
These are linearly equivalent via
(0.0 1,2
1 Vār -0
=
60,(2,8,1) =
1
22 (8,
V1-
(129)
41

---

Published conference paper at ICLR 2026
Moreover, since 2 Varzo ar, a simple algebraic cancellation yields the exact enror identity
6- (0) = Vān (s, - =00)
(130)
vI-a; aq
and therefore
||e 6 c (2, 1 āt
a1 (8, xo =00l?
(131)
Taking expectations the training distribution (at, e, shows that (for each t)
at E | c (2, 011]] 1 ae
R lllaaa (s,(),(,))]1)
(132)
Thus the uniform-action denoising loss used in Jeet is eractly (up to the known scalar factor a ás)) the uniform reconstruction error of actions under the to-paarameterizaton.. No approximation or extra constant is needed: (132) is an identity following directly from the lin- ear change of variables in (129). Consequently, minimizing J,en is equivalent (modulo a known, fixed schedule weight) to minimizing the MSE between uniform target actions and the network's reconstructed actions. As shown in subsequent steps (via Wasserstein/ ieterstttii/vvaanneee variance inequalities), reduc- ing this reconstruction error increases dispersion (trace and spectral floors) of the induced action distribution, which-in turn-raiscs entropy lower bounds through (124).
Step 3: From reconstruction error to dispersion and entropy. Let Po denote the uniform distribution on A and Po the distribution of reconstructed actions åg. (8, z). By the T2 transport inequality, the squared 2- Wasssrssteiin distance controlled by the mean squared reconstruction error:
WP.,P. Po) I B 11..
(133)
On compact action domain A with diameter Di, the dual formulation of Wasserstein distance implies that for any L-Lipschitz i,,
Er, Fell & LWi(Po, Po) s LW2(Po, Pu).
(134)
Choosing ((a) = ||a me 11? where me = [a] is the uniform mean, we have ip is 2D,-Lipcchitz.
Thus,
Ep, ||a - 1 - lla we 111 < 2D, (P).
(135)
Expanding both sides using variance docomposition,
Epzlaa 1 Tr Di + -111 moll?, Epg ||a
pe 12 Tr Ei.
(136)
Therefore,
Tr Ez Tr Eg
2D, Wo Pv).
(137)
42

---

Published conference paper at ICLR 2026
Since W Pv) Bglsae a²ll2 (Step 2), it follows that Tr Mg. 2 Ir Ir 2D. 0²112.
(138)
Thus minimizing Jm controls the dispersion Tr Sg keeping it close to that of the uniform distribution. By Step 1, this ensures strictly positive entropy floor.
Step 4: Multi- agent compatibility and additivity. For a factorized joint policy a(a s) IT (a s), the chain rule of entropy yields
Hia s)= Σ H(a' s).
(139)
==1
Therefore, if each agent i enforces the entropy lower bound, then summing gives
H(a 8) 2 2 log (2rro
(140)
Hence, joint exploration grows additively with the number of agents, ensuring that global explo- ration is not bottlenecked by the collapse of a single agent's entropy.
Step 5: Role and scheduling of aq Each agent solves regularized optimization of the form
('(T') as
J.e (T"),
(141)
where L enforces advantage alignment while Jont enforces entropy. The coefficient as thus deter- mines the tradeoff between exploitation and exploration:
Tf a; + 0, optimization dominated by L leading to greedy exploitation but possible collapse of entropy.
If a; large, the entropy term dominates, effectively pushing Po z Pv, encouraging uniform coverage of the action space.
principled schedule sets a adaptively to maintain dispersion above a threshold:
Tr Eg 2 n Tr Ej K (0,
(142)
For instance, if Tr Σ3 falls below the threshold, increase a; until dispersion recovers. This ensures that entropy never collapses and maintains robust multi-agent exploration.
Putting the steps together: by Step 2 the surrogate controls reconstruction error; Step 3 translates this into dispersion and covariance guarantees; Step 1 links covariance to entropy: Step ensures joint entropy additivity; Step governs the trade-off with exploitation. Hence (9) provides principled, diffusion-compatiiin entropy surrogate.
43

---

Published conference paper at ICLR 2026
Remark A.15.
Uniformm sampling over state agnostic actions Uniform U(A') mazimizzes dispersion on bounded domain and provides simple, state-agnostic prozy. In safety- or feasibility- constrained settings, one can replace U by any full-support baseline (e.g., trancated wni- form or mialure 3 - B)maia) without changing the W2 dispersion argument.
Implicit entropy bonus without log The W2 bound in Step 3 shows that small Jm keeps Po close to broad baseline, which lower-bounds H(a" 8) even though log xO is intractable. This is the diffusion counterpart to the explicit -aH(T) regularizer in mar- entropy RL.
Interplay with QV surrogate. J. pulls probability moss toward high-odvantage re- gions, while J.me prevents over-concentration. Together, they implement a trust-region, entrogy-regularized mirror step that remains policy- improvinng (ef. Theorem 5.4) and avoids mode collapse.
Guarantee scope. The urgament procides a lower bound on entropy (preventing collapse), not on upper bound. complements, rather than replaces, lask-driven shaping from J.Q and the trust-region construint.
A.9 PROOF OF THEOREM 5.4
MONOTONE IMPROVEMENT OF HAQO
Proof. We index one full round of sequential updates by m I,... ..... n using a permutation ipsin write T(m) for the joint policy after the m-th agent is updated (so T(0) sold and T(n) Tnew) and set im to be the agent that updates at step m.
Step 1: A sharp sequential TR bound for one micro-update. Recall the standard performance-difference identity and its trust-region relaxation (TRPO-type bound). For any two joint policies x' and
J(x')-((T) 1 Randdr (x, 2 B Bant (A((s, C C('''m, m), (143)
where d=' is the (normalized) discounted occupancy of n', p is the unnormalized occupancy of ,, and (by bounded rewards) one may take
47
C
(1 (1- Rmaxx
<(T',T) = Espp' (Dr((T() m(-1 [s)]]
(See Pinsker's inequality Dry S / DKL control by expected KL.) Apply (143) the m-th micro-update ((m-1) ((m) while keeping track of which agent changed. Using the sequential advantage decomposition (Lemma A.6) and the per-agent surrogate definition,
((T())) ((( 2 E Ssnp
((m-1) ((n
Ey fm nor
E a mo A (8, atkmm)) C C'Ems
= Lm (m/m))) C Cm.
(144)
Because we enforce trust region for agent imm e.g. (m DKL.. incquality yields c. S /2m
(f/m DKL.. T(m)) di... Pinsker's
44

---

Published conference paper at ICLR 2026
Quadratic remainder via line search. Following the TRPO trust-region line-search argument, we scale the step along the (natural))rradient that the local quadratic model is valid and the model- improvement dominates the vō penalty. Using the elementary bound vō S 1 + 20 for any 1) > 0, 2nj
the Vi.. term can be upper-bounded by a constant plus quadratic term in ōi,,- The constant is removed by backtracking until the mopre-emproveeent condition is met (cf. Step 5 the TR argument). Hence there exists a curvature constant C.m. = = 2022 such that
((T()) J (T) 2 (L (Tm)) - cm
(I fm (mm-)))) 1))) Ci., 02...
(145)
Zero baseline at the old policy. By construction of L (conditional increment has zero mean under the old n²), we have
(x( 0
for the step where only agent i moves.
(146)
Therefore (145) simplifies to
(TT)) - J (m(m-11) 2 bm (5((m) Cim 8²
Telescoping over one round. Summing m to n gives
[(Tnww) - (Tooa)
(' Σ G 67.
(147)
This the sequential surrogate improvement bound.
Step 2: Relating L' the implementable diffusion loss J.QY. From the QV lower-bound proposition (Proposition 5.2) proved via the conditional denoising/Fiite arguments (Steps 2-4 in that proof), for any feasible n' and associated diffusion parameters 8¹ we have
('(7)) 2 J. &,, & O(e))
(148)
The slack & arises solely from critic bias (it perturbs both the weights w' and the conditional increments Am), and it is additive because all terms enter linearly inside expectations and the signal uniformly bounded (A (Assumption A.1).
Apply (148) al the new iterate for agent. i:
L (Feww) 2 J.Q (8hew) E..
Step 3: Adding the (nonnegative) entropy surrogate. The entropy surrogate J!")((") constructed as a (reweighted) uniform-action denoising risk. As shown in the entropy analysis, is nonnegative and pushes up spectral dispersion, thus guarding against collapse. Consoquently,
45

---

Published conference paper at ICLR 2026
adding it can only help in a lower-bound direction:
C (T
J.0% (Pn(oeol + (0
Ei.
(149)
both computable at the new iterate
Insight. We do not need JH to approximate L'; we only use that J: 2 0, so including it on the RHS preserves a valid lower bound.
Step 4: Collecting all terms and finishing. Plug (149) into the round-wise bound (147) and sum over agents:
(TTww) - J (Tola)
2
Σ
J.QV CORV(Oy (Obew) + -J. (Dnew))) -.) - Σ C
==1
J. (Oeew) (ooe((2eew)
Σ
C2 O(eg):
where absorbed E into ((Eq) and recalled C. 0 07tmnss from Step
Remorks on neighborhoods/ /drifts. The drift D' specifies a feasible neighborhood around vold whose size we calibrate to enforce the expected KT, trust radius 6, Any positive drift functional (zero at the current policy, nonnegative elsewhere, with zero Gâteaux derivative the origin) admits such calibration; the line-search rescales the step to stay inside this neighborhood while guaranteeing model-validity and yielding the quadratic remainder in 8,.
A.10 VARIANCE AND SAMPLE EFFICIENCY
While the QV surrogate aligns diffission policies with policy gradients theory, in practice diffusion objectives exhibit notoriously high variance, stemming from stochasticity in both the denoising process and the action sampling procedure. In multi-agent. settings this challenge compounds, as variance scales with the number of agents and can quickly overwhelm learning dynamics. To address this, HAQO employs K-candidate sampling strategy, whereby each agent generates K candidate actions from its diffusion policy, evaluates them using the critic, and selects the most promising sample for training. This design preserves unbias in the large- K limit while substantially reducing variance at practical values of K. Proposition A.16 formalizes this intuition, showing that the variance of the QV surrogate scales down by a factor of 1 providing theoretical justification for candidate sampling as variance-reduction technique. Thus, K-candidate sampling is not merely a heuristic but an essential component that makes diffusion-based HAQO updates tractable and stable in multi-agent settings.
Proposition A.16 (Variance reduction via K-candidates). Let 9 denote the stochastic gradient estimator of (11). Suppose at each update, agent somples K condidate actions {a) from T and selects the top-Kr candidates according to Q'. Then
Var[gk] S R Var[gı]
lim Var[gk] K-00
(150)
4 or K 8 suffices to significantly stabilize training. This variance-control is particularly important in diffusion models, where the stochasticity of e sampling can overwhelm sraantgg--wiihheen signals. Thus the K-candidate trick not only a theoretical guarantee but also a crucial choice for scalability. Moreover, the concentration view clarifies the trade-off: larger K reduces variance sublincar computational cost, yiclding near- -incar improvements in stability.
In practice, =
46

---

Published conference paper at ICLR 2026
A.11 ADDITIONAL EXPERIMENTAL RESULTS
A.11.1 ENVIRONMENTS
Multi-Agent Particle Environment. (MPE) (Mordatch & Abbeel, 2017). Agents interact in a simple 2D physics world with continuous observation and action spaces. Observations typically inchude relative positions, velocitics, and landmark locations, while actions correspond to continuous movement vectors. By assigning asymıetric roles (e.g., speaker vs. listener, pursuer vs. evader), we induce heterogeneity that tests stability under sequential updates.
StarCraft Multi-Agent Challenge (SMAC) and version2 (SMACv2) (Vinyals et al., 2017; Ellis et al., 2022). In these benchmarks, each agent controls StarCraft II unit under partial observability, with local obscrvations including unit health, relative distances, and visible enemy information. Actions are discrele, consisting of unit movements, allacks, and ability usage. SMAC emphasizes heterogeneous roles, such as melee vs. ranged units or healers vs. damage dealers, making coordination essential for success. SMACv2 extends this setup with larger teams, more diverse maps, and stochastic enemy strategies, increasing difficulty by demanding robustness across asymmetric and dynamic scenarios.
Google Rescarch Football (GRF) (Kurach al., 2020). GRF is a high-dimensional environ- ment that blends low-level motor control (e-g., dribbling, tackling, intercepting) with high-level strategic coordination (e.g., passing, positioning, counter-attacking). Agents observe structured state features such as player and ball positions, velocities, and possession indicators, and act through a discrete action space that includes both movement primitives and soccer-specific skills. The bench- mark is particularly challenging because success requires discovering multimodal strategies rather than converging to single dominant behavior. For example, teams must balance dribblc-oriented play with coordinated passing or shooting strategies, depending on the evolving game state. This makes GRF an ideal lestbed for evalualing the expressiveness of diffusion-based policies and the im- portance of entropy regularization, as unimodal Gaussian policies often collapse to narrow tactical pattorns, whereas HAQO can sustain diverse, complementary strategics.
Multi-MuJoCo (Kurach et al., 2020). In this benchmark, multiple agents jointly control different body parts of a single MuJoCo robot. Each agent receives continuous proprioceptive observations, such as joint angles, velocities, and contact lorces, and oulputs continuous lorque commands to actuators associated with its assigned body segment. The environment naturally induces strong coupling and heterogeneity, since effective locomotion manipulation requires tightly coordinated actions across distinct subsystemss Small deviations in one agent's behavior can destabilize the entire robot, making this domain a stringent test of stability and scalability in sequential multi- agent updates. Furthermore, because action spaces differ in scale and influence across body parts, Multi- highlights the need for expressive policies that can adapt to hetcrogencous roles within a single coherent control system.
Bi-DextcrousHands (Bi-D) (Zhong et al., 2024). This benchmark fcatures two multi-fingered robotie hands that must cooperate to grasp, reorient, and manipulate objects in a coordinated manner. Each hand receives high-dimensional proprioceptive observations, including joint positions, velocities, and tactile foedback, while producing contimuous torque commands across dozens of actuators. The task requires fine-grained, asymmetric coordination between the two hands, as successful manipulation often depends on complementary but distinct roles. This environment poses extreme challenges in heterogeneity and action dimensionality, making it a critical testbed for expressive multimodal policics.
47

---

Published conference paper at ICLR 2026
Table Performance Comparison Multiple Particle Environments
Multi- Agent Particle Environments
Environment
HAA2C
MAPPO
HATRPO
НАРРО
HAQO
Reference (continuous)
36.2 (1.3)
-10.1 (0.2)
-13.7 (1.2)
-9.1 (0.8)
-9.6 (0.2)
Speaker Listener (continuous)
-13.7 (6.4)
-8.3 (0.4)
-8.1
-8.5 (0.2)
-9.2 -9.4
Spread (continuous)
-98.8 (2.6)
-69.2 (1.2)
-71.3 (0.6)
-64.3 (7.2)
-60.2 (3.2)
Reference (discrete)
- 14.5 (0.8)
-14.7 (0.3)
-34.7 (2.4)
-13.7 (1.3)
-10.2 (0.3)
Speaker Listener (discrete) Spread (discrete)
9.6 (0.2)
-15.2 (1.2)
-10.7 (0.9)
-9.2 (2.1)
-8.2 (1.3)
68.2 (8.2)
-54.6 (0.1)
-64.8 (1.1)
-22.4 (1.3)
-00.3 (2.1)
Nole: Values represent mean returns with standard devialions in parentheses, Blue indicales other algorithms achieve best performance, red indicates our HAQO method achieves best performance.
Table Performance Comparison on SMAC and SMACv2 Environments
Environment
QMIX
MAPPO
HATRPO
HАPPO
HAQO
SMAC Maps
Sm _s 9m (H)
92.2 (1.0)
87.5 (4.0)
92.5 (3.7)
83.8 (4.1)
90.7 (1.3)
25m (H)
89.1 (3.8)
100.0 (0.0)
100.0 (0.0)
95.0 (2.0)
100.0 (0.0)
5m vs 6m (H)
77.3 (3.3)
75.0 (18.2)
75.0 (6.5)
77.5 (7.2)
93.6 (2.7)
8s5z (H)
89.8 (2.5)
96.9 (0.7)
93.8 (1.2)
97.5 (1.2)
98.2 (0.8)
10m vs 11m (H)
95.3 (2.2)
96.9 (4.8)
98.8 (0.6)
87.5 (6.7)
92.4 (3.1)
MMM2 (S)
87.5 (2.5)
93.8 (4.7)
97.5 (6.4)
88.8 (2.0)
99.2 (0.1)
3s5z 3s6z (S)
87.5 (12.6)
70.0 (10.7)
72.6 (11.7)
66.2 (3.1)
92.4 (1.3)
27m vs 30m (S)
45.3 (14.0)
80.0 (6.2)
93.8 (2.1)
76.6 (1.3)
97.5 (2.3)
corridor (S)
82.8 (4.4)
97.5 (1.2)
88.8 (2.7)
92.5 (13.9)
93.7 (3.1)
6h vs 82 (S)
92.2 (26.2)
85.0 (2.0)
78.8 (0.6)
76.2 (3.1)
96.3 (1.4)
Cv2 Maps
protoss 5 vs
65.6 (3.9)
65.2 (3.2)
50.0 (2.4)
57.5 (1.2)
78.2 (6.2)
terran 5 vs
62.5 (3.8)
53.1 (2.7)
56.8 (2.9)
57.5 (1.3)
64.3 (1.3)
zerg vs
34.4 (2.2)
40.6 (7.0)
43.8 (1.2)
42.5 (2.5)
40.8 (3.7)
zerg
40.6 (3.4)
37.5 (3.2)
34.6 (0.2)
28.4 (2.2)
48.7 (1.2)
zerg vs
25.0 (3.9)
29.7 (3.8)
19.3 (2.1)
16.2 (0.6)
41.3 (11.2)
Note: (H) denotes Hard difficuly,, (S) denotes Super Hard difficulty.
A.11.2 ADDITIONAL RESULTS
In this subsection provide extended results across MPE, SMAC/SMACV2, GRF, and Bi- DexterousHands benchmarks. These complement Section 6.2 and illustrate that the advantages of HAQO generalize across domains requiring different forms of coordination and expressiveness.
Multi-Agent Particle Environment (MPE). Table 3 shows that HAQO achieves stronger stability than policy-gradient baselines in MPE tasks. In cooperative navigation with multiple landmarks, unimodal Gaussian policies used by MAPPO and HAPPO tend to collapse onto subsets of goals, leaving others unexplored. HAQO's diffusion-based policies maintain diverse visitation densities and balanced coverage across landmarks. Sequential updates further reduce run-to-run variance, producing smoother convergence curves. These findings highlight that expressiveness and sequential coordination are both essential to avoid mode collapse in simple yet illustrative tasks.
StarCraft Multi-Agent Challenge (SMAC/SMACv2). Table 4 summarizes results on heterogeneous-umig combat scenarios. HAQO consistently obtains higher win rates than MAPPO, HAPPO, and HATRPO, especially in SMACv2 maps with specialized unit roles. Expressive diffu- sion policies enable diverse tactical behaviors such kiting, zoning, and flanking, which Gaussian
48

---

Published conference paper at ICLR 2026
baselines fail sustain. Sequential updates also prove crucial: simultaneous updates in baselines often dostabilize training, whereas HAQO converges more reliably with lower variance across sceds.
Table 5. Performance Comparison on Google Research Football Enviromments
Google Research Football
Environment
QMIX
MАPPO
НАРРО
HAQO
Pass and shoot with keeper Run pass and shoot with keeper
8.05 (5.58)
94.92 (0.85)
96.93 (1.11)
92.14 (2.13)
8.08 (3.29)
76.83 (3.57)
77.30 (7.40)
80.39 (3.87)
3 with keeper
8.12 (4.46)
88.03 (4.15)
94.74 (3.05)
97.27 (1.07)
Counterat lack (easy)
15.98 (11.77)
87.76 (6.40)
92.00 (1.62)
90.87 (3.37)
Counterat tack (hard)
3.22 (4.39)
77.38 (10.95)
88.14 (5.77)
91.72 (6.27)
Table 6. Performance Comparison Bi-DextterousHands Environments
Bi- Hande
Enviromment
PPO
MAPPO
НАРРО
QVPPO
ShadowHandd ver ShadowHand
28.3 (6.3)
23.7 (0.3)
30.2 (2.3)
34.7 (3.2)
27.8 (0.2)
18.2 (1.7)
28.3 (3.9)
31.3 (4.7)
97.1 (31.2)
57.9 (29.3)
179.8 (5.2)
148.32 (12.8)
Google Research Football (GRF). Table reports average returns on strategic GRF tasks. HAQO significantly outperforms unimodal Gaussian policies, which collapse brittle single-tactic behaviors. By contrast, diffusion-based HAQO agents sustain multimodal tactical strategies in- cluding short passes, delayed shols, and dribble pass combinations. Beyond higher mean returns, HAQO shows reduced variance across runs, confirming that expressive policies can be integrated sta- bly when combined with sequential updates and entropy regularization. These results remonstrate that HAQO captures the diversity required for realistic high-dimennioonll coordination without sac- rificing stability.
BidDextcrousSaands (Bi-D). Table presents results on bimanual manipulation. Baselines without sequential updates frequently destabilize, with agents dropping objects or failing to coordi- nate grasps. HAQO's sequential design mitigates conflicting gradients, allowing each hand to adapt to its partner's 's most recent policy. Coupled with expressive diffusion actors and entropy regulariza- tion, HAQO achieves both higher asymptotic returns and lower variance across seeds. The results show that. HAQO enables robust, dexterous manipulation high-dimennsonal continuous control, where stability and expressiveness are both indispensable.
A.12
PRACTITIONE''S GUIDE AND SENSITIVITY ANALYSIS
A.12.1 ANALYSIS OF ENTROPY REGULARIZATION (ax)
The entropy coefficient a serves a fundamental structural role beyond standard regularization. It functions critical constraint the policy's optimization manifold. As derived in Proposition 5.3, minimizing the entropy surrogate enforces a spectral floor on the action covariance matrix Σ. This theoretical property is indispensable for diffusion-based actors. Unlike Gaussian policies, which have explicit variance parameters, diffusion models define distributions implicitly through the reverse SDE. Without this spectral constraint, the denoising process becomes prone overfitting carly, noisy advantage signals, which leads to structural collapse into zero-width Dirac distributions.
49

---

Published conference paper at ICLR 2026
Our sensitivity analysis on Ageti-Agent MuJoCo identifies three distinct operational regimes for O4- In the Sub-Critical Regime (a; + 0), the spectral floor vanishes and allows the Q-weighted objective JQV to dominate the gradient. This configuration results in mode collapse, where the agent behaves deterministically and fails to coordinate in heterogeneous scenarios requiring flexible adaptation. In contrast, the Super-Critical Regime (a, - 1) exhibits the opposite behavior: the uniform reconstruction loss overwhelms the advantage signal and degrades the signal-to-noise ratio of the update, which results in high-variance random walks. The Optimal Regime (o, 0.01-0.05) creates a "soft" energy landscape that maintains sufficient dispersion in the generative process to cover multiple high-value modes simultancously. This balance ensures stable multimodality and allows agents to discover and maintain diverse st strategies, such as stable alternating gail, cycles in Humanoid locomotion, which are inaccessible to unregularized policies.
Table 7. Behavior Regimes and Performance Metrics
Regime
Mechanism
Behavior
Return
Cov.
Sub-Critical (a 0.001)
Spectral floor vanishes;
Mode collapse; local op- tima; coordination failure Stable multimodality; di- verse strategies Random walk; high vari- ance; no convergence
4500 + 500
20%
J'QV dominates
Optimal (a*= 0.01-0.05)
Soft landscape maintains dispersion
7013 + 311
95%
Super-Critical (.5)
Reconstruction
loss
1200 1 800
100%
washes out gradient
A.12.2 SELECTION OF Q-WEIGHTING SCHEMES
The selection of the Q-weighting scheme, w", represents a fundamental decision regarding gradient signal fidelity versus optimization landscape smoothness. This choice dictates how the algorithm manages the bias-variance tradc-off inherent in advantage estimation while satisfying the non- negalivity constraint required for Proposition 5.2. We characterize two primary schemes based on domain affinity. The geut (Hard Filtering) scheme functions as a high-pass filter and zeros out gradients where A < c. This mechanism aggressively climinates policy noise, which consists of updates derived from uncertain or marginally suboptimal actions. Such aggressive filtering theoretically preferable for high-variance discrete environments (e.g., SMAC) where suppressing stochastic noise is critical for stability. In contrast, the lado (Soft Calibration) scheme acts normalizer by recentering the advantage (A- 11-A1 A). This approach preserves the relative ranking of actions and maintains à continuous gradient flow even for small advantages. This property makes Godv essential for continuuus control tasks (e.g., MuJoCo), where fine-grained manipulation requires a smooth, dense gradient signal guide the diffusion trajectory without inducing abrupt discontinuities in the policy update.
A.12.3 TRADE-OFFS IN DIFFUSION TIMESTEPS (T)
The number of diffusion timesteps T governs a unique trado-off between the resolution of the policy distribution and computational latency. From a theoretical perspective, T determines the discretization eгror of the reverse stochastic differential equation (SDE). Lower values of T (e.g., T = 5) introduce significant discretization error and potentially violate the assumption that the learned denoiser accurately approximates the true score function. This results in a noisy policy that fails to capture fine-grained multimodal structures. On the other hand, extremely high values (e.g., 100) yield diminishing returns, as performance plateaus when the total approximation error bocomes dominated by critic bias rather than SDE discrctization. Based these considcrations,
50

---

Published conference paper at ICLR 2026
Table Signal Processing Mechanisms and Domain Suitability
Scheme
Signal Mechanism
Processing
Domain Suitability
Performance
SMAC: 92.4%
Geut (Hard Filtering)
Zeros out gradients where A < Eliminates "policy noise" Sassee-udatess from certain or marginally sub- optimal actions.
High-Pass Filter:
High- Variance / Discrete (e-g., SMAC). Critical for stochastic environments; trades sample efficiency high-precision updates.
MuJoCo: 4800
SMAC: 65.0%
Gado (Soft Calibration)
Normalizer: Recenters advantage (A-A) to pre- serve relative action rank- ing. Maintains continu- ous gradient flow even for small advantages.
Contimuous Control (e.g., MuJoCo). Essential for fine-grained manipulation where "slightly better" tions must provide gradi- ents.
MuJoCo: 5874
our profiling suggests a sweet spot typically between 20 and T 50. In this range, the generative fidelity is sulficient to capture complex dependencies and multimodal behaviors (as seen in Bi- (e-DetrrrousHndss without incurring the prohibitive wall-clock overhead associated with higher step counts.
Table 9. Impact of Diffusion Steps Performance and Efficiency
Steps (T)
Discretization Error
Fidelity
Perf.
Time
T' 5
High. Score matching as- sumption violated Moderate. Acceptable for simple tasks
Low (Noisy)
0.45
1.0x
T 20
Medium
0.88
1.8x
= 50 (Ours)
Low. Stable SDE approx- imation
High (Multimodal)
1.00
3.5x
T = 100
Negligible. Diminishing returns
High
1.02
6.8x
A.12.4 THE HINGE ON KL RADII (delta;) AND CRITIC BLAS (EQ)
To validate that the assumptions supporting Theorem 5.4 hold in practice we establish a proto- col for actively controlling and monitoring the KL Divergence (Assumption A.5) and Critic Bias (Assumption A.4).
Active Control of KL Radius (ōÆl): Unlike standard PPO implementations that roly on passive clipping, HAQO treats the trust region as a hard constraint. We employ an adap- tive line-search mechanism that dynamically rescales the policy update step to strictly enforce (TDei(Thell/ataa) < dxz- As shown in our diagnostic profiling (Table R1), raw gradient steps frequently propose updates that exceed the safety threshold (e.g., Raw KL s 0.045 > 0.01). Our mechanism successfully constrains these to the Realized KL of 0.01. This ensures that the sequential update assumption remains valid and prevents the catastrophic drift. that destabilizes simultancous updates.
51

---

Published conference paper at ICLR 2026
Quantification of Slack via Critic Bias (EQ): The tightness of our Q-weighted variational lower bound depends the critic bias eQ, which introduces additive slack term O(cq) (Proposition 5.2). While the true bias is measurable only with oracle, the Temporal Difference (TD) error serves as robust runtime proxy. Empirical data demonstrates a clear transition from high initial TD error (loose bounds) to a stable, low-magnitude steady state (tight bounds). This convergence provides practical verification that the variational lower bound becomes increasingly tight during training and thus supports the monotonic improvement guarantee.
Table 10. Validation Theoretical Constraints (Ant-v2 4x2, 0.01)
Stage
Raw KL
Real. KL
TD Err.
Slack
Status
Early (1M)
0.045 + 0.02
0.009 + 0.002
1.45 = 0.85
-0.82 (Loose)
Exploration
Mid (10M)
0.028 + 0.01
0.010 = 0.001
0.35 0.12
-0.15 (TTgghtenig)
Stabilization
Late (50M)
0.015 + 0.01
0.008 + 0.003
0.12 0.03
-0.04 (Tight)
Convergence
A.13 COMPUTATIONAL AND MEMORY ANALYSIS
The additional computational cost in HAQO arises from two core design choices: (1) the diffusion- based expressive actor introduces multi-step denoising and Q-weighted training, and (2) the K- candidate evaluation and sequential per-agent update mechanism ensures stable improvement under heterogeneous coordination. To provide a comprehensive analysis, we conducted detailed profiling study comparing HAQO to MAPPO and HAPPO under the exact hyperparameters employed in our experiments, including a lightweight denoiser architecture, diffusion timesteps E 8, 16, candidate count 4, and batched candidate ovaluation.
Table 11. Computational Analysis of HAQO vs MAPPO/HAPPO Baselines
Component
Baseline
Conservative
Practical
Notes
Actor forward (sin-
1.0
6.0x
4003.0x
Diffusson cost T x FLOPs. Conservative: 1T50-100 Practi- cal: T=5 20, small denoiser.
gle sample)
Actor backward (per update)
1.0
8.0x
2.5-4.0x
Backprop through denoiser and Q-weighted VLB, FP16 + check- pointing reduce cost.
K-candidate sam- pling (critic evals)
1.0
2.5 8.0
1.5-3.0x
Critic evals increase with K; torize to keep overhead modest.
Critic update (TD
1.0
1.1-16x
1.1x
Extra candidate targets (Ki Kb) slightly increase cost.
largets)
Sequential update wall-clock
1.0
up
1.2-1.8x
Worst-case scales with agents. Practice: overlap sampling, vec- torize, pipeline.
Peak GPU memory
1.0
1.5 3.0x
1.2 1.6x
Denoiser K candidates la- tents. FP16 reduces substantially.
End-to-end per iter- ation
1.0
2.5-6.0x
1.5-2.5x
Conservative: large T, unvec- torized. Practical: ~1.5-2.5x. GPU-hours appendix.
52

---

Published conference paper at ICLR 2026
Table 11 summarizes the computational and memory characteristics across representative com- ponents of the training pipeline. Across multiple benchmark tasks, HAQO exhibits wall-clock overhead of approximalely 1.5 to 2.5 times that of the baselines, with memory usage increasing by a factor of 1.2 to 1.6 times. These measurements are substantially lower than the conservative the- orctical upper bounds typically associated with diffusion models. The observed differences reflect well-bounded computational requirements: the practical configuration adopted in HAQO, which features small diffussin timesteps, compact denoiser network, vectorized critic evaluations, and pipelined sequential updates, successfully contains the computational burden while preserving the theoretical guarantees that necessitate likelihood-free expressive policies and per-agent trust-region updates. The modest increase in computational overhead is counterbalanced by substantially im- proved training stability and sample efficiency in heterogeneous multi-agent environments, where HAQO matches or surpasses HAPPO and MAPPO performance while requiring fewer environment interactions to reach comparable performance levels.
Table 12. Wall-clock timing and peak GPU memory comparison between HAQO and baseline methods. Measurements were obtained from training on a single NVIDIA RTX 4090 GPU for the SMAC "MMM2 (a super-hard scenario)" benchmark under standard
hyperparameter settings.
Metric
Bascline HAAPPO/HAPPO
HAQO Practical 2.76 hours (L.69x)
Wall-clock per 1e6 steps
1.66 hours
Wall-clock
5e6 env steps (Lypical set-
~7.91 hours
12.72 hours (1.61x)
ting)
Peak GPU memory
~ 13.6 GB
22.7 GB (1.67x)
In addition, Table 12 presents detailed wall-clock timing and peak GPU memory measurements obtained from training on a single NVIDIA RTX 4090 GPU for the SMAC "MMM2" benchmark under standard reinforcement. learning training configurations. These measurements correspond to the multipliers derived from our profiling analysis. The additional computational overhead, while present, remains bounded and is effectively amortized through HAQO's superior stability and sample efficiency characteristics. When HAQO reduces the number of required environment steps by more than the measured 1.69x overhead factor, the method achieves faster wall-clock convergence to target performance levels. Our learning curves across multiple environments demonstrate that this condition is consistently satisfied in practice, HAQO reaches comparable superior performance using substantially fewer training iterations than the baselines.
It is important to note that the central strength of HAQO lies in its ability to transform the in- creased per-step computational overhead associated with diffusion modeling into substantially su- perior sample efficiency. This efficiency emerges from HAQO's stable sequential trust-region design (Proposition 5.1), which effectively mitigates non-stationarity and enables aggressive sample reuse during optimization without inducing catast divergence. The maintenance of this stability allows the policy to fully exploit the multimodal expressiveness of the diffusion policy, and capture complex strategics that remain inaccessible to unimodal basclines. We further provide empirical evidence through Fig. 6 to demonstrate this efficiency gain. In "super-hard" SMAC benchmarks, HAQO consistently reaches the converged performance thresholds of competing methods (HAPPO, HATRPO, MAPPO, and QMIX) substantially earlier in the training process and achieves higher final win rates. These results demonstrate that the increased per-step computational cost is offsct
53

---

Published conference paper at ICLR 2026
Performance 20m
Performance XOMNZ
Baticarant Blge (1aB)
Eagistert
Figure Sample efficiency comparison across challenging SMAC benchmarks. HAQO consistently reaches bascline convergence thresholds substantially carlier in training and achieves higher final performance, demonstrating that increased per-step cost offset by reduced total environment interactions required for convergence.
by a significantly reduced total computational budget, as fewer environment steps are required to achieve and surpass state-of-the-art performance. This trade-off between per-step complexity and overall sample efficiency substantiates the practical viability of the proposed approach.
A.14 LIMITATIONS
While HAQO provides both theoretical guarantees and strong empirical performance, has several limitations. First,, the sequential update scheme, although stabilizing, introduces additional wall- clock time compared to fully parallel updates, which may hinder scalability in very large teams. Second, the use of expressive diffusion-based policies increases computational cost relative to uni- modal Gaussian policies, which could limit adoption in resource-constraiee settings. Addressing these limitations is an important direction for future work, particularly in scaling HAQO to even larger and more complex world domains.
54


Reviews:
[{"review_id": "vQcar4Tv16", "reviewer": "Reviewer_ADDX", "summary": "This paper proposes the Heterogeneous-Agent Q-weighted Policy Optimization framework to address the imbalance between stability and expressiveness. HAQO integrates three key components: Sequential Advantage-Aware Updates, Q-weighted Variational Surrogates, and Entropy Regularization. Through systematic theoretical analysis, HAQO extends the Trust-Region Policy Optimization (TRPO) framework to diffusion-based policies, ensuring monotonic improvement even when the log-likelihood is intractable. Empirically, HAQO achieves higher returns and lower variance across multiple heterogeneous multi-agent benchmarks, demonstrating its strong balance between stability and expressiveness.", "questions": "Please refer to the Weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper provides rigorous proofs for monotonic improvement in heterogeneous-agent settings, generalizing TRPO theory to diffusion-based policies. It also introduces the Q-weighted variational lower bound and the monotonic improvement theorem.\n    \n    - The introduction of a Q-weighted variational bound bridges generative modeling and reinforcement learning by effectively connecting diffusion models with the RL framework.\n    \n    - The experiments cover five major multi-agent reinforcement learning benchmarks with heterogeneous settings. The results demonstrate higher mean returns and lower variance across multiple tasks, validating the effectiveness of the proposed HAQO framework.", "weaknesses": "- Diffusion-based actors are computationally heavy. The paper does not discuss runtime, convergence speed, or scalability.\n\n   -  All theoretical guarantees rely on the assumptions of bounded critic bias and small trust-region radii, yet the paper does not verify whether the neural critics used in experimental results section satisfy these conditions.\n\n   -  The baselines in Table 1 focus on traditional policy-gradient methods (MAPPO, HAPPO), but omit recent diffusion-based RL or normalizing flow MARL approaches.\n\n   -  Although sequential updates are emphasized, there is no quantitative study showing the effect of agent update order or critic noise magnitude on convergence behavior. Only a single comparative curve of “Sequential vs. Simultaneous Update” is presented in the Figure 3.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eVDEpFO7eF", "reviewer": "Reviewer_V9WW", "summary": "The paper introduces a new multi-agent reinforcement learning (MARL) algorithm for heterogeneous cooperative MARL settings, called HAQO (Heterogeneous-Agent Q-weighted Policy Optimization). The algorithm aims to reconcile stability (monotonic improvement under non-stationarity) with expressiveness (multimodal policies needed for heterogeneous coordination). It combines:\n1. Sequential, advantage-aware per-agent updates within CTDE to avoid simultaneous-update instabilities;\n2. A Q-weighted variational surrogate that lets diffusion-based actors (with intractable log-likelihoods) optimize for return; and\n3. An entropy surrogate that enforces exploration/anti-collapse without requiring log-densities.\n\nIt also provides bounds showing monotone joint-return improvement under bounded critic bias and trust-region drift, and an algorithm instantiation using KL/PPO-style constraints. Empirically, on Multi-Agent MuJoCo and other suites, the proposed HAQO generally matches or outperforms MAPPO/HAPPO/HATRPO baselines (from both value-based and policy-based paradigms), and ablations attribute gains to (i) sequential updates, (ii) expressive (diffusion) policies, and (iii) the entropy surrogate.", "questions": "- What is the additional computational overhead (training time, memory) of HAQO relative to HATRPO or MAPPO? Can you add results for it?\n- The QV surrogate objective in equation (5) is conditioned on the already updated policies of other agents, but this is not possible in practice for all agents, does it affect the method's performance, or how does violating that affect the theoretical guarantees presented?\n- In 6.1, it was mentioned that all baselines were run with consistent hyperparameters. Does this mean the same values were used for all methods or the values recommended for each method?\n- In the sequential update ablation, can you mention more details on that? Are both methods HAQO, and if so, was it just replacing new policies with old policies for all agent updates?\n- When computing the KL divergence, is it enough for stability to just take it individually for each agent? Shouldn’t it matter how the new joint policy is different compared to the old joint policy? In other words, does ensuring that individual policies don’t move too far ensure that the joint policy also stays in the trust region?\n- In line 229, where is the a-i used inside the expectation?\n- In Fig. 1, were all methods initialized identically? If not, how should differences in the first column be interpreted?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 1, "strengths": "- The paper is generally well-organized and provides insight into how heterogeneous agents and diffusion-based policies interact in cooperative MARL.\n- The theoretical derivations are carefully laid out and formally sound given the stated assumptions.\n- The experimental evaluation covers a wide set of benchmarks and includes ablations that test the contribution of each design component.", "weaknesses": "### Motivations\n\n- The central motivation—that MARL methods face a tension between stability and expressiveness—is not well substantiated. The paper provides no prior evidence or citations that improving one necessarily degrades the other. This makes the claimed tradeoff appear somewhat self-imposed; ideally, both properties are desirable.\n\n- The problem formulation for the cooperative setting is unclear. In a purely cooperative objective (shared reward), optimization resembles a joint minimization $min_{x_1, \\dots, x_a} \\ f(x_1, \\dots, x_a)$. Without coupling terms between agents, sequential vs. simultaneous updates should be equivalent. If the authors mean practical instability from non-stationary policies, this distinction should be stated more formally.\n\n- The introduction conflates non-stationarity (inherent to MARL) with the effect of simultaneous updates (lines 80–82). Sequential updates may alleviate, but not eliminate, non-stationarity—particularly with replay buffers.\n\n\n### Non-self-sustained Background\n\n- The background for diffusion policies and heterogeneous-agent settings is insufficient for a general ML audience. The paper does not clearly specify whether agents share a global objective or if heterogeneity also includes distinct observation or action spaces. For example, it is not very clear if the objective in heterogeneous settings is shared between agents (as in a potential game setup) or how exactly the problem is defined\n\n\n- Simple toy examples, similar to those in Zhong et al. (JMLR 2024), would clarify how simultaneous updates lead to instability and why sequential updates are beneficial. I had to check that paper first for understanding.\n\n\n- Some terms introduced early (e.g., “unimodal policies”) are not defined, making the abstract and introduction harder to follow for readers unfamiliar with this subarea.\n\n\n\n### Computational aspects\n\n- As authors discuss, using diffusion policies can be computationally expensive. As shown in ablations, this can make a large improvement, but it is not always the case (for example, in Multi-MuJoCo table, the improvements are not that significant)\n\n\n### Novelty and theoretical contribution\n\n-  Algorithmic novelty is limited. Each component—sequential trust-region updates (HAPPO/HATRPO), Q-weighted diffusion objectives, and entropy surrogates—exists in prior work. The main contribution is their combination and an accompanying monotonic improvement analysis.\n\n\n- The cooperative extension introduces moderate technical difficulty, mainly because diffusion actors lack explicit log-likelihoods. The proposed Q-weighted variational surrogate addresses this but relies on assumptions similar to prior single-agent Q-weighted diffusion methods and HATRPO-style drift bounds.\n\n\n- The cooperative setting adds little new theoretical challenge beyond bookkeeping of per-agent drift. The paper would benefit from more explicit discussion of how inter-agent coupling or heterogeneity complicates the guarantees.\n\n### Additional feedback and minor typos\n\nAbstract:\n- The term “unimodal policies” in the abstract is not defined; readers unfamiliar with diffusion or Gaussian policy classes may not immediately know this refers to simple parametric (e.g., Gaussian) action distributions lacking multimodality.\n- The phrase “monotone improvement guarantees” should specify what is guaranteed to improve (e.g., expected joint return / performance objective under the centralized critic). Clarifying the exact measure of improvement would help.\n- The initial paragraph states there is a “tension between stability and expressiveness.” This is somewhat self-imposed; in principle, we desire both. The text could better motivate why these two properties conflict in practice (e.g., why expressive multimodal policies break standard monotonicity proofs).\n- Minor stylistic issue: the abstract’s first few sentences read slightly overloaded, introducing both the “tension” and the contribution before defining key terms; consider re-ordering or simplifying for readability.\n\nOther:\n- The introduction begins after a figure, unusual and slightly disruptive.\n- Multiple claims lack citations; for instance, there are a lot of claims in lines 55-61, and references must be given for each point\n- Several figures appear redundant. (e.g., Fig. 1 could omit the first columns; Fig. 2 repeats similar content).\n- In Table 1, isn't HAQO also the best performer in Walker 6x1?\n- Line 267: transformation\n\n\n—--\n\nReferences:\n[1] Yifan Zhong, Jakub Grudzien Kuba, Xidong Feng, Siyi Hu, Jiaming Ji, et al. Heterogeneous-agent reinforcement learning. In Journal of Machine Learning Research (JMLR), 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qWHpBsikXO", "reviewer": "Reviewer_7FA2", "summary": "This paper aims to resolve the conflict between learning stability and policy expressiveness in heterogeneous MARL. It combines HATRPO's sequential policy update mechanism and QVPO's weighted variational objective function policy expression as a lower bound for policy improvement, providing a monotonically increasing guarantee and analysis of the team reward optimization objective for online MARL learning with diffusion-style policy expressions. To prevent mode collapse even for policies with strong expressiveness, the authors designed an entropy surrogate objective. By injecting actions sampled from a uniform distribution into the diffusion model, they ensure policy distribution diversity, encouraging exploration and providing a corresponding lower bound.\n\nExperimentally, the authors validated the algorithm in several challenging MARL environments. Results show that HAQO significantly improves performance and stability compared to state-of-the-art baseline algorithms such as MAPPO and HAPPO. Extensive ablation experiments also validated the necessity of the three core components mentioned above.", "questions": "Q1: Algorithm 1 in the appendix mentions using ppo-clip for trust domain policy constraints. However, with non-Gaussian policies, the likelihood function of $\\pi_i$ is difficult to calculate. How can this ratio be calculated for a diffusion model in practice?\n\nOther Problems mentioned in the weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper is the first to combine a diffusion model strategy with a theoretically guaranteed heterogeneous MARL framework, and provides a complete theoretical analysis (monotone improvement guarantee) for this combination—a solid contribution. The paper provides a complete derivation chain, including the performance difference, the sequential update bound, and the monotone improvement guarantee. This provides a strong theoretical foundation for the effectiveness of the method.\n\nWhen a diffusion model is used as an actor, it is often impossible to obtain a precise expression of the policy distribution, limiting the use of entropy regularization. The proposed method, which reconstructs a surrogate objective of uniformly sampled actions, is clear in its approach and simple to implement. Theoretically, it is demonstrated that this method can guarantee the \"spectral floor\" of the action distribution covariance, thus effectively preventing mode collapse.\n\nThe experimental setup in this paper is comprehensive, covering various scenarios such as continuous control, discrete actions, and high-dimensional policies. Comparisons with mainstream and stable baselines in MAPPO, HAPPO, and other fields make the experimental results more convincing. Ablation experiments clearly demonstrate the indispensable roles of the three components: serialization update, expressive strategy, and entropy regularization.", "weaknesses": "W1: Compared to Gaussian strategies, the sampling and training of diffusion models are extremely time-consuming, and combining this with sequence update processes significantly increases computational costs. The paper mentions this limitation but does not provide any quantitative comparisons of training time or computational resources in the experimental section. Adding a wall-clock time comparison for MAPPO/HAPPO would be more helpful in evaluating its feasibility in practical applications.\n\nW2: The experimental details disclosed in this paper are limited. The framework involves several key hyperparameters and network structures, such as the entropy regularization coefficient, the number of steps in the diffusion process, the chosen base network architecture of the diffusion model, and a performance comparison between qadv and qcut. The paper does not discuss the sensitivity of these hyperparameters or how to tune them, which may pose difficulties for reproduction and application.\n\nMinor Problems: In line 266, there is a repetition in the paragraph.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "89bBalW7zb", "reviewer": "Reviewer_S3C5", "summary": "The paper introduces **HAQO (Heterogeneous-Agent Q-weighted Optimization)**, a multi-agent RL framework for heterogeneous teams (agents with different observations, actions, and roles). The goal is to get both (i) stable training and (ii) expressive, multimodal policies in cooperative tasks.", "questions": "1. Can the authors estimate or upper-bound ($\\epsilon_Q$) online (e.g., via TD-error calibration or ensemble dispersion) to quantify the slack in the QV lower bound during training? \n2. How are per-agent ($\\delta_i$) chosen across heterogeneous roles, and how sensitive are results to these settings? A small sweep/heuristic would help practitioners. \n3. When do you prefer each weighting? Please characterize the stability/variance trade-offs and default choices across benchmarks.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. The synthesis of heterogeneous sequential trust-region updates with a Q-weighted diffusion surrogate is well-motivated and technically nontrivial, enabling expressive policies while retaining improvement structure.  \n2. A per-agent sequential bound (Prop. 5.1) and a global Theorem 5.4 establish monotone improvement with explicit dependence on KL radii and critic bias ($\\epsilon_Q$).  \n3. Results cover discrete/continuous and partially observable regimes, with tables for SMAC/SMACv2, GRF, and Multi-MuJoCo; comparisons use consistent hyperparameters and three seeds.  \n4. Algorithm 1 operationalizes the full pipeline (critic updates, permutation, QV/entropy/drift) and links theory to practice.", "weaknesses": "1. Guarantees hinge on KL radii ($\\delta_i$) and critic bias ($\\epsilon_Q$); the paper lacks practical diagnostics/tuning guidance for monitoring these quantities during training.  \n2. Diffusion actors and sequential updates plausibly increase wall-clock and compute; the paper itself acknowledges added cost but does not report GPU-hours or env-steps-to-target. \n3. On HalfCheetah-v2 (2×3), HAQO trails HAPPO (6873 vs. 7024), suggesting sensitivity or robustness issues that merit analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)