Distributionally Robust Cooperative Multi-agent Reinforcement Learning with Value Factorization
Authors: Chengrui Qu
Christopher Yeh
Kishan Panaganti
Eric Mazumdar
Adam Wierman
Venue: ICLR 2026 Poster (2026)
Keywords: distributionally robust RL
cooperative multi-agent RL; Centralized training decentralized execution
Primary area: reinforcement learning
TL;DR: We develop distributionally robust cooperative MARL algorithms based on a novel value factorization principle that accounts for environmental uncertainty.

Abstract: Cooperative multi-agent reinforcement learning (MARL) commonly adopts centralized training with decentralized execution, where value-factorization methods enforce the individual-global-maximum (IGM) principle so that decentralized greedy actions recover the team-optimal joint action. However, the reliability of this recipe in real-world settings remains uncertain due to environmental uncertainties arising from the sim-to-real gap, model mismatch, system noise. We address this gap by introducing Distributionally robust IGM (DrIGM), a principle that requires each agent's robust greedy action to align with the robust team-optimal joint action. We show that DrIGM holds for a novel definition of robust individual action values, which is compatible with decentralized greedy execution and yields a provable robustness guarantee for the whole system. Building on this foundation, we derive DrIGM-compliant robust variants of existing value-factorization architectures (e.g., VDN/QMIX/QTRAN) that (i) train on robust Q-targets, (ii) preserve scalability, and (iii) integrate seamlessly with existing codebases without bespoke per-agent reward shaping. Empirically, on high-fidelity SustainGym simulators and a StarCraft game environment, our methods consistently improve out-of-distribution performances.

Published as a conference paper at ICLR 2026
DISTRIBUTIONALLY ROBUST COOPERATIVE MULTI- AGENT REINFORCEMENT LEARNING VIA ROBUST VALUE FACTORIZATION
Chengrui Qu' Caltech
Christopher Yeh Caltech
Kishan Panaganti Tencent AI Lab
Eric Mazumdar' Caltech
Adam Wierman' Caltech
ABSTRACT
Cooperative multi-agent reinforcement learning (MARL) commonly adopts tralized training with decentralized execution, where faltorffctorizatiion methods enforce the individual-global-aas moxximmm (IGM) principle that decentralized greedy actions recover the caam-optimal joint action. However, the reliability of this recipe in real-world settings remains unreliable due to environmental uncer- tainties arising from the sim-to-real gap, model mismatch, and system noise. We address this gap by introducing Distributionally robust IGM (DrIGM), a principle that requires each agent's 's robust greedy action to align with the robust team-optimal joint action. We show that DrIGM holds for novel definition of robust individual action values, which is compatible with decentralized greedy execution and yields a provable robustness guarantee for the whole system. Building on this founda- tion, derive DrIGM-compliant robust variants of existing value-factorization architectures (e.g., VDN/QMIX/QTRANN that (i) train on robust Q-targets, (ii) preserve scalability, and (iii) integrate seamlessly with existing codebases with- out bespoke per-agent reward shaping. Empirically, on high-fidelity SustainGym simulators and StarCraft game environment, our methods consistently improve out-of-distribution performance.
INTRODUCTION
Multi-agent reinforcement learning (MARL) a popular framework for studying how multiple agents compete or cooperate in complex environments such video game playing (Vinyals et al., 2019), economic policy design (Zheng et al., 2022), wireless network communications (Qu et al., 2020), and power grid control (Gao et al., 2021), among others. In this work, we focus on the cooperative MARL setting, where each agent can only observe its local history, and agents must collaborate to achieve joint goal. To address partial observability and reduce real-time communication costs, widely used paradigm is centralized training with decentralized execution (CTDE) (Oliehoek et al., 2008). During training, agents may aggregate global information, coordinate credit assignment, and learn team structure; at deployment, each agent must act myopically based on its own local history.
The CTDE paradigm is typically realized through value factorization methods (e.g.. VDN (Sunehag et al., 2017), QMIX (Rashid et al., 2020), QTRAN (Son et al., 2019)). A key concept that underpins the success of these methods is the mndividual-lloaal-aaiii
(IGM) principle (Son al., 2019), which aligns each agent's ' greedy action with the leam-optimal joint action via a suitable value factorization. However, most examples where the success of this principle demonstrated are in virtual tasks (e.g., games (Vinyals et al., 2017) and grid worlds (Leibo et al., 2017)). It remains unclear whether this principle maintains its reliability in real-world domains, where modeling imperfect and execution is noisy.
In practice, a major obstacle facing cooperative MARL is environmental uncertainty (Shi et al., 2024): team performance can drop sharply when the deployed environment deviates from the training
Department of Computing Mathematical Sciences, Califomia Institute of Technology, CA 91125, USA Tencent AT Lab in Bellevue, WA, USA

---

Published as a conference paper at ICLR 2026
environment due to model mismatch, system noise, and the sim-to-real gap (Zhang al., 2020b; Balaji et al., 2019). While environmental uncertainty presents challenges in single-agent RL settings, it is more significant hurdle in cooperative MARL, where partial observability and inter- agent coupling can cause small mismatches to cascade into coordination failures (Capitan et al., 2012; He al., 2022).
In single-agent RL, uncertainty in the environment is commonly addressed by distributionally robust RL (DR-RL) techniques (Wiesemann et al., 2013; Taori et al., 2020; Nilim & El Ghaoui, 2005; Panaganti & Kalathil, 2021a; Shi et al., 2023) which seek policies that perform well under adversarial perturbations of a nominal environment model. Single-agent DR-RL is well-explored, however extending DR-RL to the cooperative MARL setting is fundamentally more challenging. In partieular, each agent acts on a local history yet shares a team reward coupled with teammates' actions, making it nontrivial to define individual robust Q-functions that both evaluate worst-case outcomes and remain compatible with IGM for decentralized greedy execution. Reward engineering can help empirically, but only narrow class of shaping functions can provably preserve optimality (Foerster et al., 2016), even in the single-agent setting. Thus, we seek a principled route to distributional robustness for cooperative MARL that remains compatible with decentralized greedy execution.
Contributions. In this paper, we introduce family of distributionally robust cooperative MARL algorithms for the CTDE setting. Our central technique is Distributionally robust IGM (DrIGM), a robustness principle that requires each agent's robust greedy action to coincide with the robust team-optimal joint action, thereby preserving decentralized greedy execution.
We first show, via a concrete counterexample, that naïvely adopting individual robust action-value functions from single-agent DR-RL, where each agent considers its own worst case, does nor guarantee decentralized alignment (IGM) in the cooperative multi-agent setting. We then provide a sufficient condition under which DrIGM holds: when individual robust action-value functions defined with respect to the worst-case joint action-value function, DrIGM is guaranteed.
Next, we derive DrIGM-compliant robust variants of existing value factorization architectures (VDN, QMIX, and QTRAN), by training on robust Q-targets while retaining the CTDE information structure. The resulting methods are scalable, easy to implement on top of existing codebases, and maintain robustness at execution without requiring bespoke individual robust value design.
Finally, we evaluate our DrIGM-basee algorithms on a realistic simulation an HVAC control task in SustainGym (Yeh et al., 2023), as well as on SMAC, a StarCraft II-dased multi-agent game-playing environment (Samvelyan et al., 2019). Across out-of-distributtion settings, our methods outperform non-robust value factorization baselines and recent robust cooperative MARL baseline, consistently mitigating sim-to-real degradation on operational metrics.
Brief discussion of related work. Robustness in cooperative MARL has been studied along several axes: adversarial or heterogencous teammates (Li et al., 2019; Kannan et al., 2023; Li et al., 2024), state/observation and communication perturbations (Guo al., 2024; Yu et al., 2024), risk-sensitive (tail-aware) objectives under a fixed model (Shen et al., 2023), and explicit model uncertainty (K wawk et al., 2010; Zhang al., 2020b; 2021; Liu et al., 2025). Most of the works on model uncertainty adopt distributionally robust optimization viewpoint and targets Nash solutions with provable algorithms, often assuming full observability or individual rewards (Zhang et al., 2020a; Kardeş et al., 2011 1; Ma et al., 2023; Blanchet et al., 2023; Shi et al., 2024; Liu et al., 2025). In this work, we focus on the cooperative CTDE regime with partial observability and a single team reward, providing a systematic framework for robustness to model uncertainty without real-time communication. Due to space constraints, we provide an extended discussion of related works Appendix A.
2 BACKGROUND AND PROBLEM FORMULATION
Notation. For set X, |X| denotes its cardinality and A(X) the probability simplex over X. We write II, X; for the Cartesian product. For N E N, we let [M] == (1,...,N) Let [x] = max x).
Cooperative Dec-POMDPs. A cooperative multi-agent task with N agents is modeled Decen- tralized Partially Observable Markov Decision Process (Dec-POMDP):
G = -S, (A)) P.r, (0.) (a N
2

---

Published as a conference paper at ICLR 2026
with joint action space A := [IEENN] A,. At time t, each agent obtains individual observation of == (s") from its observation space O, chooses an action at A,,, a bounded joint reward r(s", a²) is received, where a' :- (af..... d'y) E A is the joint action, and then the state evolves via 8 P(- ,", Here we assume the joint observation (01,..., ,0x) can recover the full state. Each agent i acts using history-based policy a.(- ke) with h == a .........."" o
_.-".of the = (TI...... nT). We use H to denote the space of possible histories for agent up to time t. In the following sections, we will omit the superscript to avoid notational clutter. The joint cotte--observation history denoted E H := [[ H. Given the current state 8, the joint history and the joint action a, we denote the joint action-value function under policy и by Qioi (h, a), which can be reduced to IS x A dimension as we assume the joint observation can recover the full state. We use Qioc (h, a) = max, Qioi (h,a) a) to denote the optimal joint action value.
joint policy
CTDE. Centralized training with decentralized execution (CTDE) leverages global information during learning while executing individual policies from individual histories. A common CTDE approach is value factorization, which learns optimal joint action-value Q (h, a) and individual action-value functions [Q"" (hu, ai)lkE[N] that satisfy the following individual sriiiduul-globl-mmae glooal--axaaaa (IGM) principle (Son et al., 2019).
Definition 1 (IGM). We say that individual action-value functions (0 H. x A, - R. RlEE(N) satisfy the aadividual-globad max (IGM) principle for optimal joint action-value function Q - under joint history (lu,,..., ,NN) E H if
arg max ( (11, arg max Q (hx..ax) S arg max Q (h, a).
NN
IGM ensures that greedy individual actions are jointly optimal w.r.t. Qlee enabling decentralized exccution without test-time communication.
Robust Dec-POMDPs. To capture model error and deployment shift, consider an uncertainty sei P of environment models around a nominal pi In Dec-POMDPs, we work with a history-based view: let a.) (-) denote the transition kernel over next joint histories h', given the current joint history h and the joint action a. We assume a history-action rectangular uncertainty set.
P
II
Ph... Pr,a S CAH),
(1)
(b,a)EXXA
e.g., balls around Ph. under a probability metric with radius > O. This rectangularity assumption widely adopted DR-RL literature (Blanchet et al., 2023; Shi al., 2024; Ma et al., 2023), and ensures that robust policy exists.
Given a function Q H. x A + IR, define the robust Bellman operator T
(TQ) a) = rss, +7 inf Pra
max Q(h', a²) a'EA
(2)
Under standard assumptions (bounded rewards, E 1)) and rectangularity in Eq. (1), T is a y-contraction on the space of bounded Q, it has unique fixed point Q (Iyengar, 2005) which satisfies
Q (h, a) inf (for PEP
V(h,a) A.
We call Qhe the optimal robust joint action-value function for the Dec-POMDP, and it admits a deterministic robust greedy joint policy π (h) E arg max, P. (h,
Robust Cooperative MARI.. We study robust cooperative MARI under the CTDE setting. Given a model uncertainty set P, our goal is to learn decentralized policies that maximize Q. That is, we aim to find [m H4 A.lie[N), such that
( ..... mNv arg max ( a).)
ILN O is injective. Equivalently, there exists (deterministic) decoding map g [hIG O₁ such that g(a(s)) for all E S.
Formally, we assume a (a1 (oi().....oN) .o. (.)) :
3

---

Published as a conference paper at ICLR 2026
Specifically, seek value factorization method that automatically generates robust individual action values, thereby enabling a decentralized policy. This is non-trivial for two reasons. First, no individual reward signals available, so robust individual action values are ill-defined priori. Second, directly defining robust individual action values from the single-agent DR-RI. literature can break standard value factorization. As demonstrate in Example 1, robust individual actions may not align with the robust joint action. These challenges motivate the central question of our work: Can we construct robust individual utilities and mixing scheme such that decentralized greedy actions recover the joint maximizers of Quar thereby enabling a robust CTDE framework?
3
DISTRIBUTIONALLY ROBUST IGM (DRIGM)
To address the question above, we propose a novel principle for robust value factorization that builds upon the IGM principle while explicitly incorporating robustness.
3.1 DISTRIBUTIONALLY ROBUST IGM (DRIGM) PRINCIPLE
Definition 2 (DrIGM). Given an uncertainty set P. we say that robust individual action-value functions [Q His x A; + []] satisfy the Distributionally robust IGM (DrIGM) principle for the optimal robust joint action-value function Qint H x A (h1,..., hN) E H if
+
R under joint history h
arg max Q7o (h1, a1), arg max
D (hN, ax))
S
arg max Q.at (h, a).
aN
DrIGM extends classical IGM to the robust setting by requiring that the robust joint greedy action induced by Qio factorizes into robust individual greedy actions from ( Note that when P = [P] is singleton (i.e., there no uncertainty), then DrIGM equivalent IGM.
Satisfying DrIGM is nontrivial. In particular, the single-agent definition Qrob(s, a) inf (,, a). commonly adopted in the DR-RL literature, does not ensure DrIGM when applied with global uncertainty set. As shown in Example in Appendix B, an adversarial model P E P that minimizes one agent's value need not coincide with the adversarial model P' E P that minimizes the joint value. As a result, robust individual greedy actions may fail align with the robust joint greedy action. Similar inconsistencies arise even under agent-wise uncertainty sets defined in Shi al. (2024) for essentially the same reason. This highlights the need for a new formulation of robust individual action values to support consistent robust CTDE framework.
In robust cooperative MARL, the primary concern is the robustness of the entire system, as opposed to robustness of individual agents. Thus, it is sufficient to consider the worst case for the joint action value, rather than independently for each agent. Motivated by this idea, we show that the robust individual action value defined under global worst-case model can guarantee DrIGM.
Theorem 1. Given a global uncertainty set d defined in Eq. (1), suppose for all P E P, there exist keeN) satisfying IGM for Qiot under joint history h (h1,..., E H. Let
pworst (h, a) arg inf Q. Quor (h, a),
(4)
PEP
a arg max Q (h. a),
(5)
a denote the global worst-case model and the robust joint greedy action, respectively, For each agent i [N], define the robust individual action-value functions Qoob as
(ha o.) == = pwores (h,ā) (hi, az).
(6)
Then, [Qro")eE(M) satisfy DrIGM for Por under joint history h.
The proof of Theorem 1 can be found in Appendix C.I. Theorem demonstrates that by anchoring individual robust action values to the global worst-case model evaluated at the robust joint greedy action, individual robust greedy actions become aligned with the robust joint greedy action. This construction resolves the misalignment problem that occurs when individual adversaries differ from the global adversary. More broadly, the result highlights that robust CTDE is achieved not by independently robustifying each agent, but by coordinating all agents against a shared adversarial model tied to the team's 's worst-case joint outcome. This perspective offers principled foundation for designing robust value factorization methods that maintain decentralized execution while ensuring robustness guarantees.
4

---

Published as a conference paper at ICLR 2026
Common factorization methods satisty DrIGM. Having established that robust individual ac- tion values can be consistently defined via Theorem 1, we now examine whether these values are compatible with standard factorization methods used in cooperative MARL.
Theorem 2. Given P defined in Eq. (1), for joint history h E H. suppose for all P E EP, there exist individual action-value functions [Q] [Ke[N] satisfying one of the following conditions for all
(01,-.., AN) A:
Vior (h,
Σ
o (hs, ai),
(VDN)
SE[M]
8Q (h, a)
2 0,
ViE
(QMIX)
aQ, (bs; ai)
Σ (hi iE[M]
Qiae (h, a) + Viot(h)
00
a =
(QTRAN)
(2
220, 0,
a + a,
where := [āilKE[N] with ā. == arg Q, ar) and Vior (h) == max, Qior (h, a)
EienNy Q (2 with a = [a.]kEIN] Then (ePe) as defined in Eq. (6) satisfy DrIGM for Qio under joint history h.
The proof of Theorem can be found in Appendix C.2. Theorem 2 shows that when the underlying individual Q-functions satisfy the structural conditions of VDN (Sunehag et al., 2017), QMIX (Rashid et al., 2020), or QTRAN (Son et al., 2019), the robust individual action values [Qrob] JEINN] constructed from Eq. (6) automatically satisfy DrIGM. This result ensures that robust CTDE can be realized directly within widely used value factorization frameworks, enabling principled distributionally robust extensions of existing algorithms. Moreover, as long as the test environment lies within the prescribed uncertainty set, this approach yields provable robustness guarantee, as formalized in the next theorem.
Theorem 3. Given P defined in Eq. (1), suppose the robust individual action values Quob satisfy Definition 2. the test environment model Prest is included in the uncertainty set (i.e., Piest P), then the robust joint action values provably lower bound the real joint action values in Ptesti
Qh5 (h. a) S Qfae (h,a), vh H, A.
The proof of Theorem 3 can be found in Appendix C.3.
3.2 ROBUST BELLMAN OPERATORS UNDER SPECIFIC UNCERTAINTY SETS
To design training loss functions, we next present the DrIGM- based robust Bellman operators for two common uncertainty designs: p-contamination and total variation (TV), which are well-studied in the single-agent distributionally robust RL literature (Yang et al., 2022; Panaganti & Kalathil, 2021b; Xu al., 2023; Dong et al., 2022: Liu & Xu, 2024; Panaganti al., 2022; Wang & Zou, 2022; Zhang et al., 2024). Both types of uncertainty sets consider perturbations of size p E (0, 1] around nominal model po
The p-contamination uncertainty set is defined as (for all h E H. and a E A)
Ph.a = (P (PE((H)) Ph,. (1 (1--ph Pp. + plh,a V E A(H) arbitrary).
with corresponding robust Bellman operator
(TQP))) (0)
r(s,a) a) + 7(1 - p) pg
max QP (Ki...., Nw at) as).a) + 71 - pp) Eg [PPa (M (H....., Ny. ai d'......'''... dxx)]
a'Ed
(8)
arg where a = arg max Q1th(h(.. Here, (a) follows from robust Bellman operator as the single- agent setting due to the h x a-rectangularity from Eq. (1), (b) follows from the DrIGM principle where robust individual greedy actions are aligned with the robust joint greedy action.
Similarly, the TV-uncrrtantty set defined as (for all h E H and a A)
Ph.a {P A(H) TV(P, Ph $ p).
5

---

Published as a conference paper at ICLR 2026
update 0
MLP
LSTM
MLP
Due (h,
rubust Lip
po
Pin aa; )))
Ayy
MLP
LSTM
MLP
saggst
rebustness parameter
(Q,"" hajM satisty DrIGM wast The
P
(ho, 4..6) CLA.
(M, (; a.A.
Figure Overview of our robust value factorization algorithms. Because the robust individual action-value functions satisfy DrIGM, greedy actions can be computed efficiently in decentralized manner while the function parameters are trained with a robust TD loss based on global reward.
with corresponding robust Bellman operator,
(TQr)) a) r(ss a)
inf nE(0, (.
? Bhin (-
(1 p)m((s
+ ((s,,a) - Phor(M(..... dy......... dxx)
(10)
where v185, a) is the dual variable.
The derivation of the robust Bellman operators relies on "fail state" assumption (Assumption 1); the proof is provided in Appendix D. In the next section, we show how DrIGM leads to practical robust value factorization algorithms.
ALGORITHMS: ROBUST VALUE FACTORIZATION
Overall framework. Guided by DrIGM, we develop six robust value factorization algorithms by combining two types of uncertainty sets (p-contamination, TV-uncertainty) with three different value factorization architectures (VDN, QMIX, QTRAN). The overall framework is illustrated in Algorithm 1 and Fig. 1, while detailed pseudocode for each variant can be found in Appendix E. Concretely, we collect trajectories using e-greedy exploration and train the robust individual action- value network using TD- TD-learning (Sutton & Barto, 2018). For stability, robust one-step targets evaluated using target networks, which are updated periodically.
Robust individual action-value networks. Each agent i uses DRQN (Deep Recurrent Q Network)-style network that maps its local history h; (observations and past actions) to action- values (a a;), following MLP encoder + LSTM core + MLP output architecture. For our training procedure, we follow the approach from Hausknecht & Stone (2015). We sample mini- batches of sub-trajectories from the replay buffer D and use bootstrapped random updates. We use burn-in steps warm-start the LSTM state and only take the last step output to calculate the loss and update the networks. This procedure computationally and memory efficient while achieving performance comparable sequential updates from the start of each episode.
Factorization networks. We instantiate three networks for robust value factorization:
1. VDN factorizes the robust joint action-vall as the of robust per-agem values,
N
P..
P.DD (h,a) =
Q (hi, ax).
2. Beyond direct summation, QMIX uses monotone mixing network
(11) P.QMIXX (h, = fo(Qrob(hn (1)...),.... Q (Nw, (N,,a),,,) s),
where is the global state. A lightweight hypernetwork takes input and outputs the layer weights of fo; to ensure aQP.QMIS P.QIIX /aQrob > 0 (the QMIX monotonicity constraint),

---

Published as a conference paper at ICLR 2026
Algorithm Robust value factorization
1: Input robustness parameter target network update frequency f. and
2: Initialize replay buffer 'D
3: Initialize robust individual action-value networks [Q KEIN] with random parameters 0
4: Initialize factorization networks that produce Q with random parameters
5: Initialize target parameters 0 =
6: for episode e ..... E do
7: Observe initial state s and observation of - (s()) for each agent i.
8: fort ...... T do
9:
Each agent i chooses its action of using e-greedy policy.
10:
Take joint action a', observe the next state s reward r' and observation of 0;(8tt1)
for each agent i Store transition Sample mini-batch of transitions (h, a, h') from Calculate TD loss LTD using Eq. (14)
11:
"r.,r"''t l in replay buffer D
12:
13:
14:
Update 9 by minimizing LTD Update 8- = / with frequency f
15:
16:
end for
17:
end for
we enforce elementwise nonnegativity on these weights via an absolute-value (or softplus) transform. Biases remain unconstrained.
3. QTRAN learns a separate joint action-value value function Q.o Q..ATRAN (h, a) and baseline Viot (h). For efficiency and scalability, the joint network shares the encoder/head with the individual DRQN modules. In addition to the robust TD loss, QTRAN imposes two consistency terms to align the factorized and joint values:
STRAAN (h, a) Vior (h))² Lopt (Q.0 VDN (h, a) P.
(12)
Lnopt = ( [min[QP.
VDN (h, a) - Q..
S..RTRAN (h, + Vior 01)²
(13)
where the Q is the detached Q value for training stability. Intuitively, Lope enforces equality at the (robust) greedy joint action, while Tnapt penalizes positive slack elsewhere, recovering the QTRAN constraints in our robust setting.
TD Loss. Given the robust Bellman operator T defined in Eq. (2) for a Dec-POMDP setting, the generic form of the TD-loss is
LTD (Q (h,
(14)
where 9 is the network parameters, and A- the target network parameters for training stability. Specifically, for p-contamination uncertainty sets, by the robust Bellman operator in Eq. (8), have
LTD (Q (h, a; 0) (r(s, a) + +1 p) En Q (h', (h',a';'0)))
(15)
For TV uncertainty sets, by the robust Bellman operator in Eq. (10), we have
LTD (0 (h, a; (h,
r(s, a) + E - (1 ())(s,)))))))
+ [n(s, a) - Q.a (h', a'; a2;00)1+)
(16)
where S A + RR is calculated by minimizing the following empirical loss:
(Inss.a) - max Q (h', a/)] (1-p) -))n(s,a))
Tedual(1) (1, Q.ao)
Σ
(17)
(h.aDh')D
7

---

Published as a conference paper at ICLR 2026
5 EXPERIMENTS
5.1 SUSTAINGYM
We evaluate our proposed robust value factorization methods on SustainGym (Yeh et al., 2023), recent benchmark suite designed to simulate real-world control tasks under distribution shift. We focus on multi-agent environments for smart building HVAC control, which inherently involve stochastic dynamics, distribution shifts, partial observability, and inter-agent coupling. These environments are particularly well-suited to test robustness, as the environmental models can vary across days and building locations (thus climate conditions). More details about the environment and the experiment setup are provided in Appendix F. Our code can be found at https: /ttpttttttttb/// com/ crqu/ robust-coMi coMARL.
Evaluation protocol. To assess generalization under distribution shift (i.e., model uncertainty), we adopt the following protocol. In the training phase, each algorithm is trained on a single environment. For robust MARL baselines that require multiple environments, we follow their standard protocol and train them on a fixed set environments. In the evaluation phase, trained policies are deployed on unseen configurations that differ from those used in training, simulating realistic deployment where distribution shifts inevitably arise. This design allows us to explicitly measure robustness to changes in environment dynamics rather than simple memorization of training conditions.
Baselines.
We compare our robust value factorization methods against:
Non-robust factorization methods: VDN, QMIX, and QTRAN trained without robustness considerations, representing the standard CTDE paradigm.
Existing robust CTDE baseline: the multi ageti- group distributionally robust algorithm from Liu et al. (2025), which we refer to as "GroupDR". While the original work used only the VDN architecture, we extend the algorithm to QMIX and QTRAN for completeness.
Q
QTRAN
TV uncortainty
p-contamination
Non-robust
GroupDR (baseline)
Robust (ours)
car. shift level +
shift level
shift level
Figure 2: Normalized performance (averaged over 5 independent training runs, with error bars showing standard error) across different environment configurations for robust MARL algorithms and other baselines. Each panel corresponds to one value factorization method. Robustness gain the difference reward (shaded area) between Robust (ours) and Non-robust, which shows the out-of of-distribution performance improvement from the robust training.
Experiment 1: climatic shifts. We first test robustness under shifts induced by changes in climate conditions. Results (averaged five seeds) are shown in Fig. 2. Our robust MARL algorithms con- sistently outperform both non-robust counterparts and the group DR baseline. Notably, performance degradation scales with the severity of the shift (c.g., env deviates most from the training environ- ment, env_1) 11), but our methods maintain relatively high returns. In contrast, the GroupDR baseline exhibits little sensitivity to shift severity, reflecting its reliance on worst-case rewards computed only from configurations encountered during training.
Experiment 2: seasonal shifts. We next evaluate robustness to seasonal shifts, training algorithms season data and evaluating on season 2. Results reported in Table 1, showing mean
8

---

Published as a conference paper at ICLR 2026
and standard error of normalized episodic returns. The results show that robust value factorization algorithms with TV uncertainty set achieve consistent robustness gain against seasonal shifts.
Factorization Methods
VDN
QMIX
QTRAN
Non-robust
0.877 = 0.012
0.895 + 0.008
0.816 + 0.036
baseline (GroupDR)
0.624 0.040
0.499 0.022
0.508 + 0.048
Robust (TV-unccrtainty)
0.898 = 0.008
0.916 0.006
0.861 = 0.006
Robust (p-contamination)
0.869 = 0.013
0.911 + 0.005
0.825 = 0.028
Table 1: Final Performances under seasonal shifts for robust MARL algorithms and other baselines (mean + standard error over independent training runs). Values outperforming both the non-robust and group DR baselines are highlighted in bold.
Experiment 3: climatic and seasonal shifts. Finally, we test on the most extreme case, where have distribution shifts arising from climatic and seasonal shifts. The results are presented Table with robust MARL algorithms achieving 10-40% higher average reward than the non-robust baseline. Notably, QTRAN-bacs based robust MARL algorithms demonstrate strong ouitto-distribution performance and stability.
Factorization Methods
VDN
QMIX
QTRAN
Non-robust
0.440 0.040
0.478 0.052
0.654 + 0.066
baseline (GroupDR)
0.624 + 0.056
0.383 0.053
0.520 + 0.049
Robust (T (TV-uncertainty)
0.627 = 0.049
0.520 + 0.048
0.733 + 0.026
Robust (p-contamination)
0.551 0.039
0.500 0.075
0.682 + 0.026
Table 2: Final Performances under climatic and seasonal shifts for our robust MARL algorithms and other baselines (mean + standard error over 5 independent training runs). Values outperforming both the non-robust and group DR baselines are highlighted in bold.
Choice of p. Theoretically, p should be chosen based on prior estimation the model uncertainty level. Practically, select p by training env_1 and validating env_2 and env_3, which yields stable performance without overfitting to a single shift.
Robustness in cooperative MARL. A noteworthy finding is that robustness in cooperative MARL does not necessarily entail reduced performance the training environment. Unlike in single-agent robust RI,, where conservatism often penalizes in-distribution returns, explicitly modeling robustness here mitigates errors from partial observability and decentralized execution. In several cases, robust training even improves in-distribution performance relative to non-robust baselines, suggesting that robustness can simultaneously enhance stability and adaptability in multi-agent systems.
5.2 STARCRAFT II
We additionally conduct experiments SMAC (Samvelyan et al., 2019), a well-known benchmark consisting of two teams of agents engaged in cooperative combat scenarios based StarCraft II. We focus on the hard 1s_vs_5z map. In the test environment, we introduce distribution shift by adding noise to each agent's observation of every enemy unit's normalized position, sampled from (O 0.75²). Since QTRAN has been shown to perform poorly on SMAC (Son et al., 2020), we report results for the pccontmminttoon uncertainty set using only VDN and QMIX (averaged over five seeds) in Fig. 3. The results demonstrate that for small values of our robust MARL algorithms significantly improve outtof-distribution performance.
We further compare the final performance of our algorithms against their non-robust baselines, reporting the improvement in the test win rate for different choices of p relative to the baseline. The results (averaged over five seeds) are shown in Fig. 4. Interestingly, the test win rate first increases as grows, and then decreases. This observation aligns with our theory: when p small relative to the shift level, explicitly modeling distribution shift during training yields improved of-iistribution performance. However, when & becomes large, the robust MARL algorithms become overly conservative, leading to degraded performance.
Robustness parameter evaluation.

---

Published as a conference paper at ICLR 2026
10-
10-
Row862
Rc-630
Rate
Win
22
022
Steps
Stops
Robust VDN
(b) Robust QMIX
Figure 3: Performance of our robust MARL algorithms and their non-robust baselines in SMAC (3s vs. 5z map). Each algorithm evaluated every 10,000 environment steps, with each evaluation averaged over 32 episodes. Shaded regions denote the standard error across 5 random seeds. For small ( the robust algorithms significantly outperform their non-robust counterparts.
Improvement in Test Win Rate
Improvenne improvement in Test Win Rate
0.4
0.0
0.0
-0.4
6.00
0.22
0.04
0.28
5.08
0.10
3.50
692
0.06
5.06
0.08
5.30
rho
rho
(a) Robust VDN
(b) Robust QMIX
Figure 4: Improvement in final test win rate of our robust MARL algorithms over their non-robust baselines in SMAC (3s vs va.52 5z map) for different values of p. Error bars denote the standard error across 5 random seeds.
CONCLUSION
In this work, we introduce Distributionally robust IGM (DrIGM), robustness principle for cooper- ative MARL that extends the classical IGM property to settings with environmental uncertainties. Whereas naïvely "robustifying" individual agent policies fails to align robust individual policies with the joint robust policy, the DrIGM offers principled framework for constructing robust individual action values that remain aligned with the joint robust policy, thereby enabling decentralized greedy execution under uncertainty.
Building this foundation, derive DrIGM-based robust value factorization algorithms for VDN, QMIX, and QTRAN, trained via robust Bellman operators under standard uncertainty sets (p- contamination and total variation). Empirically, a high-fidelity building HVAC control benchmark, our methods consistently mitigate out-of-distribuiig performance degradation arising from climatic and seasonal shifts. On a StarCraft II game-playing benchmark, our methods likewise improve out-of-disttrbuttin performance under added observation noise. Unlike single-agent robust RL, where conservatism often harms in-distriuution returns, we find that robustness in cooperative MARL can simultaneously enhance stability and adaptability.
While we introduced the DrIGM framework for global uncertainty set, we believe may be possible to further extend this framework. Future work includes developing DrIGM-compliant algorithms under agent-wise uncertainty sets and exploring additional training paradigms (c.g., decentralized training) further broaden applicability.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
We release github repository containing all code, configuration files, and scripts needed to reproduce our results, including data generation and figure plotting, All proof's for the main paper are stated in Appendix C. Algorithm pseudocode is also provided in Appendix E.
ACKNOWLEDGMENT
KP acknowledges support from the 'PICCO Postdoctoral Fellow in Data Science' fellowship at the California Institute Technology. The major part this work was done while KP was at the California Institute of Technology. EM acknowledges support from NSF award 2240110. This work acknowledges support from NSF CNS-2146814 CPS-2136197, CNS-2106403, NGSDI-2105648, and funding from the Resnick Sustainability Institute as well Quad Fellowship.
AUTHOR CONTRIBUTIONS
Chengrui Qu led the technical development of the project, driving the theory and core methodology, running and analyzing the experiments, and taking a leading role in writing the paper.
Christopher Yeh co-developed the project and contributed in all aspects of experiments and writing.
Kishan Panaganti co-developed the project direction and prepared the initial proof of concept, advised CQ and CY, and played a significant role in paper writing.
Eric Mazumdar co-developed the project direction, advised CQ and KP.
Adam Wierman co-developed the project direction, advised CQ2 CY, and KP, and played significant role in paper writing.

---

Published as a conference paper at ICLR 2026
REFERENCES
Bharathan Balaji, Sunil Mallya, Sahika Genc, Saurabh Gupta, Leo Dirac, Vineet Khare, Gourav Roy, Tao Sun, Yunzhe Tao, Brian Townsend, Eddic Calleja. Sunil Muralidhara, and Dhanasekar Karuppasamy. Deepracer: Educational autonomous racing platform for experimentation with sim2real reinforcement learning. arXiv preprint arXiv:/911.015000 2019.
Dimitris Bertsimas, Vishal Gupta, and Nathan Kallus. Data-driven robust optimization. Mathematical Programming, 167(2)223522)) 292, 2018.
Jose Blanchet and Karthyek Murthy. Quantifying distributional model risk via optimal transport. Mathematics of Operations Research, 44(2):655-600, 2019. 17
Jose Blanchet, Miao Lu, Tong Zhang, and Han Zhong. Double pessimism is provably efficient for distributionally robust offline reinforcement learning: Generic algorithm and robust partial coverage. Advances in Neural Information Processing Systems, 36, 2023. 2, 17
Albert Bou, Matteo Bettini, Sebastian Dittert, Vikash Kumar, Shagun Sodhani, Xiaomeng Yang, Gianni De Fabritiis, and Vincent Moens. Torchrl: data-driven decision-making library for
pytorch. arXiv preprint arXivv2306.00577 2023. 26
Alexander Bukharin, Yan Li, Yue Yu, Qingru Zhang, Zhehui Chen, Simiao Zuo, Chao Zhang, Songan Zhang, and Tuo Zhao. Robust multi-agent reinforcement learning via adversarial regularization: Theoretical foundation and stable algorithms. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Jesus Capitan, Matthijs T.J. Spaan, Luis Merino, and Anibal Ollero. Decentralized multi-robot cooperation with auctioned pomdps. In 2012 IEEE International Conference Robotics and Automation, pp. 3323-3328, 2012.
doi: 10. 109/ICRA..012.2224911
Esther Derman and Shie Mannor. Distributional robustness and regularization in reinforcement learning, arXiv preprint arXiv:2003.02894 2020. 17
Esther Derman, Daniel J Mankowitz, Timothy A Mann, and Shie Mannor. Soft-robust actor-critic policy-gradient. In AUAI press for Association for Uncertainty in Artificial Intelligence, pp- 2018. 17
Jing Dong, Jingwei Li. Baoxiang Wang, and Jingzhao Zhang. Online policy optimization for robust MDP. arXiv preprint arXiv:2209. 13841, 2022. 17
John Duchi and Hongseok Namkoong. Learning models with uniform performance via distributionally robust optimization. arXiv preprint
2018. 17
Jakob N. Foerster, Yannis M. Assael, Nando de Freitas, and Shimon Whiteson. Learning to com- municate with deep multi-agent reinforcement learning. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS' 16, pp. 2145-2153, Red Hook, NY, USA, 2016. Curran Associates Inc. ISBN 9781510838819. 2
Diego Fonseca and Mauricio Junca. Decision-dependent distributionally robust optimization. arXiv preprint arXiv:2303.0383 033971, 2023. 17
Rui Gao. Finite-sample guarantees for wasserstein distributionally robust optimization: Breaking the
curse of dimensionality, arXiv preprini arXiv:2009.04382 2020. 17
Yuanqi Gao, Wei Wang, and Nanpeng Yu. Consensus Multi-Agent Reinforcement Learning for Volt-VAR Control in Power Distribution Networks. IEEE Transactions on Smart Grid, 12(4): 3594-3604, July 2021. ISSN 1949-3061. doi: 10.1009/SSG.2021.33358888 Conference Name:
IEEE Transactions on Smart Grid. 1
Vineet Goyal and Julien Grand-Clement, Robust markov decision processes: Beyond rectangularity.
Mathematics of Operations Research, 2022. 17
Weiran Guo, Guanjun Liu, Ziyuan Zhou, Ling Wang, and Jiacun Wang. Enhancing the robustness of qmix against ttate-adversarill attacks. Neurocomputing, 572:127191, March 2024. ISSN 0925-2312. doi: 10. newcom.203..122177 2, 17
12

---

Published as a conference paper at ICLR 2026
Songyang Han, Sanbao Su, Sihong He, Shuo Han, Haizhao Yang. and Fei Miao. What is the solution for atvte-adversarial multi multi-agent reinforcement learning? arXiv preprint arXiv:2212.02705, 2022.
17
Matthew J. Hausknecht and Peter Stone. Deep recurrent q-learning for partially observable mdps. CoRR, abs/ abs/1507. 157 006770657 2015. 6, 26
Keyang He, Prashant Doshi, and Bikramjit Banerjee. Reinforcement learning in many- agent settings under partial observability. In James Cussens and Kun Zhang (eds.), Proceedings of the Thirty- Eighth Conference on Uncertainty in Artificial Intelligence, volume 180 of Proceedings of Machine Learning Research, pp. 780-789, PMLR, 01-05 Aug 2022. 2
Sihong He, Songyang Han, Sanbao Su, Shuo Han, Shaofeng Zou, and Fei Miao. Robust multi-agent reinforcement learning with state uncertainty. Transactions on Machine Learning Research, 2023.
17
Chin Pang Ho, Marek Petrik, and Wolfram Wiesemann. Fast bellman updates for robust mdps. In International Conference on Machine Learning, pp. 1979-19989 PMLR, 2018. 17
Chin Pang Ho, Marek Petrik, and Wolfram Wiesemann. Partial policy iteration for 11-robust markov decision processes. Journal of Machine Learning Research, 22(275): 2021. 17
Shariq Iqbal, Christian A. Schröder de Witt, Bei Peng, Wendelin Boehmer, Shimon Whiteson, and Fei Sha. Randomized entity-wise factorization for multi-agent reinforcement learning. In ICML, 2021. 17
Garud N Iyengar. Robust dynamic programming. Mathematics of Operations Research, 30(2):
257-280, 2005. 3, 17
Syyun-Chhool Shyam Sundar Kannan, Vishnunandan LN Venkatesh, and Byung- Cheol Min. Smart-llm: Smart multi-agent robot task planning using large language models. arXiv preprint arXiv:2309.10662 2023. 2, 17
Erim Kardeş, Fernando Ordóñez, and Randolph W Hall. Discounted robust stochastic games and an application to queueing control. Operations research, 59(2):365-382, 2011. 2, 17
David I. Kaufman and Andrew J Schaefer. Robust modified policy iteration. INFORMS Journal on Computing, 25(3):396-400, 2013. 17
Jun- Young Kwak, Rong Yang, Zhengyu Yin, Matthew E. Taylor, and Milind Tambe. Teamwork and coordination under model uncertainty dec-pomdps. Proceedings of the 3rd AAAI Conference on Interactive Decision Theory and Game Theory, AAAIWS' 10-03, pp. 30-36. AAAI Press, 2010. 2, 17
Joel Z. Leibo, Vinicius Zambaldi, Mare Lanctot, Janusz Marecki, and Thore Graepel. Multi-agent reinforcement learning sequential social dilemmaa. In Proceedings of the 16th Conference on Autonomous Agents and MulriAgen gent Systems, AAMAS '17, pp. 464-47,, Richland, SC, 2017. International Foundation for Autonomous Agents and Multiagent Systems.
Shihui Li, Yi Wu, Xinyue Cui, Honghua Dong, Fei Fang, and Stuart Russell. Robust multi-agent reinforcement learning via minimax deep deterministic policy gradient. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 4213-4220, 2019. 2, 17
Simin Li, Jun Guo, Jingqiao Xiu, Ruixiao Xu, Xin Yu, Jiakai Wang, Aishan Liu, Yaodong Yang, and Xianglong Liu. Byzantine robust cooperative multi-agent reinforcement learning as a bayesian game. In The Twelfth International Conference on Learning Representations, 2024. 2, 17
Guangyi Liu, Suzan Iloglu, Michael Caldara, Joseph W Durham, and Michael M. Zavlanos. Distribu- tionally robust multi-agent reinforcement learning for dynamic chute mapping. In sortyd International Conference on Machine Learning, 2025. 2, 8, 17,
Zhishuai Liu and Pan Xu. Distributionally robust off-dynamics reinforcement learning: Provable efficiency with linear function approximation. arXiv preprint arXiv:2402. 15399, 2024. 5, 17
13

---

Published as a conference paper at ICLR 2026
Miao Lu, Han Zhong, Tong Zhang, and Jose Blanchet. Distributionally robust reinforcement learning with interactive data collection: Fundamental hardness and near-optimal algorithms. In The Thirty-eighth Annual Conference Neural Information Processing Systems, 2024. 21
Shaocong Ma, Ziyi Chen, Shaofeng Zou, and Yi Zhou. Decentralized robust v-learning for solving markov games with model uncertainty. Journal of Machine Learning Research, 24(371):1-40) 2023. 2, 3, 17
Daniel J Mankowitz, Nir Levine, Rae Jeong, Yuanyuan Shi, Jackie Kay, Abbas Abdolmaleki, Jost To- bias Springenberg, Timothy Mann, Todd Hester, and Martin Riedmiller. Robust reinforcement learning for continuous control with model misspecification. arXiv preprint 1en.::996.77555 2019. 17
Peyman Mohajerin Esfahani and Daniel Kuhn. Data-driven distributionally robust optimization using the Wasserstein metric: Performance guarantees and tractable reformulations. Math, Program., 171(122)115-166, September 2018. ISSN 0025-5610.
doi: 10.10077/10107-017--
17
Amab Nilim and Laurent EI Ghaoui. Robust control of Markov decision processes with uncertain transition matrices. Operations Research, 53(5):780- 2005. 2
Nilay Noyan, Gábor Rudolf, and Miguel Lejeune. Distributionally robust optimization under decision-dependent ambiguity set with applications to machine scheduling and humanitarian logistics. INFORMS Journal Computing, 34(2).729-751, 2022. doi: 10. 10.1287/jjo::021110099 17
Frans A. Oliehoek, Matthijs J. Spaan, and Nikos Vlassis. Optimal and approximate q-value functions for decentralized pomdps. J. Artif. Int. Res., 32(1):289-335 May 2008. ISSN 1076-9757.
Kishan Panaganti and Dileep Kalathil. Robust reinforcement learning using least squares policy iteration with provable performance guarantees. In International Conference on Machine Learning (ICML), pp. 51 511-520, 2021a. 2. 17
Kishan Panaganti and Dileep Kalathil. Sample complexity of model-based robust reinforcement learning. In 2021 60th IEEE Conference on Decision and Control (CDC), pp. 2240-2245, 2021b. 5, 17
Kishan Panaganti, Zaiyan Xu, Dileep Kalathil, and Mohammad Ghavamzadeh. Robust reinforcement learning using offline data. Advances in Neural Information Processing Systems (NeurIPS), 2022. 5, 17, 21,21,22 22, 26
Chengrui Qu, Huiwen Jia, and Pengeheng You. Decision-dependent distributionally robust optimiza- tion with application to dynamic pricing. In 2025 IEEE 64th Conference Decision and Control (CDC), pp. 693-698, 2025. doi: 10. 11110/9 13125144 17
Guannan Qu Yiheng Lin, Adam Wierman, and Na Li. Scalable Multi-Agent Reinforcement Learning for Networked Systems with Average Reward. In Advances in Neural Information Processing Systems, volume 33, pp. 2074-2086. Curran Associates, Inc., 2020.
Hamed Rahimian and Sanjay Mehrotra. Distributionally robust optimization: A review. arXiv preprint 10080:9998.00653 2019. 17
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Monotonic value function factorisation for deep multi-agent reinforcement
learning. Journal of Machine Learning Research, (118))) 2020. 1, 5, 17, 26
Aurko Roy, Huan Xu, and Sebastian Pokutta. Reinforcement learning under model mismatch. In Advances in Neural Information Processing Systems, pp. 3043-3052, 2017. 17
Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim G. J. Rudner, Chia-Man Hung, Philip H. S. Torr, Jakob Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS '1, pp. 1186-2188, Richland, SC. 2019. International Foundation for Autonomous Agents and Multiagent Systems. ISBN 978 170363396,
2, 9
14

---

Published as a conference paper at ICLR 2026
Siqi Shen, Mengwei Qiu, Jun Liu, Weiquan Liu, Yongquan Fu, Xinwang Liu, and Cheng Wang. Resq: A residual function-basee approach for multi ageet reinforcement learning value factorization. In NeurIPS, 2022. 17
Siqi Shen, Chennan Ma, Chao Li, Weiquan Liu, Yongquan Fu, Songzhu Mei, Xinwang Liu, and Cheng Wang. Riskq: resk-sensitive multi-agent reinforcement learning value factorization, Advances in Neural Information Processing Systems, 36:34791-34822 2023. 2, 17
Laixi Shi, Gen Li, Yuting Wei, Yuxin Chen, Matthieu Geist, and Yuejie Chi. The curious price of distributional robustness in reinforcement learning with a generative model. Advances in Neural Information Processing Systems, 36, 2023.
Laixi Shi, Eric Mazumdar, Yuejie Chi, and Adam Wierman. Sample-efficient robust multi-agent reinforcement learning in the face of environmental uncertainty. arXiv preprint arXiv:2404. 18909, 2024. 1, 2, 3, 4, 17
Elena Smirnova, Elvis Dohmatob, and Jérémie Mary. Distributionally robust reinforcement learning. arXiv preprint (XX::/992000700 2019. 17
Kyunghwan Son, Dacwoo Kim, Wan Kang. David Earl Hostallero, and Yung Yi. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In International conference on machine learning, pp. 5887-5896. PMLR, 2019. 17, 26
Kyunghwan Son, Sungsoo Ahn, Roben Delos Reyes, Jinwoo Shin, and Yung Yi. Qtran++: Im- proved value transformation for cooperative multi-agent reinforcement learning. arXiv preprint arXiv:2006. 12010, 2020. 9
Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Mare Lanetot, Nicolas Sonnerat, Joel Leibo, Karl Tuyls, et al. Valmosoecooooositoon networks for cooperative multi-agent learning. arXiv preprint arXiv:1766.085 2017. 5, 17, 26
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Aviv Tamar, Shie Mannor, and Huan Xu. Scaling up robust mdps using function approximation. In International Conference on Machine Learning, pp. 181-189, 2014. 17
Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. Measuring robustness to natural distribution shifts in image classification. Advances in Neural Information Processing Systems,
2020. 2
Daniel Vial, Sanjay Shakkottai, and R Srikant. Robust multi-agent bandits over undirected graphs. Proceedings of the ACM on Measurement and Analysis of Computing Systems, 6(3):1-57, 2022. 17
Oriol Vinyals, Timo Ewalds, Sergey Bartunov, Petko Georgiev, Alexander Sasha Vezhnevets, Michelle Yeo, Alireza Makhzani, Heinrich Küttler, John Agapiou, Julian Schrittwieser, John Quan, Stephen Gaffney, Stig Petersen, Karen Simonyan, Tom Schaul, Hado van Hasselt, David Silver, Timothy Lillicrap, Kevin Calderone, Paul Keet, Anthony Brunasso, David Lawrence, Anders Ekermo, Jacob Repp, and Rodney Tsing. Starcraft ii: A new challenge for reinforcement learning. arXiv preprint arXiv: 04782, 2017. 1
Oriol Vinyals, Igor Babuschkin, Wojeiech M. Czamecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H. Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo Danihelka, Aja Huang, Laurent Sifre, Trevor Cai, John P. Agapiou, Max Jaderberg, Alexander S. Vezhnevets, Rémi Leblond, Tobias Pohlen, Valentin Dalibard, David Budden, Yury Sulsky, James Molloy, Tom L. Paine, Caglar Gulcehre, Ziyu Wang, Tobias Pfaff, Yuhuai Wu, Roman Ring, Dani Yogatama, Dario Wünsch, Katrina McKinney, Oliver Smith, Tom Schaul, Timothy Lillicrap, Koray Kavukcuoglu, Demis Hassabis, Chris Apps, and David Silver. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature, 575(7782): 350-354, November 2019. ISSN 1476-4687. doi: 10. 10.1038/44158-000 1724-2. 1
Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. Qplex: Duplex ducling multi-agent learrnnng. In ICLR, 2021. 17
15

---

Published as a conference paper at ICLR 2026
Yue Wang and Shaofeng Zou. Policy gradient method for robust reinforcement learning. In Interna- tional Conference on Machine Learning, pp.
2022. 5, 17
Wolfram Wiesemann, Daniel Kuhn, and Berç Rustem. Robust Markov decision processes. Mathe- matics of Operations Research, 38(1):153-183, 2013. 2
Eric M Wolff, Ufuk Topcu, and Richard Murray. Robust control of uncertain markov decision processes with temporal logic specifications. In 2012 IEEE 5 IEEE Conference on Decision and Control (CDC). pp. 3372-3379, IEEE, 2012. 17
Huan Xu and Shie Mannor. Distributionally robust Markov decision processes. Mathematics of
Operations Research, 37(2))288-30M 2012. 17
Zaiyan Xu, Kishan Panaganti, and Dileep Kalathil. Improved sample complexity bounds for distribu-
tionally robust reinforcement learning. arXiv preprint arXiv:2303.022883 20277
2023, 5, 17
Wenhao Yang, Liangyu Zhang, and Zhihua Zhang. Toward theoretical understandings of robust Markov decision processes: Sample complexity and asymptoties. The Annals of Statistics, 50(6): 2022. 5, 17
Yaodong Yang, Jianye Hao, Ben Liao, Kun Shao, Guangyong Chen, Wulong Liu, and Hongyao Tang. Qatten: A general framework for cooperative multiagent reinforcement learning. CoRR, abs/2002.03939, 2020. 17
Christopher Yeh, Victor Li, Rajeev Datta, Julio Arroyo, Nicolas Christianson, Chi Zhang, Yize Chen, Mehdi Hosseini, Azarang Golmohammadi, Yuanyuan Shi, Yisong Yue, and Adam Wierman. SustainGym: Reinforcement learning environments for sustainable energy systems. In Thirty- seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, New Orleans, LA, USA, 12 2023. 2, 8,
Lebin Yu, Yunbo Qiu, Quanming Yao, Yuan Shen, Xudong Zhang, and Jian Wang. Robust commu- nicative multi- agent reinforcement learning with active defense. In Proceedings the Thirty-Eightt AAAI Conference on Artificial Intelligence and Thirty-Sixti Conference on Innovative Applica- fions of Artificial Intelligence and Fourteenth Symposium on Educational Advances in Artificial Intelligence, AAAI'24/IAA ATT//AAAAAA AAAI Press, 2024. ISBN 978-1-57735-88779 doi:
10.1609/aaai.v336
17
Huan Zhang, Hongge Chen, Duane S Boning, and Cho-Jui Hsich. Robust reinforcement learning on state observations with learned optimal adversary. In International Conference on Learning
Representations, 2020a. 2,
Huan Zhang, Hongge Chen, Duane Boning, and Cho-Jui Hsich. Robust reinforcement learning on state observations with learned optimal adversary. arXiv preprint arXhv:2101.08452. 2021.
Kaiqing Zhang, TAO SUN, Yunzhe Tao, Sahika Genc, Sunil Mallya, and Tamer Basar. Robust multi- agent reinforcement learning with model uncertainty. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 10571-10588
Curran Associates, Inc., 2020b. 2, 17
Zhengfei Zhang, Kishan Panaganti, Laixi Shi, Yanan Sui, Adam Wierman, and Yisong Yue. Dis- tributionally robust constrained reinforcement learning under strong duality. arXiv preprint M.Xi:224061158888 2024. 5, 17, 26
Zhili Zhang, Yanchao Sun, Furong Huang, and Fei Miao. Safe and robust multi-agent reinforce- ment learning for connected autonomous vehicles under state perturbations. arXiv preprint
arXiv:2309. 11057, 2023. 17
Stephan Zheng, Alexander Trott, Sunil Srinivasa, David C. Parkes, and Richard Socher. The AI Economist: Taxation policy design via two-level deep multiagent reinforcement learning. Science Advances, 8(18))ebb,2607 May 2022. doi: 10.1 I 22b/
Ziyuan Zhou and Guanjun Liu. Robustness testing for multi-agent reinforcement learning: State perturbations on critical agents. arXiv preprint wrXiv:2306.06636 2023. 17
16

---

Published as a conference paper at ICLR 2026
A RELATED WORK
Single-agent Distributionally Robust RL (DR-RL). The single-agent setting is typically for- malized as a robust Markov decision process (MDP). A substantial literature studies finite-sample guarantees for distributionally robust RI., exploring a variety of ambiguity-set designs (lyengar, 2005; Xu & Mannor, 2012: Wolff et al., 2012; Kaufman & Schaefer, 2013: Ho et al., 2018: Smirnova et al., 2019; Ho et al.. 2021; Goyal & Grand-Clemenn 2022; Derman & Mannor. 2020; Tamar et al., 2014; Panaganti & Kalathil, 2021a; Roy et al., 2017: Derman et al., 2018; Mankowitz al., 2019). Most relevant to our work are tabular robust MDPs with (s, recttagggl uncertainty sets defined by total-variation balls (Yang et al., 2022; Panaganti Kalathil, 2021b; Xu et al., 2023; Dong et al., 2022; Liu & Xu, 2024; Panaganti et al., 2022) or p-contamination models (Wang & Zou, 2022; Zhang et al., 2024), for which minimax dynamic programming and learning algorithms admit provable performance bounds.
Value factorization methods for cooperative MARL. Value factorization is the standard mech- anism for scalable cooperative MARL under CTDE. Early work adopts simple additivity (VDN (Sunchag al., 2017)), while QMIX (Rashid et al., 2020) learn a state-conditioned monotone com- biner to enlarge the function class without violating the IGM requirement. QTRAN (Son et al., 2019) further relax the monotonicity assumption with consistency constraints. Other approaches include attention-bee based mixers (e.g.. QAtten (Yang et al., 2020), REFIL (Iqbal et al., 2021)), dueling-style decompositions (QPlex (Wang et al., 2021)) and residual designs (ResQ (Shen et al., 2022)). Building on this body of work, we develop robust aalue-fcttorizatio algorithms with provable robustness guarantees under model uncertainty, enabling robust decentralized exccution in partially observable cooperative settings.
Robustness in MARL. In general MARL, robustness is typically studied within Markov games, where uncertainty can be modeled in different components, such as the state space (Han et al., 2022; He et al., 2023; Zhou & Liu, 2023; Zhang et al., 2023), other agents (Li et al., 2019; Kannan et al., 2023), and environmental dynamics (Zhang et al., 2020a; Liu et al., 2025). We refer readers to Vial et al. (2022) for an overview. This work considers robustness to model uncertainty, primarily studied via distributionally robust optimization (DRO) (Rahimian & Mehrotra, 2019; Gao, 2020; Bertsimas et al., 2018: Duchi & Namkoong. 2018; Blanchet & Murthy. 2019; Fonseca & Junca, 2023; Qu et al., 2025; Mohajerin Esfahani & Kuhn, 2018; Noyan al., 2022), where most prior efforts target Nash equilibria and provide provable factor-critic/ / Q-learning) algorithms (Zhang et al., 2020a; Kardeş et al., 2011: Ma al., 2023; Blanchet et al., 2023; Shi et al., 2024; Liu et al., 2025), often under full observability or individually rewarded settings. We complement this line by addressing the cooperative, partially observable CTDE regime, where agents receive single joint reward and act only local observations.
In cooperative MARL, robustness has been modeled along several complementary axes, including adversarial (Byzantine) teammates (Li al., 2024), state/observation disturbances (Guo al., 2024), communication errors (Yu et al., 2024), risk-sensitive objectives that guard against tail events under fixed model (Shen et al., 2023), and explicit model uncertainty Kwak al. (2010); Zhang et al. (2020b). Focusing on the last category, Kwak et al. (2010) address model uncertainty with sparse, execution-time communication, whereas Zhang al. (2020b) study settings in which each agent observes the full state and receives individual reward. Similarly, Bukharin et al. (2023) also considers settings where each agent receives individual reward, and they achieve robustness by controlling the Lipschitz constant of each agent's policy. In contrast, our work targets robustness to model uncertainty in the cooperative CTDE setting, complementing prior approaches by providing systematic framework that does not require real-time communication and operates under partial observability with a single team reward.
B EXAMPLES
Example (Naive single-agent robust action values cannot guarantee DrIGM). Consider robust cooperative two-agent task (illustrated in Fig. 5) with action spaces A = A = = (1, state space S {so, $1, 82, $3, $1), and uncertainty set P = {PI, Let su be the initial state, and let $2 and Sa all be absorbing states with zero reward. We assume each agent observes the full state. For Pt, all the transitions are fully deterministic. P2 differs from P1 only in transitions on joint
17

---

Published as a conference paper at ICLR 2026
actions (1 and (2, 1), given by:
1
P(S2 1,2)
IP(53 1,2) 3
3"
2
P(S2 2,
IP(53 2, = 3'
3'
As shown in Fig. 5, the optimal joint action value function at state 8y is (we omit the r/(1 - 7) factor for clarity)
Q (5o, 1, = 0.7, Qiái (su, 1,1) 0.7,
Q (so, 1,2) = 0.4, Q (50, 2,1) = 1.0,
Q (so, 2,2) = 0.7;
Q (su. 1, 008, Qii (su. 2.1) = 0.6,
p (so. 2,2) = 0.7.
Therefore, the robust joint action-value function Q (8, a) inP Q (8, a) is given by:
Quae (50, 1, 1) = 0.7, Phat (s0, 1,2) = 0.4, Q.ot (so, 2. = 0.6, Q 2.2) = 0.7,
is straightforward to check that the following individual action-value functions (Q Ji.jE14 satisfy Rea (8,01, 01, (1,02))) 02) = (x, + Q (s, (2)2)) all 8 E S and P E P, which is a special case of the classical IGM property:
Qf 1) 0,
2f0 2) = -03,, (so: 1) 0.7, (fu = 0.1, Q (s0: -)
0₂ P (so, 2) = =0.4,
Qf 1) 0.2,
2 (Su, 2) = 0.6.
Suppose the robust individual action-value function is defined (rob(s, a) inf Of (8, a), as in the single-agent DR-RL literature. Therefore, the robust individual action-value functions are given by:
Q (o. 1) 0,
Q7 (so. 2) = =0.1, (50. 1) = 0.5,
15 (50, 2)= = 0.4,
So, these robustifications fail to satisfy DrIGM:
(2, 1) arg max Q100 (50, a1), arg max Q (so, d2) $ arg max Q (50, a) 1),, (1,
St
1
0.7
0.4
0.7
0.4
(1,1)
a (1,2)
1)
(2.1) or (1,2)
Sn
0.0
0.0
12,
a (2,2)
a (1,2) or (2,
(2,2)
Sq
S
1
== 1.0
== 0.7
r= 1.0
0.7
(a)
(b)
Figure 5: Fig. is the MDP under transition kernel Pt. Fig. 5b is under Ps. The two differ in their transition probabilities to 82 and $3.
Example (DrIGM can address cases where IGM fails.). Consider a similar robust cooperative two-agent task (illustrated in Fig. 6) with action spaces A₁ Ag {1,2), state space S {su, $1, $2, $3, $1), and uncertainty set P P.). Let P1 be the training and testing environment. For Pt. all the transitions are fully deterministic, the optimal joint action value function state sy (we omit the 7/(1 7) factor for clarity):
Qids (so, 1,
0.7, Q'à (so, 1,2) 0.4, (0o 2,1) 1.0, Q.3 (50, 2, 0.5.
Pz differs from P1 in that all actions leads to Sa. Therefore, the optimal joint action value function at state so is (we omit the 71(1- - 7) factor for clarity):
QP' (Su. 1, 0.4, Q.a (su. 1, 2) 0.4, Qaa (su, 2, 0.4, Q (sv. 2,2) 2) 0.4. It can be verified that Pt does not admit VDN-style value decomposition, but the worst case, Pa. admits a feusible VDN-style value decomposition.
18

---

Published as a conference paper at ICLR 2026
S
Sg
S r= = 0.7
Sp = 0.4
== 0.7
r= 0.4
(1,1)
(1,2)
all actions
So
Sn 0.0
0.0
(2)
(2,2)
Ss
Sq r= 1.0
S - 0.5
r= 1.0
0.5
(a)
(b)
Figure 6: Fig. 6a is the MDP under transition kernel Pt, Fig. 6b is under Pg.
C PROOFS
C.I PROOF OF THEOREM 1
Proof. Recall that for all a E A, we have
Pot (h, a)
inf PEP
Q (h, a)
(Eq. (3))
pwarst (h, E arg inf Q (h,a).
(Equation (4))
PEP
Thus, Qior
paoore (h.a)
(h a)
Qlat (h, a).
Equation (4), we also defined a
arg aA Pot (h, a). Since pworst (h,a) P, by assumption there exist 1Q p (h,ä)
that satisfy IGM for 2uo (hara(hha) under h. Therefore,
arg max Q (h1, a1)....... arg max Q (hN,
arg max Q
pworst (h,a)
(1):01).... ... arg max QN @
pworst (h.a) (b, ax)
(Eq. (4))
ax
C arg max Peo
pworst (h,a)
(h, a)
(IGM)
arg max Quat pworst (h.a) (h.
(Eq, (4))
arg max Q (h. a);
which shows that [Q [Qrob1ee[N] satisfy DrIGM under h.
Remark 1. The statement of Theorem I assumes that for all P E P there exist IeEM satisfying IGM for Qe, under joint history h E H. However, we note here that this assumption can be relaxed only requiring that there exist [Qi]sE[N] satisfying IGM for Q under h.
Remark 2. As shown in Example 1, an adversarial model P E P that minimizes one agent's 's value need not coincide with the adversarial model pr that minimizes the joint value. Theorem I circumvents this problem by directly considering the global worst case, i.e.,
Qoo0 (h,, a.) == Q, pwarat (h.a) (fu, a:).
(18)
Given this definition, a robust joint action is given by a = (1, or (2, 2), and there exist robust individual action-value functions given by:
(so 1) 0.2, Quob(so; 2) 0.1, @so 1) 0.7,
@ 2) 0.4,
where the robust individual action is given by 01 1, ag Alternatively, another set of individual action- value functions are given by:
Qrob (su.
( 0.3, Q (su, 1) 0.5,
Q (So. 2) 0.6, 2, 02 2. Either way, the robust individual
where the robust individual action is given by actions are aligned with the joint actions.
19

---

Published as a conference paper at ICLR 2026
C.2 PROOF OF THEOREM 2
Proof. We proceed by proving that IGM holds for under each of the three conditions given in Theorem 2. By Theorem this suffices to show that DrIGM holds for under each of the three conditions given in Theorem 2.
VDN condition.
Given a joint history E H. for any P E P, we have
Q (h, a)
Σ iE[N]
Of
Va (a1....,,y) E A.
Let a arg max,; (,,, a;) for i [N], and let a [ālkee[N]
Then, for any A,
(h, E (hs,
iE[N]
s Σ
of (hs,
(Definition of a.)
IE[N]
Q (h, a).
This implies that
arg.......mmmaaa arg max Qf (h1, a.),..., arg max Q (hv, ax) S arg max Q (h, a),
so 1QfkeeN satisfy IGM for Qia under h. Therefore, by Theorem 1, [Qroh ke[N] satisfy DrIGM for Pot under h.
QMIX condition.
Given a joint history h e H. for any P. suppose the following monotonicity
property holds;
eQ (h, a) 2Q a;)
2 0,
Vi [N], (a....., aN) E A.
Let a; arg max,, O (h;, az) for E [M], and let
[ā.leE[N]- Given that aQ (h, a)/0Q////// (h1, an) 2 and ā = arg max,, Of (h1 01). we have (for any a A)
P S Pfo ã1, ................................ aN).
Applying the same logic to all E [N] yields that for any a E A.
0 (h, a) x Uia (h, āΙ, "i.,a.,...,
(19)
P (h, a).
(20)
This implies that
arg max Of (h1, a1)............................. arg max Q (hy, ax)) C arg max Q (h, 0x
a1
so 1Q KE[N] satisfy IGM for Of under h. Therefore, by Theorem 1, [Q:ob] kE[N] satisfy DrIGM for Poo under h.
QTRAN condition. Given a joint history h E H. for any E P, we have (for all a =
(o1....,aN) E A):
[0 a a 2 0,
Σ
of (hi,, Q (h, Viot (h)
(21)
a + a.
where á = [askEEN with à = arg max O (h,,, az) and Viot (h) = max P (h, a) IN (h a.). Therefore,
N
Q (h, a) = Σ Q (h,, ā,) + (h)
(Eq. (21))
20

---

Published as a conference paper at ICLR 2026
max Qiot (h, a),
(Definition of V(h)
This implies that
arg max Qf (h1, a.)..... arg max Q (hN. - arg max Q (h, a), a1
ox
so [Q kEIN satisfy IGM for Quo under h. Therefore, by Theorem 1, [Qr ke[[N] satisfy DrIGM for Peo under h.
Combining the three cases concludes the proof of Theorem
C.3 PROOF OF THEOREM 3
Proof. Recall that given an uncertainty set P, the robust joint action value is defined as,
Plo (h, == inf a (h, a), a(h,a) E H x A.
(22)
PEP
Given that Ptest E P, we directly have:
Q (h, (h, vh E H. E A.
(23)
This concludes the proof.
D ROBUST BELLMAN OPERATORS
We start by introducing the assumptions needed to derive the robust bellman operators. Assumption (Fail-state (Panaganti et al., 2022)). The robust Dec-POMDP has a fail state 8y such that
r(s,,a) a) 0 and (sy,a(s))=1 1,
A,
(24)
This requirement is mild, as fail states naturally arise in both simulated and physical systems. For example, in robotics, configuration where the robot falls and cannot recover, whether in simulators such as MuloCo or in real hardware, serves as a natural fail state. We can further relax it to the following assumption.
Assumption (Vanishing minimal value (Lu et al., 2024)). The underlying RMDP satisfies
min V (s) = 0.
(25))
sES
Without loss of generality, we also assume that any initial state & arg min ViP (s).
This assumption states that the lowest achievable robust value across all states is normalized zero. The exclusion of the minimizing state as the starting point rules out the degenerate case where the agent begins with zero guaranteed return.
p-contamination uncertainty set. Given p-contamination uncertainty set defined in Eq. (7), the robust bellman operator can be expanded as:
(T.Q)(h,a) = r(s, a) + inf
max Q(h', a') n'CA
(26)
Ph.aEPi.a
r(ss a) - 7(1 - p) Enr po
max Q(h' a²)
(27)
A'EA
+ min ViP (s'). ES
(28)
Under Assumption (Assumption 2), obtain that:
(h,) r(ss + 7(1 p) P
PR...
max Q(h',a')
(Assumption (Assumption 2))
a'EAA
=)) + +1(1- -p) En'~po
[Q(h'aa))
(Definition 2)
where df = arg max,; (lob(1(,d4)
21

---

Published as a conference paper at ICLR 2026
TV-uncertainty set. Leveraging Panaganti al. (202s)Prooositiion 1], given TV-uncertainty set defined in Eq. (9), the robust bellman operator can be expanded as:
(
[m(issa)
(TQto))), r(s,a)
inf nel0, x Y R BA Ph
inf Vior e" voor(1")
11
((ssa)) max Qiot (h', (h',aa) a'EA
(29)
Under Assumption (Assumption 2), we obtain that:
(TQio) a) r(s,a)
inf ne(o n 7 Pp.
( (11--))) -0--p0o1 (p))(ss))))
+ (((8,a)
max Q (h', a²) a'EA
(Assumption (Assumption 2))
r(s,a) a)
inf nelo,pt1_n)
? En Epr
- -(1 ()))))
m(s,a) max Q Piot (1/1 Ny.....,,,y,, Ny "y, ",y a'EA
(Definition 2)
E ALGORITHMS
We offer a full description of our algorithms in this section, presented in Algorithms 2 to 7.
Algorithm 2 Robust VDN with p-contamination uncertainty set
Input robustness parameter p. target network update frequency f and
2: Initialize replay buffer D
3: Initialize [Qro*] [E[N] with random parameters
4: Initialize target parameters f- =
5: for episode = 1,... E do
6: Observe initial state s and observation of = ()) for each agent i.
7: for 1,..., do
8: Each agent i choose its action of using e-greedy policy.
9:
Take joint action a', observe the next state s++1 reward r' and observation of (,(t1)1)
for each agent i Store transition (h', ah' a", "",'",hh"" htth+tt) ) in replay buffer D
10:
Sample mini-batch of transitions (h, a, h') from Set a' [arg max,; Quob(hi, af; 0 )iE[N
12:
Set Q P.VDN (h, a; 0) = Lie[N] Q 03;
13:
Set Q P.VDN (h', a';0-) ) = Eie(N Q. prob(h,,/4 d/;0-)
14:
Set y target = + 7(1 - pp P.V.N (h', a²;0-)
15:
16:
VDN Calculate TD loss LTD = (Q
(h, a: 0) target
Update 0 by minimizing LTD Update 0- = 0 with frequency f
18:
end for
20:
end for
F EXPERIMENT DETAILS
F.1 TASK DESCRIPTION
We test our algorithms and baseline algorithms in Bui BuildingEnn in Yeh et al. (2023). This environment considers the control of the heat flow in a multi-zone building so as to maintain a desired temperature setpoint. Building temperature simulation uses first-principled physics models, to capture the worldworld dynamics. The environmental model and reward functions can differ from three climate types and locations (San Diego, Tueson, New York), which jointly decide the climate.
22

---

Published as a conference paper at ICLR 2026
Algorithm 3 Robust QMIX with p-contamination uncertainty set
Input robustness parameter p. target network update frequency f and
2: Initialize replay buffer 'D
3: Initialize [Q;0* and mixing network fo with random parameters 0
4: Initialize target parameters 0- =0
5: for episode = 1,..., E do
6: Observe initial state 8 and observation of = (8") for each agent i.
7: for -,...... do
8:
Each agent choose action of using e-greedy policy.
9:
Take joint action a', observe the next state s*+1 reward r' and observation of o,(st+1)
for each agent i Store transition an' ss,rr', h + in replay buffer D Sample mini-batch of transitions (h, a, s, h s') from D
10:
11:
12:
Set = (arg max (,o o/; 0 [))]]([N]
13:
Set Q P. QMIX
(h, a; = fo((Qu (he, a;; ())
tor Set Q P PMQ (h' a';0)) = --- ((Qrob (1 "):;0))))(())s---
14:
Set y paaget = + 7(1 p)Q QMIX (h', a²;
15:
Calculate TD loss LTD (Q PMMIX (h, a; f) y jageet)3
16:
17:
Update 8 by minimizing LTD
18:
Update 0- = 0 with frequency f
19:
end for
20:
end for
Algorithm 4 Robust QTRAN with p-contamination uncertainty set
1: Input robustness parameter target network update frequency f and
2: Initialize replay buffer D
3: Initialize [Qr
[E[N))
P..TTRNN and y.P.TTAN with random parameters tnt
4: Initialize target parameters 0- =0
5: for episode 1, ... E. do
6: Observe initial state s and observation of = or(s") for each agent i
7: for l,..,, do
8: Each agent i choose its action ot using e-greedy policy.
9:
Take joint action a'. observe the next state s *+1 reward r' and observation of 0,(st+1)
for each agent i Store transition (h', ahf r1,h++1 ) in replay buffer D Sample mini-batch of transitions (h, a, h') from Set a' (arg max,; Qrab a/; 0 [iEEN
10:
12:
Set y target = + p)Q P.P.P.TTn QTRAN (h', a':'''0-)
13:
Caleulate TD loss LTD (Q .IRRAAN (h, a: 0) target
14:
15:
Calculate Teopt using Eq. (12)
16:
Calculate Teopp using Eq. (13)
17:
Update 0 by minimizing L = LTD + Lopt + Lnopt
Update 0- = 0 with frequency f
19:
end for
end for
23

---

Published as a conference paper at ICLR 2026
Algorithm 5 Robust VDN with TV uncertainty set
1: Input robustness parameter target network update frequency f and
2: Initialize replay buffer D
3: Initialize [Qro with random parameters 0
4: Initialize dual network li with random parameters
5: Initialize target parameters f =
6: for episode e = 1,..., E do
7: Observe iniiil state s and observation of os(s)) for each agent i.
8: for tort=11 fort.,.... do
9:
Each agent i choose its action of using e-greedy policy.
10:
Take joint action a', observe the next state s*+1, reward pf and observation of+1 0,(st11)
for each agent i Store transition (h', a h=+1) in replay buffer D Sample mini-batch of transitions (h, a, s,,,,h
12:
Calculate dual loss Lduat using Eq. (17)
14:
Update & by minimizing Ldual Sample another mini-batch transitions (h, s,r, h') from D
16:
Set a' = [arg max,; Qrob (Mi a4; 0 )(E[]
17:
Set Q P.VD (h, a; 0) EiE(N) 2, (hs,
0i a;;
Set Q PR.DN, (h',a'; a'; EiE(N) (h/ 0):0))
18:
19:
Set y Jarget = + 71 p) ( a)
(s, a) Qoo P.VDN (h',a';0-)
VDN (h, a: 0) grarget)z Calculate TD loss LTD (Qt
20:
21:
Update 8 by minimizing LTD
22:
Update 8- f with frequency f
23:
end for
24:
end for
Algorithm 6 Robust QMIX with TVV uncertainty set
1: Input robustness parameter p. target network update frequency f and
2: Initialize replay buffer D
3: Initialize [Qro and mixing network fo with random parameters 0
4: Initialize dual network 77g with random parameters &
5: Initialize target parameters 0 =
6: for episode = 1,..., F do
7: Observe initial state 3 and observation of = (s9)) for each agent i.
8: for 1, T do
9:
Each agent choose action af using e-greedy policy.
10:
Take joint action a', observe the next state sh+1, reward r and observation of a,(stt1)
for each agent i Store transition (h', s',"" 1) in replay buffer D Sample mini-batch of transitions (h, (h,a, h')
11:
12:
13:
Calculate dual loss Ledual using Eq. (17)
14:
Update & by minimizing Laual Sample another mini-batch of transitions (h, a, s, ht) from D
15:
Set a' [arg max,; Q.o (h/, at; 0 )lEEN
Set Q QMIX (h, a; 0) = fal(Qrob(hi az (aaa;)))) ())
Set Q QMIX (h', ā'; 0- = fo ((Qrob (h/, af; ()) )ie((,),'))
18:
Set y target + 7(1 (sn a) - (s, (s, a) PMQMIXX (h'a''00))
Calculate TD loss LTD = (Q P.P.QMIX,, (h, a; 0) ja
20:
21:
Update 0 by minimizing LTD
22:
Update 0- = with frequency f
end for
24:
end for
24

---

Published as a conference paper at ICLR 2026
Algorithm 7 Robust QTRAN with TV uncertainty set
1: Input robustness parameter p. target network update frequency f and
2: Initialize replay buffer DD
3: Initialize [Q] Qeoi Q and V. ...TTAN with random parameters
4: Initialize dual network Ti with random parameters
5: Initialize target parameters =
6: for episode e ...., E do
7: Observe initial state s and observation of - ()") for each agent i.
8: for ...... T do
9:
Each agent i choose its action af using e-greedy policy.
10:
Take joint action a', observe the next state s²+1, reward r' and observation of o,(stt1)
for each agent i Store transition (h' a',"s",',, h²+1) Sample mini-batch of transitions (h, a, 8')
11:
I in replay buffer D
12:
13:
Calculate dual loss Liual using Eq. (17)
14:
Update & by minimizing Laual Sample another mini-batch of transitions (h, a, s, h1) from D
15:
16:
Set = aarg m 15a (h/, 0 O
17:
Set y target = r + 7(1 - () a) (1, a) Q.POTRAN "e'
(h', )]+
Calculate TD loss LTD (Q P.QTRAN (h, a: - y
18:
tot
Calculate Lopt using Eq. (12)
20:
Calculate nopt using Eq. (13)
Update 0 by minimizing T= LTD + Topt + Tnopt
22:
Update 0- = with frequency f
23:
end for
24:
end for
Episode. In Bui Iggnnn, each episode runs for day, with 5-minute time intervals. That is, the horizon length H = 288, and the time interval length 5/60 hours. We set the discount factor y = 0.997 by using H the effective horizon length H = -- =
State Space. For building with N indoor zones, the state contains observable properties of the building environment at timestep t:
(()) = (7(0),..,,T(())) TE(), QCT (1))())))
(30)
where Ti(l) denotes zone i's temperature at time step I, (L( is the heat acquisition from occupants' (I) is the heat gain from the solar irradiance, and (l) and (l) denote the ground and outdoor environment temperature.
activities, QQHI
Observation Space.
For agent i, given the current state s(t), its observation (t(t) is given by:
(t) = (s(())) (E)) (t) (t)).
(31)
That is, agent can observe the tempature at zone i, the heat acquisition from occupants' activities, the heat gain from the solar irradiance, and the ground and outdoor environment temperature.
Action Space. At time t, agent i's action is scalar (i(t) E [-1, which sets the controlled heating supplied to zone i. The joint action a(t) (a.(t)...,,x(())
The objective is to reduce energy consumption while keeping the temperature within given comfort range. Therefore, the reward function is weighted average of these two goals:
Reward Function.
r() = -1 3)] - 3|| -
(32)
where Tagget = (Trarget (t),... T the target temperatures and T(t) (t). (t)) are the actual zonal temperature, IN ||2 denote the l2 norm. We use the same default hyperparam- eter /3 across all experiments.
25

---

Published as a conference paper at ICLR 2026
Environmental Uncertainty. The environmental model and reward functions differ from three climate types and locations (San Diego, Tucson, New York), which jointly decide the climate. Besides, Bui MuildingEEn contains distribution shifts in the ambient outdoor temperature profile Tr incurred by seasonal shifts.
F.2 EXPERIMENTS SETUP.
We implement distributionally robust algorithms with two types of uncertainty sets (p-contamination Zhang et al. (2024), V--unertaintyPanngaa et al. (2022)) and three different value factorization methods (VDN (Sunehag et al., 2017), QMIX (Rashid et al., 2020), QTRAN (Son et al., 2019)). We also implement the GroupDR MARL algorithm from (Liu al., 2025), also with these three different value factorization methods. Each experiment is run independently with 5 different random seeds.
Hyperparameters. Training lasts 600 episodes with parameter updates every 2 steps and target updates every 25k steps. Replay buffer size is 10k; batch size is 64, We use s-greedy exploration with annealed from 1.0 0.01 over 120k steps. Hidden layer size 64.
Environment configurations. The environment configurations we use throughout the three experi- ments are shown in Table 3 where Env_1 is the training environment, and the other environments numbered in order of how much they differ from Env.l.
Environment
Weather
Location
Env_l
Hot_Dry
Tucson
Env_2
Hot Humid
Tampa
Env_3
Very Hot Humid
Honolulu
Env_4
Warm Wry
El Paso
Env_5
Cool_ Marine
Seattle
Env_6
Mixed_Humid
New York
Table 3: Environment configurations for SustainGym BuildingEny
Robust individual Q-networks. Each agent employs recurrent Q-network (Hausknecht & Stone, 2015), encoding local observation concatenated with the previous action (one-hot). Features pass through a fully connected layer with ReLU, followed by single-layer LSTM (hidden size 64). The final hidden state projected into Q-values. We optimize with RMSprop (Ir x 10-4).
Mixing networks (QMIX). Following TorchRL (Bou et al., 2023), we use hypernttwrrks to generate state-dependent mixing weights while enforeing monotonicity. Per-agent Q- Q-values are embedded, passed through ELU, and projected to scalar joint Q. A state-conditioned bias is added via a two-layer MLP. Optimizer: RMSprop (Ir =
5 x 10 4).
QTRAN Networks. We implement joint action-value and state-value networks in Son et al. (2019). The joint action is encoded by concatenated one-hots, projected with ReLU, and optionally combined with agent features. The value network processes the global state and summed agent features in parallel, concatenates them, and outputs a scalar. Both use Adam (Ir = 10 10-3).
Dual networks (TV uncertainty). The global state and joint action (concatenated one-hots) separately embedded via ReLU MLPs, concatenated, and passed through another ReLU layer before final linear projection scalar. Optimizer: Adam (Ir 10 10-3).
GroupDR training. GroupDR first fits a contextual-bandee bandbtanddddsssseeeeeee bbsed worst-case reward estimator to estimate the worst-case reward, using data from Env_1 Env_5 collected under behavior policy. Using this estimator, we then train the individual Q-networks by randomly sampling episodes from Env_l Env_5. (For fairness, Env_6 is never used for training by any method.)
Normalized Return. Because the reward in Bui BuildingEns is negative, we normalize the returns to [0, 1] by the following transformation:
Return + 9000
Normalizod return
(33)
9000
26

---

Published as a conference paper at ICLR 2026
Experiment 1: elimatic shifts. During training, we train each algorithm in Env_1 by sampling episodes from Day I to Day 200. During evaluation, we evaluate each algorithm in Env_1 to Env_6, by sampling 50 episodes from Day 1 to Day 200 in each environment. This setup demonstrates distribution shift induced by climate differences.
Experiment 2: seasonal shifts. During training, we train each algorithm Env_1 by sampling episodes from Day to Day 200. During evaluation, we evaluate each algorithm in Env 1,, by sampling 50 episodes from Day 400 to Day 600 each environment. This setup demonstrates distribution shift induced by seasonality within the same location.
Experiment 3: climatic and seasonal shifts. During training, we train each algorithm in Env_1 by sampling episodes from Day to Day 200. During evaluation, we evaluate each algorithm in Env_6, by sampling 50 episodes from Day 400 to Day 600 in each environment. This setup demonstrates the combined effects of climate and seasonal shifts.
G DISCLOSURE OF LLM USAGE
To improve clarity and readability, we used large language model (LLM) to assist polishing the writing. The LLM was only employed for language refinement (e.g., grammar, style, and conciseness) and was not involved designing methods, experiments, or drawing conclusions.
27


Reviews:
[{"review_id": "YIT0Fxxfk7", "reviewer": "Reviewer_FAoh", "summary": "This paper aims to achieve robust multi-agent reinforcement learning value factorization under environmental transition kernel uncertainty. It proposes a novel history-based DrIGM principle and theoretically demonstrates that the proposed optimal robust joint action-value and the robust individual action-value satisfy the DrIGM principle. Accordingly, two new loss functions are designed to learn these robust action-values under two different uncertainty sets. The proposed approach is integrated with three typical value factorization methods, and its effectiveness is validated through experiments in high-fidelity SustainGym simulators.", "questions": "1. As indicated in Reference [2], there exists a bias between Q(h,a) and Q(s,h,a). Could this bias compromise the validity of GrIGM?  \n2. Can the proposed algorithm effectively resolve the problem presented in Example 1?\n\n[2] On Stateful Value Factorization in Multi-Agent Reinforcement Learning. arXiv 2024", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Addressing environmental uncertainty is crucial for improving policy robustness in real-world applications. \n2. This paper introduces the DrIGM principle, a novel concept in cooperative Multi-Agent Reinforcement Learning.\n3. To realize the DrIGM principle, this paper proposes two loss functions that can be easily integrated into various value factorization algorithms.", "weaknesses": "1. To achieve provable robustness guarantees, the test environment must be included within the uncertainty set. This requirement may limit the method's performance on unseen environmental models.\n2. There is a lack of sensitivity analysis regarding the algorithm's performance across different ranges of uncertainty. It remains unclear to what extent of uncertainty the algorithm can effectively handle.\n3. The experiments lack comparisons with important baseline algorithms, such as ERNIE [1], which addresses changing transition dynamics by adversarial regularization.\n\n[1] Robust multi-agent reinforcement learning via adversarial regularization: Theoretical foundation and stable algorithms. NIPS 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8Dz0DALeSL", "reviewer": "Reviewer_Nb62", "summary": "This paper addresses a critical, often overlooked challenge in cooperative Multi-Agent Reinforcement Learning (MARL): ensuring reliable decentralized execution when the deployment environment deviates significantly from the training environment. The core paradigm in cooperative MARL is centralized training with decentralized execution (CTDE), relying on value factorization methods (like VDN/QMIX/QTRAN) and the Individual-Global-Maximum (IGM) principle. The authors demonstrate that naive robustification of individual Q-functions, similar to approaches in single-agent Distributionally Robust RL (DR-RL), breaks the IGM alignment needed for decentralized execution.\nTo fix this, the paper introduces the Distributionally Robust IGM (DrIGM) principle. DrIGM requires that robust individual greedy actions align with the robust team-optimal joint action. Theoretically, this is achieved by defining robust individual action values based on the global worst-case model ($P_{worst}$) for the joint value function, rather than assuming independent per-agent worst cases. Building on this robust factorization theory (Theorems 1, 2), the authors derive DrIGM-compliant algorithms compatible with VDN, QMIX, and QTRAN, training them using robust Bellman operators based on standard uncertainty sets (ρ-contamination and Total Variation). Experiments on high-fidelity SustainGym simulators, modeling HVAC control under climatic and seasonal shifts, show that the DrIGM-based methods consistently mitigate out-of-distribution performance degradation compared to non-robust and baseline robust MARL algorithms.", "questions": "1.  The experimental environment (HVAC control) uses continuous actions. Since the DrIGM principle (Definition 2) and the derived robust Bellman operators (Eqs. 7, 9) rely fundamentally on the argmax operation over the action space, how exactly was the robust joint greedy action calculated in the continuous action setting for VDN, QMIX, and QTRAN? Was the action space discretized, or were alternative continuous control techniques (like optimization or actor-critic methods) employed for the argmax step, and if so, how does that impact the theoretical guarantee of DrIGM?\n\n2. Theorem 1 relies on the existence of individual Q-functions under $P_{worst} (h, \\overline{a})$. In the robust TD loss for both ρ-contamination and TV uncertainty, the expectation is calculated only over the nominal transition model $P^0$ (or implicitly factored through η in the TV case). Can you elaborate on how the implicit selection of $P_{worst}$ is handled during the training updates via function approximation, especially given that $P_{worst}$ itself depends on the optimal robust joint action $\\overline{a}$?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- Originality\n\nPrevious value factorization methods rely entirely on the classical IGM principle to ensure that decentralized greedy actions recover the team-optimal joint action. The authors correctly identify that the reliability of this recipe is uncertain in real-world settings plagued by environmental uncertainties like the sim-to-real gap or system noise. Extending DR-RL techniques to cooperative MARL is fundamentally non-trivial because individual agents act on local histories but share a single, team-coupled reward.\n\nCrucially, the authors provide a concrete counterexample (Example 1) showing that simply adopting robust per-agent action value formulations from the single-agent DR-RL literature breaks the decentralized alignment required by IGM in the multi-agent cooperative setting. This realization is a key theoretical novelty. The derivation that DrIGM is guaranteed when robust individual value functions are defined with respect to the global worst-case joint action-value function is a principled solution to this misalignment problem. This approach as depicted in Eq. 5, ensures robustness while maintaining the decentralized execution structure of CTDE. Furthermore, the work successfully derives DrIGM-compliant robust variants of VDN, QMIX, and QTRAN, showing the broad applicability of the new principle to established architectures. This systematic framework for model uncertainty in the CTDE regime under partial observability is a substantial theoretical advance over related work which often targets Nash solutions, assumes full observability, or requires individual rewards.\n\n\n- Quality\n\nThe foundation of the work rests on solid theoretical results, specifically Theorems 1, 2, and 3. Theorem 1 formally establishes the sufficient condition under which DrIGM holds, tying the robust individual Q-function to the global worst-case model. Theorem 2 further proves that this DrIGM condition is compatible with the structural constraints of all three canonical value factorization methods: VDN (additive factorization), QMIX (monotonic factorization), and QTRAN (consistency constraints). This demonstrates the generality and foundational nature of DrIGM.\n\nThe use of Distributionally Robust Optimization (DRO) techniques is appropriate for modeling environmental uncertainty. The paper successfully derives corresponding robust Bellman operators for two standard uncertainty sets, ρ-contamination and Total Variation (TV), which are well-studied in the single-agent DR-RL literature. These operators are then integrated into the practical TD-loss formulation (Eqs. 14, 15) for training deep recurrent Q-networks (DRQN-style networks).\n\nEmpirically, the evaluation is strong, moving beyond typical simplified MARL benchmarks. The authors use the SustainGym benchmark focused on multi-agent HVAC control, which intrinsically involves stochastic dynamics, partial observability, and inter-agent coupling, making it a high-fidelity environment suitable for testing robustness. The evaluation protocol is designed explicitly to measure generalization under distribution shift, simulating realistic deployment scenarios where unseen configurations arise. The consistency of the results across multiple factorization architectures (VDN, QMIX, QTRAN) and both uncertainty sets (ρ-contamination and TV) provides compelling evidence that the DrIGM framework effectively mitigates performance degradation under climatic and seasonal shifts.\n\n\n- Clarity\n\nThe problem definition is clear. The introduction of DrIGM (Definition 2) is logically built upon the classical IGM principle (Definition 1), clearly showing the extension to the robust setting. The immediate inclusion of Example 1 serves as a powerful illustration, preventing ambiguity by clearly showing why naive application of single-agent DR-RL techniques fails in the cooperative domain. This preemptive clarification is helpful in understanding the rest of the paper.\n\n\n- Significance\n\nThe practical utility of MARL policies is often hampered by environmental uncertainty, which can cascade into coordination failures due to partial observability and inter-agent coupling. By introducing DrIGM, the paper provides a crucial theoretical mechanism to maintain decentralized execution reliability even under distribution shifts. The resulting algorithms integrate seamlessly with existing CTDE codebases (VDN, QMIX, QTRAN) without requiring bespoke per-agent reward shaping or changes to the decentralized execution mechanism. This makes DrIGM a readily adoptable approach for improving robustness in many existing CTDE systems.", "weaknesses": "- Originality\n\nWhile the formulation of DrIGM is novel for cooperative MARL, the theoretical and algorithmic tools leveraged to derive the robust Bellman operators are essentially direct adaptations of established single-agent Distributionally Robust RL (DR-RL) techniques. Specifically, the paper relies on ρ-contamination and Total Variation (TV) uncertainty sets, which are standard in the robust MDP literature. The extension relies fundamentally on the assumption of a history-action rectangular uncertainty set P (Eq. 1). This rectangularity assumption is known to simplify the robust Bellman operator to maximize over actions before minimizing over the transition probability P. Although this is necessary for the current theoretical construction, relying on this rectangular structure limits the sophistication of the uncertainty models that can be handled by DrIGM.\n\n\n- Clarity\n\nthe precise implementation of the core DrIGM concept needs improvement in clarity for the algorithms section. The definition of DrIGM relies on the robust individual action value $Q_i^{rob}$ being derived from the value function under $P_{worst}(h, \\overline{a})$, where  $\\overline{a}$ is the robust optimal joint action. However, the resulting robust Bellman operators (Eqs. 7 and 9) replace the term $max_{a′} Q_{tot}^P (h′, a′)$ with $Q_{tot}^P (h′, \\overline{a}′)$, where  $\\overline{a}'$ is derived from the greedy robust individual actions $ \\overline{a}'_i  = argmax a'_i Q_i^{rob} (h'_i, a'_i)$. This is a crucial step that relies on the DrIGM principle to align execution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "im5rmrCVvX", "reviewer": "Reviewer_pUrR", "summary": "This paper proposes Distributionally Robust IGM (DrIGM) — a principled framework extending the classical Individual–Global–Max (IGM) principle in cooperative multi-agent reinforcement learning (MARL) to environments with distributional uncertainty.\nThe authors formalize DrIGM within the Dec-POMDP framework, proving that robust individual action-value functions can align with the robust joint action-value function under specific rectangular uncertainty sets.\nThey then instantiate DrIGM-based variants of VDN, QMIX, and QTRAN, design robust Bellman operators under ρ-contamination and total variation (TV) uncertainty models, and empirically evaluate the methods on the SustainGym HVAC control benchmark.\nResults show modest but consistent improvements over non-robust baselines and the existing.", "questions": "1. In line 189. The optimal joint action value should not produce eight values.Given there are only two scenarios, there should be two corresponding joint Q-values, not eight.\n\n2. In line 203. My understanding is that DrIGM should ensure that, under the worst-case environment, the optimal individual actions equal the globally optimal joint action.However, Example 1 instead shows that the worst individual actions do not equal the globally worst joint action, which corresponds to a different principle (worst-vs-worst, not best-under-worst).\n\n3. DrIGM is essentially equivalent to IGM under the worst-case model?\n\n\n4. Could you supplement this by reproducing it on at least one standard MARL benchmark (such as SMAC) and performing sensitivity/ablation analysis on ρ and providing a significance test?\n\n5. How does the DrIGM method compare to the non-robust version in terms of training time, sample complexity, and memory overhead? Please provide training curves and statistical cost (or complexity estimate).", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper formally defines DrIGM and provides a proof, logically connecting robustness with value factorization.\n\n2. DrIGM can be directly embedded into mainstream architectures such as VDN, QMIX, and QTRAN, allowing for the reuse of existing code in engineering projects.", "weaknesses": "1. DrIGM is essentially equivalent to \"performing IGM under a global worst-case model.\" In other words, the paper does not propose any new algorithmic ideas fundamentally different from existing Distributionally robust RL or MARL value decomposition methods; it simply incorporates the \"worst-case model\" into the IGM framework. This point needs to be explicitly acknowledged in the manuscript, along with a discussion of its limitations and added value (e.g., in which situations is this worst-case model approach more advantageous than other robustness strategies).\n\n2. Example 1's presentation is particularly critical: the current description of the \"selection process for individual optimal actions and joint optimal actions\" is not detailed enough, making it difficult for readers to trace why taking the worst value for each agent individually leads to inconsistencies with the joint worst value.  The symbols and variables are quite messy. It is recommended to provide intuitive annotations or diagrams where they first appear. η(s, a) first appears on line 312, but is not explained until line 375, which disrupts readability and understanding of its role in the proposed framework.\n\n3. Only the SustainGym HVAC environment is used; no tests on SMAC, MPE, or adversarial settings.Baselines are too few (only GroupDR and non-robust variants).Reported gains are small (2–5%) and lack significance tests or ablation analysis (e.g., sensitivity to ρ).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)