Empirical Study on Robustness and Resilience in Cooperative Multi-Agent Reinforcement Learning
Authors: Simin Li
Zihao Mao
Hanxiao Li
Zonglei Jing
Zhuohang bian
Jun Guo
Li Wang
Zhuoran Han
Ruixiao Xu
Xin Yu
Chengdong Ma
Yuqing Ma
Bo An
Yaodong Yang
Weifeng Lv
Xianglong Liu
Venue: NeurIPS 2025 poster ()
Keywords: Multi-Agent Reinforcement Learning
Trustworthy AI
Primary area: reinforcement_learning
TL;DR: We conduct a large scale empirical evaluation to study the effect of robustness and resilience in MARL

Abstract: In cooperative Multi-Agent Reinforcement Learning (MARL), it is a common practice to tune hyperparameters in ideal simulated environments to maximize cooperative performance. However, policies tuned for cooperation often fail to maintain robustness and resilience under real-world uncertainties. Building trustworthy MARL systems requires a deep understanding of \emph{robustness}, which ensures stability under uncertainties, and \emph{resilience}, the ability to recover from disruptions—a concept extensively studied in control systems but largely overlooked in MARL. In this paper, we present a large-scale empirical study comprising over 82,620 experiments to evaluate cooperation, robustness, and resilience in MARL across 4 real-world environments, 13 uncertainty types, and 15 hyperparameters. Our key findings are: (1) Under mild uncertainty, optimizing cooperation improves robustness and resilience, but this link weakens as perturbations intensify. Robustness and resilience also varies by algorithm and uncertainty type. (2) Robustness and resilience do not generalize across uncertainty modalities or agent scopes: policies robust to action noise for all agents may fail under observation noise on a single agent. (3) Hyperparameter tuning is critical for trustworthy MARL: surprisingly, standard practices like parameter sharing, GAE, and PopArt can hurt robustness, while early stopping, high critic learning rates, and Leaky ReLU consistently help. By optimizing hyperparameters only, we observe substantial improvement in cooperation, robustness and resilience across all MARL backbones, with the phenomenon also generalizing to robust MARL methods across these backbones.

Empirical Study on Robustness and Resilience in Cooperative Multi-Agent Reinforcement Learning
Simin Li', Zihao Mao', Hanxiao Li', Zonglei Jing', Zhuohang Bian', Jun Guo', Li Wang' Zhuoran Han', Ruixiao Xu', Xin Yu', Chengdong Ma', Yuging Ma², Bo An Yaodong Yang" >Weifeng Lv Xianglong Li,
State Key Laboratory of Complex & Critical Software Environment, Beihang University, China ZZoogguuancun Laboratory, China ?Institute of Artificial Intelligence, Peking University, China "Institute Data Space, Hefei Comprehensive National Science Center, China 'Institute of Artificial Intelligence, Beihang University, China Nanyang Technological University, Singapore
Abstract
In cooperative Multi-Agent Reinforcement Learning (MARL), it is common practice to tume hyperparameters in ideal simulated environments to maximize co- operative performance. However, policies tuned for cooperation often fail to main- tain robustness and resilience under real-world uncertainties. Building trustworthy MARL systems requires deep understanding of robustmess, which ensures stability under uncertainties, and resilience, the ability to recover from disruptions con- cept extensively studied in control systems but largely overlooked in MARL. In this paper, we present a large-scale empirical study comprising over 82,620 experiments evaluate cooperation, robustness, and resilience MARL across real-world environments, 13 uncertainty types, and 15 hyperparameters. Our key findings are: (1) Under mild uncertainty, optimizing cooperation improves robustness and silience, but this link weakens as perturbations intensify. Robustness and resilience also varies by algorithm and uncertainty type. (2) Robustness and resilience not generalize across uncertainty modalities or agent scopes: policies robust to action noise for all agents may fail under observation noise on a single agent. (3) Hyper- parameter tuning critical for trustworthy MARL: surprisingly. standard practices like parameter sharing, GAE, and PopArt can hurt robustness, while early stopping, high critic learning rates, and Leaky ReLU consistently help. By optimizing hyper- parameters only, we observe substantial improvement in cooperation, robustness and resilience across all MARL backbones, with the phenomenon also generalizing to robust MARL methods across these backbones. Code and results available at https:://githue http:://gttthbb.. com BUAA /ttp://AAAUTTtuttttthhaaa////// marl, benchmark
Introduction
10 has demon- strated significant success across various complex simulation environments, including StarCraft [1]. turbulent flow modeling [1]. and network control
Cooperative Multi-agent reinforcement learning (MARL)
[ However, real-world deployment of MARL systems introduce uncertainties unseen in simulation environments. These uncertainties spans from observation errors [4 15), action perturbations 17), and environmental unpredictability [[8 [9]
which degrade the performance of MARL significantly.
In this paper, we study two fundamental ways deal with uncertainties in real-world deployment: robustness and resilience. Robustness measures the ability of MARL to withstand uncertainties in the
"Corresponding Authors. E-mails: yaodong.yang @ pku.edu.cn, Iwf@ buaa.edu.cn. xlliu @ buaa.edu.cn.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

real-world, enabling MARL systems to maintain functionality despite partial system failures [20]. Resilience, on the other hand, is the ability of MARL recover from disruptions. For example, robot swarms may lose coordination due to communication interference, or smart grids may collapse during natural disasters. After the disruption, the system should be able to recover from such external shock. While robustness and resilience are complementary, their distinction has been extensively studied in fields such as control theory [21], ecology [22]. and economics (23), but remains underexplored in MARL. Current MARL research often conflates these concepts (24 25 26 neglecting their critical differences and failing to account for resilience in evaluation.
Another challenge in dealing with real-world uncertainties lies in understanding the role of hyperpa- rameters. While often overlooked in theoretical analyses, careful selection of hyperparameters can play more critical role than the algorithms themselves. In single-agent RL. hyperparameters account for most the observed empirical differences between TRPO and PPO [27]. Similarly, MARL, MAPPO achieves state-of- of-thee-rt performance largely due to its emphasis on hyperparameters ( including parameter sharing and five PPO-specific choices. Consequently, it is common practice for MARL researchers to tune hyperparameters to optimize cooperative performance.
However, such tuning effective only in the ide- alized environments on which it is trained, and its effects on policy robustness and resilience remain largely unexplored. As shown in Fig.
Task: Dexhands HanddSssonsSssssorss
200
Optimizing Implemenatioon Cheioe
Episodo Reward
I while changing hyperparameters maximizes cooperative rewards, it significantly reduces ro- bustness and resilience under uncertainties. Fur- ther analysis shows that hyperparameters in- fluence cooperation, robustness and resilience more strongly than the choice of MARL algo- rithm out of 18 tasks via two-way ANOVA (p < 0.001; see Appendix. [3.4for full results), To evaluate and understand the role of robust- ness and resilience in MARL, conduct large-scale empirical study involving 82,620 ex- periments across 4 real-world environments, 13 uncertainty types, and 15 hyperparameters. Our main findings are as follows:
1.50
100
Cooperation
Robustness
Resilience
Figure I: While optimizing hyperparameters to im- prove cooperation, the algorithm gets significantly less robust and resilient when uncertainty occurs.
1. While improving cooperation can enhance robustness and resilience under mild uncertainty, this relationship weakens as perturbations intensify. This trend holds across diverse uncertainty types and agent configurations. Moreover, algorithm sensitivity varies: MADDPG is more robust action noise, whereas MAPPO and HAPPO perform better under observation uncertainty.
2. Robustness in MARL can not generalize across uncertainty types or scopes. Policies trained to handle action noise may still fail under observation and environment shifts, and defenses against group-level perturbations may break down under attacks against single agent. Trustworthy MARL. must account for diverse uncertainty modalities and agent-level effects.
Hyperparameter tuning plays a critical role robustness and resilience. Surprisingly, common practices such parameter sharing, GAE, and PopArt can hurt performance under uncertainty, while techniques like early stopping, critic-dominant learning rates, and Leaky ReLU consistently observe average improvement of 52.60%
improve it. By combining these hyperparameters, in cooperation, 34.78% in robustness and 60.34% in resilience. This same set of hyperparameters generalizes robust MARL methods on the same backbones, yielding average improvements of 89. 99.43% on cooperation, 65,83% on robustness, 82.96% on resilience.
Related Work
Uncertainties in RL and MARL Robustness against uncertainties has been extensively studied within the framework of robust RL and MARL, as summarized in (28 In reinforcement learning, uncertainties deployment are typically categorized into state 29
action [32] and envi-
ronment uncertainties 33 [8] MARL follows similar taxonomy but incorporates the multi-agent dynamic. For observation uncertainties, noise can be introduced each agent
15| 35 or applied
to specific agents [14. Action uncertainties often arise when one more agents deviate from their

---

optimal policy [6 36 37 17
Environment uncertainties in MARL are typically analogous to those 38 The primary focus of robust RL and MARL is to make algorithms
seen in single-agent RL robust against the worst-case adversaries chosen in the uncertainty set. In our paper, we select set of representative methods derived from these studies for robustness evaluation. For resilience, the MARL literature often overlook its distinct role from robustness, using the term resilience and In this paper, we draw in fields such as control theory [21, ecology [22], and economics 23 to formally define resilience in MARL.
robustness interchangeably [25] 39
26
40
The Importance of hyperparameters. It a well-known fact that implementation matters for RL and MARL. In RL, [27] highlighted the suprising finding that variations in performance among RL algorithms often stem from hyperparameters rather than fundamental algorithmic differences. This notion was further explored by [4], who provided a comprchensive set of recommendations for RL implementation through large-scale experimentation. In MARI., Epymarl 42 introduced the first extensive benchmark suite, while MAPPO [ significantly boosted Epymarl's performance by simply optimizing the implementation. Similarly, Pymarlv2 4 demonstrated that state-of-the atet-art results in the SMAC environment can be achieved by fine-tuning QMIX [2]. [24] and 44] offer preliminary evaluations of RL/MARL under uncertainty, but are limited to simple simulation experiments in a small scale. The works most similar to ours are RRLS 45 and Robust Gymnasium [46]. which provide integrated codebases for evaluating the robustness of single- and multi-agent RL. Our work differs by presenting the relations between cooperation, robustness and resilience under multiple real world environments, algorithms, and diverse uncertainties based on over 82,620 experiments.
Robustness and Resilience
Preliminaries. We formulate MARL decentralized partially observable Markov decision process (Dec-POMDP) [7]. defined as a tuple:
G = W.S,,,,,,A,P.RR,
Here N {I,.., N is the set of N agents, S is the global state space, O Xi is the observation space, O is the observation emission function. At each timestep, each agent selects an action a² AA', with = X is the joint action space, forming joint action a E A. The environment proceeds following the state transition probability P S x A + ASS), and each agent receives shared reward function r(s, a) : S + R in the training environment, Y E [0. 1) the discount factor. Each agent owns a state-action trajectory T' E = (O' x AA'", and make decisions using a (stochastic) policy N' <a'('r=") T - ((A ). Let po be the initial state distribution, the goal of the joint policy is to maximize the objective J(x) = Ro [R_ (Eto 7'rx|so]]-
Motivating Example: Robustness versus Resilience. We use metropolitan power grid to illustrate robustness and resilience under both small and large uncertainties. Under normal conditions, the grid faces minor perturbations such sensor noise, voltage fluctuations, or local demand changes, where it should exhibit robustness, maintaining stable operation despite small disturbances. Yet even mild perturbations can temporarily degrade performance, requiring resilience to recover normal functionality. In contrast, rare but severe events like earthquakes or major short circuits may cause large-scale blackouts. Here, robustness measures how much performance is retained under extreme conditions, while resilience captures the grid's ability to re-stabilize and resume operation after recovery. Together, they describe a system's capacity to withstand and recover from perturbations.
Robustness. Robustness has been a central concept in control systems, which refers to the stability of the algorithm under uncertainties 48). In MARL, the study of robustness relies on defining an uncertainty set U/ in the decision process. During the deployment of MARL policies, such uncertainty set l/ can be defined as a distribution over uncertainty realizations, where each u Ul represents perturbation on observation [4], action [34] 40) or environments [38. Let π be fixed policy, and Po the initial state distribution, we define the robustness of x as:
jrobust (T) = En--z
Exgrpo Er,y
Σ
7 rr
Resilience. When uncertainties cannot be handled by algorithms alone, robustness alone may become insufficient. In such cases, resilience--iz -ie., , the ability of systems to recover from external shocks 49 becomes crucial. As illustrated in Fig. resilience and robustness complementary: while
3

---

Per-Step Reward
Cooperation
Robustness
Resilience Recover After Uncertainties
Without Uncertainties
Resist During Uncertainties
Timestep
Figure 2: Relation between cooperation, robustness, and resilience under uncertainty. Cooperative MARL is trained without perturbations, but must be robust and resilient when they occur.
robustness allows a system to maintain functionality under small perturbations, resilience ensures recovery when perturbations exceed the limits of robustness [50]. This complementary relationship has been extensively explored across various fields, including control theory 21 ecology [22]. economics [23] and complex networks [51..
Despite this, the MARL literature often conflates resilience with robustness, overlooking their distinct roles. For instance, [24) frame resilience as an inherent feature of robustness, while [25 39 label their approaches resilient MARL but ground their methodologies in robust RL. Similarly, 226 and
[40] use the terms resilience and robustness interchangeably. failing to distinguish between the two. To explicitly define resilience as the capacity recover from uncertainties, we propose a framework in which an episode begins at perturbed state S, at timestep fu. where the perturbation is induced by uncertainty U. From this perturbed state, the system evolves without additional uncertainties. Formally, let Su ~ Pa denote the initial state distribution, we define the resilience of π as:
jresilience (T) Ru
Bas Ppp E
r'ra
80
8u
We view resilience a practically grounded form of generalization, emphasizing recovery and stabil- ity under unknown perturbations. In reinforcement learning, generalization measures policy's 's ability sustain performance across variations in initial states or environment dynamics, typically bounded or randomized during training (52). In contrast, resilience concerns recovery after unexpected disturbances, characterized by a post-uncertainty state distribution s" that arises from real-world perturbations and often lies beyond standard generalization settings. Thus, resilience addresses narrower yet practically crucial challenge for deployment and opens algorithmic directions toward policies that actively adapt or recover from disturbed states.
Difference Between Robustness and Resilience. We conduct two studies illustrate the distinction between robustness and resilience in multi-agent reinforcement learning. A system is resilient but non-robust if it recovers well from a variety of initial states, including those following adversarial perturbations, but fails under persistent uncertainties such as external forces or observation noise. such cases, the policy may succeed at high-level task execution but lacks the fine- fine-grained control needed for precise behavior, as small perturbations consistently mislead action selection. Conversely, a system robust but non-resilient if it performs reliably under continuous uncertainty, yet fails to recover when initialized in perturbed or atypical ttates-ofte due to limited exposure to such states during training. In both cases, we observe evidence of state aliasing, where similar observation histories require subtly different responses, but the policy fails to distinguish them, resulting in degraded performance. Additional examples, results, visualizations, and analysis are provided in Appendix B.2
4 Experiment Procedure
In this section, we describe the hyperparameters, types of uncertainties, environments, algorithms used for our evaluation. Our full experiment takes ~230K GPU hours, measured in GTX 4090.
Algorithms. To align with the continuous control requirements of our real-world environments, we three widely used MARL algorithms that support continu-
evaluate the impact of hyperparameters ous control: MADDPG [1] MAPPO [3]. and HAPPO 4 with HAPPO representing heterogeneous- agent-based methods. Our study focuses on commonly used algorithms ensure broad applicability while prioritizing an in-depth analysis of hyperparameters. Our modular codebase enables easy integration of new algorithms via predefined interfaces, supporting future extensions.

---

Hyperparameters. For generality, we pick hyperparameters that are used
General hyyperparameters
Choices
Choice Kange
by most methods. Specifically, we
Network Hidden Size
[64, 128, 256]
Discount Factor (7)
10.9, 0.95, 0.99)
consider both general and algorithm- specific hyperparameters. General hy- perparameters includes network hidden size, discount factor, activation func- tion, initialization method, neural net- work type. learning rate, critic learn- ing rate, feature normalization, share parameters and early stop. For MAD- DPG, algorithm-specing cludes N-step TD) and exploration noise. For MAPPO and HAPPO, algorithm- specific choices includes entropy coeffi- cient, Use GAE and Use PopArt. The de- scriptions of each hyperparameters and the reasons for selecting them are de- ferred to Appendix [.I.I]] The values of each hyperparameters are shown in Table. [ We use bold font to denote the default choices, which are shared for all algorithms and environments. This leads to 15 different hyperparameters. In each time, we vary one different hyperparameters to test its effect, resulting in 34 models with different implementations. Other hyperparameters used in our experiments are listed in Appendix. A.1.2
Activation Function
[ReLU, Leaky RcLU, SELU, Sigmoid. Taanh]
Initializaion Method
[Orthorgonal, Xavier]
Neural Network Type
[MLP. RNN!
1 leamnin Rate (I.R)
[5e-5, Se-4, Se-3]
Critic Learning Rate
[5e-5, Se-4, 5e-3]
Feature Normalization
Truee False]
Share Parameters
[True. False)
Early Stop
[Truc False)
MADDPG Specific hyperparameters
TD Steps (N Step)
[5. 20, 25]
choices in-
Exploratory noise
[0.001, 0,D1, 0.1,0.5. I]
MAPPOНAAPO Specilie hyperparameters
Entropy Cocfficient
[0.0001, 0,001, 0.11
Use GAE
True. False]
Use PopArt
True, False]
Table 1: General and algorithm-specifi hyperparameters shared for all methods. Default choices are shown in bold font, which is shared by all algorithms and environments.
Table 2: Overview of evaluated environments.
Environment
Task Type
Control Mode Episude Len. Engine/Source
Data Source
Challenge
DexHand
Mulli-robot Manipalation
Continuous
880
Issac Gym
Real-world Robots
Precise Control
Quads
Multi-rohol Navigation
Continuous
1600
OpenAl Gym
worldworld Robots
Long-range Task Assigmment
Traffic
Network Control
Discrete
~ 1000
SUMO
Real-world Traffic
Long-range range Contro
Voltage
Network Control
Continuous
200
IEEE Standard Real-world
Grid Complex and Noisy Dynamics
Real-World Environments. Unlike conventional benchmarks that consider solely simulation en- vironments, our study incorporates four
environments encompassing diverse task types, control modes, episode lengths, simulation engines, data sources, and control challenges, as summa- rized in Table. 2 The first two environments, Dexterous Hand Manipulation (Dexhand) [53] and Quadrotor Swarm Control (Quad) [54] are grounded in real-world applications, allowing policies learned in simulation to be directly transferred to physical robots with the same dynamic. We use these environments for tasks that requires robustness in precise control. We select six tasks in each environment. The remaining two environments, Intelligent Traffic Control (Traffic) [55] and Active Voltage Control (Voltage) (56) are constructed from real-world data, ensuring high-fidelity replication of real-world dynamics. We use these environments for tasks that requires robustness in adaptive control when cooperators are unavailable. We select all three tasks with homogeneous observation and action space for Traffic and all three tasks for Voltage, resulting in 18 tasks in total. See detailed descriptions of environments and tasks in Appendix. A.2
Types of Uncertainties. We evaluate the robustness and resilience of MARL algorithms under 13 distinct uncertainties spanning observation, action, and environment. For observation uncertainties (obs), we consider three types: Gaussian noise, greedy worst-case attacks using white-box, gradient- based methods to generate perturbations [30 and learned optimal attacks via MARL [4 15]. These uncertainties are applied either to all agents with a small perturbation budget (e = 0.1) or to single agent with a large budget (e = 0.2), resulting in six scenarios. For action uncertainties (act), we assess three types: random policies, greedy worst-case policies 34 and learned optimal policies [6]. Action perturbations are modeled as eी + (1-e)) e),,, where से represents the perturbed policy. Similar to observation uncertainties, these are applied across all agents (E = 0.1) to single agent (e = 0.2), yielding six scenarios. For environmental uncertainties (env), following [57] 58 59 60), we sample 50 rollouts uniformly from uncertainty sets over environmental hyperparameters (e.g., mass, velocity) and report the worst-case result. In total, our study includes observations, actions, and 1 environmental uncertainty, totaling 13 distinct scenarios. We assess both robustness and resilience under these uncertainties, resulting in 27 evaluation settings: 1 cooperative baseline, 13 robustness evaluations, and 13 resilience evaluations. Further details are provided in Appendix. AA.3
Experiment Procedures. In each experiment, we first train all algorithms and tasks using default hyperparameters. Then, for each hyperparameter in Table. I we vary one setting at time to create
5

---

212
unge
1X1
1M
6m
091
011
30
000
354
100
1m
6:14
094
011
022
004
0000
034
022
145
6.2
004
4.20
All
036
211
Figure 4: The extent cooperation is corre- Figure 3: Correlation between cooperation, robustness, and lated to robustness and resilience linearly depends on the severity of the attack.
Madean (1aN
HAT0
resilience under uncertainty types and algorithms.
set of cooperative models. To evaluate robustness, fix all cooperative models and measure their performance under uncertainties. Finally, we evaluate resilience by initializing a new episode from the perturbed state reached by cooperative models after encountering uncertainties. We then assess their performance from this state, measuring their ability to recover from uncertainties. We train all cooperative models using 5 random seeds, and report the robustness and resilience results on the same five seeds as the cooperative model. In total, our study includes (random seeds) x 27 (uncertainty settings) 18 (tasks) x 34 (hyperparameters) 82620 experiments.
Architecture. For practitioners to evaluate the robustness and resilience in their own settings, our codebase allows users to easily integrate custom algorithms and environments by adhering to a well- defined interface. This flexible architecture ensures that users can seamlessly adapt our framework to their specific requirements. We defer these details to Appendix. A.4
5 Experiment Results and Takeaways
In this section, we thoroughly analyze the data gained in our experiments and summarize our key findings as takeaways for MARL practitioners and researchers. A concise summarization of our finding is shown in Table. B To account for variations in reward scaling across tasks, we use normalizations for different analysis, with details in Appendix. B.] We report statistically significant results to mitigate the randomness introduced by diverse test conditions, while noting that these findings reflect general trends and may not hold in every individual case.
Table 3: Recommended practices for handling different scenarios in MARL.
Scenario
Suggestion
Reference
Small uncertainties
Optimize cooperation
Section 5.1
Large uncertainties
Requires dedicated rolustnessrrsilliccce strategics
Section 5
uncertainties
MAPPOHAPPO best
Section 38
Action uncertainties
MADDPG best
Section 5.1
Modulities
Evaluate obs/env/action sepurately
Section 5.2
Agent scopes
Test both individualgllohal perturbations
Section 57
Early stop
On
Section 55
Critic LR
Actor LR
Soction 5.5
Activation
Leaky RtLU
Section 5.5
Ofl OIT
Section 3.5
PopArt
Section 5.1
Param sharing
On (homogeneous), Off (heterogeneous)
Soction
5.1
Exploration
High (MAPPOHHAPO)) Moderate (MADDPG)
Section 5.3
5.1 Are Cooperation, Robustness, and Resilience Correlated?
Our first evaluation investigates the relationship between cooperation, robustness, and resilience. We compute the Pearson correlation among these metrics to determine whether maximizing cooperation naturally leads to higher robustness and resilience. The results are shown in Fig. 3
1. The correlation of cooperation with robustness, and resilience depends on the severity of the attack. At the first glance, robustness and resilience seems highly correlated with cooperation (Fig. 3. particularly under observation and environmental uncertainties. However, further analysis reveals that this correlation is strongly influenced by the severity of the uncertainty. Specifically, Fig. illustrates the relationship between correlation strength and the percentage of performance degradation under varying uncertainties. The results show a clear linear trend: as the severity of the attack increases, the Pearson correlation between cooperation and robustness (r = 0.85, p < .001) and between cooperation and resilience (r = 0.76. p < .001) weakens significantly. This suggests that mild uncertainties in MARL can be countered by simply maximizing cooperative performance, but their impact becomes critical when perturbation is severe. Similarly, while robustness and

---

on pessis sugs
Comelation Between Imppcct Unnertatrties Robustness
Comelation Betweet Impaact Uscemaintes on Resilience
on pornto
6028
4:28
0.21
1.00
LSV
6.56
gronn
4:22
6:38
4:20
0.31
4424
031
037
XXT
(100
6:54
0.66
6:30
0:44
8.55
de spindl grece
6:21
6:24
6:4
6:37
0.2
60M
ABC
6:21
6:10
0311
6-41
3.18
60
C
6.4
4.5
4.28
002
037
1.90
6:55
0.2
0:44
8.5
0.9
prmvy congle
0.5
3.38
327
6.58
AI5
unt spetes
41
614
C13
614
4:00
086
031
945
344
0:44
6.4
944
OUT
RUAS
0.17
-39
A24
0111
3.55
836
SUY
654
4:55
X25
325
A29
6/4
0.31
6:71
6.77
300
000
0.44
de pestoe alogl
als Secormaa siught
en eesoemm stogle speaeol
1,00
de
Figure 5: The self- correlattl between 13 types of uncertainties in terms of robustness and resilience. Correlations in uncertainties differ significantly across modalities (observations, actions, environ- ments) and scope of agents (applied to individual or all agents).
resilience seem correlated, such correlation is small under large uncertainties, indicating that these two properties behave independently and should not be treated interchangeably.
2. Linearity across uncertainty types, agent scopes, and attack strategies. A closer look reveals strong correlations under random, greedy, and optimal attacks, as well as under observation uncertainty and perturbations affecting either single or all agents (r 2 0.6, p < .05). In the case of environment uncertainty, correlations remain strong (r 2 0.85) but lack statistical significance. Under action uncertainty, MADDPG exhibits greater robustness than MAPPO and HAPPÓ, resulting in weak overall correlations. When MADDPG is excluded, MAPPO and HAPPO show strong linearity with robustness under action uncertainty (r 2 0.69. p < .05). See full results in Appendix. B.3
3. Algorithms exhibit similar overall robustness and resilience, but differ in sensitivity to uncer- tainty types. While no significant differences are observed in overall robustness and resilience across algorithms, we find that MADDPG is more resilient to action uncertainties, whereas MAPPO and HAPPO perform better under observation uncertainties. This discrepancy stems from their training methods: MAPPO and HAPPO leverage centralized critic, which improves sample efficiency and robustness against noisy observations. While these on-policy methods experience high randomness for max-entropp learning at the beginning of training, the policy becomes more deterministic as policy converge, making it still non-robust against action noise. As for MADDPG, the exploration noise is maintained throughout training. As training progresses, MADDPG continues to inject noise in the action space, which contributes to its greater robustness to action perturbations, especially in late-stage policies that retain some level of stochastic behavior.
Takeaway. Cooperation improves robustness and resilience under mild uncertainty, but this corre- lation weakens as attack severity increases. The phenomenon holds across most uncertainty types, agent scopes, and attack strategies. MADDPG is preferable for action uncertainties, while MAPPO and HAPPO are better suited for observation uncertainties.
5.2 Uncertainty Diversity Matters
We next examine the relationships among 13 types of uncertaintics to determine whether robustness against a single uncertainty type generalize to all other types. To achieve this, we compute the Pearson correlation between all pairs of uncertainties, conditioned on robustness and resilience. The key findings are summarized below:
Observation, environment, and action uncertainties are uncorrelated. As shown in Fig. 5 compared to the 6 subtypes of uncertainties within observation and action categories, the three modalities of lations. A one-way ANOVA confirms this effect is statistically significant for robustness 153- = 9.53, p- < .001) and resilience (F(2, 153)= = 14.51, p < .001). This indicates that robustness against one modality does not imply robustness against other modalities. Such phenomenon is observed in
environment, and action-exhibit significantly lower corre-
7

---

Figure 6: Percentage change in cooperation, robustness, and resilience caused by varying hyperpa- rameters. 5% winsorization is applied to limit extreme value within two standard deviations.
EIR-MAPPO 40 where policies trained to be robust against action uncertainties remain vulnerable
to observation uncertainties.
2. Uncertainties applied to individual agents and all agents are uncorrelated. Similarly, we find that the correlations between uncertainties applied to individual agents and those applied to all agents are significantly different. A one-way ANOVA confirms this effect for robustness (F(1, 142) = 4.36, p .05) and resilience (F(1, 142) = 7.00, .05). This explains the issue encountered in M3DDPG [4], where agents trained to be robust against small uncertainties applied uniformly across all agents fail when exposed to large uncertainties applied to individual agents.
Takeaway. Robustness and resilience MARI. can not generalize across uncertainty modalities or agent scopes. Trustworthy MARL systems must therefore evaluate against diverse types of uncertainty, and account for both individual and group-level perturbations.
5.3 What hyperparameters are effective?
In this section, we examine the impact of hyperparameters on cooperation, robustness, and resilience. This is motivated by our preliminary study, where two-way ANOVA shows hyperparameters have stronger effect than MARL algorithms in 9 of 18 tasks (p < 0.001), highlighting the need to quantify their influence on trustworthy MARL. Due to space constraints, detailed results are presented in Appendix. B.4 To identify broadly effective hyperparameters, we assign equal weight to each task and apply 5% winsorization to limit extreme values within two standard deviations. Figure. [o)pootss the percentage change in cooperation, robustness, and resilience due to change of hyperparameters. The results are summarized as follows.
1. Early stopping is generally effective. During MARL training, after the convergence of cooperative performance, robustness and resilience may continue to evolve. As shown in Fig. 7 we implement an early stopping criterion that saves models based on their combined performance in cooperation, robustness, and resilience. This ensures the saved model performs no worse than the model the final training round. In contrast, existing codebases 42 3 save model in the final timestep, which is significantly worse than our early stopping criterion (t(161) 6. 6.16,p < 0001, paired t-test).
2. Use a higher learning rate (LR) for the critic. Using learning rate for the critic higher than the actor often improves coopera- tion, robustness, and resilience simultaneously (1(161) = 10.02, p < .001, paired t-test). This aligns with theoretical studies on the con- vergence of actor-critic algorithms using two-
Visualization of Eady Stop
140
Early Stop
100
Eytood Rewar
40
timescale stochastic optimization [6] 62 63
Cooperation Robustness
Stable convergence is achieved where a faster LR for the critic ensures it is essentially equi- librated, while the actor's policy updates at slower rate and is quasi-static. Despite this well-established theoretical insight, most MARI. codebases set equal LRs for the actor and the critic [42 B..
Resilince
0.0
0.2
0.4
0.6
6.5
1.0
Tincskop
Figure 7: The effect of early stop. While coop- eration performance converge. robustness and re- silience continue to evolve. We use early stop to select the best model.
8

---

3. Use Leaky ReLU for activation. Leaky
ReLU consistently outperforms ReLU in cooperation, robustness, and resilience (f(161) 6.31. .001, paired t-test). For cooperation, Leaky ReI.U avoids the dead neuron problem inherent in ReI.U, enabling better representation learning. For robustness and resilience, ReL.Us dead neurons can become unexpectedly activated under distribution shifts caused by uncertainties, introducing errors into the network. In contrast, Leaky ReLU maintains stable gradients, preventing such issues.
4. GAE and PopArt do not improve performance. Despite being standard in modern RL and MARL, GAE (1() 7.44, p < .001) and PopArt (t(107) 4.84. p < .001) reduce cooperation, robustness, and resilience in our benchmark (paired t-test). To verify this counterintuitive finding, we conduct additional experiments on several widely used MARL benchmarks (Appendix [B.5]. confirming that while effective in Multi-Agent MuJoCo. these methods have limited impact in MPE and SMAC. We hypothesize that GAF performs well in MuJoCo due to dense and stable reward structure, whereas in real-world environments, sparse and highly variable rewards amplify bootstrapping errors for GAE See Appendix. [B.6 an example. As for PopArt, its normalization benefits may be unnecessary when reward scales remain stable, leading to inconsistent effectiveness across different tasks.
5. Parameter sharing does not improve overall performance. Although parameter sharing is highlighted by MAPPO as a key technique for its success [3), our findings show that it often reduces cooperation, robustness, and resilience (t(161) 7.01, < .001, paired t-test). Further evaluation on MPE, SMAC and Multi-Agent Mujoco reveals that improvements from parameter sharing are limited to SMAC tasks (see Appendix B.5 for details). While parameter sharing improves sample efficiency, it hinders the learning of agent-specific features, particularly in environments like Voltage and Traffic, where heterogeneous nodes require distinct strategies. Additionally, parameter sharing assumes that all agents follow the same policy, making the shared policy more vulnerable to attacks when other agents are impacted by uncertainties. Our claim can be reinforced by HATRPO [4) where construct a negative example showing parameter sharing brings exponentially worse performance.
6. Exploration benefits MAPPO and HAPPO. Increasing exploration improves the cooperation, robustness and resilience of PPO-based methods (t(107) = 6.64. p < ,001, paired t-test). The entropy bonus in these methods naturally arises as a consequence of the optimal solution in maximum entropy RI. (64, and provably enhances robustness against environmental uncertainties 65 Greater exploration also increases state space coverage, which enhances resilience. Our results confirm these benefits in MARL. For MADDPG, however, does not enjoy such theoretical advantages and the impact of increased exploration is task-dependent. These differences reflect distinct exploration schemes. In MAPPO/HAPPO, entropy regularization acts as a soft constraint, which encourage exploration at the beginning, while the policy can still converge to a relatively deterministic policy to reach optimal cooperation. In contrast, MADDPG relies on additive action noise that typically persists throughout training. With large exploratory noise, well-trained policy can be perturbed to low-value or unstable regions, degrading policy quality.
7. Discussions on task-specific choices. We do not find universal preference for learning rates, hidden state sizes, discount factors, use RNN, or network initialization methods. However, these choices may significantly impact performance and should be tuned based on the task applied.
Takeaways. We recommend following our guidance when applying MARL to new environments with uncertainties. Notably, several findings challenge established assumptions in RL/MARL literature, including the use of parameter sharing, GAE, and PopArt. These discrepancies likely stem from the narrow task and environment focus of previous studies, highlighting the task-dependent nature of MARL performance.
5.4 Improving Robustness and Resilience
In this section, we demonstrate that robustness and resilience can be significantly improved through hyperparameters alone. For each task, we identify the best set of hyperparameters that maximize the combined performance of cooperation, robustness, and resilience. Again, we apply a 5% win- sorization to suppress outliers. As shown in Fig. B, optimizing hyperparameters during training leads to substantial performance improvements, with increases of 52.60% cooperation, 34.78% in robustness, and 60.34% in resilience.

---

Bos Implemoraiiee Choeecs in Dethand
Bes implementtiion Chozees is Indiic
200%
Roe implenemmicc Chicco in Velagg
50%
Bed Implerettaion Choico Quad
100%
200%
150%
10%%
100%
2%
-20
SEXPPO
RAPPD
MARINN
MAPPO
RAPPO
S4,000
RARPO
MADDIG
MARO
RAPRO
Figure 8: Percentage improvement for each environments using the best tricks for each task. Cooper- ation, robustmess and resilience be obtained altogether by simply tuning hyperparameters.
tellgge
Baclllnn- - ERNIE Defans
Badibinn- - ERNIE Delane
Backbote- ERNIE Delane
Buckbosz ERNIE Defense
.00%
158%
30%
300%
100%
18%
100%
58%
0%
-00%
-00%
SUADDRG
MAPRO
IAAPP
MANERG
MARDO
-0%%
MADOPG
MA00
MA200
-00%
XADBBG
HAPPO
Figure 9: Generalization of the effect of hyperparameter tuning. We report percentage improvement
on ERNIE defense with diverse backbones and environments.
To show the generality of hyperparameter tuning, we train ERNIE 666, general purpose robust MARL method on all backbones, using default hyperparameters and our selected best hyperpararme- ters on non-robust backbones above. As shown in Fig. 9 the improvement of hyperparameter generalize to robust MARL methods under various backbones and environments, with an average improvement of 89.43% on cooperation, 65.83% robustness, 82.96% resilience.
general-purpose
Post-hoc analysis of the interactions among hyperparameters was conducted using ordinary least squares regression, statistical method that quantifies the significance of each factor's contribution. Results show it beneficial use parameter sharing for homogeneous agents (p .05), smaller discount factors for short episodes (p < .001), higher critic LR in policy gradient methods (p < .05) and always use early stop (p < .001). See full results in Appendix, B.7
6 Conclusions
In this paper, we conducted a large-scale empirical study on the robustness and resilience of MARL, yielding several meaningful insights: (1) Optimizing for cooperation improves robustness and resilience under mild uncertainty, but this relationship weakens as perturbations intensify, with performance differing across algorithms and uncertainty types. (2) Robustness and resilience do not transfer across uncertainty modalities or agent scopes; a policy robust to action noise affecting all agents may still fail under observation noise targeting single agent. (3) Hyperparameter tuning plays a central role trustworthy MARL. Surprisingly, widely used practices like parameter sharing, GAE, and PopArt can degrade robustness, while strategies such as early stopping, higher critic learning rates, and Leaky ReLUU offer consistent improvements. By optimizing hyperparameters only, observe substantial improvement in cooperation, robustness and resilience across all MARL backbones, with the phenomenon also generalizing to robust MARL methods across these backbones. A limitation of our paper is the focus of MARL algorithms based policy gradient, since many environments requires continuous control. This can be mitigated by integrating new algorithms into codebase, which supports custom environments and algorithm integration.
Acknowledgements
This work was supported by National Natural Science Foundation of China (62476018).
10

---

References
[1] Ryan Lowe, Yi Wu, Aviv lamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperatie-compelttiie environments. Advances in neural information processing systems, 30, 2017.
[2] Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. In International conference on machine learning, pages 4295-4304. PMLR, 2018.
[3] Chao Yu, Akash Velu, Eugene Vinitsky, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo in cooperative, multi-agent games. arXiv preprint S.Xix:2103.09955 2021.
[4] Jakub Grudzien Kuba, Ruiqing Chen, Muning Wen, Ying Wen, Fanglei Sun, Jun Wang. and Yaodong Yang. Trust region policy optimisation in multi-agen reinforcement learning, arXiv preprint arXiv:2109. 112.5 2021.
[5] Rongye Shi, Xin Yu, Yandong Wang. Yongkai Tian, Zhenyu Liu, Wenjun Wu, Xiao-Ping Zhang, and Manuela M Veloso. Symmetry-informed marl: A decentralized and cooperative uav swarm control approach for communication coverage. IEEE Trunsactions on Mobile Computing, 2025.
[6] Xin Yu, Rongye Shi, Pu Feng, Yongkai Tian, Simin Li, Shuhao Liao, and Wenjun Wu. Lever- aging partial symmetry for multi-agent reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 17583-17590, 2024.
[7] Xin Yu, Yongkai Tian, Li Wang, Pu Feng, Wenjun Wu, and Rongye Shi. Adaptaug: Adap- tive data augmentation framework for multi-agen reinforcement learning. In 2024 IEEE International Conference Roboties and Automation (ICRA), pages 10814-10820. IEEE, 2024.
[8] Pu Feng, Junkang Liang, Size Wang. Xin Yu, Xin Ji, Yiting Chen, Kui Zhang, Rongye Shi, and Wenjun Wu, Hierarchical consensus-based multi-agent reinforcement learning for multi-robot cooperation tasks. In 2024 IEEE/RS/ International Conference Intelligent Robots and Systems (IROS), pages 642-649. IEEE, 2024.
[9] Pu Feng, Rongye Shi, Size Wang. Qizhen Wu, Xin Yu, and Wenjun Wu. Lyapunov-informed multi-agent reinforcement learning for multi-robot cooperation tasks. IEEE Transactions on Automation Science and Engineering, 22:20263-20279, 2025.
[10] Pu Feng, Size Wang, Yuhong Cao, Junkang Liang, Rongye Shi, and Wenjun Wu. Neural algorithmic reasoners informed large language model for multi-agent path finding. arXiv preprint 250 2025.
[11] Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Jun- young Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level starcraft using multi-agent reinforcement learning. Nature, 575(7782):350-354 2019.
[12] H Jane Bae and Petros Koumoutsakos. Scientific multi reinforcement learning for wall- models of turbulent flows. Nature Communications, 13(1):1443, 2022.
[13] Chengdong Ma, Aming Li, Yali Du, Hao Dong, and Yaodong Yang. Efficient and scalable reinforcement learning for large-scale scale network control. Nature Machine Intelligence, pages 1-15, 2024.
[14] Jieyu Lin, Kristina Dzeparoska, Sai Qian Zhang, Alberto Leon-Garcia, and Nicolas Papernot. On the robustness of cooperative multi-agent reinforcement learning. In 2020 IEEE Security and Privacy Workshops (SPW), pages 62-68, 2020.
[15] Sihong He, Songyang Han, Sanbao Su, Shuo Han, Shaofeng Zou, and Fei Miao. Robust multi-agent reinforcement learning with state uncertainty. arXiv preprint 220.12230..16212. 2023.
11

---

[16] Adam Gleave, Michael Dennis, Cody Wild, Neel Kant, Sergey Levine, and Stuart Russell. Adversarial policies: Attacking deep reinforcement learning, arXiv preprint 1XXi:1905.11615. 2019.
[17] Simin Li, Jun Guo, Jingqiao Xiu, Ruixiao Xu, Xin Yu, Jiakai Wang, Aishan Liu, Yaodong Yang, and Xianglong Liu. Byzantine robust cooperative multi-agent reinforcement learning as bayesian game. arXiv preprint arXiv:2305. /2872, 2023.
[18] Annie Xie, Shagun Sodhani, Chelsea Finn, Joelle Pineau, and Amy Zhang. Robust policy learning over multiple uncertainty sets. International Conference on Machine Learning, pages 24414-24429, PMLR, 2022.
[19] Laixi Shi, Eric Mazumdar, Yuejie Chi, and Adam Wierman. Sample-efficient robust multi- agent reinforeement learning in the face of environmental uncertainty. arXiv preprint arXiv:2404.18909 2024.
[20] Gunnar W Klau and René Weiskircher. Robustness and resilience. In Network analysis: Methodological foundations, pages 417-437. Springer, 2005.
[21] Quanyan Zhu and Tamer Başar. Robust and resilient control design for cyber-physical systems with an application to power systems. In 2011 50th IEEE Conference on Decision and Control and European Control Conference, pages 4066 4071, 2011.
[22] Crawford S Holling et al. Resilience and stability of ecological systems. 1973.
[23] Paolo Di Caro and Ugo Fratesi. Regional determinants of economic resilience. The Annals of Regional Science, 60:235-244 2018.
[24] Vahid Behzadan and William Hsu. RI-based method for benchmarking the adversarial resilience and robustness of deep reinforcement learning policies. In Computer Safery, Reliability, and Security: SAFECOMP 2019 Workshops, pages 314-325. Springer, 2019.
[25] Thomy Phan, Thomas Gabor, Andreas Sedlmeier, Fabian Ritz, Bernhard Kempter, Cornel Klein, Horst Sauer, Reiner Schmid, Jan Wieghardt, Marc Zeller, et al. Learning and testing resilience in cooperative multi-agent systems. In Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems, pages 1055-1063, 2020.
[26] Lanting Zeng, Dawei Qiu, and Mingyang Sun. Resilience enhancement of multi-agent reinforce- ment learning-basee demand response against adversarial attacks. Applied Energy, 324:1 19688, 2022.
[27] Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. Implementation matters in deep policy gradients: A case study on ppo and trpo. arXiv preprint 20XX::20055.2299 2020.
[28] Inaam Ilahi, Muhammad Usama, Junaid Qadir, Muhammad Umar Janjua, Ala Al-Fuqaha, Dinh Thai Hoang, and Dusit Niyato. Challenges and countermeasures for adversarial attacks on deep reinforcement learning. IEEE Transactions on Artificial Intelligence, 3(2):90-109, 2021.
[29] Sandy Huang, Nicolas Papernot, Ian Goodfellow, Yan Duan, and Pieter Abbeel. Adversarial attacks on neural network policies. arXiv preprint arXiv:1702.02288. 2017.
[30] Huan Zhang, Hongge Chen, Chaowei Xiao, Bo Li, Mingyan Liu, Duane Boning, and Cho- Jui Hsieh. Robust deep reinforcement learning against adversarial perturbations state observations. Advances in Neural Information Processing Systems, 13:21024-21037, 2020.
[31] Huan Zhang, Hongge Chen, Duane Boning, and Cho-Jui Hsieh. Robust reinforcement learning on state observations with learned optimal adversary. arXiv preprint
2021.
[32] Chen Tessler, Yonathan Efroni, and Shie Mannor. Action robust reinforcement learning and applications in continuous control. In International Conference on Machine Learning, pages 6215-6224, 2019.
12

---

[33] Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforcement learning. In International Conference on Machine Learning, pages 2817-2826.
PMLR, 2017.
[34] Shihui Li, Yi Wu, Xinyue Cui, Honghua Dong, Fei Fang, and Stuart Russell. Robust multi agent reinforcement learning via minimax deep deterministic policy gradient. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 4213-4220, 2019.
[35] Chuangchuang Sun, Dong-Ki Ki Kim, and Jonathan P How. Romax: Certifiably robust deep multiagent reinforcement learning via convex relaxation. In 2022 International Conference on Robotics and Automation (ICRA), pages 5503-5510, 2022.
[36] Eleni Nisioti, Daan Bloembergen, and Michael Kaisers. Robust multi-agent q-learning in cooperative games with adversaries. In Proceedings of the AAAI Conference on Artificial Intelligence, 2021.
[37] Lei Yuan, Ziqian Zhang, Ke Xue, Hao Yin, Feng Chen, Cong Guan, Lihe Li, Chao Qian, and Yang Yu. Robust multi-agent coordination via evolutionary generation of auxiliary adversarial attackers. In Proceedings of the AAAI Conference Artificial Intelligence, volume 37, pages 11753-11762, 2023.
[38] Kaiqing Zhang, Tao Sun, Yunzhe Tao, Sahika Genc, Sunil Mallya, and Tamer Basar. Robust multi-agent reinforcement learning with model uncertainty. Advances in Neural Information Processing Systems,
2020.
[39] Thomy Phan, Lenz Belzner, Thomas Gabor, Andreas Sedlmeier, Fabian Ritz, and Claudia Linnhoff-Popier Resilient multi-agent reinforcement learning with adversarial value decom- position. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pages 11308-12 1131,,,,,,1111 2021.
[40] Simin Li, Jun Guo, Jingqiao Xiu, Ruixiao Xu, Xin Yu, Jiakai Wang, Aishan Liu, Yaodong Yang, and Xianglong Liu. Byzantine robust cooperative multi-agent reinforcement learning as a bayesian game. arXiv preprint arXiv:2305. 2023.
[41] Marein Andrychowicz, Anton Raichuk, Piotr Stañczyk, Manu Orsini, Sertan Girgin, Raphael Marinier, Léonard Hussenot, Matthieu Geist, Olivier Pietquin, Marcin Michalski, et al. What matters in on-policy reinforcement learning? large-scale empirical study. arXiv preprint arXiv:2006.05999, 2020.
[42] Georgios Papoudakis, Filippos Christianos, Lukas Schäfer, and Stefano V Albrecht. Benchmark- ing multi-agent deep reinforcement learning algorithms in cooperative tasks. arXiv preprint 0777722006.07869, 2020.
[43] Jian Hu, Siyang Jiang, Seth Austin Harding, Haibin Wu, and Shih wei Liao. Rethinking the implementation tricks and monotonicity constraint in cooperative multi-agent reinforcement learning. arXiv, 2021.
[44] Jun Guo, Yonghong Chen, Yihang Hao, Zixin Yin, Yin Yu, and Simin Li. Towards comprehen- sive testing on the robustness of cooperative multi-agent reinforcement learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 115-122, 2022.
[45] Adil Zouitine, David Bertoin, Pierre Clavier, Matthieu Geist, and Emmanuel Rachelson. Rris: Robust reinforcement learning suite. arXiv preprint arXiv:2406. 08406, 2024.
[46] Shangding Gu, Laixi Shi, Muning Wen, Ming Jin, Eric Mazumdar, Yuejie Chi, Adam Wierman, and Costas Spanos. Robust gymnasium: A unified modular benchmark for robust reinforcement learning. Github, 2024.
[47] Frans Oliehoek and Christopher Amato. A Concise Introduction to Decentralized POMDPs. Springer, 2016.
[48] Da-Wei Gu, Petko Petkov, and Mihail M Konstantinov. Robust control design with MATLAB@. Springer Science & Business Media, 2005.
13

---

[49] Giliberto Capano and Jun Jie Woo. Resilience and robustness in policy design: A critical appraisal. Policy Sciences, 50(3):399-4266 2017.
[50] Quanyan Zhu and Tamer Başar. Disentangling resilience from robustness: Contextual dualism, interactionism, and game-theoretic paradigms. IEEE Control Systems Magazine, 44(3):95-103, 2024.
[51] Oriol Artime, Marco Grassia, Manlio De Domenico, James P Gleeson, Hernán A Makse, Giuseppe Mangioni, Matjaž Perc, and Filippo Radicchi. Robustness and resilience of complex networks. Nature Reviews Physics, (2):114-131,, 2024.
[52] Marlos C. Machado, Marc G. Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the arcade learning environment: evaluation protocols and open problems for general agents. J. Artif. Int. Res., 61(1))523-562, 2018.
[53] Yuanpei Chen, Tianhao Wu, Shengjie Wang. Xidong Feng, Jiechuan Jiang, Zongqing Lu, Stephen McAleer, Hao Dong, Song-Chun Zhu, and Yaodong Yang. Towards human-level bimanual dexterous manipulation with reinforcement learning. Advances in Neural Information Processing Systems, 35:5150-5163, 2022.
[54] Sumeet Batra, Zhehui Huang, Alcksei Petrenko, Tushar Kumar, Artem Molchanov, and Gaurav Sukhatme. Decentralized control of quadrotor swarms with end-to-end deep reinforcement learning. In Conference on Robot Learning, pages 576-586. PMLR, 2022.
[55] Tianshu Chu, Sandeep Chinchali, and Sachin Katti. Multi-agent reinforcement learning for networked system control. arXiv preprint arXiv:2004.01339, 2020.
[56] Jianhong Wang, Wangkun Xu, Yunjie Gu, Wenbin Song, and Tim C Green. Multi-agent reinforcement learning for active voltage control on power distribution networks. Advances in Neural Information Processing Systems, 34:3271 14:3271-32885 -284,, 2021.
[57] Esther Derman, Daniel J Mankowitz, Timothy A Mann, and Shie Mannor. Soft-robust actor- critic policy-gradient. arXiv preprint arXiv: 183.:0488, 2018
[58] Daniel Mankowitz, Nir Levine, Rae Jeong, Yuanyuan Shi, Jackie Kay, Abbas Abdolmaleki, Jost Tobias Springenberg, Timothy Mann, Todd Hester, and Martin Riedmiller. Robust re- inforcement learning for continuous control with model misspecification. arXiv preprint 1rXiv:190.007566 2019.
[59] Esther Derman, Daniel Mankowitz, Timothy Mann, and Shie Mannor. bayesian approach to robust reinforcement learning. In Uncertainty in Artificial Intelligence, pages (48-658. PMLR, 2020.
[60] Annie Xie, Shagun Sodhani, Chelsea Finn, Joelle Pineau, and Amy Zhang. Robust policy learning over multiple uncertainty sets. In International Conference on Machine Learning, pages 24414-24429, PMLR, 2022.
[61] Vivek S Borkar. Stochastic Approximation: A Dynamical Systems Viewpoint, volume 48. Springer, 2009.
[62] Yue Frank Wu, Weitong Zhang, Pan Xu, and Quanquan Gu. A finite-time analysis of two time-scale actor-critic methods. Advances in Neural Information Processing Systems, 33:17617- 17628, 2020.
[63] Markus Holzleitner, Lukas Gruber, José Arjona-Medina, Johannes Brandstetter, and Sepp Hochreiter. Convergence proof for actor-critic methods applied to ppo and rudder. In Trans- actions on Large-Scale Data-and Knowledge-Centered Systems, pages 105-130. Springer, 2021.
[64] Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review. arXiv preprint a 100000000..0000 2018.
[65] Benjamin Eysenbach and Sergey Levine. Maximum entropy rl (provably) solves some robust problems. arXiv preprint arXiv:2103.06257, 2021.
14

---

[66] Alexander Bukharin, Yan Li, Yue Yu, Qingru Zhang, Zhehui Chen, Simiao Zuo, Chao Zhang. Songan Zhang, and Tuo Zhao. Robust multi-agent reinforcement learning via adversarial regularization: Theoretical foundation and stable algorithms. Advances in Neural Information Processing Systems, 66:68121-6818 2023.
[67] Yifan Zhong. Jakub Grudzien Kuba, Xidong Feng, Siyi Hu, Jiaming Ji, and Yaodong Yang. Heterogeneouus-agenn reinforcement learning. Journal of Machine Learning Research, 25(32):1- 67, 2024.
[68] Mesut E Baran and Felix F Wu. Network reconfiguration in distribution systems for loss reduction and load balancing. IEEE Transactions on Power delivery, 4(2):1401-149)) 1989.
[69] HM Khodr, FG Olsina, PM De Oliveira-De Jesus, and JM Yusta. Maximum savings approach for location and sizing of capacitors in distribution systems. Electric Power Systems Research, 190)::1199-223,, 2008
[70] Aleksander Madry. Towards deep learning models resistant to adversarial attacks, arXiv preprint arXiv:1706.06083 2017.
[71] Anay Pattanaik, Zhenyi Tang, Shuijing Liu, Gautham Bommannan, and Girish Chowdhary. Robust deep reinforcement learning with adversarial attacks. arXiv preprint arXiv: 1712.03632, 2017.
[72] Yen-Chen Lin, Zhang-War Wei Hong, Yuan-Hong Liao, Meng-Li Shih, Ming- Yu Liu, and Min Sun. Tactics of adversarial attack on deep reinforcement learning agents. arXiv preprint 17Xiv:1703.06748, 2017.
[73] Jernej Kos and Dawn Song. Delving into adversarial attacks on deep policies. arXiv preprint arXiv: /705v 06450 2017.
[74] Huan Zhang, Hongge Chen, Duane Boning, and Cho-Jui Hsieh. Robust reinforcement learning on state observations with learned optimal adversary. arXiv preprint FrXXx:2101.0845: 2021.
[75] Yanchao Sun, Ruijic Zheng, Yongyuan Liang, and Furong Huang. Who is the strongest enemy? towards optimal and efficient evasion attacks in deep rl. arXiv preprint arXiv:2106.00 05087, 2021.
[76] Simin Li, Jun Guo, Jingqiao Xiu, Yuwei Zheng, Pu Feng, Xin Yu, Aishan Liu, Yaodong Yang, Bo An, Wenjun Wu, et al. Attacking cooperative multi-agent reinforcement learning by adversarial minority influence. arXiv preprint arXiv:2302.0333 033222 2023.
[77] Arna Nilim and Laurent El Ghaoui. Robust control of markov decision processes with uncertain transition matrices. Operations Research, 53(5):780-798, 2005.
[78] Garud N Iyengar. Robust dynamic programming. Mathematics of Operations Research, 30(2):257-280, 2005.
[79] Jiaming Ji, Mickel Liu, Josef Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. Beavertails: Towards improved safety alignment of Ilm via a human- preferrnce dataset. Advances in Neural Information Processing Systems, 36:24678 -247444 2023.
[80] Jiaming Ji, Boyuan Chen, Hantao Lou, Donghai Hong, Borong Zhang, Xuehai Pan, Tianyi Alex Qiu, Juntao Dai, and Yaodong Yang. Aligner: Efficient alignment by learning correct. Advances in Neural Information Processing Systems, 37:90853-90899 2024.
[81] Jiaming Ji, Kaile Wang. Tianyi Qiu, Boyuan Chen, Jiayi Zhou, Changye Li, Hantao Lou, Juntao Dai, Yunhuai Liu, and Yaodong Yang. Language models resist alignment: Evidence from data compression. arXiv preprint arXiv:2406.061 06144, 2024.
[82] Anna Kochan. Shadow delivers first hand. Industrial Robot: An International Journal, 32(1):15-
16, 2005.
[83] Julian Förster. System identification of the crazyflie 2.0 nano quadrocopter. B.S. thesis, ETH Zurich, 2015.
15

---

[84] Mikayel Samvelyan, Tabish Rashid, Christian Schroeder De Witt, Gregory Farquhar, Nan- Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip HS Torr, Jakob Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. arXiv preprini arXiv: 1902.04043, 2019.
[85] Benjamin Ellis, Jonathan Cook, Skander Moalla, Mikayel Samvelyan, Mingfei Sun, Anuj Mahajan, Jakob Foerster, and Shimon Whiteson. Smacv2: An improved benchmark for cooperative multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
[86] Bei Peng, Tabish Rashid, Christian Schroeder de Wit, Pierre-Alexandre Kamienny, Philip Torr, Wendelin Böhmer, and Shimon Whiteson. Faemac: Factored multi-agent centralised policy gradients. Advances in Neural Information Procexsing Systems, 34: 12088-12222, 2021.
[87] Karol Kurach, Anton Raichuk, Piotr Stañczyk, Michal Zajkac, Olivier Bachem, Lasse Espeholt, Carlos Riquelme, Damien Vincent, Marein Michalski, Olivier Bousquet, al. Google research football: A novel reinforcement learning environment. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 4501-4510 2020.
16

---

APPENDIX FOR "EmPiRiCal STUDY ON ROBUSTNESS AND RESILIENCE IN COOPERATIVE MULTI-AGENT RE- INFORCEMENT LEARNING"
A Appendix for Experiment details
A.1
Additional Details on hyperparameters
In this section, we first discuss the key hyperparameters that significantly affect the performance of the MARL algorithms evaluated in our benchmark. Next, we provide an overview of all hyperparameter settings used across the different algorithms and highlight configurations specific each algorithm.
A.1.1 Introduction on hyperparameters
The following subsections provide brief description of the critical hyperparameters and settings that impact the implementation of the MARL algorithms. As no universal set of hyperparameters exists for multi-agent reinforcement learning (MARL) algorithms, our selections are informed by established practices in reinforcement learning, with particular reference to the codebases of MAPPO 3 and HAPPO [67]. These codebases provide carefully designed hyperparameter configurations that perform well across environments such as MPE, SMAC, and MAMujoco, though the optimal settings vary by task. Building on this, we conducted a pilot study to identify a subset of hyperparameters that consistently work well across most of our real-world tasks, while adopting alternative configurations for the remaining parameters as needed.
1. Network Hidden Sizes: This refers to the size of the actor networks and critic networks used in the MARL algorithms, which determines the number of neural units per layer. The choices are [64, 128, 256), with the default value being 128.
2. Discount Factor: The standard parameter in RL, which controls the importance of future rewards during training, with a choice set (0.9, 0.95, 0.99). The default value is 0.99.
3. Activation Function: Various activation functions in neural networks, including ReLU, Leaky ReLU, SELU, Sigmoid, and Tanh. The default function is ReLU.
4. Initialization Method: The initialization of network weights is conducted using either
Orthogonal or Xavier initialization. Orthogonal initialization is the default choice.
5. Neural Network Type: This parameter denotes the network architecture of actor and critic in MARL algorithms, with options including MLP (Multi-Layer Perceptron) and RNN (Recurrent Neural Network). The default architecture is MLP.
6. Learning Rate; The learning rate for training the actor network is selected from the set (5e-5, 5e-4, 5e-3), with default value of 5e-4.
7. Critic Learning Rate: The critic network learning rate chosen from the same set as the actor network, 55e-5, 5e-4, 5e-3), 3,, with 5e-4 as the default value.
8. Feature Normalization: This applies Layer Normalization to standardize the features of each sample, ensuring they have a mean of 0 and a variance of 1. This helps stabilize training and improve convergence. Feature normalization is enabled by default.
9. Share Parameters:
This parameter specifies whether multiple agents share the same neural network parameters for their policy functions. The default setting is False. Note that HAPPO inherently assumes heterogeneous agents, and do not support parameter sharing.
10. TD Steps (MADDPG): TD steps determine the number of steps used Temporal Difference (TD) learning for updating the value estimates. The choices available are (5, 20, 25), with default value of 20.
11. Exploratory Noise (MADDPG): This parameter controls the scale of noise added to the actions generated by the actor network, order to encourage exploration during training. The options available are (0.001, 0.01, O. 0.5, 1), with a default value of 0.
17

---

12. Entropy Coefficient (MAPPO/HAPPO): Controls the entropy regularization strength in the MAPPO/HAPPO objective, which affects exploration. The selection is f0.0001, 0.001, 0.01, 0.1). with default value of 0.01.
13. Use GAE HTAPPO/HAPPO::Whhthee Generalized Advantage Estimation (GAE) is used for estimating the advantage of state-action pair. The default is True.
14. Use PopArt (MAPPO/HAPP)): PopArt is normalization technique used to rescale value targets dynamically to improve training stability. The default choice is True.
We use early stop as a model selection criterion. each evaluation period, we evaluate the cooperation, robustness and resilience of algorithms, and save the model with maximum cooperation + robustness + resilience. Baseline model save the model their last round of evaluation. The default choice is False.
Early Stop:
A.1.2 Other Default Hyperparameters
The following table summarizes the default hyperparameters used across the various MARL, algo- rithms. We first describe the general hyperparameters that are common across all algorithms, followed by algorithm-specifie parameters.
Table 4: Hyperparameters MARL algorithms.
Number
Hyperparameter
MAPPO
НАPPO
MADDPG
General Hyperparameters
Neural network type
MLP
MLP
MLP
2
Network hidden sizes
[128, 128]
[128, 128]
[128, 128]
3
Activation function
ReLU
ReLU
ReLU
4
Discount factor (n)
0.99
0.99
0.99
5
Actor learning rate
0.0005
0.0005
0.0005
6
Critic learning rate
0.0005
0.0005
0.0005
7
Feature normalization
True
True
True
8
Share parameters
False
False Orthogonal
False
Weight initialization method Recurrent layers (if use RNN)
Orthogonal
Orthogonal
10
11
Use max gradient norm
True
True
True
12
Max gradient norm
10.0
10.0
10.0
13
Linear learning rate decay
False
False
False
MAPPO and HAPPO Shared Hyperparameters
14
Entropy coefficient
0.01
0.01
N/A
15
Use PopArt
True
True
N/A
16
Use GAE
True
True
N/A
17
GAE parameter (A)
0.95
0.95
N/A
18
PPO epochs
5
5
N/A
19
Critic epochs
5
5
N/A
20
Clip parameter
0.05
0.1
N/A
21
Use clipped value loss
True
True
N/A
22
Value loss coefficient
1.0
1.0
N/A
Product of probabilities True
Product of probabilities True
23
Action aggregation
N/A
24
Use huber loss Huber delta
N/A
25
10.0
10.0
N/A
Algorithm-Specific Hyperparameters
26
Exploration noise
N/A
N/A
0.1
27
TD steps
N/A
N/A
20
28
Buffer size
N/A
N/A
5000
29
Batch size
N/A
N/A
1000
30
Polyak averaging factor
N/A
N/A
0.005
Continued on next page
18

---

Table 4: Hyperparameters for all MARL algorithms.
Number
Hyperparameter
MAPPO
HАPPO
MADDPG
31
Warmup steps
N/A
N/A
50000
32
Update per training step
N/A
N/A
33
Final activation function
N/A
N/A
Tanh
A.2
Additional Details on Environments
Our benchmark study is grounded on four real-world environments, as illustrated in Fig. 10 The first two, Dexterous Hand Manipulation (Dexhand) [53] and Quadrotor Swarm Control (Quad) [4 are rooted in real-world robotics, enabling seamless policy transfer from simulation to physical systems. In dexterous hand control and quadrotor swarm control, the MARL algorithm needs to control the low-level actions made by robot, where small variation might leads to large change in robot actions, requiring robust control in precise manipulations. Each environment comprises six tasks. The other two environments, Intelligent Traffic Control (Traffic) (55] and Active Voltage Control (Voltage) (56), are based on real-world data, are derived from real-world data, providing high-fidelity simulations of complex systems. In intelligent traffic control and active voltage control, each agents in MARL algorithm are highly responsible for the performance of the overall system, requiring adaptive control with uncertainties. These environments feature four tasks and three tasks, respectively, resulting in total of 19 tasks across all environments (6 for Dexhand + for Quad + for Traffic + for Voltage 18 tasks). The following subsections offer more detailed description of the environments and tasks included in our benchmark.
(a) Dexterous Hand Manipulation
(b) Quadrotor Swarm Control
Repalvalent E Elettri Chreat
Local Earg Paner System
DitcDratie
(c) Traffic Control
(d) Active Voltage Control
Figure Illustration of four real-world environments in the benchmark.
A.2.1 Dexterous Hand Manipulation
The Bimanual Dexterous Hand Manipulation environment (DexHand) is comprehensive simulation platform designed to mimic human dexterity through reinforcement learning. Using two anatomically realistic Shadow Hands, each with 24 degrees of freedom (DoF), tackles tasks ranging from basic object manipulation to advanced bimanual coordination like stacking, catching, and reorientation. These tasks are designed to match different levels of human motor skills according to cognitive science literature, and demand precise control, adaptability, and synchronized multi-agent cooperation. Powered by the Isaac Gym simulator, Bi-Dex Dandaads combines physical realism with challenging, multi-agent tasks, presenting high level of difficulty for learning and mastering these tasks.
19

---

In this work, six typical tasks are considered for the deployment of all the MARL algorithms and attacks in our benchmark. These tasks include: PushBlock. SwingCup, Re- Re-Oriintation, BlockStack, HandCatchUnderarm
Below, we provide brief description of these tasks. For more details, we direct readers to the official project page of Bi-Dex https: // thub. wwt. WWw.-//ARLLDeeeerrrssiiiaaae
PushBlock: This task requires both hands to touch the block and push it forward.
SwingCup: This task requires two hands to hold the cup handle and rotate it 90 degrees.
Re-Orientation: This task involves two hands and two objects. Each hand holds an object and we need to reorient the object to the target orientation.
BlockStack: This task involves dual hands and two blocks, and we need to stack the block as a tower.
this problem, two shadow hands with palms facing upwards are controlled to pass an object from one palm the other. What makes it more difficult is that the hands' translation and rotation degrees freedom are not frozen but are added into the action space.
HandCatchUnderanme
HandCatchhvvr22nndraa In this task, the object needs to be thrown from the vertical hand to the palm-up hand.
A.2.2 Quadrotor Swarm Control
The Quadrotor Swarm Control (Quad) environment is designed to train RL policies for controlling quadrotor swarms. In this environment, each quadrotor is controlled by an individual RL agent, en- abling the development of sophisticated swarm behaviors. These include performing tight maneuvers in formation, avoiding collisions, adapting to dynamic obstacles, and collaborating in pursuit-evasion tasks. The environment simulates realistic quadrotor flight physics, accurately capturing the complex dynamics of aerial movement and multi-agent interactions. This ensures that the policies learned within the simulator can be generalized to real-world systems.
The environment includes several key scenarios to assess the quadrotors' abilities in different contexts. These tasks focus on the control, coordination, and safety of the swarm under vari- ous challenging conditions. Our benchmark experiments consist of six tasks: static static_diff goal, static same_ same_goa,, o __ttaic_same_____________ee_____ goane_ 0_ random, o swap_ggals, and swarm_ pas_vss_sanaa__ where o_ denotes environments with moving obstacle. A more detailed description of these six tasks is provided below, and additional information can be found on the official project page at https: htaps:///ssttsss www.tees.gooo..... www.coooviiiiee vwwe//swanwrwlll
static_diff diif__goal:: In this task, quadrotors are trained fly in close formation while maintain- ing a cohesive structure and avoiding collisions. The target formation, which remains fixed throughout the episode, can take various geometric shapes (e.g., 2D grid, circle, cylinder, and cube). The separation r between goals within the formation is randomly chosen.
platccssme__am gameegoal: This is special case of the static_dift ditee_ddf[gaaall scenario, where the separa- tion between goals is zero (r = 0). In this case, the goal locations for all quadrotors coincide, creating dense formation with high probability of collisions. This task requires enhanced coordination and decentralized control to ensure safe flight compared to the static _iiff_iffee_eee goal task.
o static same _oal: This task is similar the static sarric_sameeeeee goal task but takes place in an environment with dense cylindrical obstacles. The quadrotors must maintain their formation while avoiding collisions with the obstacles.
o random: In this task, each quadrotor's position in the formation is randomly sampled within the environment, which contains dense cylindrical obstacles. Every quadrotor needs to reach its own target position as fast as possible.
o_ o_swap_goaal This is a kind of dynamic formation control task. Given a predefined formation, the target positions of the quadrotors are randomly swapped multiple times during the episode. Additionally, the quadrotors must avoid collision with the dense obstacles in the environment.
20

---

swarm vs swarm: In this task, the swarm is split into two groups, then the target formations of quadrotors in these two groups are swapped several times per episode, which requires two teams of quadrotors to fly through each other while avoid head-on collisions at high speed.
A.2.3 Intelligent Traflic Control
The Intelligent Traffic Control (Traffic) environment focuses on training MARL agents to manage networked traffic systems. It supports various real-world traffic tasks, including Adaptive Traffic Signal Control (ATSC) and Cooperative Adaptive Cruise Control (CACC). In this environment, each agent represents either a traffic signal or a vehicle. Agents adjust their policies based on local observations and messages from neighboring agents, aiming to optimize overall traffic flow and safety, For ATSC tasks, agents control traffic signals at intersections, dynamically adjusting green/red light timings in response to local traffic conditions (e.g., vehicle density, wait times) and messages from adjacent intersections. In CACC tasks, agents simulate cooperative vehicle control, adjusting their speeds to either follow or close the gap with a leading vehicle, ensuring smooth, coordinated traffic flow.
This environment is capable of simulating both synthetic and real-world traffic networks, offering challenging scenarios for traffic signal and cooperative vehicle control tasks. Its versatility makes it an ideal platform for capturing the complexities of real-world traffic systems, such as partial observability, -on-stationary dynamics, and decentralized control. In our benchmark, we evaluate performance across four tasks within this environment: ATSC Grid, ATSC Monaco, CACC Catch-up, and CACC Slow-down. A detailed description of these tasks can be found in the original paper (55 and on the official project page https: //gi thub. com/ cts 198899/
ATSC Grid:
This task simulates traffic signal control within synthetic 5x5 grid-based traffic network. Each intersection in the grid is controlled by an agent, which adjusts its signal timings based on local traffic conditions (e.g., vehicle density, wait times) and messages from neighhoring agents. The task aims to optimize traffic flow and reduce congestion in this synthetic environment.
ATSC Monaco:
This task models adaptive traffic signal control in real-world 28- intersection network, specifically based on the traffic system of Monaco city. Agents control traffic signals in this dynamic environment, addressing complex traffic patterns and interactions between intersections. The learned policies must effectively manage these com- plexities, focusing reducing congestion, enhancing traffic throughput, and ensuring safe driving conditions. However, this task requires agents to have heterogeneous observation and action spaces. We thus omit this in our evaluation.
In this task, agents simulate CACC systems in a network of 8 vehicles. The goal is for each vehicle follow and "catch up" with a leading vehicle by adjusting its speed based on the relative position and velocity of the vehicle ahead. This task emphasizes the importance of coordination between agents to maintain efficient vehicle spacing and flow.
CACC Catch-up:
CACC Slow-down: This task simulates a CACC system in which agents follow a leading vehicle and slow down appropriately. Agents adjust their speeds based on the behavior of the leading vehicle, ensuring safe distances while reducing speed when necessary. This scenario highlights the challenges of safely managing the traffic flow in situations where deceleration required.
A.2.4 Active Voltage Control
The Active Voltage Control (Voltage) environment provides simulation platform for training RL policies to manage voltage in power distribution networks. It specifically targets the challenges posed by the integration of distributed energy resources, such as rooftop photovoltaics (PVs), into power grids, where excess power generation can cause voltage fluctuations. These fluctuations may exceed acceptable grid limits, as outlined by [68 69), necessitating effective voltage regulation.
The primary objective of the Voltage environment is to mitigate these voltage fluctuations by con- trolling the reactive power generated by PV inverters. Each agent in the system corresponds to a PV inverter, which adjusts the reactive power to regulate the voltage at its respective bus. However, since the voltage at each bus is influenced by the power at all other buses, and not all buses equipped
21

---

with PV inverters, agents must collaborate to ensure that the voltage across the entire network remains within a safe range. Given that each agent has limited visibility and can only observe the state of the local zone, the problem is naturally framed as a Decentralized Partially Observable Markov Decision Process (Dec- POMPP), requiring coordinated decision-making under uncertain conditions.
The Voltage environment includes three distinct scenarios based on real public data, each with different scales, ranging from small systems with 6 agents to larger systems with 38 agents. These scenarios vary in complexity, and our benchmark tests the robustness and scalability of multiple MARL algorithms across all three. The scenarios are referred to as the 33-Bus System, 141-Bus System, and 322-Bus System. A brief overview of each follows, with further details regarding the configurations and complexities available in the original paper [56]]
33-Bus System: This scenario models small-scale distribution network with 6 PV agents, which cooperate to regulate the voltage across 32 loads distributed over regions. The challenge is to maintain voltage within a safe range while minimizing power loss in system with relatively fewer agents and simpler interactions.
141-Bus System: This scenario involves a medium-scale distribution network with 22 PV agents that must coordinate to control voltage across 84 loads in 9 regions. The agents must manage more complex interactions and larger variations in power generation, requiring advanced coordination strategies.
This scenario simulates a large-scale distribution network with 38 agents controlling voltage across 337 loads in 22 regions. This complex environment emphasizes scalability and robustness, as agents must manage a larger network with greater uncertainty. Coordinating actions efficiently maintain voltage stability is critical, despite the challenges posed by a large number of loads and agents, along with more dynamic power fluctuations. This scenario tests the performance and adaptability of MARL algorithms in large-scale control tasks.
322-Bus System:
A.3
Additional Details on Type of uncertainties
In this section, we detail the type of observation, action and environment uncertainties considered in our paper.
A.3.1 Observation uncertainty
Observation uncertainty arises when agents' sensing capabilities are flawed, inaccurate, or maliciously
perturbed by attackers [29 14 15].
We consider the following types of uncertainties:
Gaussian noise. Gaussian noise is ubiquitous in real-world systems, arising from sources such as thermal fluctuations, sensor inaccuracies, and environmental errors. While simple and relatively easy to mitigate, Gaussian noise serves benchmark. Depending on the settings, we either apply small perturbations to all agents larger perturbations single agent, denoted as Gaussian-all and Gaussian-single, respectively.
essential baseline for our
Greedy worst-case attacks. Early research in robust RL and MARL introduced heuristic- based attacks to exploit vulnerabilities in agents" policies at individual timesteps. These methods specify a detrimental policy distribution and use gradient- PGD (70). to generate observation perturbations that mislead agents. For instance, [29] proposed maximizing the KL divergence between the original policy and the perturbed policy at given timestep. Other works explored alternative heuristics [71] 72 73). SA-
approaches, such
MDP 30 formalized this problem as a state-adversarial MDP and proved the existence of a worst-case adversary. In our study, we adopt the Maximal Action Difference (MAD) attack from SA-MDP B0), which generalizes (29 by perturbing observations to maximize the KL. divergence, max; D (-18)), where á is the perturbed state. For MARL, as only observations are available, perturbations are applied to the observation space. Attacks targeting all agents are denoted as Greedy-all, while attacks targeting single agent denoted as Greedy-single.
Optimal learned attacks. Moving beyond heuristic approaches, subsequent research developed adversarial policies using RL to optimize long-term term attack efficacy. ATLA [74]
22

---

pioneered this area by learning policy (ā]s) to generate perturbed states from the current state. PA-AD [75] improved ATLA by introducing two-step process: first learning a target action for the perturbation to induce, and then applying gradient- methods to craft state perturbations that guide the policy towards the target action. This approach is advantageous because actions often have lower dimensionality than states, simplifying the RL learning process, Similar to greedy attacks, we denote attacks against all agents as Optimal-all and those targeting single agent as Optimal-single.
A.3.2 Action uncertainty
Action uncertainties are a critical area of study in MARI. because, multi-agent systems, individual agents often take actions during deployment that deviate from optimal actions defined in simulation environments. These deviations can arise from various factors, such as external forces causing abrupt action changes, slight mismatches in robot weight compared to simulation models, or loss of control due to adversarial interference. We consider the following types of action uncertainties in MARL:
Random policies. Agents may adopt random policies due to actuator errors, defaulting to random actions under limited computational resources, communication failures, or power fluctuations. To evaluate the robustness of MARL systems, we use random actions as a baseline. Depending on the setting, all agents may take random actions with a small probability, denoted as Random-all, or single agent may take random actions with large probability, denoted as Random-single.
Greedy worst-case policies. Extending the concept of greedy worst-case perturbations in observations, we define greedy worst-case policies as those leading to the worst-case outcomes. Unlike observation perturbations, policy manipulations directly alter the action distribution, allowing for more targeted interventions. When the Q-function is available, we perturb policies to take actions that minimize the Q-value. If the Q-function is unavailable, policies are perturbed to take actions with the lowest probability. Depending on the setting, all agents may adopt greedy worst-case policies with small probability, denoted Greedy- all, or a single agent may do so with a large probability, denoted as Greedy-single
Optimal learned policies. Since policy perturbations naturally align with the RL/MARL framework, it is feasible to train a parameterized worst-cas cast--case agent to generate perturbed policies. [32] formulated this as an action-robust MDP and used RL algorithms to learn worst-case perturbations. [16] studied this in two-agent zero-sum games, demonstrating how adversarial policies can easily exploit opponents. In MARL, 76 40] learned worst-case policies to minimize cooperative rewards. Depending on the setting, all agents may take learned worst-case policies with small probability, denoted as Optimal-all, or single agent may do so with a probability of I, denoted as Optimal-single.
A.3.3 Environment uncertainty
Environment uncertainty has been critical area of research since the inception of the robust RL framework [77 78). Given the inevitable discrepancies between simulation environments and real-
world conditions, addressing uncertainties in environmental dynamics is long-standing challenge in RL and MARL. In MARL, theoretical studies on this topic have been conducted by [38] [9]. though these works lack empirical evaluation. To systematically evaluate the impact of environment
uncertainties in MARL, we adopt approaches commonly used in RL [57] 58 59 60). Specifically, we define uncertainty sets over key environment parameters such as velocity and mass to simulate
potential environmental variations. The uncertainties we use are listed in Table. 5
The following provides explanation of the meanings of these uncertainty sets within their respective environments:
Quadrotor Swarm Control:
quads_com _oadd_collision_ collision_hitbox caciooo__rduu:- This parameter defines the effective radius used for collision detection in quadrotor swarms, ranging from [1.5, 2.5]. When two quadrotors come within this distance, they are considered to have collided, prompting evasive maneuvers to ensure the swarm's 's safety.
23

---

Table 5: Uncertainty sets for environment uncertainties
Environment
Uncertain Set
Range
Quadrotor Swarm Control
_ittooo___ttot__eee bitbox_radius
1.5, 2.5
quads_collision
radius
0.7,
quads_obst_demns _bessty
-0.2, 0.2]
quads_obst_sizz
0.7, 1.3]
ftadd_collision_eee
-0.3, 0.3]
guads_collision.n saamooth_maxppemaaee max__pnnatty
7.13]
quads
guads_episode episode_ dutlleinuriinnnn
10.20]
quads obst collision
collision_rewart reward
-0.3.
replay buffer sample_ prob
0.
Active Voltage Control
pv DV_scale
0.7,
demand_seae
0.77, 1.3]
Y_ V_upper
00.70,, 1.305]
y_lower
00.65, 1.25]
q weight
0.,0.2
action bias
[-0.5, 0.5]
action_scale
0.44, 1.2)
demand_scale
0.5, 1.5
Dexterous Hand Manipulation
StartPositionNoise transition _sclee orientation scale
-0.005, 0.005]
0.22, 0.8)
-0.2,
rotRewardScale
0.7, 1.3]
startRotationNNisse
-0.5.0.5
resetPositionNoise resetRotationNoise
-0.3, 0.3)
0.0,
reselDoiPoRRandomnntevall
0,0.4]
reseDDorVelRanddmmnnevvaall
-..2
Intelligent Traffic Control
norm_ wave
0.7,
norm_ wait init_density coop_gamma headway_min
-1.2, 0.8)
0.5, 0..3,
1.5
headway_5. st
4.5, 6.5
headway go
30. 40
speed max
25, 35
accel_ _ma accel_min
2.3
3. 2
quads collissoooliiinn___e falloff radius::- The falloff radius determines how collision effects diminish with increasing distance between quadrotors, with a range of [0.7, 1.3). Within this radius, the interference forces decrease gradually, affecting flight dynamics and reducing abrupt changes in quadrotor behavior during near collisions.
quads_obee obst_ density: This parameter, ranging from [-0.2, 0.2), represents the relative density of obstacles in the environment. Positive values indicate denser obstacle field, increasing the difficulty of navigation, while negative values suggest sparser environment, facilitating smoother flight.
quads obst size: The size of obstacles is represented by this parameter, with range of [0.7, 1.3]. Larger obstacles necessitate more substantial avoidance maneuvers, whereas smaller obstacles, despite being easier evade individually, may pose challenges when present in large numbers.
quads coaliissooooooonooee reward: This parameter defines the reward value associated with collisions between quadrotors, with a range of [-0.3, 0.3). Negative values penalize collisions, encouraging the control algorithm to prioritize collision avoidance. The magnitude of the reward/penalty influences the aggressiveness of the avoidance maneu- vers.
24

---

quads colllisoooloioonne smooth max penalty: This parameter specifies the maximum penalty for non-smooth collision avoidance maneuvers, with range of [7, 13]. It ensures that the quadrotors' movements remain smooth and controlled, even when avoiding collisions. Higher penalties encourage more gradual and stable maneuvers, reducing the likelihood of abrupt changes in velocity or direction.
quads epissdepisde___e duration: This parameter defines the duration of each control episode, ranging from [10, 20] seconds. It determines the time horizon over which the control algorithm must manage the quadrotor swarm. Longer durations require more robust and sustained control strategies, while shorter durations allow for more aggressive maneuvers.
collision_reward:: This parameter defines the reward value associated with collisions between quadrotors and obstacles, with a range of [-0.3, 0.3). Negative values penalize such collisions, encouraging the control algorithm to prioritize obstacle avoidance. The magnitude of the reward/penalty influences the aggressiveness of the avoidance maneuvers.
quads amads_obst____e ceassts_ollsion____ reward:
replay buffer sample, 1ampple_eoob:::_111111111111 This parameter defines the probability of sampling from the replay buffer in reinforcement learning algorithms, with a range of [0.5, 1]. It affects how frequently the algorithm revisits past experiences stored in the replay buffer. Higher sampling probabilities ensure that the algorithm makes better use of historical data, improving its learning efficiency and ability to generalize from past experiences.
Active Voltage Control:
pv_scale: This parameter adjusts the proportion of photovoltaic (PV) power generation in the grid, with values ranging from [0.7, 1.3]. A value above I indicates an increased PV capacity, potentially enhancing voltage support, whereas values below 1 suggest reduced PV contributions, requiring supplementary power sources.
demand_seed seale: The scaling factor for electricity demand ranges from [0.7, 1.3]. Higher values signify increased power demand, complicating voltage regulation, while lower values ease the control requirements due to reduced demand.
t_upper: This parameter sets the upper voltage limit, with a range of [0.705, 1.305]. Ensuring grid voltage stays below this threshold is critical to prevent overvoltage-related equipment damage and maintain grid stability.
Fllower: The lower voltage limit ranges from [0.65, 1.25). Maintaining voltage above this threshold is essential to prevent equipment malfunctions and avoid voltage collapse, thus ensuring reliable grid performance.
(_weight:: This parameter represents the weighting factor for reactive power control, with values ranging from [0, 0.2). higher weight emphasizes the importance of reactive power management in the control strategy, helping to maintain voltage stability through reactive power compensation.
action_bias: The bias term for control actions ranges from [-0.5, 0.5]. This parameter introduces a baseline offset in the control actions, which can be used to adjust the default behavior of the control algorithm. Positive values shift the control actions towards more active interventions, while negative values reduce the aggressiveness of the control actions.
action bias:
action scale: This parameter scales the magnitude of control actions, with values ranging from (0.4, 1.2]. A higher scale factor increases the impact of control actions on the system, allowing for more significant adjustments to voltage levels. Conversely, a lower scale factor results in more conservative control actions.
demand scale: The scaling factor for electricity demand ranges from [0.5, 1.5]. This updated range indicates a broader variation in power demand, with higher values representing significantly increased demand scenarios that pose greater challenges for voltage regulation. Lower values, closer to 0.5, suggest reduced demand conditions that may simplify control requirements.
Dexterous Hand Manipulation:
startPositionNoise: This parameter introduces initial position noise in dexterous hand manipulation tasks, ranging from [-0.005, 0.005). It simulates minor deviations in the hand's starting position, increasing the complexity of the task and requiring precise control adjustments.
25

---

The scaling factor for state transitions ranges from [0.2, 0.8). Higher values result in more dynamic state changes, demanding robust control strategies, while lower values yield smoother transitions, facilitating easier manipulation.
transition scale:
orientation scale: This parameter, ranging from |-0.2, 0.4], adjusts the amplitude of object orientation changes. Positive values enhance orientation variations, challenging control precision, while negative values dampen these changes.
roteewardScale: The rotation reward scaling factor, with a range of [0.7, 1.3), in- fluences the prioritization of rotational actions in control algorithms. Higher values incentivize exploring rotational strategies, potentially improving manipulation perfor- mance.
startRotationNNois:: This parameter introduces initial rotation noise in the hand's orientation, ranging from 0.5,, 0.5]. It simulates variability in the starting orientation of the hand, adding complexity to the task and requiring the control algorithm adapt to different initial conditions.
This parameter specifies the noise added to the hand's position during resets, with values ranging from [-0.3, 0.3]. It introduces randomness in the hand's position the beginning of each trial, making the task more challenging and requiring the control algorithm to handle a wider range of initial positions.
NeseooositionNNois::
This parameter defines the noise added to the hand's rotation during resets, with values ranging from [0.0, 0.01]. It introduces small perturbations in the hand's orientation at the start of each trial, requiring the control algorithm to correct for these initial deviations.
orssetootationNoie
DosseDooPosRandooo Interval:: This parameter specifies the random interval for re- setting the degrees of freedom (DoF) positions, ranging from [0, 0.4]. It introduces variability in the initial joint positions of the hand, making the task more realistic and challenging. as the control algorithm must adapt different starting configurations.
resseDoFVeRRandmmInner
This parameter defines the random interval for resetting the degrees of freedom (DoF) velocities. with values ranging from [-0.2, 0.2). It introduces variability in the initial joint velocities of the hand, adding an additional layer of complexity to the task and requiring the control algorithm to manage both position and velocity adjustments.
Intelligent Traffic Control:
norm wave: This parameter represents normalized traffic flow fluctuations, ranging from [0.7, 1.3). Larger values indicate greater variability in traffic flow, increasing the complexity of traffic management, while lower values suggest more stable traffic patterns.
norm_ wait: The normalized waiting time parameter ranges from [-1.2, -0.8). Nega- tive values denote reduction in average vehicle waiting times, typically resulting from optimized control measures, thereby improving overall traffic efficiency.
density: This parameter, with a range of [-0.3, 0.3). defines the initial traffic density. Positive values indicate congested conditions that require advanced control strategies, whereas negative values suggest lighter traffic, allowing simpler interven- tions.
init_density
gamma: This parameter, ranging from [0.5, 1.1], represents the cooperation factor in traffic control. Higher values indicate stronger cooperation among vehicles traffic control systems, which can improve traffic flow efficiency. Lower values suggest more independent behavior, potentially leading to less coordinated traffic patterns.
headway min: The minimum headway distance between vehicles ranges from [0.5, 1.5] meters. This parameter defines the smallest safe distance that should be maintained between vehicles avoid collisions. smaller minimum headway allows for higher traffic density but increases the risk of accidents.
headway st: The standard headway distance ranges from [4.5, 6.5] meters. This parameter represents the typical distance maintained between vehicles under normal driving conditions. It affects the overall traffic flow and capacity of the road.
thedwwy_ggo The headway distance for vehicles in motion ranges from [30, 40] meters. This parameter is particularly relevant for high-speed traffic and defines the
26

---

Environment Class
Robusiness
Agent Class
Supports Real- World Applications
Model Weights
</>
Implementations
Customized
Model
Replication Real- World Dynamies
Resilience
Custom Agest Training Interface
Collect
Prep training(
Sample()
Commonly Used Benchmnurks
SMAC
SMAC12 Multi-Agent Mujoco
MPE GRF
Train()
Save()
Forward()
Trajectorics
Environment Customization
Custom Agent Evaluation Interface
resel()
step()
Hyperparameters
Actions
Restore()
Prep rollout()
Perform()
Evaluation
Observation Unnzrtaainties
Affected Agents
Action
Alfected Agenis
Enveromment Uncertainties
Uncertainty Set
Perturbttion Type
Uncertainties
Perturbation Type
MARL Configuration
Implementation
Environment
Uncertainty
Robustness Resilieene
Choice
Configurations
Configurations
Evaluation Settings
Figure 11: Architecture of codebase. The framework supports customized environments and algorithms, provided they implement the required interfaces. The evaluation of hyperparameters on robustness and resilience integrated into an automated workflow.
safe distance required to maintain smooth traffic flow while allowing for safe stopping distances.
speed_me max:
The maximum allowable speed ranges from [25, 35] meters per second (or approximately 90 to 126 kilometers per hour). This parameter sets the upper limit for vehicle speeds, balancing traffic flow efficiency with safety considerations.
accel_ _aa_:mmaaam---_1111 The maximum acceleration ranges from [2, 3] meters per second squared. This parameter defines the highest rate at which vehicles can increase their speed, affecting traffic dynamics and the ability to manage traffic flow efficiently.
accel min: The minimum acceleration (or maximum decelcration) ranges from [-3, 2] meters per second squared. This parameter defines the highest rate which vehicles can decelerate, which is crucial for maintaining safe distances and avoiding rear-end collisions.
These parameters are sampled randomly reflect real-world variability. In MARL scenarios, as all agents are equally influenced by environmental changes, we denote this type of uncertainty as emv.
A.4 Additional Details on Architectures
In this section, detail the overall architecture of algorithm, shown in Fig. 11 Our framework supports customized environments and algorithms, provided they implement the required interfaces. The evaluation of hyperparameters workflow. Our architecture design are also inspired by existing prominent codebase, including
robustness and resilience is integrated into automated
79 80 81.
A.4.1
Environment Architecture
To ensure the comprehensiveness and utility of our codebase, we integrate three types environments to support diverse applications and research needs.
The first category includes environments that support real-world applications. For instance. in Dexterous Hand Manipulation (53]. two agents collaboratively control 24-DoF Shadow Hand
82 to perform complex tasks similar to those learned by infants. Similarly, Quadrotor Swarm Control [54), multiple Crazyflie quadrotors [8] are trained to execute advanced swarm robotics tasks. Policies trained in these simulation environments can be directly deployed real-word robots with
minimal Sim2Real gaps.
27

---

The second category comprises environments constructed from real-world data to ensure high- fidelity replication of real-world dynamics. For example, Intelligent Traffic Control [55], agents optimize adaptive traffic signal control (ATSC) using real-world traffic data from Monaco and perform cooperative adaptive cruise control (CACC), a critical task in autonomous driving. In Active Voltage Control, agents regulate voltage fluctuations to remain within IEEE grid standards, as described by
[68. 669. These environments provide high-fidelity simulations of real-world traffic systems and voltage control scenarios.
The third category includes widely used benchmark environments in MARL research. These include SMAC [84], SMAC v2 [85]. MPE M. Multi- Agent Mujoco [86], and Google Research Football [87]. We believe these benchmarks are representative of environments commonly used in MARL studies, offering researchers solid foundation for their work on robust MARL.
Finally, recognizing the rapid emergence of new MARL.environnen industrial environments, we designed our codebase to support customization. Our environment interface accommodates custom environments as long as they implement two core APIs: reset () and step (). The reset () function initializes a new episode, while step () advances the environment to the next timestep based on agent actions. To evaluate the robustness and resilience against environment uncertainties, our interface also take custom environment hyperparameters as inputs.
and the existence of proprietary
A.4.2 Agent Architecture
As for the architectures of each MARL agent, we aim to make our design general and adaptable. accommodating customized code implementations and model weights. Our architecture is follows:
To evaluate the robustness and resilience of MARL algorithms, we introduce Agent class that disentangles code implementations and model weights from the evaluation process. This approach differs from previous codebases such Epymarl [42] and MAPPO [3, which separate agent functionality into three classes; actor, critic, and mixer. Such separation complicates integration, making the pipeline less user-friendly, particularly for non-MARL researchers.
In our codebase, we abstract all functionalities required for MARL agents into a single Agent class. This design allows users to integrate their custom code implementations and model weights into our framework by implementing APIs that satisfy our requirements. Specifically, the Agent class represents MARL agents interacting with the environment by receiving trajectories and outputting joint actions. If training is not required during evaluation, users need only upload their code implementation and model weights, and implement the following three APIs:
restore() (): Loads model weights into the code implementation.
prep_re reppotoloutt/ () Initializes the decision-making agent, including clearing the hidden state of RNNs and resetting the action distribution.
perform() (): Outputs the agent's action distribution during testing.
If training with customized code implementations required, users need additionally implement the following six APIs:
collect O) Collects trajectory data for all agents, including state, individual observations, and rewards.
prep_training(> Initializes the decision decision-making agent for training, clearing RNN hidden states and resetting the action distribution. Note that some algorithms, such as MADDPG, use different action distributions for training and testing, which this API supports.
sample (): Samples action from the model output. This varies by algorithm (e.g.. QMIX, MAPPO, MADDPG).
train (): Trains the model using data collected from the environment. While training meth- ods differ across algorithms, all training is encapsulated within this single API, simplifying integration.
save (): Saves the trained model weights.
forward (): Processes observation input and outputs an action distribution for all agents. For Q-learning algorithms, this API returns Q-values for each action and agent.
28

---

Although similar APIs exist in previous codebases, our work is the first to propose abstracting an Agent class to enable seamless integration of customized implementations and model weights. This which required the separate implementation of actor, critic, and mixer classes. By consolidating functionalities, our framework enhances usability and flexibility for both researchers and practitioners.
abstraction simplifies previous approaches 42 3
A.4.3 Uncertainty Architecture
Next, we describe the architecture for applying uncertainties to MARL systems. The primary design principle is ensure that uncertainties are simple to implement and operate in parallel with the existing Environment and Agent classes, without interfering with their original functionalityy
Uncertainties are introduced during the interaction between the Environment and Agent classes. As illustrated in Fig. the three types of uncertainties are highlighted in orange and applied follows:
Observation uncertainty: This added to the observation inputs of all agents. The perturbations are applied after the agents collect trajectories from the environment.
Action uncertainty: Actions, generated by the Agent class and passed to the Environment class, are perturbed after the victim agent outputs its action probabilities.
Environment uncertainty: These uncertainties are sampled from predefined external uncertainty sets and applied by modifying the environment's hyperparameters.
By structuring the addition of uncertainties in this manner, we achieve concise and modular implementation that integrates seamlessly with the existing architecture.
A.4.4 Efficient Evaluation Workflow
Given the extensive range of uncertainties, hyperparameters, tasks, and algorithms integrated into codebase, efficient evaluation workflow is essential. Our system allows users to specify the desired uncertainties, hyperparameters, tasks, and algorithms for evaluation. Once configured, the workflow automatically generates a comprehensive set of shell commands, facilitating large-scale, automated evaluations. This significantly reduces the time and effort required by users, streamlining the evaluation process and enabling rapid experimentation at scale.
For example, to train and evaluate all hyperparameters under different uncertainties, one sample code for generating all bash commands is:
Listing 1: Code for generating evaluation workflow.
python generate. py eval dexhands ShadowHandSwinggCup mappo extra=" hhealees- --stage
This automatically generates 256 commands for execution.
Listing 2: Generated bash commands for training and evalution.
CUDA CUDA_
WISIBLE python -u /root/ade httpa/dddaaaa_beeeeee bencharbnncmarr///// single_train --algo mappo --env dexhands --env. task ShadooHaandSvvingcupp --aggo usa_eval True --algo log_dir ./
results reaulta--algoosliicc True l-add_
-0000--024-1122---------00---0 --ex_nnamee
perturbation www....uu.evv.steps... --algo pertuu..prruurrtteee. 0
saaloe.adapttve_aalph. False targggo. attack False
headless hatpl//aaanrr////aaaaaaaaoooaonn
http//aapppp/////// /efautt/
random_noise. 2>41
DIL..IISBBLEEEEVIIEE- python -u /roor//dffnaarllbbmceeeee
single_train.py --algo mappo --env dexhands --env task
ShadorHandSvingCup --algo use_evl True .ogg ./
results -ago False - --lad_.--vvdiiiict stradd..itiinnrrsuussssss..
/results//exhandds
oec.:0000--00011111----------9-3 -xop_ann
perturbation
29

---

algo num_eny_steps --algo perturb_iters --algo.
seaptiveaaphie False --algo targeted False headless
out /ee.oo///eexaand//// hag.aaaaannaaaccccoooaaaaaaaaaaa occliatto..fuucclaaaeeeeeeaaaaaa
2>&1
CUDA_
WAA.VLIIIIBLE_EVVCSSSSSS python -u /root/aw,.na1__ecchmar/ dexhands --env task
single_train. --algo mappo
ShadosHandSringgCup -lal use_eval False --algo loggo.lo_dir ./
results -eaulta False --load_victin
-:002--002-11188----------------
petr.o0..sha.eeerrcrraaop.pp.ppp -urum perturbation
10 --algo.
adaptive_alpha True tat......a..etee..tttta..e
attackk False --adv_
False --adv_eps 0.1 --aaallsss het//logs/dehhanee
ShadowHaadSSinggpppaaaaa
pert_obs_sin. log 2>&1
B Additional Details on Experiments and Takeaways
B.1 Details of Reward Normalization Methods
In reinforcement learning, the scale of episodic rewards can vary substantially across tasks. This variability complicates the consistent evaluation of RL algorithms under different experimental settings. To enable robust and systematic comparisons, adopt three reward normalization methods. These approaches ensure meaningful performance assessments across diverse uncertainties and hyperparameters.
Z-Score Normalization: To address task-specific variations in episodic rewards (R), we employ 7-score normalization on per-task basis. Each reward value is transformed by subtracting the mean reward for its respective task and dividing by the standard deviation of the task's s rewards. The normalized reward is defined as:
R
Prask
Rnorm
G mask
where R is the original reward, Pask denotes the mean reward for the task, and dusk the standard deviation of the task's ' rewards. These statistics are calculated over the full reward distribution for the task, encompassing rewards from both untrained models and models trained with all algorithms under all uncertainties and hyperparameters in the benchmark. In our analysis, unless otherwise specified, z-score normalization applied wherever rewards from different tasks aggregated into single distribution, such as in the correlation analysis presented Scction 5.1 and the uncertainty diversity analysis in Section [.2
Performance Normalization for hyperparameters: To examine the impact of hyperpa- rameters on each task (as discussed in Section B.4), we rescale the rewards to quantify the variance caused by different hyperparameters Specifically, for each task, we normalize the rewards using the formula:
Rimpl Runtrain
R
Rdefault Runtrain
where Rimpl is the episodic reward obtained under a specific hyperparameters. Runrain is the reward the untrained model, and Ratfautt the reward achieved using the default hyperparameters for the task.
After normalizing the rewards, we aggregate the rewards across algorithms, uncertainties. and hyperparameters for each task. The resulting reward distribution is then averaged over the dimensions algorithms and uncertainties, ensuring that the remaining variance in the reward distribution is solely attributed to differences in hyperparameters. Then, the impact of hyperparameters can be quantified by computing the 95% % confidence interval of the normalized reward distribution for each task.
30

---

Per-Step Reward
Cooperation
Robustness
Resilience
Timestep
(a) Evaluation curve for cooperation. robustness and resilience.
Fail maintain
precise control
(b) Cooperation.
(c) Robustness.
(d) Resilience.
Figure 12: Case study on agents that are robust but non-resilient, Evaluated in task static_diff goal in Quads environment, under the uncertainty setting obs pos_grred_._. all.
Relative Performance Change by hyperparameters: To further examine the efficacy of specific hyperparameters (see Section 5.3) we calculate the relative performance change of each hyperparameters with respect the default implementation. This metric captures the degree to which an hyperparameters improves or degrades task performance. The normalized reward is defined as:
Rampt Ratefault
Reomm
Raefautt Runuzain
In this equation, Rimpt is the reward achieved with specific hyperparameters, Ritefauli is the reward obtained using the default implementation, and Runtrain the reward of the untrained model.
By explicitly quantifying relative performance changes, this method highlights hyperparam- eters that significantly enhance or impair task performance. offers a clear and interpretable metric for evaluating the effectiveness of various implementation strategies under different experimental conditions.
B.2 Examples on Differences Between Robustness and Resilience
In this section, present two representative cases to illustrate the distinction between robustness and resilience. Specifically, we provide examples of agents that are robust but not resilient, and agents that are resilient but not robust. To ensure visual clarity, restrict analysis the Quads environment. Other environments, such as Traffic and Voltage, are excluded due to limitations in rendering quality, while the fine-grained control of each finger joint required in Dexhands difficult to interpret intuitively.
Robust but non-resilient. We examine a case of robustness without resilience in the task static diff goal within the Quads environment, under the uncertainty setting obs obs_ sooos_greey____ all. This task involves two quadrotors, each assigned distinct goal state. The objective is navigate each quadrotor to its respective goal. The uncertainty obs blos__geeey_____ee all introduces small observation noise all agents, perturbing their perceptions toward adversarial directions, described in 30
As shown Fig. 12a the system performs comparably well under cooperative and resilient conditions, yet fails under robustness evaluations. In particular, in Fig. 12b and Fig. 12d agents behave similarly
31

---

when initialized in cooperative configurations or when recovering post-attack, with both quadrotors successfully navigating to their respective goals. However, under observation noise, as shown in Fig. 12c the agents are still able to reach approximate goal regions but fail to maintain precise control. The continuous perturbations in their observations lead to control errors that gradually displace the quadrotors from their true goals. This inability to maintain fine control reflects lack of robustness. In contrast, a truly robust agent would complete the task accurately despite the presence of such uncertaintics- an essential requirement for real-world deployment.
Resilient but non-robust. We now consider a case that exhibits resilience but lacks robustness, under the task o static same goal in the Quads environment, with uncertainty env. This task features two quadrotors. multiple obstacles, and shared goal. The uncertainty env modifies the environment dynamics by increasing the number and size of obstacles.
As shown in Fig. 13b and T3c the policy enables the agents reach the goal even in the pres- ence of added obstacles, albeit with minor collisions. This suggests a degree of robustness to environment-levee perturbations. However, when evaluating resilience in isolation (i.e., in the absence of uncertainty), as shown in Fig. [3d the quadrotors often take suboptimal trajectories, drifting away from the goal and failing to reach it precisely. We attribute this failure to a mismatch in state initialization: during resilience evaluation, both quadrotors begin in close proximity to the goal (Fig. D)), whereas in cooperative and robustness evaluations, they are initialized farther away (Fig. 1) and 13c 1)). This discrepancy leads to differing dynamics during execution.
13d
13b
In cooperative and robustness settings, the greater distance from the goal encourages smooth accel- eration and stable convergence. In contrast, during resilience evaluation, the short initial distance prompts one quadrotor to accelerate too quickly, overshooting the target and needing reverse course (Fig. [3d 2)). Upon returning, both quadrotors struggle to stabilize at the goal-an issue not observed under other evaluation conditions. This behavior consistently emerges across multiple rollouts. We attribute this to state aliasing: the agents encounter observation histories that closely resemble those seen during training, but differ in nuanced ways that demand distinct responses. The policy, unable to disambiguate these subtle variations, fails to adapt its behavior accordingly accoriingly-resulingg in degraded fine-graine tne-grained control and impaired resilience performance (Fig. [3d 3)).
B.3 Additional results on generality of linearity and further discussions
In this section, we discuss the linearity of correlation of performance degradation with robustness and resilience, under different uncertainty types, agent scopes, and attack strategies. As shown in Fig. [4] while linearity does not hold universally, it is statistically significant (p .05) in most cases includdin random, greedy, and optimal attacks, as well as observation uncertainty and perturbations applied to single or all agents. For environment uncertainty. linearity holds (r = 0.99 for robustness, r = 0.85 for resilience) but lacks statistical significance. Nonetheless, cooperation strongly correlates with both robustness and resilience under enviromment uncertainty (r > 0.88 and r > 0.95,p p < .001), suggesting that improving cooperation enhances robustness in this modality. In contrast, action uncertainty shows weak linearity (e.g., r 0.37, p 0.19 for robustness), largely due to MADDPG's s high resilience to single-agent perturbations. When these outliers are removed, linearity improves significantly (r 0.69, p .05 for robustness; 0.91, p < .001 for resilience), reflecting MADDPG's 's off- off-policy training and robustness to exploratory noise.
B.4 How Important Are hyperparameters?
In this section, we highlight the importance of hyperparameters in MARL by analyzing their impact across tasks. For each task, we quantify this impact as the 95% confidence interval of relative performance changes due hyperparameters (See details in Appendix B.I. Our key findings are:
1. hyperparameters affect tasks unevenly. As shown in Fig. 15 the impact of hyperparameters varies widely. Cooperative performance can change by 190%, robustness by 1 1123, and resilience by 1058% with a single change in hyperparameters. This underscores the need of practitioners to identify and focus on tasks highly sensitive to such changes.
hyperparameters often outweigh algorithms. Fig. 15] we highlight tasks in blue where a two-way ANOVA shows that changing single hyperparameters has more significant effect than
32

---

Per-Step Reward
Cooperation
Robustness
Resilience
Timestep
(a) Evaluation curve for cooperation, robustness and resilience.
1)
2)
3)
Cooperation. Quadrotors reach the goal consistently.
1)
2)
3)
X
Collision
Collision
(c) Robustness.
Quadrotors reach the goal despite several collisions.
1)
2)
Closer
Overshoot The Target
Imprecise Control
Initialization
(d) Resilience. Quadrotors are initialized an unseen state closer to goal, making acceleration and overshooting the target. This overshooting brings state aliasing in subsequent histories, resulting in imprecise control.
Figure 13: Case study on agents that are Resilient but non-robust. static_ sameic_same__a goal in Quads environment, under the uncertainty setting env.
Evaluated in task
33

---

Cooperation Batustere: Corvlation vs Attack
Medaliika
Coppration Robeatnes Corvlation Aftark Altacking
Atrckig--ttacligg SiateAAl
Cosperation Bohminn Carndation Atork Severit; Diflerest Altsk Serenties Oveall f AA5--181-l
Ohersl d85p Adlyds
Ovceel a- LAl--
Cogeratee Woheaanos Coerelitte
1250
Roheston 04
Uhe potteii cl-33
Maak Reder Sedeer/oltiippttee Oolcut soll
Alad. Dp
Der Sold
Perfermane Uncersintio (%)
Degrolation (1e)
10
Perforasnce Under
1he 21 21
(S)
Cosperation Resilinnee Corretation Attack
Casprration Reillerre Corvlation Attack
Cosperation- - Rediliane Curviatiin ve Attack
Severity: intleeres Attack Medalliio
Sevrrti:: Attarking SinckeAll
Semetty: Different Actark Smvritis
(hervalt 4.7% 588is
Casperston Moddang Corveitiin
10
Aack type
Ncc. pel Ho41
Stal Juish prd 15-3
Cridy beatlipeeeee
Optiat p122600
Peffrmaan Degrsaaiann Under Uncermieddm (%)
Pethrmmac Degradation Under Uscerstttioo (%)
Porfemaane Degnddation Under Lncertitedts
Figure 14: Linearity of robustness and resilience correlations under different uncertainty types, agent scopes, and attack strategies.
Task Sensitivity Implementatiin Choices
Standard Dev iiiiiii Value
Cosperation
Rebrainess-
0.3
0.49
0.22
0.2
9:34
10.31
voltage
volage
volluge
guads
Quaals
randoon
quads
oharmy
Maric Misoo Svall
Madee daft gpaal
Staads
Simuds segt Jinsss
peterool
ratelog
slowdowwr
matrrock
Cacktnduaam
Aunads
Red ReOrimaan
CaickOveca Undecarm
dinkkndss
hannds
dess shaness
Karte
dustus Shringgt
O. sanic sast
Tasks
Figure 15: The impact of hyperparameters varies across different tasks. Task names in blue indicate cases where altering a single hyperparameters has greater effect on performance than changing the MARL algorithm itself (two-way ANOVA, < .001).
Table 6: Examining the effect on the use of parameter sharing, GAE and PopArt on commonly used benchmarks, including MPE, SMAC and Multi-Agent Mujoco (MA-Mujoco). In fact, while these
tricks are effective on SMAC, the performance gain is not consistent across all environments.
Environment
Task
Default Use_ BAe_ False
Use_ Lise_PopArt_Fall False
Share Param False
Speaker Listener
11.48
-1.39
-11.39
-11.40
MPE
Spread
22.56
26.12
-31.86
-27.25
3m
18.62
18.80
18.75
11.58
SMAC
2s32
28.74
28.07
28.80
25.21
Ant-4x2
12256.34
8252.60
87 8715.61
17260.66
MA-Mujoco HalfCheetah-6x
19272.95
6628.80
8876.22
28280.03
switching algorithms (p < .001). Remarkably, in 9 out of 18 tasks, the performance improvement achieved by altering single hyperparameters surpasses that of switching between MARL algorithms.
Takeaway. Practitioners should evaluate task sensitivity to hyperparameters and prioritize tasks with high variability, as hyperparameters can have a profound impact on MARL performance.
34

---

Task: MaMujoco Ant-4x2
Task: Dexhand ShadowHanndCCtchhhhddmmnm
14
0.4
12
Per-Step Reward
Per-Step Reward
10
0.3
0.2
0.1
0.0
750
1750
Timestep
Timestep
Figure 16: Verifying our hypothesis that GAE is ineffective. GAE works best in Multi-Agent Mujoco with dense and predictable reward, while in Dexhand, the reward is less predictable.
B.5 Evaluating the Effectiveness of hyperparameters Other Environments
To verify the counterintuitive conclusion that the use of parameter sharing, GAE and PopArt is not generally effective hyperparameters, check the performance of these tasks in three commonly used benchmarks, MPE. SMAC and Multi-agent Mujoco using MAPPO. We select two task for each environment including speaker listener and spread for MPE, 3m and 2s3z for MAPPO, and Ant-4x2, HalfCheetah6x1 for Multi-Agent Mujoco. We report the results in 5 random seeds.
As shown in Table 6 confirm MAPPO's finding that parameter sharing is critical for SMAC tasks. However, parameter sharing offers no benefit in Multi-Agent Mujoco, where the task involves controlling highly diverse joints, and its impact is mild in MPE. Additionally, while GAE and PopArt significantly improve performance in Multi-Agen Mujoco, they have negligible or even slightly negative effects in MPE and SMAC. These findings support the statements in our main paper: (1) parameter sharing is effective for SMAC and not universally beneficial, and (2) the effectiveness of GAE and PopArt task desedeeenddent and not universally applicable.
B.6 Verifying the hypothesis of the ineffectiveness of GAE
In this section, we conduct a preliminary analysis to our hypothesis that GAE improves per- formance in Multi-Agent MuloCo but is less effective for real-world tasks. As shown in Fig. 16 the rewards in Multi-Agent MuloCo are dense, stable, and predictable within an episode. Conse- quently, GAE effectively stabilizes learning and reduces variance. In contrast, Dexhand presents more complex task with highly variable and less predictable rewards over time. As a result, GAE may introduce significant errors in estimating 8, leading to larger bootstrapping errors during value function approximation.
B.7 Analyzing the synergy of hyperparameters
In this section, we analyze how individual hyperparameters interact when jointly applied, as shown in Fig. and identify those that remain consistently beneficial in combination.
To quantify their joint effect, we leverage individual impact of each hyperparameter from Fig. (denoted as h.) and the combined performance when applying the best group of hyperparameters from Fig. (denoted as j). We fit an ordinary least squares (OLS) regression model, assigning a weight wy to each individual hyperparameter Vi, and minimize the squared error (j E, h,w; - +, where & is a random noise. While OLS shares similar formulation of standard linear regression, additionally enables significance testing by adding random noise, allowing us to filter out spurious correlations and identify hyperparameters that offer consistent synergistic effects.
We apply this OLS analysis separately for cooperation, robustness. resilience, and the combined dataset, using 5% winsorization to suppress outliers. higher weight w; indicates a stronger contribution of the corresponding hyperparameter when used in combination. Note that the learned weights not reflect global average performance, rather the effectiveness of each hyperparameter variation within its specific task. As result, some variations may not perform well on average but
35

---

Table OLS Regression Coefficients for Each Hyperparameter Across Metrics, with statistical test results denoted by x. * denotes p < .0,, ** denotes < .001.
Hyperparameter usc_popart_Pai False
Cooperation
Robustness
Resilience
metrics
0.6688
1.667"
1.1007
2.0496"
gamma 0.95
2,8845"
1.1021
1.0255
1.5002
gamma 0.9
2.7529
0.5963
1.0765
1.3207
activation func selu
3.7292
0.2915
1.2928
1.0862
share_parane Truc
0,8536
0.8999
0.7935
0.8932
hidden_sizes_tne 2_04666444446
1.3729**
0.7576
0.1808
0.8097
critic_Ir_ 00.000050
0.9467
0.4166'
0.8912
0.7751
hidden_sizen 226___26____ _56__56
-0.6873
0.9917
0.5153
0.7496
aottalization_metteee _avveu___aee____e uniform
-1.1196
0.6602
0.5090
0.7130
cettrpyy_cef__eee 10000
2.0065
0.6207
0.2690
0.6972
n_step_5
-1.4511
0.3846
0.3337
0.5887
expl_niss 0.01
-3.1952
0.7608
-1.3955
0.5136
entropy_come coef 0.0001
0.3350
0.9233
0.1412
0.4918
use feature normalization False
0.4857
0.4671
0.4711
0.3944
peccurrent_pooiey_Tnee
0.7096
0.5041
-0.2559
0.0402
expl_noise_1.0
0.0000
0.0000
0.0000
0.0000
ietrratio_funessignne
1..4466
0.7742
0.0398
0.0698
n n_steep_ 25
4..9211
0.2335
0.2881
0.0762
activation func _anh
-0.447
0.1760
0.1597
0.2164
activation_ Laapa________eeeeeeeeeee leaky_relu relu
0.9681
0.0915
0.1567
0.2396
expl_noise 0.001
0.7439
-0.1964
1.1622
-0.3811
use_gae_ galse
-0,2794
0.1803
00.7211*
-0.6672*
entropy_com care__oee___ee 0.1
-.1006
1.2337
0,1215
-0.6374
expl_noise_0..
-0.7096
0.2485
-1.7288
00.311
Ir_5e-05
-0.4499
0.4082
-0.1224
-0.9115
Ir_0.005
-2.9178
-0.3520
-0.7945"
-0.9277*
critic 5e-05
-1.6991"
007381
-0.9523*
-0.9954
can be highly beneficial particular environments. As shown in Table. 7 results are sorted by the weights learned from the combined dataset. The findings are as follows.
1. share param true is effective for homogeneous agents. In earlier analysis, parameter shar- ing, as a hyperparameter originally emphasized by MAPPO as their core trick, appeared ineffective when applied in isolation across all tasks. However, proves beneficial in the Traffic environment, where agents are homogeneous and share similar observations and objectives. Since we use the best-performing hyperparameter set per task, parameter sharing consistently yields strong results in this setting (p < .05). In contrast, in environments with heterogeneous agents, such as Dexhand, Quads, and Voltage, agents have specialized roles, making parameter sharing less effective.
2. Small discount factor y benefits tasks with short episodes. Although smaller discount factors show mixed results when applied individually, we find they perform well in combination with other hyperparameters (p < .001 for both & = 0.9 and y 0.95). Further analysis reveals that they are particularly effective in tasks like Dexhand and Voltage, which have shorter episode lengths (80 and 200 steps, respectively). In such settings, lower y downweights distant rewards and promotes faster convergence. In contrast, environments with longer episodes sonodees-uchh Traffic (1000 steps) and Quads (1600 bbeef) from higher 7 values to better account for long-term dependencies.
3. Higher critic learning rate brings consistent benefit. While beneficial as standalone trick, using a higher learning rate for the critic also consistently improves performance when combined with other variations (p < .05). This result further supports the efffcctiveness of the two-timescale actor-critic framework, where a faster-updating critic stabilizes training for a slower-updating actor.
4. The benefit of early stop remains consistent in hyperparameter synergy. Although early stop effective as standalone technique, we find it remains consistently beneficial when combined with other hyperparameters. Notably, we exclude it from T as we save both the final model and the best-performing checkpoint, allowing us to directly assess its impact. On average, early stop improves cooperation by 2.85% robustness by 13.62%, and resilience by 2.48%, with the improvement statistically significant (paired I-test, p < .001). Its effectiveness is intuitive-earl stopping selects the best perforenn model, making it generally applicable hyperparameter for trustworthy MARL.
36

---

Takeaway. Use parameter sharing for homogeneous agents, smaller discount factors for short episodes. Always use higher critic LR and early stop.
37

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We have clearly stated all claims.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Yes, in conclusions.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
38

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
Code and data provided.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
39

---

Answer: [Yes]
Justification:
We have provided the code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ccpppi.ccppulli//uuuiu wwwti//gguie///eeeeceoocc CoodeSubeiiie ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We have provided code and data.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We have reported test results
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
40

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have reported them.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: htt.://ppp............/cccccaaaa
Answer:
[Yes]
Justification:
We follow code of ethies.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Yes, in main paper.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
41

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper focus trustworthy MARL, possible risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We have cited related assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppswpptseeee wwworeeswthhhe....c. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
42

---

Answer: [Yes]
Justification:
We have open-sourced them.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/codd/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
43

---

Justification:
We use LLM to polish writing only.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy htte:s/s//eerrpppoccc///o/rreeep for what should or should not be described.
44


Reviews:
[{"review_id": "sePFndUSVp", "reviewer": "Reviewer_W4zq", "summary": "This paper explores the importance of tuning hyper parameters in MARL systems with the goal of understanding their roles in cooperation, robustness and resilience. \n\nIt makes the clear distinction between that robustness is ability to deal with constant perturbations while resilience is the ability to recover from disruptions. \n\nThey test a combination of 82,620 different experiments. Varying hyper parameters, model designs and environments.\nThe statistical analysis identifies a few compelling ideas and correlations.\n\n1.\tOptimizing cooperation under mild uncertainty enhances both resilience and robustness, though this relationship diminishes as uncertainty grows.\n\n2.\tThe modality of the uncertainty affects different policies in different ways. For example, MAPPO and HAPPO are more robust to observation uncertainties and MADDPG is better with action uncertainty. (This posses the question of whether there is a model that can incorporate the strengths of both of these.)\n\n3.\tParameter sharing appears to reduce cooperation, robustness, and resilience, except for improvements in SMAC.\nAn interesting question to ask is why does parameter sharing help in SMAC but hinder performance in MPE and Multi-Agent Mujoco? Is this because the agent  roles and objectives are the same for some agents in SMAC but not in the others? (example, zealots have the same role and objective and will likely benefit from the same parameters, while a particle in MPE has a different objective to another particle and thus benefits from a different policy entirely.)\n\n4.\tGAE, and PopArt can hurt robustness, as they found in MPE and SMAC, but helps in Multi-agent Mujoco. They suggest that it’s because of the dense and stable reward structure. Does this not just mean it is a design choice we should be tuning if we know about the reward structure?\n\n5.\tThey find that early stopping helps. Suggesting saving a model at a point where it is optimal over cooperation, robustness and resilience is better than saving the final model in many cases. This begs the age old question, at what point should we be stopping?\n\n6.\tHigh critic learning rates (higher than actors) and using Leaky ReLU help with robustness. The critic updating faster than the actors is suggested to have more robust results, and Leaky ReLU is suggested to solve the dead neuron problem that arises with ReLU. Are there environments that this might not be the case?", "questions": "1. Is there possibly a more principled approach to the hyper parameter search than just checking over a range? This feels like it would depend heavily on the environment and the model being checked.\n\n2. Are you sure your general statements about GAE, PopArt, etc... can be generalized? Especially considering the dense reward in Mujoco seemingly benefiting from it.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1.\tThe comprehensive statistical analysis on the different experiments are thorough and do highlight factors we should consider.\n\n2.\tThe work covers many different hyper parameters that can be chosen and shows reasons why one size fits all can’t be the solution for MARL (However I am not convinced that the correct conclusion is always drawn. The authors even find exceptions to their claims, eg parameter sharing being bad except in SMAC).\n\n3.\tThere is a code base that can help anyone tune their models. Which is useful since most studies don’t focus enough on the hyper parameter tuning.\n\n4.\tThe paper correctly distinguishes between robustness and resilience, a concept that many MARL works lack clarity on.\n\n5.\tThe insight that GAE and PopArt are probably only useful in high density reward and harmful at other times is useful.\n\nWeaknesses:\n\n1.\tThe paper generalizes claims about concepts, while directly giving a contradictory example. See the example with parameter sharing and SMAC, or GAE and PopArt with Mujoco. I believe there is likely a more principled manner to make these claims, since the exceptions aren't always explained.\n\n2.\tThe hyper parameter search appears to just be a sweep over a certain range, is there not a more principled manner to do this?\n\n3.\tMinor: Typically, the related work is stated at the start of the paper to help frame the work.", "was_revised": true, "final_justification": "I will keep my score.\nI lean toward acceptance based on the share magnitude of the experiments run, and I believe there are good insights to be drawn from these experiments, with the code base also allowing for ease of implementation for future researchers.", "rebuttal": "We thank Reviewer W4zq for appreciating the novelty, large-scale empirical study, codebase contribution, statistical contributions and findings of our paper. A detailed response is provided below.\n\n> Q1: The paper generalizes claims about concepts, while directly giving a contradictory example. Likely a more principled manner to make these claims, since the exceptions aren't always explained.\n\n1. When and why parameter sharing do not work?\n\nWe believe parameter sharing helps in environments where agents are more homogeneous like SMAC, where agents are of the same type (eg, zealot, marine). However, it do harm performance for Voltage, Traffic, MPE and MAMujoco since in these environments, agents are heterogeneous (Line 307-311). For Dexhand and Quads, the effect is mixed and non-significant. While we do believe agents benefit from parameter sharing when they are homogeneous, there do not exist a quantitative way to measure how \"homogeneous\" the environment is since it is highly task-specific. A safe conclusion is to treat parameter sharing as a design choice, rather than default implementation, especially in environments with heterogeneous agents.\n\nOur claim is reinforced in HATRPO [1], where they show parameter sharing may result in suboptimal actions since it pose an important constraint that the parameters of all agents are identical. The Proposition 1 made by HATRPO even show parameter sharing brings exponentially worse outcome by constructing a negative example. On MAMujoco environment, they also find parameter sharing \"introduces extra variance to training, harms the monotonic improvement property, make algorithm converge to suboptimal policies, and the problem gets more severe in the task with more agents\".\n\n[1] TRUST REGION POLICY OPTIMISATION IN MULTI-AGENT REINFORCEMENT LEARNING. ICLR 2022.\n\n2. GAE/Popart. is it just a design choice if we know the reward structure?\n\nWhile we do find GAE and PopArt helps in MAMujoco with dense and stable reward structure, and can have negative impact in our 4 real world environment, SMAC and MPE with statistical significance, we shall additionally remind readers that such claims are made empirically and statistically, which will not hold for every case. However, the statistical significant result do suggest a potential direction for tuning, which have high probability (ie, statistically significant) to yield positive results.\n\n3. While early stopping helps, at what point we should stop?\n\nIf we have a perfect measure of model performance, we should stop exactly at that point. However, in MARL community, a common way is to optimize cooperative performance greedily, which we find it will hurt robustness and resilience after a certain point due to overfitting. As such, while our focus on cooperation, robustness and resilience may not be a perfect measure, it can be a good enough starting point to start with in real world, and is much better than optimizing cooperation performance only. In practice, engineers can estimate the range of uncertainties in their own tasks, and perform early stop accordingly.\n\n4. While high critic learning rates and leaky relu works better, are there any environment not the case?\n\nIn fact, there are environments and uncertainties that are not the case. For example, higher critic LR slightly hurts resilience in Dexhand and Leaky ReLU slightly hurts resilience for MADDPG. However, in all experiments we conduct, higher critic LR works better in 153/161 cases, and leaky relu works better in 148/161 cases. As a consequence, trying it first is likely to return positive result on average.\n\n\n> Q2: The hyperparameter search appears to just be a sweep over a certain range, is there not a more principled manner to do this? it will heavily depend on the environment and models.\n\nTo the best of our knowledge, there do not exist a theoretically rigorous way to define ideal hyperparameter sweeps (otherwise we can tune MARL optimally and analytically!). Thus, we select parameters that are commonly used in existing codebases, including the codebase of MAPPO [1] and HAPPO [2]. These codebases includes a set carefully designed hyperparameters that works well for MPE, SMAC and MAMujoco, which varies in different tasks. Among all hyperparameters in these codebase, we conduct a pilot study to select hyperparameters that works well in most of our real-world tasks, and select other hyperparameters as alternate solutions.\n\n[1] The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games. NeurIPS 2022.\n\n[2] Heterogeneous-agent reinforcement learning. JMLR 2024.\n\n> Q3: the related work is stated at the start of the paper to help frame the work.\n\nThanks for the suggestion. We will move that section in final version.\n\n> Q4: Can findings on general statements be generalized?\n\nWe believe our findings can be generalized as follows. Our findings suggests certain tuning directions that are statistically significant (i.e., likely to work on average), including higher critic LR, early stopping and Leaky ReLU. Moreover, while parameter sharing, GAE and PopArt are previously regarded as panacea to cooperative MARL, previous researchers may treat this as default. Our finding shows it is merely a design choice, revealing blind spots that previous researchers may overlook. We do not claim our findings to be a silver bullet that helps any task and any algorithm under any random seed, but it is a promising direction for optimization.\n \nAs for the claims in Section 4.1 and 4.2, we believe those evidence are general enough to guide the design of robust MARL algorithms. In Section 4.1, we suggest focusing on the more harmful attacks. In Section 4.2, we suggest focusing on attacks over all modality and agent scopes. We summarize our general suggestions as follows:\n\n\n| Scenario / Task                          | Suggested Practice                                         | Reference     |\n|------------------------------------------------|----------------------------------------------------------------|-------------------|\n| Small uncertainties                            | Optimizing for cooperation is sufficient                       | Section 4.1       |\n| Large uncertainties                            | Requires dedicated robustness or resilience strategies         | Section 4.1       |\n| Robustness to observation uncertainties        | MAPPO / HAPPO perform best                                     | Section 4.1       |\n| Robustness to action uncertainties             | MADDPG performs best                                           | Section 4.1       |\n| Uncertainty across modalities                  | Evaluate observation, environment, and action perturbations separately | Section 4.2       |\n| Uncertainty across agent scopes                | Test both individual-agent and global-agent perturbations      | Section 4.2       |\n| Early stopping                                 | Should always be enabled                                       | Section 4.3       |\n| Critic learning rate                           | Should be higher than actor learning rate                      | Section 4.3       |\n| Activation function                            | Use Leaky ReLU                                                 | Section 4.3       |\n| GAE (Generalized Advantage Estimation)         | Disable                                                        | Section 4.3       |\n| PopArt                                         | Disable                                                        | Section 4.3       |\n| Parameter sharing                              | Use for homogeneous agents; disable for heterogeneous agents   | Section 4.3       |\n| Exploration strategy                           | Use high entropy bonus for MAPPO/HAPPO; moderate noise for MADDPG    | Section 4.3       |"}, {"review_id": "hMZv3uL5Q9", "reviewer": "Reviewer_bhkV", "summary": "This paper presents a large-scale and comprehensive empirical study on robustness and resilience in cooperative MARL. The authors clearly distinguish and formalize robustness (the ability to withstand uncertainties) and resilience (the ability to recover from severe disruptions). Through 82,620 experiments across four real-world-inspired environments, the study systematically evaluates three widely used continuous-control MARL algorithms (MAPPO, MADDPG, HAPPO) under 13 types of uncertainty and 15 hyperparameters. The key findings are: (1) while optimizing for cooperation can improve robustness and resilience under mild uncertainty, this correlation weakens as perturbations intensify; (2) robustness and resilience do not generalize across uncertainty modalities or agent scopes; and (3) hyperparameter tuning can have a greater impact on these properties than the choice of MARL algorithm, with some standard practices (such as parameter sharing, GAE, and PopArt) sometimes harming robustness, while strategies like early stopping, higher critic learning rates, and Leaky ReLU are consistently beneficial.", "questions": "1. To what extent do the main findings generalize to other MARL algorithms, such as value decomposition or graph-based methods? Have the authors considered extending their analysis to these additional families?\n2. Many results are shown across four environments, but how confident are the authors that key observations would hold in other domains, such as additional robotics, control, or RL tasks (e.g., gym benchmarks)?\n3. Have similar sensitivities to hyperparameter tuning been observed in more recent robust MARL methods (e.g., distributional RL, opponent modeling, etc.)? Would direct empirical comparison with these advanced baselines alter the main conclusions?\n4. Could the authors provide further justification for the chosen hyperparameter ranges, and clarify whether the reported benefits and negative effects persist under different normalization or reward structures?\n5. Have the authors considered whether noise effects accumulate in later-collected samples as training progresses? Is there any empirical evidence or analysis regarding this potential issue?\n6. The study primarily uses 5 random seeds per experiment. Would increasing this number improve the statistical confidence of the results, especially for small effect sizes?\n7. Given the experimental scale, what are the computational requirements for reproducing the main findings? Are all tested environments and uncertainty settings easily accessible and reproducible via the released code?\n8. There are repeated references to the code link and several instances of redundant information. Would the authors consider streamlining these sections for clarity and conciseness?", "limitations": "Yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**: The greatest strength of this paper lies in the scale, rigor, and comprehensiveness of its empirical investigation—spanning over 82,000 experiments across four real-world-inspired environments, 13 types of uncertainty, and 15 hyperparameters. This enables robust, statistically meaningful conclusions and demonstrates a level of methodological thoroughness rarely seen in MARL research. The authors provide clear and rigorous distinctions between robustness and resilience, offering a practical framework and formal metrics that fill an important gap in the literature. Their systematic evaluation covers multiple algorithms and environments, with a modular codebase that enhances reproducibility and extensibility for practitioners. The study’s concrete findings, such as the outsized impact of hyperparameter tuning (often exceeding that of algorithmic choice), and the counterintuitive result that standard practices like parameter sharing, GAE, and PopArt can be detrimental under uncertainty, offer actionable guidance for the community.\n\n**Weaknesses**\n1. The paper does not introduce new algorithms, training paradigms, or theoretical guarantees. Its contributions primarily lie in systematic evaluation and formalization, which limits the originality and theoretical depth.\n2. The study centers on three mainstream MARL algorithms (MADDPG, MAPPO, HAPPO), and while these are representative, the results may not generalize to alternative MARL paradigms (e.g., value decomposition, graph-based, or distributional RL methods).\n3. Some conclusions may be specific to the chosen environments or implementation details, such as the negative effects of parameter sharing, GAE, and PopArt, or the universality of certain tuning “tricks”.\n4. There is limited presentation of raw numerical results, confidence intervals, or effect sizes in the main text; detailed per-task outcomes are relegated to the appendix, reducing transparency about the practical impact on individual tasks.\n5. The empirical comparison does not include more recent robust MARL methods (e.g., distributional RL, adversarial or opponent modeling).\n6. The large breadth of experiments and the dense presentation may overwhelm readers, and some methodological choices (such as rationale for selected hyperparameter ranges or interaction effects) could be discussed more clearly for accessibility.", "was_revised": true, "final_justification": "The authors do not propose any new algorithms or theories, which is my main hesitation. However, the empirical study is undeniable. Therefore, the final decision depends on whether the empirical study alone is sufficient.", "rebuttal": "We thank Reviewer bhkV for appreciating the scale, rigor, comprehensiveness, codebase contribution, statistical test and actionable guidance of our paper. A detailed response is provided below.\n\n> Q1: The paper do not introduce new algorithms and new theoretical guarantees.\n\nThank you for the comment. It is true that our paper does not propose a new algorithm or theoretical guarantee. However, the core aim of this work is to provide a comprehensive empirical study on robustness and resilience in MARL, which, to our knowledge, is still under-explored.\n\nRather than introducing a single new algorithm, we contribute a set of key empirical findings and actionable insights that can guide the development of an entire class of more robust and resilient MARL algorithms. We believe this kind of contribution can be equally, if not more, valuable to the community, as it identifies where and how current methods fail and what design principles matter most under different uncertainty conditions.\n\nFor example, in Section 4.3, we give suggestions on hyperparameters that improves cooperation, robustness and resilience together. In Section 4.4, we show applying these hyperparameters can improve MARL performance largely on both purely cooperative, and robust MARL. Similarly, in Section 4.2, we should robust MARL should be evaluated across uncertainty modalities and agent scopes, which highlight limitations of current evaluation and point to new algorithmic opportunities.\n\n> Q2: The results may not generalize to alternative MARL paradigms (e.g., value decomposition, graph-based, or distributional RL methods)\n\nThank you for the valuable feedback. While our focus is on policy-based MARL, many of our findings remain relevant to alternate MARL paradigms, and we identify this as a promising direction for future work.\n\nFirst, our real-world environments requires continuous control without graph structure. Thus, mainstream value decomposition based method like QMIX are not applicable and the graph sturcture is not available. We acknowledge the importance of these works and will discuss them in the related work.\n\nSecond, while our study does not directly evaluate these methods, many of these paradigms are built on policy gradient frameworks. Thus, many insights should still transfer, such as the benefits of higher critic learning rates, parameter sharing, GAE, PopArt, and exploration strategies.\n\nThird, value decomposition methods assume all agents contribute positively, which fails in adversarial settings with compromised agents. Graph-based MARL may be more vulnerable to critical agents due to inter-agent connectivity, while distributional RL may offer greater robustness via stochasticity. However, evaluating these would require ~80,000 additional experiments, which is beyond our current scope.\n\nFinally, our codebase (Appendix A.4) support new algorithms that meets our interface requirement and can be used to evaluate alternative MARL paradigms under adversarial perturbations. Our codebase support QMIX which is based on value decomposition currently. We will include this as a limitation and an invitation for follow-up work in the final version.\n\n> Q3: Some conclusions may be specific to the chosen environments or implementation details.\n\nWe believe our findings are general enough for the following reasons. While no benchmark can fully represent MARL, our environments span over diverse and representative tasks of MARL.\n\n| Environment | Task Type   | Control Type  | Episode Length | Engine/Source | Data Source  | Goal  | Challenge |\n|------|-----|-------|------|------|-------|---|----|\n| DexHand | Multi-robot Manipulation  | Continuous| ~80  | Isaac Gym | Real-world Robots | Hand Manipulation  |  Precise Control   |\n| Quads | Multi-robot navigation  | Continuous | ~1600| OpenAI Gym | Real-world Robots | Reach Target & avoid collision |  Long-range multi-robot task assignment|\n| Traffic| Network control   | Discrete| ~1000| SUMO  | Real-world traffic | Optimize throughput  | Long-range control of multiple agents   |\n| Voltage| Network control   | Continuous | ~200| IEEE Standard | Real-world power grid  | Voltage Management |  Complex and noisy dynamics|\n\nUnder diverse environments and uncertainties, our findings suggests certain tuning directions that are consistent and statistically significant across all settings, including higher critic LR, early stopping and Leaky ReLU. Moreover, while parameter sharing, GAE and PopArt are treat as default by previous researchers, our finding shows it is merely a design choice, revealing blind spots that researchers may overlook. While these findings may not help for any task, algorithm and random seed, they are a promising direction for optimization and helps at high probabilities.\n\n> Q4: There is limited presentation of raw numerical results, confidence intervals, or effect sizes in the main text. Per-task outcomes are relegated to the appendix.\n\nWe will share the raw reward score under each configurations and uncertainties after acceptance of our paper and include additional raw numerical results in Appendix for further analysis. Since we are sharing insights of our evaluation, the raw records may be messy and requires time and effort to understand.\n\nAs for confidence intervals and effect sizes, we will add them in main paper in final revision.\n\n> Q5: The empirical comparison does not include more recent robust MARL methods (e.g., distributional RL, adversarial or opponent modeling).\n\nFirst, please allow us to clarify we have included comparison with ERNIE in Fig 9, a general-purposed robust MARL method. The result shows our selcted hyperparameters also works for ERNIE. Given the fact that many robust MARL algorithms are based on MARL backbones like MAPPO, our result helps these robust MARL methods as well.\n\nSecond, robust MARL is complex. Most algorithms are specific to their own type of perturbation and their backbone MARL algorithms. For example, EIR-MAPPO [11] works only on MAPPO backbone with action uncertainty applied to one agent, M3DDPG [24] works for MADDPG backbone only with action uncertainty applied to all agents. Similarly, in observation uncertainty, noise can be applied to each agent [8] or all agents [9]. Moreoverr, as reported by EIR-MAPPO [11], since their method is designed against action uncertainties, they are not robust to obervation uncertainties. To the best of our knowledge, there do not exist a single work on robust MARL that (1) can be applied to any backbone, (2) works for perturbations on one, several or all agents and (3) works for perturbations over observation, environment and actions. As such, we compare ERNIE, which is the robust MARL algorithm that have the highest potential to deal with all types of uncertainties.\n\n> Q6: The large breadth of experiments and the dense presentation may overwhelm readers, and some methodological choices (such as rationale for selected hyperparameter ranges or interaction effects) could be discussed more clearly for accessibility.\n\nWe will include a takeaway table for readers to summarize our main points below:\n\n|Scenario|Suggestion|Ref.|\n|---|------|-----|\n|Small uncertainties|Optimize cooperation|§4.1|\n|Large uncertainties|Use robustness/resilience methods|§4.1|\n|Obs. uncertainties|MAPPO/HAPPO best|§4.1|\n|Action uncertainties|MADDPG best|§4.1|\n|Modalities|Evaluate obs/env/action separately|§4.2|\n|Agent scopes|Test individual/global perturbations|§4.2|\n|Early stop|On|§4.3|\n|Critic LR|> Actor LR|§4.3|\n|Activation|Leaky ReLU|§4.3|\n|GAE|Off|§4.3|\n|PopArt|Off|§4.3|\n|Param sharing|On (homogeneous), Off (heterogeneous)|§4.3|\n|Exploration|High (MAPPO/HAPPO), Moderate (MADDPG)|§4.3|\n\nWe select hyperparameters that are commonly used in existing codebases, including the codebase of MAPPO [1] and HAPPO [2]. These codebases includes a set carefully designed hyperparameters that works well for MPE, SMAC and MAMujoco, which varies in different tasks. Among all hyperparameters in these codebase, we conduct a pilot study to select hyperparameters that works well in most of our real-world tasks, and select other hyperparameters as alternate solutions.\n\n[1] The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games. NeurIPS 2022.\n\n[2] Heterogeneous-agent reinforcement learning. JMLR 2024.\n\nFor interaction effects, we conduct this anaysis to check if the hyperparameters are still effective when combining with each other. we will move some details in Appendix B.7 to main paper for increased clarity. \n\n> Q7: Have the authors considered whether noise effects accumulate in later-collected samples as training progresses?\n\nThe training progress is done in ideal simulation environment, mimicing current MARL training paradigm so we do not add noise in sample collection. Adding noise during training is often framed as backdoor or poisoning attacks, which follow a different theoretical paradigm and is out of the scope of our paper.\n\n> Q8: Would increasing random seeds affect statistical confidence?\n\nPlease allow us to clarify that all our results are averaged across multiple runs, tasks and algorithms. Thus, while we tested for 5 random seeds, the sample sizes are more than 100 in Section 4.2 and 4.3. Thus, while we admit more random seeds are always better, we believe the number of samples are sufficient to make statistically significant claims.\n\n> Q9: What are the computational requirements for reproducing the main findings?\n\nOur full experiment takes ~230K GPU hours, measured in GTX 4090. For better reproducability, we will share the raw reward score under each configurations for further analysis.\n\n> Q10: repeated references to the code link and several instances of redundant information.\n\nThanks for the suggestion. We will reduce these redundant information in our final revision."}, {"review_id": "7KTFPPE8fh", "reviewer": "Reviewer_GvL9", "summary": "The paper studies the robustness and resilience of policy learning for Multi-agent reinforcement learning under uncertainties. It considers three types MARL algorithms: MAPPO, HAPPO and MADDPG, and investigates the effect of typical hyper-parameters on the policy robustness and resilience. The paper evaluates different combinations of hyper-parameters across a wide range of benchmark tasks. The conclusions show that the relationship between cooperation and robustness/resilience may not be strongly corrected, and the hyper-parameter tuning is crucial for policy learning under different uncertainties.", "questions": "These questions do not affect my evaluation score. \n1. “Use a higher learning rate (LR) for the critic.” It’s a bit unclear if this high learning rate is caused by the insufficient number of mini-epochs the critic is optimized with. Other factors could also affect this conclusion, e.g., the batch size. \n2. “As for PopArt, its normalization benefits may be unnecessary when reward scales remain stable, leading to inconsistent effectiveness across different tasks.” It’s a bit surprising to see PopArt leads to inconsistent effectiveness when the reward scales remain stable. I was wondering when PopArt was used whether the advantage is also scaled according in gradient estimate. \n3. the conclusion that MADDPG is not benefitting from the increased exploration is also a bit surprising to me. MADDPG has different exploration schemes than MAPPO and HAPPO. This direction comparison in terms of entropy bonus between these methods may not stand. \n4. in the definition of robustness and resilience, the inner expectations are both with $u$, which refers to some uncertainty sampled from a set. However, in the resilience description “the system evolves without additional uncertainties”, not sure if there is a typo in the inner expectation for resilience.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strength\n**Originality**: the paper clearly defines the robustness and resilience for policy performance in multi-agent reinforcement learning. It considers both general and algorithm-specific hyperparameters and their impact on the policy robustness and resilience. \n**Quality & Significance**: the evaluation is conducted on four real-world environments, with the consideration of policy transfer to physical robots and a wide range of distinct uncertainties in observations, actions, and environments. The evaluation dimensions cover the correlation among cooperation, robustness and resilience, generalization of robustness across different uncertainties, and the effective hyper-parameter tuning. \n\n### Weakness\n**Significance**: the robustness and resilience considered in this paper are both defined over a pre-defined set of uncertainties, which might limit the scope of possible application scenarios. For example, the uncertainties might switch from one type to another and might yield non-stationary property.", "was_revised": true, "final_justification": "my main concern remains as the robustness and resilience considered in this paper are defined within a limited scope. some other implementation-related issues seem not answered directly as the authors just referred them as the default setups by MAPPO. I think the paper is around the borderline.", "rebuttal": "We thank Reviewer GvL9 for appreciating the novelty, large-scale empirical study, and significance of our findings. A detailed response is provided below.\n\n> Q1: the robustness and resilience considered in this paper are both defined over a pre-defined set of uncertainties, which might limit the scope of possible application scenarios. For example, the uncertainties might switch from one type to another and might yield non-stationary property.\n\nThanks for the observation. We answer this from three perspectives. First, we control the variable of noise to limit the amount of randomness faced in our analysis. The variable control is significant for our statistical tests since only one variable is changed, which ease our effort to study the impact of each noise/algorithm/tasks/hyperparameters.\n\nSecond, using mixed type of uncertainties may introduce large variations in experiments. For example, if we select some type of uncertainties to appear after another uncertainty at some time, we must control the time, type and magnitude of uncertainty we applied, which makes the evaluation computationally intractable and introduce large variations during evaluations.\n\nThird, existing robust MARL literature use bounded noises to evaluate their performance. We select the current setting to connect with current literatures.\n\nWe fully agree that mixing these uncertainties is important for comprehensive evaluation. We will add this as a function in our codebase to support real-world evaluation of these non-stationary noises.\n\n\n> Q2: It’s a bit unclear if higher LR of critic is caused by the insufficient number of mini-epochs the critic is optimized with. Other factors could also affect this conclusion, e.g., the batch size.\n\nOur code is based on the widely used codebase of MAPPO [1], where actors and critics are optimized with the same mini-epochs. The batch size also inherits the default of MAPPO, which is the same for actors and critics. Thus, higher LR of critic indeed helps, and suggests a potential optimization direction for MARL.\n\n[1] The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games. NeurIPS 2022.\n\n\n> Q3: It’s a bit surprising to see PopArt leads to inconsistent effectiveness when the reward scales remain stable. I was wondering when PopArt was used whether the advantage is also scaled according in gradient estimate.\n\nOur implementation again follows the official implementation of MAPPO. They use PopArt to normalize the learning process of the critic. When computing the advantage for the actor, the value prediction is first denormalized by the inverse PopArt transform and then the advantage is Z-score normalized for policy gradient.\n\n\n> Q4: MADDPG is not benefitting from the increased exploration is also a bit surprising to me. MADDPG has different exploration schemes than MAPPO and HAPPO. This direction comparison in terms of entropy bonus between these methods may not stand.\n\n\nThanks for the insightful comment. We agree that MADDPG’s exploration scheme differs from the entropy-based exploration used in MAPPO and HAPPO, and that direct comparisons must be interpreted with care.\n\nIn MAPPO/HAPPO, the entropy bonus is a soft constraint, which encourage explorration at the beginning, while the policy can still converge to a relatively deterministic policy which reach optimal cooperation.\n\nIn contrast, the exploratory noise of MADDPG is additive action noise, which is enforced throughout the training process and does not decay naturally. This can lead to suboptimal behavior, especially when large noise values perturb the agent into poor or unstable regions of the state space. Empirically, we find that increasing MADDPG’s action noise did not lead to better performance. We believe this is because excessive noise degraded the quality of the agent’s behavior, making it harder to maintain meaningful strategies.\n\nWe will add the analyzation of on-policy and off-policy exploration scheme above in final revised version for further analyzation.\n\n> Q5: In the resilience description “the system evolves without additional uncertainties”, not sure if there is a typo in the inner expectation for resilience.\n\nThanks for pointing this mistake. The inner expectation should not additionally condition on u. We will correct this in final revision."}, {"review_id": "RmegjQD8uM", "reviewer": "Reviewer_PfFs", "summary": "This paper presents an extensive empirical study to motivate the difference between robustness and resilience in cooperative multi-agent reinforcement learning.\n\nFrom the introduction, both robustness and resilience are defined as the ability to cope with uncertainties.\nThe difference lies in the level of uncertainty faced by the agents, as the need for robustness becomes a need for resilience when uncertainty exceeds a certain threshold.\nA first claim is that MARL literature uses both terms interchangeably, while control theory distinguishes between them.\nThe goal of the empirical study is to understand how hyperparameter tuning can address these two situations when training agents solely in their cooperative task without uncertainty.\nThe contributions are:\n1: Cooperative MARL training leads to robustness and resilience if the uncertainty is small.\n2: Being robust to one type of uncertainty does not imply being robust to another (e.g., uncertainty in obs vs in action)\n3: Tuning the hyperparameter can improve robustness and resilience.\n\nThe classical Dec-POMDP is defined as a framework for performing cooperative tasks within an infinite horizon.\nThree objectives are then defined:\nCooperation objective: classical expected sum of discounted rewards from an initial state with an infinite horizon\nRobustness objective: Cooperative objective under uncertainty in obs, action, or environment.\nResilience objective: Cooperative without robustness' uncertainty, but with different initial state distributions.\nIt is implicit that Dec-POMDP components change when the evaluated objective is changed.\nAgents are trained to maximise cooperation and are evaluated within Robustness and Resilience configurations.\n\nThe experimental procedure is thoroughly defined, either in the paper or in the appendix for additional information, including algorithms (3 popular CTDE algorithms), values of hyperparameters studied, real-world environments tested, types of uncertainties tested, and a complete code base is released.\n\nResults analysed offer conclusions:\nTraining for cooperation can deal with small uncertainty. \nAlgorithms are distinguished based on their robustness and resilience against different types of uncertainty.\nDealing with one uncertainty does not imply the algorithm can deal with different ones.\nSome hyperparameters are better to tune than others, and sometimes it is not the ones that are claimed in the literature to be the most effective.\nHyperparameter tuning can improve all three objectives at once.\n\nRelated work is finally presented, focusing on robustness and hyperparameter tuning in RL/MARL.\nSome resilience-related works are mentioned with the definition of the resilience objective but not in this section.\n\nThe conclusion summarizes the claims and findings.", "questions": "1. From the introduction and Figure 2, resilience seems to evaluate the capacity of agents to recover from an unknown state after many noisy transitions, leading agents in unknown, or less explored, states.\nDon't you think it is counterintuitive to define resilience as agents evolving in an environment with a new initial state distribution?\nI guess that it makes sense formally, but it does not help the intuition behind.\nMaybe you should emphasize that cooperation, robustness, and resilience configurations can be thought of as new episodes and not a single evolving one.\nWhat about finite-time horizons?\n\n2. Aren't resilience and generalisation the same?\nAside from RL and control, in machine learning, generalization refers to the model's ability to handle new data.\nThis seems to be really close, if not the same, as the resilience definition that is to have a good policy in new states.\nThis can also be said from robustness, tho, with more nuance.\nI feel a statement on generalisation vs resilience can be interesting.\nEspecially when considering other methods than hyperparameter tuning to deal with it. I am thinking about regularisation from classical ML or all the methods in sim2real works.\n\n3. About the hyperparameter. It is always difficult to choose them and their values. Maybe you can comment on the default choices and why you selected them. How did you choose the other values? Why not change the number of layers instead of hidden sizes? In TD-step, why 5 and 25 to compare with 20? Why not 5-20-35 or 15-20-25?\n\n4. How would you define/compute the threshold in uncertainty when robustness becomes resilience?\n\n5. It would be beneficial to include insights on how MADDPG and MAPPO/HAPPO behave differently facing different types of uncertainty.\nDon't you think that simply the fact that the former has a discrete policy while others have a stochastic policy is the answer?\n\n3. Why not train agents in robust and resilient configurations to investigate whether agents can learn them directly? \nThis could also improve the claim that being robust for one type of uncertainty does not imply being robust to all, and to improve all claims anyway.", "limitations": "In an empirical study, it is always possible to test more parameters/configurations, but I don't think it is a limitation in this paper.\n\nTo me, there is one limitation in the definition of resilience, how it relates to generalisation and the ambiguity between the uncertainty threshold and the new initial state distribution.\n\nThe main limitation is that hyperparameter tuning should not be the single or best option. I feel this is the message of the paper. There are many ways we can modify simulators or methods to address uncertainty. While some works addressing uncertainties with RL are clearly identified, the message that hyperparameter tuning on training agents for the task without uncertainty is the single solution to deal with uncertainty should be nuanced. It could be interesting to compare hyperparameter tuning for robustness without uncertainty at training against training agents with default hyperparameters, but in an environment with uncertainties.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "In general, I would say that the quality of the study is higher than the impact of the claims.\n\nQuality:\n\n(+) The methods and parameters studied are well aligned with the state of the art.\n\n(+) The empirical study and the details provided are of high quality.\n\n(+) Technically, everything makes a lot of sense. \n\n(-) A lot of math is defined to illustrate the study, but perhaps some developments, or an inequality between the three objectives, could improve the understanding.\n\nClarity:\n\n(+) The paper is well written, and everything is clear.\n\n(+) There is a huge amount of information in the appendix, making the paper \"standalone\", which is really appreciated.\n\n(-) Robustness and resilience are first distinguished by a threshold of uncertainty. Afterwards, they are mathematically defined as different perturbations (noise on spaces vs noise on initial state distribution). The implicit idea is that agents tend to end up in unseen states due to high uncertainties, but there are ambiguities between these two definitions. The difference between resilience and robustness (lines 111-119) could be made earlier. Appendix B2 helps a lot (there is a mismatch between some paragraphs in B2, non-robust but resilient is the first one described) (suggestion: I feel instead of repeating lines 100-103, you can save some space to explain this)\n\n(suggestion: I think it would be nice to group all the related works (l 103-107) in Section 5?\n\nSignificance:\n\n(+) Such a study is really needed in the community, it seems the first of its kind in cooperative MARL.\n\n(+) Using real-world environments for the study is a plus!\n\n(+) The study advances the field towards deploying MARL.\n\n(-) Hyperparameter tuning is presented as a single approach to address these problems, although others exist.\n\nOriginality:\n\n(+) This study provides an additional push toward deploying RL in real-world applications.\n\n(+) It is original to classify the effect of uncertainty based on the threshold.\n\n(-) Resilience and generalisation seem to be highly related, but the latter is not mentioned.", "was_revised": true, "final_justification": "The quality of the study is higher than the theoretical justifications.", "rebuttal": "We thank Reviewer PfFs for appreciating the high quality, experiment design, writing and significance of findings of our paper. A detailed response is provided below.\n\n> Q1: Too many math included. Some developments, or an inequality between the three objectives, could improve the understanding.\n\nWe will increase the clarity of these concepts by using power grid as an example:\n\nCooperation describes optimal behavior under nominal conditions. For instance, on a typical day, the power grid functions in a coordinated and efficient manner, supplying energy smoothly across all nodes. This corresponds to high expected reward under ideal settings.\n\nRobustness refers to the ability to maintain performance during a disturbance. Suppose an earthquake causes sudden shifts in energy demand or damages part of the grid. A robust power grid can tolerate such perturbations, maintaining basic functionality despite degraded performance. This corresponds to a drop in reward, but without total failure.\n\nResilience captures the system’s ability to recover after the disturbance. Once the earthquake ends and demand returns to normal, the grid must recover from a perturbed and potentially degraded state. A resilient system gradually restores optimal performance, resulting in increasing reward as it recovers.\n\nRegarding the equations, we acknowledge that too much math can hinder readability. However, since resilience has not been formally defined in prior literature, we believe the mathematical clarity is necessary, but will additionally add the example above to increase readability, such that readers can understand the concepts without relying solely on equations.\n\n\n> Q2: Ambiguities between these two definitions of robustness and resilience. The difference between resilience and robustness (lines 111-119) could be made earlier, group all the related works (l 103-107) in Section 5?.\n\nThanks for your helpful suggestion. To reduce ambiguity, we will state the difference between resilience and robustness at the beginning of Section 2, and group all related works (Line 103-107) in Section 5 as suggested.\n\n> Q3: Resilience and generalisation seem to be highly related, but the latter is not mentioned. Aren't resilience and generalisation the same?\n\nThank you for raising this important point. We agree that resilience and generalization are closely related concepts, but we deliberately distinguish them to highlight their different focuses and implications for algorithm design.\n\nIn our work, resilience refers specifically to the capability of an agent to recover performance after encountering unexpected uncertainties. We formalize this as performance under a new state distribution $s^u$, which represents the state of the system after an external perturbation has occurred. These post-uncertainty states are often far from nominal initial states and are not under the control of the agent.\n\nIn contrast, generalization in RL typically involves evaluating how well a policy performs when trained and tested on different initial states or environment dynamics. The variations are usually known or deliberately randomized (e.g., random starts or human-initialized states as discussed in [1]). The key distinction lies in the nature of the initial state distribution:\n\nIn generalization, the agent starts from a known distribution of initial states, potentially with limited or no exposure to unseen variations.\n\nIn resilience, the agent starts from an unknown, perturbed state $s^u$ induced by adversarial or stochastic uncertainties, often in parts of the state space rarely encountered during training.\n\nThus, we view resilience as a special, practically motivated subset of generalization that focuses on the real-world challenges of recovery and generalization under unknown perturbation budgets. Moreover, because the perturbed state $s^u$ is the result of certain perturbations, it opens up new algorithmic possibilities that go beyond classical generalization, such as designing policies that can actively adapt or recover given a disturbed state. We will revise the manuscript to include this discussion and clarify how resilience relates to, but differs from, standard notions of generalization.\n\n[1] Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents, JAIR 2018.\n\n> Q4: It is counterintuitive to define resilience as agents evolving in an environment with a new initial state distribution. It makes sense formally, but it does not help the intuition behind. \n\nTo reconcile the difference between formal definition and intuition, we revise resilience as starting at time t, after encountering uncertainties. Thus, the task starts at an arbitrary time t with an inferior state for recovery. The revised equation is: $J^{\\text{resilience}}(\\pi) = \\mathbb{E}\\_{u \\sim \\mathcal{U}} [ \\mathbb{E}\\_{s^u \\sim \\rho^u} \\mathbb{E}\\_{\\pi} [ \\sum\\_{t}^\\infty \\gamma^t r\\_t | s\\_t = s^u ] ]$.\n\n> Q5: It is always difficult to choose hyperparameter and their values. What are the default choices and why you selected them?\n\nCurrently, there do not exist a universal hyperparameter for MARL algorithms. In our paper, we select parameters that are commonly used in existing codebases, including the codebase of MAPPO [1] and HAPPO [2]. These codebases includes a set carefully designed hyperparameters that works well for MPE, SMAC and MAMujoco, which varies in different tasks. Among all hyperparameters in these codebase, we conduct a pilot study to select hyperparameters that works well in most of our real-world tasks, and select other hyperparameters as alternate solutions.\n\n[1] The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games. NeurIPS 2022.\n\n[2] Heterogeneous-agent reinforcement learning. JMLR 2024.\n\n> Q6: How would you define/compute the threshold in uncertainty when robustness becomes resilience?\n\nIn fact, such threshold can be hard to define mathematically, but more of an application perspective. Imagine a power grid in urban areas, in common days, there may be small perturbations to the power grid, such as sensor errors or one user request more power than usual. In such cases with small perturbations, the power grid should be robust, such that it can serve all users despite these small variations. \n\nIn rare events such as earthquake, the power grid may collapse physically and receives zero request if facing blackout, or high request if facing a short-circuit. In such cases, after engineers have fixed the power grid, it is crucial for the system to continue functioning well after recovery.\n\n> Q7: insights on how MADDPG and MAPPO/HAPPO behave differently facing different types of uncertainty. Maybe discrete/stochastic policy is the answer?\n\nPlease allow us to explain this in detail. If using discrete or stochastic policy is the answer, MAPPO and HAPPO should be robust to action uncertainties by using stochastic policies. However, our finding shows MADDPG is more robust to action uncertainties than MAPPO/HAPPO. To explain this, we empirically find the exploratory noise of MAPPO and HAPPO converge to a small value when training is over, bringing limited explorations in action space, and more deterministic policies. In contrast, in MADDPG, the exploratory noise in action space remains constant. Since the policy is constantly exploring in action space, it becomes more robust against action uncertainties.\n\n> Q8: Alternative approaches than hyperparameter tuning, like train agents in robust and resilient configurations to investigate whether agents can learn them directly? This could also improve the claim that being robust for one type of uncertainty does not imply being robust to all, and to improve all claims anyway.\n\nWe fully admit that hyperparameter tuning is not the only solution to robust and resilient MARL. The problem lies in robust MARL itself. \n\nFirst, training MARL against uncertainties is commonly referred as robust MARL, which is a complex process that requires high expertise. Existing robust MARL algorithms are specific to their own type of perturbation and their backbone MARL algorithms. For example, EIR-MAPPO [11] works only on MAPPO backbone with action uncertainty applied to one agent, M3DDPG [24] works for MADDPG backbone only with action uncertainty applied to all agents. Similarly, in observation uncertainty, noise can be applied to each agent [8] or all agents [9]. Moreoverr, as reported by EIR-MAPPO [11], since their method is designed against action uncertainties, they are not robust to obervation uncertainties. To the best of our knowledge, there do not exist a single work on robust MARL that (1) can be applied to any backbone, (2) works for perturbations on one, several or all agents and (3) works for perturbations over observation, environment and actions.\n\nThe method of training MARL against uncertainties is most similar to the work of [1], which use curriculum learning to train robust MARL against diverse uncertainties in action, observation and rewards. However, the authors of this paper found training MARL against diverse uncertainties is hard, can often harm convergence, and can be robust against at most two uncertainties. We additionally conducted a prelinminary study ourselves, and found the same difficulty on convergence.\n\n[1] Robustness to Multi-Modal Environment Uncertainty in MARL using Curriculum Learning. NeurIPS 2023.\n\nSecond, we have compared hyperparameter tuning on ERNIE (Fig 9), a general-purposed robust MARL that turns uncertainty into regularization terms, which we find to have good convergence properties empirically. While ERNIE is itself a robust MARL method, performing hyperparameter tuning further enhance its performance. Thus, we deem hyperparameter tuning and robust MARL methods are orthorgonal ways to achieve robust and resilient MARL."}]

Author rebuttal:

Meta-review: In this submission, the authors present a large-scale empirical study to evaluate cooperation, robustness, and resilience in multi-agent reinforcement learning (MARL) methods. The authors use 4 environments to perform a large-scale empirical study comprising over 82,620 experiments whereby they vary certain aspects of the experiments, such as the hyper parameters and model designs. The analysis of these experiments demonstrates that the relationship between multi-agent cooperation and robustness/resilience might not be strongly corrected, and that the hyper-parameter tuning remains key.

The reviewers highlighted several strengths of the paper. This includes the comprehensive and rigorous empirical evaluations and statistical analysis on a wide range of experiments. They also note that the manuscript provides a clear and practical distinctions between robustness and resilience that offers a practical formal metrics for future work. All reviewers agree that the paper is well-written and easy to follow. During the rebuttal, the authors addressed many of the reviewers concerns. While some reservations remain around limited methodological novelty, I believe that such a detailed and rigorous study would be highly appreciated by the multi-agent RL community at NeurIPS. I therefore recommend it for acceptance. I’d suggest that the authors use the reviewers' comments to improve the manuscript for the camera-ready submission.
Decision: Accept (poster)