Title: Empirical Study on Robustness and Resilience in Cooperative Multi-Agent Reinforcement Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Multi-Agent Reinforcement Learning

Trustworthy AI
Abstract: In cooperative Multi-Agent Reinforcement Learning (MARL), it is a common practice to tune hyperparameters in ideal simulated environments to maximize cooperative performance. However, policies tuned for cooperation often fail to maintain robustness and resilience under real-world uncertainties. Building trustworthy MARL systems requires a deep understanding of \emph{robustness}, which ensures stability under uncertainties, and \emph{resilience}, the ability to recover from disruptions—a concept extensively studied in control systems but largely overlooked in MARL. In this paper, we present a large-scale empirical study comprising over 82,620 experiments to evaluate cooperation, robustness, and resilience in MARL across 4 real-world environments, 13 uncertainty types, and 15 hyperparameters. Our key findings are: (1) Under mild uncertainty, optimizing cooperation improves robustness and resilience, but this link weakens as perturbations intensify. Robustness and resilience also varies by algorithm and uncertainty type. (2) Robustness and resilience do not generalize across uncertainty modalities or agent scopes: policies robust to action noise for all agents may fail under observation noise on a single agent. (3) Hyperparameter tuning is critical for trustworthy MARL: surprisingly, standard practices like parameter sharing, GAE, and PopArt can hurt robustness, while early stopping, high critic learning rates, and Leaky ReLU consistently help. By optimizing hyperparameters only, we observe substantial improvement in cooperation, robustness and resilience across all MARL backbones, with the phenomenon also generalizing to robust MARL methods across these backbones.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents an extensive empirical study to motivate the difference between robustness and resilience in cooperative multi-agent reinforcement learning.\n\nFrom the introduction, both robustness and resilience are defined as the ability to cope with uncertainties.\nThe difference lies in the level of uncertainty faced by the agents, as the need for robustness becomes a need for resilience when uncertainty exceeds a certain threshold.\nA first claim is that MARL literature uses both terms interchangeably, while control theory distinguishes between them.\nThe goal of the empirical study is to understand how hyperparameter tuning can address these two situations when training agents solely in their cooperative task without uncertainty.\nThe contributions are:\n1: Cooperative MARL training leads to robustness and resilience if the uncertainty is small.\n2: Being robust to one type of uncertainty does not imply being robust to another (e.g., uncertainty in obs vs in action)\n3: Tuning the hyperparameter can improve robustness and resilience.\n\nThe classical Dec-POMDP is defined as a framework for performing cooperative tasks within an infinite horizon.\nThree objectives are then defined:\nCooperation objective: classical expected sum of discounted rewards from an initial state with an infinite horizon\nRobustness objective: Cooperative objective under uncertainty in obs, action, or environment.\nResilience objective: Cooperative without robustness' uncertainty, but with different initial state distributions.\nIt is implicit that Dec-POMDP components change when the evaluated objective is changed.\nAgents are trained to maximise cooperation and are evaluated within Robustness and Resilience configurations.\n\nThe experimental procedure is thoroughly defined, either in the paper or in the appendix for additional information, including algorithms (3 popular CTDE algorithms), values of hyperparameters studied, real-world environments tested, types of uncertainties tested, and a complete code base is released.\n\nResults analysed offer conclusions:\nTraining for cooperation can deal with small uncertainty. \nAlgorithms are distinguished based on their robustness and resilience against different types of uncertainty.\nDealing with one uncertainty does not imply the algorithm can deal with different ones.\nSome hyperparameters are better to tune than others, and sometimes it is not the ones that are claimed in the literature to be the most effective.\nHyperparameter tuning can improve all three objectives at once.\n\nRelated work is finally presented, focusing on robustness and hyperparameter tuning in RL/MARL.\nSome resilience-related works are mentioned with the definition of the resilience objective but not in this section.\n\nThe conclusion summarizes the claims and findings.\n\nSTRENGTHS AND WEAKNESSES:\nIn general, I would say that the quality of the study is higher than the impact of the claims.\n\nQuality:\n\n(+) The methods and parameters studied are well aligned with the state of the art.\n\n(+) The empirical study and the details provided are of high quality.\n\n(+) Technically, everything makes a lot of sense. \n\n(-) A lot of math is defined to illustrate the study, but perhaps some developments, or an inequality between the three objectives, could improve the understanding.\n\nClarity:\n\n(+) The paper is well written, and everything is clear.\n\n(+) There is a huge amount of information in the appendix, making the paper \"standalone\", which is really appreciated.\n\n(-) Robustness and resilience are first distinguished by a threshold of uncertainty. Afterwards, they are mathematically defined as different perturbations (noise on spaces vs noise on initial state distribution). The implicit idea is that agents tend to end up in unseen states due to high uncertainties, but there are ambiguities between these two definitions. The difference between resilience and robustness (lines 111-119) could be made earlier. Appendix B2 helps a lot (there is a mismatch between some paragraphs in B2, non-robust but resilient is the first one described) (suggestion: I feel instead of repeating lines 100-103, you can save some space to explain this)\n\n(suggestion: I think it would be nice to group all the related works (l 103-107) in Section 5?\n\nSignificance:\n\n(+) Such a study is really needed in the community, it seems the first of its kind in cooperative MARL.\n\n(+) Using real-world environments for the study is a plus!\n\n(+) The study advances the field towards deploying MARL.\n\n(-) Hyperparameter tuning is presented as a single approach to address these problems, although others exist.\n\nOriginality:\n\n(+) This study provides an additional push toward deploying RL in real-world applications.\n\n(+) It is original to classify the effect of uncertainty based on the threshold.\n\n(-) Resilience and generalisation seem to be highly related, but the latter is not mentioned.\n\nQUESTIONS:\n1. From the introduction and Figure 2, resilience seems to evaluate the capacity of agents to recover from an unknown state after many noisy transitions, leading agents in unknown, or less explored, states.\nDon't you think it is counterintuitive to define resilience as agents evolving in an environment with a new initial state distribution?\nI guess that it makes sense formally, but it does not help the intuition behind.\nMaybe you should emphasize that cooperation, robustness, and resilience configurations can be thought of as new episodes and not a single evolving one.\nWhat about finite-time horizons?\n\n2. Aren't resilience and generalisation the same?\nAside from RL and control, in machine learning, generalization refers to the model's ability to handle new data.\nThis seems to be really close, if not the same, as the resilience definition that is to have a good policy in new states.\nThis can also be said from robustness, tho, with more nuance.\nI feel a statement on generalisation vs resilience can be interesting.\nEspecially when considering other methods than hyperparameter tuning to deal with it. I am thinking about regularisation from classical ML or all the methods in sim2real works.\n\n3. About the hyperparameter. It is always difficult to choose them and their values. Maybe you can comment on the default choices and why you selected them. How did you choose the other values? Why not change the number of layers instead of hidden sizes? In TD-step, why 5 and 25 to compare with 20? Why not 5-20-35 or 15-20-25?\n\n4. How would you define/compute the threshold in uncertainty when robustness becomes resilience?\n\n5. It would be beneficial to include insights on how MADDPG and MAPPO/HAPPO behave differently facing different types of uncertainty.\nDon't you think that simply the fact that the former has a discrete policy while others have a stochastic policy is the answer?\n\n3. Why not train agents in robust and resilient configurations to investigate whether agents can learn them directly? \nThis could also improve the claim that being robust for one type of uncertainty does not imply being robust to all, and to improve all claims anyway.\n\nLIMITATIONS:\nIn an empirical study, it is always possible to test more parameters/configurations, but I don't think it is a limitation in this paper.\n\nTo me, there is one limitation in the definition of resilience, how it relates to generalisation and the ambiguity between the uncertainty threshold and the new initial state distribution.\n\nThe main limitation is that hyperparameter tuning should not be the single or best option. I feel this is the message of the paper. There are many ways we can modify simulators or methods to address uncertainty. While some works addressing uncertainties with RL are clearly identified, the message that hyperparameter tuning on training agents for the task without uncertainty is the single solution to deal with uncertainty should be nuanced. It could be interesting to compare hyperparameter tuning for robustness without uncertainty at training against training agents with default hyperparameters, but in an environment with uncertainties.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the robustness and resilience of policy learning for Multi-agent reinforcement learning under uncertainties. It considers three types MARL algorithms: MAPPO, HAPPO and MADDPG, and investigates the effect of typical hyper-parameters on the policy robustness and resilience. The paper evaluates different combinations of hyper-parameters across a wide range of benchmark tasks. The conclusions show that the relationship between cooperation and robustness/resilience may not be strongly corrected, and the hyper-parameter tuning is crucial for policy learning under different uncertainties.\n\nSTRENGTHS AND WEAKNESSES:\n### Strength\n**Originality**: the paper clearly defines the robustness and resilience for policy performance in multi-agent reinforcement learning. It considers both general and algorithm-specific hyperparameters and their impact on the policy robustness and resilience. \n**Quality & Significance**: the evaluation is conducted on four real-world environments, with the consideration of policy transfer to physical robots and a wide range of distinct uncertainties in observations, actions, and environments. The evaluation dimensions cover the correlation among cooperation, robustness and resilience, generalization of robustness across different uncertainties, and the effective hyper-parameter tuning. \n\n### Weakness\n**Significance**: the robustness and resilience considered in this paper are both defined over a pre-defined set of uncertainties, which might limit the scope of possible application scenarios. For example, the uncertainties might switch from one type to another and might yield non-stationary property.\n\nQUESTIONS:\nThese questions do not affect my evaluation score. \n1. “Use a higher learning rate (LR) for the critic.” It’s a bit unclear if this high learning rate is caused by the insufficient number of mini-epochs the critic is optimized with. Other factors could also affect this conclusion, e.g., the batch size. \n2. “As for PopArt, its normalization benefits may be unnecessary when reward scales remain stable, leading to inconsistent effectiveness across different tasks.” It’s a bit surprising to see PopArt leads to inconsistent effectiveness when the reward scales remain stable. I was wondering when PopArt was used whether the advantage is also scaled according in gradient estimate. \n3. the conclusion that MADDPG is not benefitting from the increased exploration is also a bit surprising to me. MADDPG has different exploration schemes than MAPPO and HAPPO. This direction comparison in terms of entropy bonus between these methods may not stand. \n4. in the definition of robustness and resilience, the inner expectations are both with $u$, which refers to some uncertainty sampled from a set. However, in the resilience description “the system evolves without additional uncertainties”, not sure if there is a typo in the inner expectation for resilience.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a large-scale and comprehensive empirical study on robustness and resilience in cooperative MARL. The authors clearly distinguish and formalize robustness (the ability to withstand uncertainties) and resilience (the ability to recover from severe disruptions). Through 82,620 experiments across four real-world-inspired environments, the study systematically evaluates three widely used continuous-control MARL algorithms (MAPPO, MADDPG, HAPPO) under 13 types of uncertainty and 15 hyperparameters. The key findings are: (1) while optimizing for cooperation can improve robustness and resilience under mild uncertainty, this correlation weakens as perturbations intensify; (2) robustness and resilience do not generalize across uncertainty modalities or agent scopes; and (3) hyperparameter tuning can have a greater impact on these properties than the choice of MARL algorithm, with some standard practices (such as parameter sharing, GAE, and PopArt) sometimes harming robustness, while strategies like early stopping, higher critic learning rates, and Leaky ReLU are consistently beneficial.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**: The greatest strength of this paper lies in the scale, rigor, and comprehensiveness of its empirical investigation—spanning over 82,000 experiments across four real-world-inspired environments, 13 types of uncertainty, and 15 hyperparameters. This enables robust, statistically meaningful conclusions and demonstrates a level of methodological thoroughness rarely seen in MARL research. The authors provide clear and rigorous distinctions between robustness and resilience, offering a practical framework and formal metrics that fill an important gap in the literature. Their systematic evaluation covers multiple algorithms and environments, with a modular codebase that enhances reproducibility and extensibility for practitioners. The study’s concrete findings, such as the outsized impact of hyperparameter tuning (often exceeding that of algorithmic choice), and the counterintuitive result that standard practices like parameter sharing, GAE, and PopArt can be detrimental under uncertainty, offer actionable guidance for the community.\n\n**Weaknesses**\n1. The paper does not introduce new algorithms, training paradigms, or theoretical guarantees. Its contributions primarily lie in systematic evaluation and formalization, which limits the originality and theoretical depth.\n2. The study centers on three mainstream MARL algorithms (MADDPG, MAPPO, HAPPO), and while these are representative, the results may not generalize to alternative MARL paradigms (e.g., value decomposition, graph-based, or distributional RL methods).\n3. Some conclusions may be specific to the chosen environments or implementation details, such as the negative effects of parameter sharing, GAE, and PopArt, or the universality of certain tuning “tricks”.\n4. There is limited presentation of raw numerical results, confidence intervals, or effect sizes in the main text; detailed per-task outcomes are relegated to the appendix, reducing transparency about the practical impact on individual tasks.\n5. The empirical comparison does not include more recent robust MARL methods (e.g., distributional RL, adversarial or opponent modeling).\n6. The large breadth of experiments and the dense presentation may overwhelm readers, and some methodological choices (such as rationale for selected hyperparameter ranges or interaction effects) could be discussed more clearly for accessibility.\n\nQUESTIONS:\n1. To what extent do the main findings generalize to other MARL algorithms, such as value decomposition or graph-based methods? Have the authors considered extending their analysis to these additional families?\n2. Many results are shown across four environments, but how confident are the authors that key observations would hold in other domains, such as additional robotics, control, or RL tasks (e.g., gym benchmarks)?\n3. Have similar sensitivities to hyperparameter tuning been observed in more recent robust MARL methods (e.g., distributional RL, opponent modeling, etc.)? Would direct empirical comparison with these advanced baselines alter the main conclusions?\n4. Could the authors provide further justification for the chosen hyperparameter ranges, and clarify whether the reported benefits and negative effects persist under different normalization or reward structures?\n5. Have the authors considered whether noise effects accumulate in later-collected samples as training progresses? Is there any empirical evidence or analysis regarding this potential issue?\n6. The study primarily uses 5 random seeds per experiment. Would increasing this number improve the statistical confidence of the results, especially for small effect sizes?\n7. Given the experimental scale, what are the computational requirements for reproducing the main findings? Are all tested environments and uncertainty settings easily accessible and reproducible via the released code?\n8. There are repeated references to the code link and several instances of redundant information. Would the authors consider streamlining these sections for clarity and conciseness?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores the importance of tuning hyper parameters in MARL systems with the goal of understanding their roles in cooperation, robustness and resilience. \n\nIt makes the clear distinction between that robustness is ability to deal with constant perturbations while resilience is the ability to recover from disruptions. \n\nThey test a combination of 82,620 different experiments. Varying hyper parameters, model designs and environments.\nThe statistical analysis identifies a few compelling ideas and correlations.\n\n1.\tOptimizing cooperation under mild uncertainty enhances both resilience and robustness, though this relationship diminishes as uncertainty grows.\n\n2.\tThe modality of the uncertainty affects different policies in different ways. For example, MAPPO and HAPPO are more robust to observation uncertainties and MADDPG is better with action uncertainty. (This posses the question of whether there is a model that can incorporate the strengths of both of these.)\n\n3.\tParameter sharing appears to reduce cooperation, robustness, and resilience, except for improvements in SMAC.\nAn interesting question to ask is why does parameter sharing help in SMAC but hinder performance in MPE and Multi-Agent Mujoco? Is this because the agent  roles and objectives are the same for some agents in SMAC but not in the others? (example, zealots have the same role and objective and will likely benefit from the same parameters, while a particle in MPE has a different objective to another particle and thus benefits from a different policy entirely.)\n\n4.\tGAE, and PopArt can hurt robustness, as they found in MPE and SMAC, but helps in Multi-agent Mujoco. They suggest that it’s because of the dense and stable reward structure. Does this not just mean it is a design choice we should be tuning if we know about the reward structure?\n\n5.\tThey find that early stopping helps. Suggesting saving a model at a point where it is optimal over cooperation, robustness and resilience is better than saving the final model in many cases. This begs the age old question, at what point should we be stopping?\n\n6.\tHigh critic learning rates (higher than actors) and using Leaky ReLU help with robustness. The critic updating faster than the actors is suggested to have more robust results, and Leaky ReLU is suggested to solve the dead neuron problem that arises with ReLU. Are there environments that this might not be the case?\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1.\tThe comprehensive statistical analysis on the different experiments are thorough and do highlight factors we should consider.\n\n2.\tThe work covers many different hyper parameters that can be chosen and shows reasons why one size fits all can’t be the solution for MARL (However I am not convinced that the correct conclusion is always drawn. The authors even find exceptions to their claims, eg parameter sharing being bad except in SMAC).\n\n3.\tThere is a code base that can help anyone tune their models. Which is useful since most studies don’t focus enough on the hyper parameter tuning.\n\n4.\tThe paper correctly distinguishes between robustness and resilience, a concept that many MARL works lack clarity on.\n\n5.\tThe insight that GAE and PopArt are probably only useful in high density reward and harmful at other times is useful.\n\nWeaknesses:\n\n1.\tThe paper generalizes claims about concepts, while directly giving a contradictory example. See the example with parameter sharing and SMAC, or GAE and PopArt with Mujoco. I believe there is likely a more principled manner to make these claims, since the exceptions aren't always explained.\n\n2.\tThe hyper parameter search appears to just be a sweep over a certain range, is there not a more principled manner to do this?\n\n3.\tMinor: Typically, the related work is stated at the start of the paper to help frame the work.\n\nQUESTIONS:\n1. Is there possibly a more principled approach to the hyper parameter search than just checking over a range? This feels like it would depend heavily on the environment and the model being checked.\n\n2. Are you sure your general statements about GAE, PopArt, etc... can be generalized? Especially considering the dense reward in Mujoco seemingly benefiting from it.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}