Title: Robust and Diverse Multi-Agent Learning via Rational Policy Gradient
Venue: NeurIPS 2025
Accepted: yes
Keywords: reinforcement learning

adversarial training

robustness

multi-agent learning

opponent shaping

general-sum
Abstract: Adversarial optimization algorithms that explicitly search for flaws in agents' policies have been successfully applied to finding robust and diverse policies in the context of multi-agent learning. However, the success of adversarial optimization has been largely limited to zero-sum settings because its naive application in cooperative settings leads to a critical failure mode: agents are irrationally incentivized to *self-sabotage*, blocking the completion of tasks and halting further learning. To address this, we introduce *Rationality-preserving Policy Optimization (RPO)*, a formalism for adversarial optimization that avoids self-sabotage by ensuring agents remain *rational*—that is, their policies are optimal with respect to some possible partner policy. To solve RPO, we develop *Rational Policy Gradient (RPG)*, which trains agents to maximize their own reward in a modified version of the original game in which we use *opponent shaping* techniques to optimize the adversarial objective. RPG enables us to extend a variety of existing adversarial optimization algorithms that, no longer subject to the limitations of self-sabotage, can find adversarial examples, improve robustness and adaptability, and learn diverse policies. We empirically validate that our approach achieves strong performance in several popular cooperative and general-sum environments. Our project page can be found at https://rational-policy-gradient.github.io.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, a method called RPO is proposed which is a formalism for adversarial optimization. The proposed method can avoid self-sabotage by ensuring agents remain rational. In detail, authors use opponent shaping techniques to optimize the adversarial objective. In summary， the contribution of this paper follows:\n(1) A novel algorithm, PRO, is proposed which can avoid self-sabotage by ensuring agents remain rational.\n(2) Authors empirically demonstrate the proposed algorithm and the proposed methos outperform existing baselines in popular cooperative environments.\n\nSTRENGTHS AND WEAKNESSES:\nThe strengths of this paper are:\n(1) The paper introduces a formalism called RPO that overcomes the issue of self-sabotage in any adversarial optimization algorithm.\n(2) The paper introduce a gradient-based deep learning algorithm called RPG.\n(3) The paper empirically demonstrates the method outperforms existing baselines.\nThe weaknesses of this paper are:\n(1) The experiments are too toy. Four primary environment such as matrix games, Overcooked, STORM,Hanabi are performed. These experiments are easy and only few agents. \n(2) Is it suitable for large number agents, or large scale Markov Game such as \"Go\" or \"MultiAgent System\",or StarCraft II?\n(3) There are some format error such as lacking \"conclusion\" at the last section of this paper.\n\nQUESTIONS:\nIn this paper, a formalism called RPO is proposed which can overcomes the issue of self-sabotage. However, there are some problems:\n (1) The experiments are too toy. Four primary environment such as matrix games, Overcooked, STORM,Hanabi are performed. These experiments are too easy and only few agents play. More complex experiments should be performed,such as Markov Games with more agents, or \"StarCraft II\"\n(2) The details of neural networks for policy network should be given and study that how neural network parameters affect the results.\n(3)  \"Conclusion\" should be added at the last section of this paper.\n(4) Only pseudocode is available. How to obtain the open-source code and reproduce the experimental results?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Rationality-preserving Policy Optimization (RPO), a new framework for adversarial optimization in multi-agent reinforcement learning that avoids the common failure mode of self-sabotage in cooperative or general-sum environments. Traditional adversarial training methods, successful in zero-sum games, perform poorly in cooperative settings because they incentivize agents to irrationally minimize each other's rewards, harming collective outcomes. To address this, the authors propose Rational Policy Gradient (RPG), a learning algorithm that trains agents to remain rational—optimal with respect to some teammate policy—while optimizing adversarial objectives through a novel “manipulator” agent structure. RPG enables robust, diverse, and self-consistent policy learning without explicit sabotage. Authors argue that RPG-based methods outperform existing baselines in robustness, diversity, and rationality across several empirical benchmark environments.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n1.  Strengthen the contribution to explicitly connect the proposed unified framework to existing adversarial optimization algorithms and clarify how different methods can be seen as special cases.\n2. The toy example is a clear and well-constructed illustration of the problem, and the insights drawn from it are compelling.\n3. The paper gives an insightful theoretical framing that distinguishes rationality constraints from conventional adversarial training.\n\n## Weaknesses\n1. Regarding Claim 3: Finding rational adversarial examples, I am not fully convinced. For example, Line 306 states that PAIRED-A-RPG is more effective than AP-RPG at identifying policy weaknesses—but it is unclear how this conclusion is drawn. Could this be due to the added noise acting as an overly strong perturbation? Instead, I suggest evaluating each policy using a learned \"worst-case but rational\" adversary, and then reporting results under both (a) this learned attack and (b) a no-attack baseline. This would offer a more rigorous assessment of policy robustness.\n\n2. I understand the constraints imposed by the page limit, but including a conclusion section is important for completeness and helps reinforce the key takeaways of the paper.\n\nQUESTIONS:\n1. Could the authors clarify what the variable n represents in Equation (4) and Algorithm 1?\n2. Are there any guidelines or intuitions for selecting the hyperparameters $\\alpha_1, \\alpha_2, \\alpha_3$ in Algorithm 1? For example, which one should be larger?\n3. There seems to be an inconsistency between the text and figure caption: Line 260 states that five policies are trained with different random seeds, while the caption of Figure 5 mentions two seeds. Please clarify the exact number of seeds used when evaluating robustness claims.\n4. In Figure 7, could you confirm whether the values under the adversarial attack columns refer to:\n(a) performance during adversarial training, or (b) performance after training without adversaries and testing with adversarial attacks? Additionally, as the content is tabular, it may be clearer to present it as a table rather than a figure.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

comments: SUMMARY:
This paper introduces a new framework, Rationality-Preserving Policy Optimization (RPO), to address the problem of self-sabotage in adversarial optimization in multi-agent reinforcement learning (MARL). The authors propose the Rational Policy Gradient (RPG) algorithm to enforce rationality constraints via the introduction of manipulator agents. They demonstrate the framework’s ability to improve robustness, avoid sabotage, and produce diverse policies in cooperative and general-sum environments.

STRENGTHS AND WEAKNESSES:
Strengths:

The paper addresses an important and understudied failure mode (self-sabotage) in adversarial MARL.

The introduction of manipulator agents and rationality constraints is novel and shows potential in expanding the utility of adversarial training beyond zero-sum games.

Experimental results are comprehensive and support the claimed improvements in robustness, diversity, and stability.

Weaknesses:

While the paper provides strong motivation and intuitive examples (e.g., Figure 2), the mathematical formulation of the core problem is vague. For instance, the exact nature of the adversarial optimization problem being generalized is not clearly defined (e.g., is it a Markov game? Is the environment shared or decentralized?). The rationality constraint (Definition 3.1) is stated informally as “πi ∈ BR(π−i′)”, but the space of allowable π−i′ is not precisely defined. Are these stationary policies, stochastic, or deterministic?

QUESTIONS:
Are manipulators optimized simultaneously with base agents, or is there an alternating schedule? Is there any stability concern when both influence each other via nested gradients?

LIMITATIONS:
No
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper shows that naïve adversarial optimisation in cooperative/general-sum MARL provokes irrational self-sabotage. It formalises Rationality-preserving Policy Optimisation (RPO) and instantiates it with Rational Policy Gradient (RPG), which inserts a “manipulator” agent and higher-order opponent-shaping gradients so each learned policy remains a best-response to some partner. RPG variants of AP, AT, PAIRED and AD are evaluated on matrix games, STORM, Overcooked and Hanabi, demonstrating (i) avoidance of sabotage, (ii) higher cross-play robustness, and (iii) discovery of genuine policy diversity.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength**\n\n- This paper is well motivated to analyze and handle self-sabotage in diverse teammate generation problem, which has been a hard problem to define and tackle.\n- Provides a principled optimisation formalism (RPO) plus a general recipe (RPG) that retrofits several prior algorithms. Conceptually this algorithm is generalizable to any reinforcement learning algorithms.\n- Extensive experiments across four benchmarks; shows superior robustness/diversity versus strong baselines (SP, AD, CoMeDi, etc.). Ablations (look-ahead depth, partner-play coeff.) give insight into stabilizing dynamics.\n\n**Weakness**\n\n- Experiments remain in small/2-agent domains, although the formulation is pretty generalized.\n- Theoretical analysis is insufficient:\n    - L123, BR is a policy right? BR = \\arg\\max U(\\pi_{i}, \\pi_{-i}) for rigorousness\n    - No clear explanation on O_i and L_dice in this paper, even though it is referred to related work.\n    - Theoretically unclear why the method can prevent self-sabotage, and the paper does not define self-sabotage formally. Significant work should put into section 3 or 4 to explain the design of the method.\n    - The method looks pretty like a fixed-size population version of CoMeDi combined with DiCE update. So I am not sure how the authors compare a fixed-size population method with an iterative-expanding population method to make sure it is a fair comparison.\n- A few more related works on adversarial diversity & self-sabotage:\n    - (This paper has a study on how to design adversarial diversity to reduce self-sabotage) Charakorn, Rujikorn, Poramate Manoonpong, and Nat Dilokthanakul. \"Generating diverse cooperative agents by learning incompatible policies.\" *The Eleventh International Conference on Learning Representations*. 2023.\n    - (This paper uses Lagrange constraint satisfaction to anneal the self-sabotage) Rahman, M., Cui, J., & Stone, P. (2024, March). Minimum coverage sets for training robust ad hoc teamwork agents. In *Proceedings of the AAAI Conference on Artificial Intelligence* (Vol. 38, No. 16, pp. 17523-17530).\n\nQUESTIONS:\nWhat is “self-sabotage”, can you give a formal definition / theorem? It was first documented in Cui et al 2023 but I think it was just described instead of officially formalized. Moreover, the self-sabotage issue described in this paper and Cui et al 2023 can be easily patched by masking out invalid actions with large negative values?\n\nLIMITATIONS:\n1. The sample variance grows with look-ahead steps. \n2. Experiments do not cover >2 agents or partially-observable continuous control, so generalizability is uncertain.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a new formalism and method to enhance the adversarial training of methods in multi-agent reinforcement learning.\nIt targets the specific cooperative settings, where adversarial agents learning to find flaws in others' strategies can end up with a self-sabotage policy.\nThis implies that the adversarial agent ends up with a policy that prevents the victim from finding another policy that will improve its result.\nThis is also referred to as non-rational behavior.\nThe goal of RPO is to constrain the maximized adversarial objective by ensuring that the adversarial policy is the best response to one of the victim’s possible responses.\nFrom this constraint, RPG is derived to improve different adversarial training algorithms by ensuring that no self-sabotage policy is achieved.\nThis new paradigm enables the development of more robust learning strategies.\n\nThe introduction reminds us of the context of adversarial training and its success in zero-sum games, inherent to the objective of each agent that benefits from the loss of others.\nThe problem of these approaches in a cooperative scenario is that agents usually learn a « self-sabotage » behavior, where the attacker agent does not cooperate anymore, and the victims are not able to find any policy leading to better performance.\nThis is claimed to be a non-rational behavior and the whole goal of the paper is to enforce rationality.\nA clear example is provided showing the difference between a rational adversarial attack and a self-sabotage policy.\nThe introduction briefly describes how this rationality constraint is incorporated into the learning framework by introducing manipulators, copies of agents trained using opponent-shaping methods, thereby introducing an additional learning objective.\nContributions are: a new formalism (RPO), a new gradient-based method to train agents (RPG), five new adversarial algorithms extending existing ones with RPG, and an empirical demonstration of the benefits of the new paradigm.\n\nPreliminaries define the stochastic game framework, where the objective of agents is to maximize the expected sum of discounted rewards dependent on the joint policy of all agents, and what constitutes a best response policy.\n\nRPO is then defined.\nA reminder of the adversarial optimization, its application in a simple cooperative game, and what is an irrational self-sabotage policy, and what is a rational one.\nThe Rational-preserving Policy Optimization is defined: for each agent i, its adversarial objective is maximized by pi_i while there must exist a joint-policy of the others, pi_-i, for which pi_i is the best response.\nIts implication for a simple cooperative matrix game is provided to help build intuition.\nAdversarial training RPO is thus a generalization of adversarial training in zero-sum games.\n\nRPG is then defined and provides a solution to incorporate the RPO constraint into existing learning algorithms.\nFor each policy pi_i, there is a manipulator pi^M_i.\npi_i maximise the classical objective U(pi_i, pi^M_-i), enforcing pi_i to be the best response to pi^M_-i.\npi^M_-i, in parallel, are trained to maximise the adversarial objective.\nHow gradients are computed, and in which order, is then described, followed by a statement on partner-play regularization.\nA representation of the algorithm is presented and then described.\nFinally, the application of RPG to five existing adversarial algorithms is described.\n\nExperiments demonstrate four claims clearly identified: RPG allows learning 1) diverse policies, 2) robust policies, 3) non-trivial adversarial examples, and 4) non-self-sabotaging policies.\nExperiments are conducted in four environments: matrix games, Overcooked, STORM, and Hanabi, all of which are 2-player cooperative games (?).\nEnvironments and parameters are detailed in the appendix.\nThe four claims are demonstrated through dedicated experiments and metrics, showcasing the power of RPG against six other methods, while applying RPG to four of them.\n\nThe paper finishes with a related work section, reminding the need to find a non-sabotage adversarial training method, claiming that to the best of their knowledge, RPG is the first solution using opponent-shaping to achieve adversarial training.\nA future work direction is proposed: to use a line of work improving opponent-shaping with a more effective method in RPG.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper provides a high-quality explanation of the problem and the solution, and appears to be the first to address it, supported by rigorously conducted experiments. However, some clarity enhancement could benefit the reader, and a consideration of limitations and extensions beyond two-player cooperative settings could be added.\n\nQuality:\n\n(+) The submission is technically sound, and all claims are well supported with examples, references, and experiments.\n\n(+) The paper is standalone, and all details are provided in the appendix.\n\n(+) Experiments are rigorous, and many details are provided to analyse experiments, supporting the claims.\n\nClarity:\n\n(+) The paper is overall clearly written\n\n(-) I believe the three joint policies (pi_adversary, pi_victim), (pi_i, pi_-i) , (pi_1,…, pi_m) are sometimes used interchangeably, making Section 3.2, among others, hard to follow. While I believe it clarifies section 3.1, using all three of them is overwhelming.\n\n\nSignificance:\n\n(+) This paper has a significant impact on the community, as it presents the first method to address the self-sabotage problem in adversarial training.\n\n(+) Many existing methods are mentioned, tested, and surpassed by RPG.\n\nOriginality:\n\n(+) Along with possibly improving some readers’ understanding of self-sabotage, the approach is clearly original in solving it.\n\nQUESTIONS:\nWhile RPO and RPG are defined for n-player stochastic games, experiments are two-player ones.\nLinked to the clarity issues of the three different joint-policies mathematical expression, wouldn’t it be clearer only to use i and -i. And maybe to define everything in a two-player game and state that it generalizes to n-player ones. Especially because pi_-i, the co-player policy, can represent a single or a joint policy without problem in the definition.\n\nWhat is the overhead of duplicating all agents and training several policies at the same time?\nI’d be happy to have an incentive for the computational cost of the approach and its scalability with more than two agents.\n\nAt the beginning of the paper, zero-sum games are not considered because self-sabotage has no meaning in that setting.\nHowever, this method could also improve the learning of policies in a competitive setting, especially when considering the first three claims empirically demonstrated.\nDid you try the algorithm in a competitive setting?\n\nLIMITATIONS:\nThe paper lacks a proper conclusion that summarizes the findings and nuances the results that RPG and its application to the five algorithms provide.\n\nSuggestion: line 35, I would add the statement of line 99: preventing meaningful learning, because there exists no policy that will improve the returns of the attacked teammate.\n\nSuggestion: Implicitly, the Hanabi games tested are two-player, but it should be stated explicitly. \n\nTypo line 160 peusocode", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-03"}