Robust and Diverse Multi-Agent Learning via Rational Policy Gradient
Authors: Niklas Lauffer
Ameesh Shah
Micah Carroll
Sanjit A. Seshia
Stuart Russell
Michael D Dennis
Venue: NeurIPS 2025 poster ()
Keywords: reinforcement learning
adversarial training
robustness
multi-agent learning
opponent shaping
general-sum
Primary area: reinforcement_learning
TL;DR: We introduce a method that allows adversarial optimization to be used in general-sum settings to train more robust and diverse policies.

Abstract: Adversarial optimization algorithms that explicitly search for flaws in agents' policies have been successfully applied to finding robust and diverse policies in the context of multi-agent learning. However, the success of adversarial optimization has been largely limited to zero-sum settings because its naive application in cooperative settings leads to a critical failure mode: agents are irrationally incentivized to *self-sabotage*, blocking the completion of tasks and halting further learning. To address this, we introduce *Rationality-preserving Policy Optimization (RPO)*, a formalism for adversarial optimization that avoids self-sabotage by ensuring agents remain *rational*—that is, their policies are optimal with respect to some possible partner policy. To solve RPO, we develop *Rational Policy Gradient (RPG)*, which trains agents to maximize their own reward in a modified version of the original game in which we use *opponent shaping* techniques to optimize the adversarial objective. RPG enables us to extend a variety of existing adversarial optimization algorithms that, no longer subject to the limitations of self-sabotage, can find adversarial examples, improve robustness and adaptability, and learn diverse policies. We empirically validate that our approach achieves strong performance in several popular cooperative and general-sum environments. Our project page can be found at https://rational-policy-gradient.github.io.

Robust and Diverse Multi-Agent Learning via Rational Policy Gradient
Niklas Lauffer!
Ameesh Shah' Micah Carroll'
Sanjit A. Seshia' *CC Berkeley "Google Deepmind
Stuart Russell* Michael Dennis
nlauffer@berkelle edu
Abstract
Adversarial optimization algorithms that explicitly search for flaws in agents' policies have been successfully applied to finding robust and diverse policies in multi-agent settings. However, the success adversarial optimization has been largely limited to zero-sum settings because its naive application in cooperative settings leads to a critical failure mode: agents are irrationally incentivized to self- sabotage, blocking the completion of tasks and halting further learning. To address this, we introduce Rationality preserving Policy Optimization (RPO), a formalism for adversarial optimization that avoids self-sabotage by ensuring agents remain rational that is, their policies are optimal with respect to some possible partner policy. To solve RPO, develop Rational Policy Gradient (RPG), which trains agents to maximize their own reward in a modified version of the original game which
pationality-presevving
opponent shaping techniques optimize the adversarial objective. RPG enables us to extend a variety of existing adversarial optimization algorithms that, no longer subject the limitations of self-sabotage, can find adversarial examples, improve robustness and adaptability, and learn diverse policies. We empirically validate that our approach achieves strong performance in several popular cooperative and general-sun sum environments. Our project page can be found at pollccy gradieet..rd.....ggg. gradient.github.ic
Introduction
A longstanding challenge in the field of multi-agent reinforcement learning (MARL) is that of learning robust behavior: individual agents should be able to adapt to a variety of different strategies that other agents might exhibit. One way to achieve robustness is by training agents to iteratively find and fix flaws in their policy. In zero-sum settings, this can be naturally achieved through self-play [Samuel 1959 Silver et al. 2016), where agents train against copies of themselves. Due to the adversarial nature of zero-sum self-play, agents will continually be encouraged to find new ways of attacking their opponents which will naturally lead to iterative improvement and robustification. In general-sum (especially cooperative) settings, however, self-play will explicitly avoid the weaknesses other players, it is harmful to the shared reward, resulting in brittle agents [Carroll et al.) 2019).
Inspired by its success in zero-sum settings, we leverage a form of adversarial optimization (i.e., we incentivize minimizing other players' rewards) to train agents to find and fix flaws in general-sum settings. However, seeking to minimize others' rewards in cooperative settings, where all agents aim to maximize a shared reward, unsurprisingly leads to self- sabotaabogggnng behavior (Cui et al.) 2023 If an agent is solely incentivized to minimize the rewards of another player, the adversary can simply learn to refuse to collaborate with that teammate. Even worse, the adversary has incentive act irrationally by actively sabotaging its teammate's s (and by extension, its own) reward, preventing meaningful learning.
"Code can found at wug.hh/.wwmiiiikaaaaffaaaaaaiaaa
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Objective
Find adversarial examples
Train robust policies
Discover diverse policies
adversurial examppe
progress blocked
cooperale
Ieam
Theum 2 sabotage
Standard crose-play diversity leads to sabotage in cross- play
Tum
Without RPG
Toan
Standard adversarial training prevents green agent from learning
Blue agent irrationally blocks the pots
botaage
cooperate
rational adversarial excample
learning to adapt
both clockwise
incompatible
Tham
Them
Rationality
With RPG (ours)
allows
learning
meaningful
Teem
diversity
Blue moving clockwise la OOD for
Rational behavior agent improve
incomapttbbee
both counter-clockwisse
Figure 1: Rational policy gradient (RPG) allows finding rational adversarial examples, robustifying behavior, and discovering diverse policies.
In order to reap the benefits adversarial optimization without incurring self sabotagnoggg behavior, we establish new paradigm for adversarial optimization called Rationality-preervvvii Policy Opti- mization (RPO). We formalize RPO as an adversarial optimization problem that requires the policy to be optimal with respect to at least one policy that other agent(s) might play. This can be thought of as enforcing the agent to be rational: the agent must be Saliittmmximizing for some choice of teammates.
The rationality constraint imposed by RPO is difficult to directly integrate into single optimization objective. To solve RPO, we introduce novel approach called rational policy gradient (RPG), which provides tradient-based method for ensuring rational learning while optimizing an adversarial objective. RPG introduces new set of agents called manipulators. one for each of the agents in the original optimization problem (which we call base agents). In RPG, the base agents only train to maximize their own reward in copy of the game (called its manipulator environment) with their teammates replaced by their manipulator counterparts this ensures that the base agents are solely learning to be rational. Each manipulator uses opponent-shaping [Foerster et al. 2017] to manipulate the base agents' learning and guide them towards policies that optimize the adversarial objective (e.g., achieving low reward with one another in the original base environment). The manipulators are discarded after training and the trained base agents give the solution to the RPO-version of the adversarial objective whether that be related to robustmess, diversity,
other objective.
As summarized in Figure
we use RPG to extend several existing adversarial optimization algorithms to find adversarial examples in pretrained policies, train more robust strategies, and fully eliminate self-sabotaging behavior exhibited by existing cross-play-based diversity algorithms an open
problem in existing literature [Charakorn et al. 2023 Cui et al., 2023 Sarkar et al. 2024 We
summarize our contributions:
We introduce formalism called RPO that overcomes the issue self-sabotage in any adversaria optimization algorithm.
We introduce gradient-based deep learning algorithm called RPG that solves RPO.
We use RPG to construct five novel adversarial optimization algorithms that find rational adversarial examples, train robust agents, and learn diverse policies.
We empirically demonstrate that our algorithms avoid self-sabotage and outperform existing baselines in popular cooperative environments.
2 Preliminaries
We consider the setting in which agents play in general-sum purtially-obserrabhl stochastic game (M,S,A R, Y, P, O) defined as follows. M =(1,-. (1....,k is set agents. is a set of states with initial distribution Sp. A A x Am is the space of joint actions; for case

---

of notation, we assume without loss of generality that the action set is identical across agents. R : S x A S - R is agent i's reward function. Y is the discount factor. P: S A S + [0. 1] is the transition function where Exes P(s,a, s') 1. f = Ui x ... x am is the joint observation space. : x Ax - [0, 1] is the observation function.
For sake of exposition, we limit ourselves to games with two players. Agents follow stochastic Markovian policies: the policy π; agent specifies a distribution over actions A, for every observation in 22 Let II denote the joint stochastic Markovian policy space for all agents, II, denote the policy space of agent i, and IT_; denote the co-policy space, the joint policy space of the co-players (all agents other than i). We note that our methods will work for history-dependent policies too (e.g., by allowing the policies to maintain external memory). We use (T; --1) to denote the expected (over stochasticity from the environment and policy) sum of discounted rewards |S] Σ, ' (st, at St+1)] for agent i from the initial distribution over states. The best-response function for agent is the set-valued function BR; II,; + P(II.) such that (T, = arg max,, n, U2( T-i), denoting the set of policies agent could play maximize the sum of discounted returns in response to т-І- We investigate the problem setting in which arbitrary optimization objectives O, are given for each agent discussed in the following section.
3 Rationaliy-PPrsseviing Policy Optimization
3.1 Adversarial Optimization Causes Self-Sabotage
We call optimization objective adversarial when some of the agents are explicitly incentivized to minimize the reward of another agent. For example, the adversarial training (AT) optimization problem GGlave et al. 2019 is defined for victim Tvictim and an adversary Tudversary in a 2-player game. The objective for the adversary is mil Uvictm (Tvictim: Fadversary) while the objective for the victim non-adversarial: max Uvietim (F Tadversar). zero-sum settings, AT has been used to train the adversary to find adversarial examples that expose robustness flaws in the victim's ' policy and have the victim learn to fix them.
Adversary
In an attempt to train an agent that is robust in cooperative setting, consider applying AT to the game between two players defined by the matrix in Figure 2 Suppose the victim is the row player and the adversary is the column player. Under the objectives specified by AT, no matter which policy Tviatim plays, the adversary will self-sabotage the game by playing policy E since this automatically minimizes Tvictim's 's reward at value of -1. Notice how a adversary will not help Tvitum improve its robustness: no policy available to Tvictim can do well if AT
C
A
1
0
Victim
B
0
Figure 2: A cooperative game.
AT
adversary
=
AT
adiversary plays action F. Motivated by such examples, we define self-sabotage as follows.
Definition 3.1 (Self-sabotage). Optimization problems in multi-agent games often include objectives distinct from the incentives in the underlying game (i.e., agents' incentive to maximize their individual reward). When the optimization problem results in policies with irrational behavior, behavior that causes an agent to act against its own incentive in the underlying game, we call this self-sabotage.
3.2 The Cicioonalitt-Prseevviie Policy Optimization Formalism
Rationality- aationality-presevving policy optimization (RPO) fixes the issue of self-sabotage in adversarial optimization problems by requiring an adversarial agent's 's policy to be rational, i.c., a best-response to at least one possible co-policy.
Definition 3.2 (Rationality- prreeerriigi Policy Optimization (RPO)). For each agent i, let Oi(T,,... ) denote its adversarial optimization objective. The RPO-version of agent i's ob- jective is given by
max O,( Tm) subject to 3m's E TT s.t. n, E BB(
Let us walk through an example of applying RPO to the adversarial training (AT) objective millwwesay Uvieti (Tiatim, Tadversry). RPO will modify AT by requiring that Wadversary is rational by serving as a best response to some co- co-policy. Notice that the RPO constraint has no affect on Tvietim since it is already optimizing to be rational.
3

---

Now, if Tvictim = A, the constraint from Theorem 3.2 will lead the adversary to find the policy "adversary = D since D valid best response if the victim plays B. Note that while playing E minimizes the adversarial objective further than playing D, does not satisfy the constraint of being a rational action. Importantly, the adversary is teaching the victim something actionable: increasing their likelihood of playing B would improve their reward. Moreover, the only equilibrium of this new optimization problem is for the victim to play uniform mixture over actions A and B. This makes the victim robust because the minimum expected reward it will obtain against any rational co-policy is 0.5.
Notice that the AT-RPG is strict generalization of AT since they are identical in zero-sum games. In the zero-sum setting, letting n Vvictim trivially satisfies the rationality constraint since BR (Tviiti)) 5aXomy Undversary (Tvictim: Tadversary) min Kubrarary Uvictim (victim Tadversary) in zero-Sum games.
Kuboaay
4 Rational Policy Gradient
Rational policy gradient (RPG) incorporates the constraint from Theorem 3.2 by introducing a new manipulator agent policy n= for each base agent policy π.. Each base agent i now ignores its original objective Of( Tm) and instead optimizes to play a best response against its manipulator to enforee the rationality constraint from Equation (T). The manipulator responsible for the original objectives O,, giving the following two objectives
Base agents: max U 4),
Manipulators: max OT
(2)
Notice that the manipulators can only influence the original objective indirectly by choosing which policy the base agents should best respond to. Since the rationality constraint has no affect on agents with non-adversarial objectives (e.g., the victim in AT), as an optimization, such base agents keep their original optimization objective: max OT( R.).
The full RPG-modified objectives of several adversarial optimization problems modified are given in Appendix D In the rest of this section, we give an overview of the principles behind the gradient update for RPG. Further details are available in Section
4.1 RPG Gradients
RPG approximates the objectives Equation g denote the parameters for policies T. and
through a policy gradient update. Let 0, and The gradient update for the base policies is
straightforward
8 6- V ((, 0M)
Building atop existing opponent shaping techniques [Foerster al. 2017], the manipulator takes higher-order gradients through their base agents' update with respect to the adversarial objective
gM + M + V AM
Omm).
(4)
Notice that O is evaluated at the base agents' parameters 0 after they have taken one gradient step into the future. The manipulators responsible for the adversarial objective but can only indirectly affect it by altering the base agent's learning process.
4.2 Estimating Gradients from Samples
In order to compute gradients in model-free deep learning setting, we approximate gradients via samples using surrogate loss that preserve higher-order gradients. Traditional RL surrogate losses are only sufficient for single-order gradients, gradients that preserves all dependencies (in our case, the manipulator's 's influence on the base agents' update). We use Loaded DiCE [Farquhar et al. 2019], which is based on DiCE [Foerster al. 2018 to define a surrogate loss that admits computing unbiased higher-order gradients through autodiff from advantage estimates. Let E be a set of rollouts between different pairs of base agents for the optimization objective O, The manipulator loss is defined as
need a different loss for the manipulator
We Σ
2'E((tff
(5)

---

Algorithm 1 RPG update with lookahead N
02
base agent update
θ'
input Policy parameters AMM 0s Vi E [m]]
base environment
1: Copy: 0g <- 0,, Vi E [m]]
manipulator environment
2: for n in
N do
3: Rollout trajectories under (0% 04)
V L
+ a1 V [(0) MM 4: 8 xx 8
U1
5: end for
6: Rollout trajectories under (0, 1,
7: 0 - 0h + 02 go - 00
mm)
ay
Lo
8: 0, - 02 + ag Vo, [(0,, 04) output M 8c vi E [m].
manipulator update
OM
Figure 3: Psuedo code (left) and visualization (right) for the RPG update. The inner box visualizes base agent's update and the full figure visualizes the corresponding manipulator update taking gradients through the base agent's update.
where s'st B.M denotes the set of actions from all players when the base agents trains with its manipulator during lookahead, w. is the weight (e.g., +1 for maximize and -1 for minimize) specified by the optimization objective, and r are the associated rewards between the two base agents specified by e. 3 is the magic box operator from [Foerster et al. 2018). The loss for each base
agent is based on a traditional RL surrogate loss L = E.EF We E. 7 log (T((els))) rr
the sum of rewards for different training partners rescaled by the training weight (e.g., We for partner-play).
4.3 Partner-play Regularization
In deep learning settings, significant distribution shifts between training and evaluation data can lead to poor performance. In case, base agents training against manipulators and then evaluated against other base agents: this shift in partner can easily put the base agent's 's policy out-of-distributtin. To minimize the impact of this distribution shift, introduce regularization term in the base agent's loss function called partner-play regularization: for each base agent i, we add to the training data some rollouts of partnered with each other base agent that i is evaluated against under O,, We scale this by small value so that it only acts as a secondary loss. This prevents i's policy from being out-o--distribution during evaluation and ensures that manipulators cannot optimize adversarial objectives by simply putting the base agents training out-of -ffiiribbiiitiion..
4.4 The Rational Policy Gradient Algorithm
Algorithm shows pseudocode for single policy update of RPO. Lines 2-5 perform N lookahead steps on the base agent policies by taking gradient steps towards being a best-response to their corresponding manipulator policies (as well as partner-play regularization). Line performs rollouts between all of the updated base agents required to evaluate the original objectives O (rollouts between all base agents are shown for simplicity). Line then applies a gradient step on the manipulator policies through the DiCE-modified [Foerster et al. 2018 loss function Cg evaluated on the rollouts from updated base agents. Line simply applies single gradient step on the base agents (identical to the update on line 4) with respect to a traditional RI. surrogate loss. We note that RPG is agnostic to the underlying RI. algorithm used to compute the gradients of the loss function.
4.5 RPG Algorithms
In this section, we give overview of the novel algorithms that introduce using RPG, allowing them to be applied to non-zero-sum games. A more detailed specification along with a visualization of each algorithm is given in Section D as well as glossary of acronyms and terms in Section
Adversarial policy (AP)-RPG used find flaws agents. AP-RPG is the RPG variant of adversarial policy (AP) Gleave et al. 2019]. AP is used in zero-sum settings to find adversarial vulnerabilities in pretrained agents. In both AP and AP-RPG, we refer to the pretrained agent as the victim. Meanwhile, the learning agent that attempts to find vulnerability in the victim is called the adversary. In the case of AP-RPG, since we want to prevent the adversary from acting irrationally, trains against an adversary manipulator that guides it towards the adversarial solutions.
5

---

Adversarial training AT-RPG is used to robustify a learning agent. AT-RPG is the RPG variant of adversarial training (AT) [Gleave et al. 2019]. AT is an extension of adversarial policy that allows the victim to simultaneously learn while the adversary attacks. AT-RPG modifies AP-RPG in the same way by allowing the victim to learn. Since the victim is not subject to any adversarial incentives, does not require a manipulator.
PAIRED-RPG is used to robustify learning agent using regret. PAIRED-RPG is the RPG variant induced regret minimization (PAIRED) Dennis et al. [020]. PAIRED is
of protagonist-antagone an algorithm used for unsupervised environment design and if naively applied to coplayer design. the adversary can simply lear to sabotage the game when matched with the protagonist. PAIRED-RPG fixes this issue by introducing manipulator for the adversary.
PATRED-AIE Attack RPG is used to find flaws that maximize an agent's regret. PAIRED-A-RPGi is a variant of PAIRED-RPG in which the protagonist is fixed, allowing to find adversarial policies that the protagonist performs well with, but the victim does not. This finds adversarial examples that do not only minimize score, but maximizes regret.
Cross-play diversity (XPD)-RPG is dual purpose: (I) finding a set of meaningfully diverse policies,
and (2) generating an cuto-currieulum to train robust policies. XPD-RPG is the RPG variant of cross-play diversity (XPD) algorithms underlying approaches such as ADVERSITY (Cui et al. 2023). LIPO [Charakorn et al. 2023]. . and CoMeDi [Sarkar et al.) 2024]. XPD algorithms work by framing diversity as the problem of finding a set of strategies that perform well in self-play but poorly in cross-play, indicating that distinct solutions have been found. However, naively training a population this way leads agents to achieve low cross-play scores by sabotaging the game rather than finding fundamentally incompatible strategies. XPD is often implemented by sequentially training agents, however, we find that simultaneously training all agents with XPD-RPG serves the dual purpose of producing an curoicurr that identifies and fixes weaknesses in the population's 's policies throughout training, providing similar benefits as self-play in zero-sum settings.
Experiments
Following a description of our experiment setup, results are organized into three subsections according to the following empirical claims, with the last one explored throughout:
Claim 1 RPG allows the learning of meaningfully diverse policies, set of policies that covers differing strategies within the space.
Claim 2 RPG algorithms train policies that are more robust to differing partners.
Claim 3 RPG algorithms find non-trivial adversarial examples in pretrained policies.
Claim 4 RPG prevents self sabotage across range of adversarial algorithms.
Our project webpage includes additional interactive visualizations, demos for playing against our trained agents, and various videos that demonstrate interesting behavior. The glossary in Section A can be useful as reminder of terms and acronyms throughout.
5.1 Experiment Setup
We evaluate our approach in four primary environments: matrix games with varying zero-sum, cooperative, and mixeddmotive payoffs; several standard Overcooked [Carroll et al. 2019) layouts, a modified version of STORM [Khan et al.] [023) that requires agents to collect either green or red coins, rewarding them when they collect the same color; and simplified 2-player version of Hanabi [Bard et al. 20200 that contains 3 colors and ranks and a version that contains colors and ranks (instead of the standard 5). See Section for descriptions and visualization of each of these environments. We use the JaxMARL [Rutherford et al. 2023) implementation for the latter three environments. Section E outlines algorithm implementation and hyperparameter details used.
To evaluate the efficacy of RPG, we compare it against a number of baselines. We include (1) self-play (SP) [Samuel 1959]. where policies are learned by training with themselves. We also compare against selection of existing adversarial learning algorithms. Specifically, we compare against (2) adversarial policy [Gleave et al. 2019), (3) adversarial training [Gleave et al.) 2019). (4)
PAIRED [Dennis et al. 2020), (5) a cross-play diversity (XPD) baseline, and (6) CoMeDi [aararr et al. | the previous SOTA extension for preventing self-sabotage in cross-play diversity. The
XPD baseline we use is very similar to LIPO Charakorn et al.) [2023] without a mutual information
term and SPWR Cui et al. 2023 In fully observed settings (c.g., Overcooked), our XPD baseline is

---

Base 0 vs. Basc 0
Algorithm
Self-play
Cross-play
Base 1 vs. Bose 1
Reward
Base O vs. Manipulator 0
CoMeDi
220
2
Base I vs. Manipulator 1
XPD
240
1.25
Base I vs. Basc 0
XPD-RPG
240
240
Base va. Basc
0
10000 20000 30000 40000 50000
Episode
Figure 4: Left: training curves for each learning agent in XPD-RPG. Self-play rewards increase
while cross-play scores decrease (but notably never reach zero). Manipulator's 's only reach the level of reward necessary to influence the base agents to be diverse. Shaded region shows 95% confidence interval. Right: self-play and cross-play rewards for algorithms based on cross-play diversity in the cramped room Overcooked layout, Self-sabotage leads to deceptively low cross-play rewards for CoMeDi and XPD while XPD-RPG maintains high score even in cross-play.
effectively identical to ADVERSITY Cui et al. 2023 See Section C for detailed discussion of these points.
5.2 Claim 1 RPG Finds Meaningfully Diverse Policies
In this section we demonstrate how RPG allows the learning of diverse (i.e, incompatible) policies by solving self-sabotage in cross-play diversity algorithms, whereas existing approaches (Cui et al.
2023
Sarkar et al. 2024) fail.
STORM. Figure|4[lefi shows the training curves for XPD-RPG in the original STORM environment. Around episode 10000, the manipulators learn to lead each base agent to gather a different colored coin, causing cross-play reward (e-g., base 0 base 1) drop. The manipulator's rollout rewards eventually plateau since further shaping would do nothing to increase the objective of diversity between base agents. Even in cross-play, agents avoid self-sabotage and continue receive a positive reward, supporting Claim 4 Qualitatively, in cross-play, the agents first each gather an opposite colored coin. Then, once they realize their incompatibility, they adapt their initial strategy, gathering a matching coin in order to receive a partial reward. This highlights that XPD-RPG only drives down value in cross-play play if it is necessary to improve self-play.
We also test CoMeDi [Sarkar et al., 2024] in the STORM environment and it fails to exhibit any attempt deviate, quickly driving cross-play reward all the way to zero, even though XPD-RPG shows that this is not necessary for high self-play reward. In order to exemplify this behavior, we modify the original STORM environment that both agents receive a reward of -0.1 every timestep either agent occupies the top-left square in the grid, giving agents an easy way to sabotage the game if desired. Figure 18 shows the training curves for CoMeDi run in this modified version of the STORM environment. The policy quickly learns to intentionally sabotage and receive a large negative reward in cross-plas play. Meanwhile, XPD-RPG learns to avoid the sabotage state in both self-play and cross-play in this modified environment.
Overcooked. Figure 4 (right) shows the self-play and cross-play rewards computed by different algorithms in the cramped room Overcooked layout. CoMeDi achieves low cross-play rewards (along with XPD), seemingly a sign of finding incompatible strategies. However, closer inspection reveals that in cross-play, CoMeDi and XPD agents will simply stand in front of the plate dispenser to prevent their partner from finishing any dishes a clear sign of sabotage. See the project webpage for videos of this behavior, Moreover, high cross-play rewards with XPD-RPG demonstrate that it able to avoid sabotage (Claim 4) and that the cramped room layout contains very little genuine diversity: agents will only achieve low cross-play reward if they are incentivized to sabotage. This unsurprising in light Lauffer et al. (2023). which showed that policies in Overcooked require trivial levels of coordination to be successtul.
5.3
Claim 2: RPG Trains Robust Policies
In this section, we show that XPD-RPG can learn policies that can generalize across partners. To do so, we train five policies with different seeds for XPD, XPD-RPG, SP (with entropy coefficient 0.01), and SP (with entropy coefficient 0.05). We found that using a large entropy coefficient with SP often encouraged policies to converge on the same strategy (therefore leading to higher robustness within its population). XPD and XPD-RPG both use population size of two policies.
7

---

SP
XPD
XPD+RPG
5P
XPD
XPO-RPG
175
200
150
125
150
XPO
XPD
100
100
50
XPD-RPG
XPD-RPG
25
(a) Cramped Room
(b) Forced Coordination
SP
XPD
XPD-RPG
SP
XPD
XPD-RPG
160
SP
140
120
200
XPO
XPD
150
60
100
XPD-RPG
XPD-RPG
50
(c) Counter Circuit
(d) Coordination Ring
XPD
XPD-APG
14
SP
12
7
10
6
5
XPD
4
2
Hanabi(3)
Hanabi(4)
Figure 5: Cross-play grids between different algorithms across environments. Each square represents the average reward from specific pair of seeds trained by one the three algorithms when paired as teammates. Standard error for Overcooked and < 0.1 for Hanabi for all values.
SP/entropy coef.=0.011.
SPeentropy coef.=0.00
XPD
XPD-RPG (ours)
Pr-Population Reward
20
250
(OverCooled)
200
12
(Hanabl)
150
100
Cramped Room
Coordination Ring
Counter Circuit
Forced Coordination
Hanabi13)
Hanabi(4)
Figure 6: Intra-population cross-play rewards for different algorithms across environments. Points represent pairs of seeds, bar charts represent means, and error bars represent 95% confidence intervals.

---

Figure 5 shows robustness results for different environments in the form of cross-play grids. The project webpage includes an interactive visualization of rollouts from these cross-play grids. In our cross-play grids, each row represents single seed trained by the specified algorithm. The first five rows represent SP with an entropy coefficient of 0.01 and the second five represent SP with an entropy coefficient of 0.05. The following ten rows represent the five pairs of policies from each run of XPD (since we use a population of size two) and the last ten rows represent XPD-RPG in the same fashion. Each column represents the same so that in diagonal entries, both players come from the same seed. Each square in the cross-play grid represents the average reward of the policies from the two associated seeds played against one another averaged over 1000 rollouts of the game. F shows the average intra-put pieraappplaaioon cross-play reward for each algorithm: the average of the cross-play scores from within the population of seeds of each individual algorithm.
Overcooked. SP policies in cramped room and forced coordination often tend to converge on a single strategy, especially with high entropy regularization (second set of 5 SP seeds). It's therefore unsurprising that these seeds perform well with one another. However, their adaptability to strategies from other algorithms is very poor they often achieve close Take forced coordination for example: even the two sets of SP seeds (with different entropy) are completely incompatible. On the other hand, XPD-RPG performs nearly perfectly with any partner as evident by the right third of the grid achieving high value. The asymmetry of the grid can be explained by the asymmetry of roles in forced coordination: the column player is responsible for picking up onions passed by their partner. XPD-RPG, as the column player, is able to adapt to any strategy for passing the onions while other algorithms are only robust to a single one.
reward with other policies.
The more complicated layouts of coordination ring and counter circuit allow multiple solutions and prove a challenge for SP (even with high entropy), achieving very low cross-play rewards. Meanwhile, XPD-RPG is able to maintain much higher gross-play rewards, evident by the higher mean bars in and Figure and the fact that it rarely achieves close to zero reward with itself in cross-play.
Hanabi. In the 3-color and 4-color version of Hanabi, SP with high entropy achieves significantly higher intra-pope ponuulaiont rewards than low entropy. We speculate that this is due to the entropy regularization causing policies to converge on very similar strategies. This is somewhat surprising given previous works [Bard et al., 2020 Hu et al. 2020) Hanabi and might result from our policies lacking depeedece or because the game is simpler with fewer colors, either of which might make SP policies less likely to develop specialized strategies.
5.4
Claim 3: RPG Finds Rational Adversarial Examples
In order to explore the effectiveness of AT-RPG, PAIRED-RPG. AP-RPG, and PAIRED-A-RPG, we use a modified version of STORM, which we call unobserved STORM, in which the agents cannot observe their partner's s position. This simplifies the strategy space by preventing agents from learning policies that react to their partner (such as how XPD-RPG adapts in cross-play). Table T shows the performance of various fixed victims against different types of adversarial attacks in unobserved STORM. The 'Victim' column indicates the algorithm used to train the victim and the 'Training' column the reward that the algorithm achieved by the end of training. The 'AP', "PAIRED-A-RPC" and 'PP- columns show the performance against the respective adversarial attacks.
These results demonstrate that (1) PAIRED-A-RUL -RPG and AP-RPG are both able to find weaknesses in fixed policies and neither is susceptible to sabotage, (2) PAIRED-RPG and AT-RPG training lead to more robust policies, indicated by high scores against both PAIRED-A-RPG and AP-RPG attacks. Unsurprisingly, every victim achieves zero reward against the AP attack since the adversary simply learns to self self-sabotage the game by collecting no coins. Likewise, the Ron-RPG variations of algorithms (AT, PAIRED, XPD) fail during training due to self-sabotage. Figures 19 to 21 in Section G.2 show AP, AP-RPG, and PAIRED-RPG attack training curves against these victims.
Overcooked. We tested the ability of AP-RPG to discover weaknesses in fixed policies in the cramped room Overcooked layout. AP-RPG is able to successfully find weaknesses in a fixed policy trained via self-play, finding an adversarial policy that achieves a reward of 4.6, even though the fixed policy achieves a reward of 240 in self-play. Figure 7 and videos on the project webpage illustrate one of these adversarial examples that AP-RPG finds: instead of simply sabotaging the game like AP, AP-RPG discovers that the victim (blue) assumes that the agents will move around each other clockwise. The manipulator in AP-RPG incentivizes the adversary (red) to instead move in a counterclockwise fashion, rational strategy, though it happens to be incompatible with the victim.

---

Adversarial Attack
Victim
Training
AP
PAIRED-A-RPG
AP-RPG
PAIRED
0.13
0.0
0.50
0.42
PAIRED-RPG
0.93
0.0
0.84
0.85
AT
0.0
0.0
0.0
0.0
AT-RPG
0.65
0.0
0.72
0.88
XPD
0.00
0.0
0.00
0.00
XPD-RPG
0.98
0.0
0.25
0.96
Self-play
0.98
0.0
0.16
0.96
Figure 7: A rational adversar- ial example found by XPD- RPG. Both agents assume the other will move out of the way, preventing progress.
Table 1: The average reward that "Victims" trained by various algorithms achieve against different adversarial attack types. The "Train" column shows the reward achieved during training and the following columns show the reward against different adversarial attack types. The AP attack trivially achieves zero reward because it suffers from sabo-aab See Section A for a glossary of acronyms.
Related Works
Prior work [Cui et al. 2023 Sarkar et al. 2024 has identified the phenomenon of self-sabotage in the context of cross-play diversity algorithms and proposed fixes based on making the observation distributions of self-play and cross-play similar. Our experiments show that the approach from Sarkar cct al. [2024] fails to prevent sabotage in all of our environments and that [Cui et al.) [023] fails in Overcooked and matrix games, since our XPD baseline is effectively identical to ADVERSITY in fully-observed settings. See Section C for a detailed discussion of why. Several simultaneous
works [Ruhdorfer et al., 2025) Waan et al. 2025 Chaudhary et al. 2025 have explored the idea of
extending the adversarial training algorithm to the cooperative setting to improve robustness against unseen partners. Specifically, |Wang et al. (2025) and ChaCCharrrry al. (2025) encounter the problem of self-sabotage and attempt to fix it by mixing state distributions across self-play and cross-play and by limiting the adversarial search space via generative model, respectively. However, none of these methods propose a complete framework for preventing self-sabotage across any adversarial optimization algorithm in the way that we do.
There are several other paradigms for training more performant or robust policies in cooperative settings. Approaches based on the setting of zero-shot coordination Hu et al. 2020). Treutlein et al. [2021). Muglich et al. [2022) aim to learn policies that do not depend on arbitrary symmetries of the game. Several other methods aim make policies robust by training them against population of
agents Vinyals et al. [2019), Rahman et al. [2022] or against numan-data inntt--iforeed policies Carrrll
al. [2019], [FIIR] Liang al. 2024]. Our work differs from all of these by directly applying different forms of adversarial optimization and does not rely on collecting human data.
A significant part of the technical backing of RPG is based on the line of work on opponent shaping [Forrster et al. 2017 Kim et al. 2021 Letcher et al. 2018). To the best of the author's 's knowledge, RPG represents the first use of opponent shaping in the context of adversarial training. A more recent line of work [Lu et al.] 2022 Khan et al. [023) aims to achieve opponent shaping without having to explicitly compute expensive higher-order gradients. Investigating whether this type of shaping could be incorporated into RPG to improve efficiency would be an exciting line of future research.
Conclusion
We introduce the problem of form of gradient-based algorithm called rational policy gradient (RPG). We demonstrate the diverse uses of RPG by constructing algorithms that robustify agents, expose rational adversarial examples, and learn genuinely diverse behaviors across general-sum settings. Our results highlight that the success of adversarial optimization general-sum setting and we aim to inspire further research in this direction. Furthermore, we hope that the generality of RPG will enable the development of entirely new classes of adversarial optimization algorithms in general-sum settings.
policy optimization and give a solution in the
zero-sum training methods can indeed be extended to the
10

---

8 Acknowledgment
This research was supported in part by a gift from the Open Philanthropy Foundation to the Center for Hluman-Compatibe AI, by DARPA Contract HR001 12490425 (TIAMAT), and the Schmidt AI 2050 Fellowship (Russell). Niklas Lauffer is supported by a National Science Foundation Graduate Research Fellowship and a Cooperative AI Foundation Fellowship.
References
Arthur L. Samuel. Some studies in machine learning using the game of checkers. IBM Journal of research and development, 3(3):210-23) 1959.
David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou. Veda Panneershelvam, Mare Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484448, 2016.
Micah Carroll, Rohin Shah, Mark K Ho, Tom Griffiths, Sanjit Seshia, Pieter Abbeel, and Anca Dragan. On the utility of learning about humans for human-Al coordination. Advances in neural information processing systems, 32, 2019.
Brandon Cui, Andrei Lupu, Samuel Sokota, Hengyuan Hu, David Wu, and Jakob Nicolaus Fo- erster. Adversarial diversity in Hanabi. In The Eleventh International Conference Learning Representations, 2023.
Jakob N Foerster, Richard Y Chen, Maruan Al-Shedivat, Shimon Whiteson, Pieter Abbeel, and Igor Mordatch. Learning with opponent-leariing awareness. arXiv preprint 115I::::77000032320 2017.
Rujikom Charakorn, Poramate Manoonpong, and Nat Dilokthanakul. Generating diverse cooperative agents by learning incompatible policies. In The Eleventh International Conference on Learning Representations, 2023.
Bidipta Sarkar, Andy Shih, and Dorsa Sadigh, Diverse conventions for human-Al collaboration. Advances in Neural Information Processing Systems, 36, 2024.
Adam Gleave, Michael Dennis, Cody Wild, Neel Kant, Sergey Levine, and Stuart Russell. Adversarial policies: Attacking deep reinforcement learning. arXiv preprint arXiv: /905. 10615, 2019.
Gregory Farquhar, Shimon Whiteson, and Jakob Foerster. Loaded dice: Trading off bias and variance in any-order score function gradient estimators for reinforcement learning. Advances in Neural Information Processing Systems, 32, 2019.
Jakob Foerster, Gregory Farquhar, Maruan Al-Shedivat, Tim Rocktäschel, Eric Xing, and Shimon Whiteson. Dice: The infinitely differentiable monte carlo estimator. In International Conference on Machine Learning, pages 1529-1538, PMLR, 2018.
Michael Dennis, Natasha Jaques, Eugene Vinitsky, Alexandre Bayen, Stuart Russell, Andrew Critch, and Sergey Levine. Emergent complexity and zero-shot transfer via unsupervised environment design. Advances neural information processing systems, 33:13049-13006 2020.
Akbir Khan, Timon Willi, Newton Kwan, Andrea Tacchetti, Chris Lu, Edward Grefenstette, Tim Rocktäschel, and Jakob Foerster. Scaling opponent shaping to high dimensional games. arXiv preprint arXiv:2312.
2023.
Nolan Bard, Jakob N Foerster, Sarath Chandar, Neil Burch, Mare Lanctot, H Francis Song, Emilio Parisotto, Vincent Dumoulin, Subhodeep Moitra, Edward Hughes, et al. The hanabi challenge: A new frontier for ai research. Artificial Intelligence, 1800103216.. 2020.
Alexander Rutherford, Benjamin Ellis, Matteo Gallici, Jonathan Cook, Andrei Lupu, Gardar Ing- varsson, Timon Willi, Akbir Khan, Christian Schroeder de Witt, Alexandra Souly, et al. Jaxmarl: Multi-agent rl environments in jax. arXiv preprint arXiv:2311.10090 2023.
Niklas Lauffer, Ameesh Shah, Micah Carroll, Michael D Dennis, and Stuart Russell. Who needs to know? minimal knowledge for optimal coordination. International Conference on Machine Learning, pages 11699118613. PMLR, 2023.

---

Hengyuan Hu, Adam Lerer, Alex Peysakhovich, and Jakob Foerster. "other-playy Sither-play" for zero-shot coordination. In International Conference on Machine Learning, pages 4399-4410. PMLR, 2020.
Constantin Ruhdorfer, Matteo Bortoletto, Victor Oci, Anna Penzkofer, and Andreas Bulling. Un- supervised partner design enables robust ad-hoc teamwork. arXiv preprini arXiv:2508.06336, 2025.
Caroline Wang, Arrasy Rahman, Jiaxun Cui, Yoonchang Sung, and Peter Stone. Rotate: Regret-driven open-ended training for ad hoc teamwork. arXiv preprint arXiv:2505.23686. 2025.
Paresh Chaudhary, Yancheng Liang, Daphne Chen, Simon S Du, and Natasha Jaques. Improv- ing human-ai coordination through adversarial training and generative models. arXiv preprint arXiv:2504.15455 2025.
Johannes Treutlein, Michael Dennis, Caspar Oesterheld, and Jakob Foerster. new formalism, method and open issues for zero-shot coordination. In International Conference on Machine Learning, pages 10413-10423. PMLR, 2021.
Darius Muglich. Christian Schroeder de Witt, Elise van der Pol. Shimon Whiteson, and Jakob Foerster. Equivariant networks for zero-shot coordination. Advances in Neural Information Processing Systems, 25:6410-6423,2022
Oriol Vinyals, Igor Babuschkin, Wojeiech M Czamecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. nature, 575(778223350-334, 2019.
Arrasy Rahman, Elliot Fosong, Ignacio Carlucho, and Stefano V Albrecht. Generating team- mates for training robust ad hoc teamwork agents via best-response diversity. arXiv preprint arXiv:2207. 14138, 2022.
Meta Fundamental Al Research Diplomacy Team (FAIR)t, Anton Bakhtin, Noam Brown, Emily Dinan, Gabriele Farina, Colin Flaherty, Daniel Fried, Andrew Goff, Jonathan Gray, Hengyuan Hu, et al. Human-level play in the game of diplomacy by combining language models with strategic reasoning. Science, (78(6624)11067-111 1074, 2022.
Yancheng Liang, Daphne Chen, Abhishek Gupta, Simon S Du, and Natasha Jaques. Learning to cooperate with humans using generative agents. arXiv preprint urXiv:2411.13955 2024.
Dong Ki Kim, Miao Liu, Matthew Riemer, Chuangchuang Sun, Marwa Abdulhai, Golnaz Habibi, Sebastian Lopez-Cot, Gerald Tesauro, and Jonathan How. A policy gradient algorithm for learning to learn multiagent reinforcement learning. In International Conference on Machine Learning. pages 5541-5550. PMLR, 2021.
Alistair Letcher, Jakob Foerster, David Balduzzi, Tim Rocktäschel, and Shimon Whiteson. Stable opponent shaping differentiable games. arXiv preprint wrXiv:1811.08444 2018.
Christopher Lu, Timon Willi, Christian A Schroeder De Witt, and Jakob Foerster. Model-free opponent shaping. In International Conference Machine Learning, pages 14398-14411. PMLR. 2022.
Logan Engstrom, Andrew Ilyas, Benjamin Chen, Axel Feldmann, William Moses, and Aleksander Madry. Optimizing mi training with metagradient descent. arXiv preprint UrXiv:2503.1375: 2025.
Diederik Kingma. Adam: method for stochastic optimization. arXiv preprint 14Xiv:1412.6990 2014.
Caroline Claus and Craig Boutilier. The dynamics of reinforcement learning in cooperative multiagent systems, AAAI/IAI,, 1998(746-752):2. 1998.
Michael Bowling and Manuela Veloso. Multiagent learning using a variable learning rate. Artificial intelligence, 136(2):215-2544 2002.
12

---

A Glossary of terms
Acronym
Term
Definition
RPO
cationality-preseervve Policy Optimization
Section 3.2
Equation
RPG
Rational Policy Gradient
Section 4
Igorithm
AP-RPG
Adversarial Policy - RPG
Section
Scction
AT-RPG
Adversarial Training RPG
Section
Section
PAIRED-RPG
Trotagonist-Antagonii Induced Regret Min. RPG
Section
Section
PAIRED-A-RPG
PATRED-Attack RPG
Section
Section
XPD-RPG
Adversarial Diversity - RPG
Section
4.5
Section
D.6
AP
Adversarial Policy
Gleave al.
2019
AT
Adversarial Training
Gleaave
2019
PAIRED
Induced Regret Minimization
Dennis a1
2020
XPD
Adversarial Diversity
C.0 el a.. 202
Carakom et al.
2023
SP
Self-play
Samucl 1959
CoMeDi
Cross-play optimized, Mixed-play enforced Diversity
Sarkar el al.
2024
Table Glossary terms and acronyms.
B Limitations
RPG relies on higher-order gradients through agents' learning updates, which introduces computa- tional overhead, especially in nigh-dimensional domains. However, this overhead is roughly linear in the number of optimizing agents in the problem and is indepedent of any other aspects of the problem. Empirically, we that cross-play diversity (XPD) is roughly two times slower (in wall-clock time) throughout our experiments since it doubles the number of agents and XPD-RPG is an extra three times slower than that through the introduction of a manipulator agent for each base agent. Otherwise, RPG algorithms experience only constant overhead as the size of the problem grows. Estimating higher-order gradients from samples has high variance, so large batch sizes are required stabilize learning in our experiments although recent works [Engstrom et al. 2025 have significantly improved the stability of meta- grtta-graieets and could be extended our work.
While RPG enforces rationality constraints by ensuring that base agents only train to maximize their utility, currently have formal guarantee of convergence truly rational strategies. However, would be useful for future work to explore under which conditions RPG ensures a solution to RPO.
C Comparison to existing attempts to solve self-sabotage
In this section, theorize why existing approaches preventing self-sabotage fail. In order match the setting of existing work on self-sabotage [Cui et al., 2023] Sarkar et al., [024]. we investigate the setting of the cross-play diversity (XPD) algorithms. XPD algorithms aims generate diverse population of policies by maximizing self-play scores and minimizing cross-play scores within the population. Our XPD baseline is very similar LIPO Charakorn et al. [2023 except we use an average instead of a max to aggregate cross-play scores and do not add any additional mutual information terms to the loss. Our XPD baseline is nearly identical ADVERSITY [Cui al. 2023) in fully-observed settings since there is no oartially-observed history to do belief reinterpretation over. The only remaining distinction that our XPD baseline rolls out SP and XP distinct rollouts instead of mixing within the same rollout as ADVERSITY does.
Now turn observing why traditional approaches to solving self-sabotage in XPD algorithms have failed. Consider running XPD with a population of size two in the following matrix game:
Player Y
C
D
E
F
A
0.9
-1
0
Player X
B
-1
0.9
1
Figure 8: The payoff matrix for cooperative game.
13

---

manpulater
manpulitt
mampulake
(1
2e)
Victm
Boss advieisary
Victim
adversary
Protagorist
Baso adversary
Antagorist
Post revad
All
Adversarial Policies - RPG
Adversarial Training RPG
PAIRED RPG
Adversary manipuiator
Manipultoor
Maripuiator
Ze)
+1
+1
2e)
(I Del
Base
Antagonistt
Bise
Rose
Dase
Prot revuut
ret.reward
PAIRED Attack RPG
Cross-play Diversity RPG
Figure 9: Graphical representation of the RPG algorithms described in section D.1 Nodes represent optimizing agents and outgoing edges represent their optimization objectives. Weights edges scale the importance of different objectives such that negative weights indicate minimization.
Running the original XPD algorithm (along with those from Cui et al. 2023 Sarkar et al. 2024]
with population of size two will lead the solution {(A, D), [B, E)}, achieving self self-play scores of 0.9 and cross-play scores of 11 Both policies maximize the objectives of their respective algorithm by irrationally playing D and F. instead of C and F, respectively. They sacrifice little bit of score in self-play to greatly decrease the cross-play score. However, notice that the actions and are strictly worse no matter what the partner plays clearly irrational choice.
As identified in [Cui et al. 2023 Sarkar et al., 2024], true diversity should only come from a decreases in eross- eross-play play scores that necessary to improve self-play scores within the population. Choosing action D over C' is the opposite: sacrificing self-play score to artificially reduce cross-play score. XPD-RPG finds the desired solution (AA C), (B, F)} because it constrains the diversity rational policies and only allows a decrease in cross cross-play it benefits self play However, just like ordinary
XPD, the algorithms presented in [Cui et al. 2023 Sarkar al. 2024 find the undesired solution (), (B, F)). Both of the algorithms presented in these works are based on the idea of confusing agents during training that they do not know whether they currently training in self-play cross-play. They do this by altering RL training so that the distributions of observations seen during self-play and cross-play are similar as possible. However, the example in Figure FigureBBdemonstrinn demonstratts that self-sabotage can happen, even if there's ' only a single observation in the game. No amount of confusion about observation distributions can prevent selo-sabo so the approaches in [Cui et 2023 Sarkar et al. 2024] fail in situations like this. The experiments shown in Figure and Figure 18 demonstrate that these approaches fail in STORM and Overcooked, too, indicating that the structure of the game in Figure B may be present in more complicated environments.
D RPO objectives
D.1 Graphical Language for RPG Algorithms
In order to simplify specifying and interpreting RPG algorithms, we introduce a graphical language. Agents (base agents and manipulators) are represented by nodes and an agent's loss function is represented by a collection of triples that can be visually represented by outgoing edges in the graph. A triple is defined by the optimizing agent (i.e. the node that the edge is coming out of) and the two agents being evaluated.
Figure shows the graph for all RPG algorithms. Let us use AP-RPG as example. It includes three agents: the victim, adversary base agent, and adversary manipulator. The victim is fixed, indicated by having no solid outgoing edges. The adversary base agent has two outgoing edges, one for training against its manipulator (with weight e), and one for partner-play against the victim base agents (with weight e). The adversary manipulator has single outgoing edge pointing to the victim's 's reward when the adversary base agents and victim base agents are evaluated against one another. The weight of 1 on this edge indicates that the adversary manipulator's loss wants minimize this value.
14

---

D.2 Adversarial Policy-RPG
The objective for adversarial policy-RPGH
min Uvictim Wadversaryy Tvictim):
Palmorry
(6)
S.t. Tadversary E BR Tadversaryy
Notice that AP-RPG is identical to AT-RPG except that the the victim is fixed.
D.3 Adversarial Training-RPG
The objective for adversarial training- RPG
min Vvictm (nnversaryy Tvictim )
aahoorry
max Uvictim (Tadversary> Tvictim): )
(7)
s.t. Tadversary E BRR(T adversary
D.4 PAIRED-RPG
The objective for PAIRED-RPG is
max Uvictim Wadverarry Wantagoniss)
Uvictim Wadversary, protaggoisst)
T M Shenayy
max
Uvictim (aaveesaay: ppootngonist):
T
(8)
max Uvictim (aadeeraar: Tantagonist);
s.t. Tadversary BR( T adversary)
D.5 PAIRED-AttackRPGG
The objective for PAIRED-A-RPG: is
max Uvictim Tadverrury, Mantagonis))
Uvictim (aavversary Trrotagunist
14
max Uvictim (Taveersary, Mantagonist);
aatiagoiis
Tadversary &BR(Tabersan Taalverssay ).
Notice that PAIRED-A-RPER is identical to PAIRED-RE RPG except that the protagonist (which acts as the victim) is fixed.
D.6 Adversarial Diversity-RGG
The objective for cross-play diversity-RPG with population size of is
A
Ue(mt, m²)
m
(10)
1E/m]
je miliji
s.t.
m. E B Vi [m],
where À weighs the relative importance of minimizing cross-play. Our experiments use À 0.25. Note that we only apply XPD-RPG to cooperative settings.
15

---

E
RPG Details
E.1
Implementation details
RL algorithm. Our implementation RPG uses actor-critic with a single epoch and update as the RI. algorithm. We initially considered trying PPO, but realized that clipping might interfere with the higher-order gradients, leaving the implementation of more sophisticated RL. algorithms to future work. For the same reason, only the manipulators use max gradient norm. We use separately learned centralized critic for each pair of agents rolled out the underlying game. All of our experiments also use entropy regularization.
Choice of optimizer. Since the manipulators need to take gradient's through the hase agent's 's update, the base agent's optimizer update must be differentiable. Many modern optimizers for deep learning incorporate methods such as momentum, which might make more difficult for the manipulator's to estimate their influence their base agent. In our experiments, show that our methods work whether we use vanilla SGD or adam optimizers [Kingma 2014).
Normalization. We also find that properly normalizing advantage estimates for each agent is important: each agent normalizes the advantage estimates associated with each of their training partners together. We also tried normalizing advantages between training partner's independently but learning proved to be more unstable.
Variable learning rates. Using different learning rates between the manipulator and the base agents often important. We find that it is often necessary to significantly larger manipulator learning rate than base agent learning rate to magnify the manipulator's s ability to influence the base agent's s learning. We also find that using different learning rate in the base agent lookahead update step is important. A larger inner base agent learning rate allows the manipulator to project the base agent's update further into the future, similar using additional lookahead step, except without the added computational burden.
its actual
Partner-play. Partner-play regularization also makes manipulator's optimization problem more difficult: if manipulator wants to minimize the score between two base agents, it cannot simply make sure its base agent's 's training trivially out- -if-o--fistibbttion We find that in the case of XPD-RPG, small partner-play (0.05 0.1) is more effective at finding genuinely diverse solutions while large partner-play (0.1 0.2) more effective robustifying policies. Too large of a partner-play eliminates the manipulator's influence over the base agent entirely, rendering RPG useless.
E.2
RPG hyperparameters
Table 3 contains the RPG hyperparemeters used in the experiments for each of our environments.
Table 3: RPG hyperparameters for different environments
Hyperparameter
Matrix
STORM
Overcooked
Hanabi
Architecture
64x64 SGD
64x64
64x64
512x512
Optimizer
Adam
Adam
Adam
Manipulator LR
10
8 x 10-3
1 1 10-3
2.5 10 10-3
Base LR
x 10-2
1 x 10-1
2 2 10-4
5 10
Base lookahead LR
10
4 10
2 10
5 10 10
Batch size
128
256
512
128
Discount factor (7)
0.95
0.99
0.99
0.99
GAE parameter (A)
0.95
0.99
0.99
0.99
Loaded DiCE coef (A)
0,95
0.99
0.99
0.99
Value function coef.
0.5
0.5
0.5
0.5
Entropy coef.
0.0
0.01
0.01, SP: 0.05
0.01, SP: 0.05
Manipulator max gradient norm
0.5
0.5
0.5
0.5
Partnerplay coefficient
0
0.15
0.1
0.1
16

---

E.3 Compute Resources
Experiments were all performed on single GPUs (a mix of A4000s and A6000s) on local SLURM cluster using 32 epu cores and 50Gb of RAM. Seeds for all experiments took between a minute up to 24 hours run.
17

---

Environment descriptions
E1
STORM
Figure 10: A visualization of the storm environment with green and red coins and the agents in orange and blue. Their 5x5 observation windows illuminated in front of them.
The Opatial-Tempora Representations of Matrix Games (STORM) (Khan et al. 2023] environment is visualized in Section The agents move around the grid and collect green and red coins while only observing a limited area directly in front of them. The agents can 'interact' by getting near each other and playing the 'interact' action. When the agents interact, coin is randomly chosen from the ones they have picked up. The agents both get a reward of 1 if they interact with the same colored coin, and 0 otherwise.
Our experiments also investigate two modifications of the original STORM environment. In the first modification, which we refer as wnobserved STORM, agents' observations longer include their partner's position. We use this modification to simplify analysis by eliminating the possibility of strategies that react to their partner's policy. In the second modification, both agents receive reward -0.1 when either agent stands in grid position (0,0). We use this modification to exemplify sabotaging behavior exhibited the CoMeDi [Sarkar et al. 2024 baseline.
F.2 Overcooked
Figure 11: Visualization of various Overcooked layouts in their initial positions. From left to right: Cramped Room, Coordination Ring, Counter Circuit, Asymmetric Advantages, and Forced Coordination.
Overcooked [Carroll al. 2019 is cooperative game where players control chefs working together in kitchen to prepare and serve meals under time constraints. The agents need to work together to gather multiple ingredients into a pot, wait for the soup to cook, and then plate and deliver the dish. We perform experiments in a variety of Overcooked layouts.
F.3
Hanabi
Hanabi [Bard et al.. 2020 is cooperative, imperfect-informaann card game that has been a corner- stone of recent coordination benchmarks. Players work together to arrange shared set of colored, numbered cards in order. Each player holds their cards facing outward, unable see their own hand but able to see others'. Players provide limited hints to teammates about their hands, constrained by finite set of clue tokens. The challenge lies deducing one's own cards and playing them correctly without direct knowledge. It is difficult to coordinate in Hanabi because strategies often rely on arbitrary signaling strategies to communicate information to teammates. Hanabi is typically played with colors and 5 ranks, but experiment with version that uses 3 colors and ranks well as a version that uses colors and ranks.
18

---

G Additional results
G.1
Matrix Games
G.1.1
RPG stabilizes learning dynamics.
1.0
Victim Reward
0.5
Lookahead Steps
0.0
0
1000
2000
3000
4000
Samples
Figure 12: larger RPG lookahead stabilizes learning dynamics and allows them to converge on an unstable equilibrium.
It well known that multi-agent problems can have unstable learning dynamics [Claus and Boutilier 1998 [Bollig and Veloso 2002]. However, a large enough number of lookahead steps in RPG helps stabilize learning, leading to convergence. Figure 12 shows the reward the learned victim policy in various of AT-RPG for the matrix game from Figure 2 The solutions converge playing equal mixture between actions A and B even though this is an unstable equilibrium. By allowing the manipulator look several steps forward through the learning of the hase agents, it can anticipate that the victim could respond to a deviation which would ultimately decrease the manipulator 's objective. We observed the effect in RPG algorithms across variety of matrix games. See Section G.T for more details.
G.1.2 Understanding how RPG behaves through various matrix games
Player Y
C
D
E
0.9
Player X
B
0
Figure 13: common-payofi game with a dominated action.
Consider the problem of finding two different pairs of diverse strategies in the matrix game in Figure 13 using pross-play diversity (XPD). Imagine for a moment that column D did not exist. Clearly, the most diverse population of successful strategy pairs would then be the solutions (A, C') and (B, E): both pairs of strategies get the maximum self-play score of and the minimum cross-play score 0. Now, consider adding column D back in: Player Y could change their action in the first pair from C' to D. This greatly improves the diversity metric by decreasing the cross-play score to -1 while only losing little bit of score in self-play. However, Player Y choosing to play action D over C is clearly a case of sabotage: not only are they intentionally reducing their score in cross-play, they doing it the cost of self-play performance! In fact, action D strictly dominated by action C: it performs worse in all scenarios. Not only do agents artificially sabotage their performance in cross-play by playing action D, they do it the cost of self-play performance. Existing work [Cui] et al. 2023 Sarkar et al. 2024) fails to prevent sabotage in these types of scenarios because it does depend on different observation distributions between cross-play and self-play.
Figure 14 shows a mixed-motive game with a sabotage option. AT and PAIRED sabotage by playing action E. AT-RPG converges playing uniform mixture across the two rational policies while PAIRED-REE RPG converges on one of the two equilibria. XPD converges on one sabotage policy and at equilibrium while XPD-RPG converges the two equilibrium.
Figure 15 shows a cooperative game. Suppose the victim AT is Player and the adversary is Player Y. In this case, AT oscillates between playing C and D to minimize the victim's 's reward. On
19

---

Player Y
C
D
E
A
3, 2
0,0
- 1, 1
Player X
B
1,1
2,3
-1,
Figure 14: A mixed-motive game with a sabotage option.
Player Y
C
D
E
A
0.9
0
1
Player X
B
0.9
1
Figure 15: cooperative game.
the other hand, since such adversary irrational, the adversary in AT-RPG will just play action E. Likewise, XPD finds artificially diverse solutions by playing (A, C)) and (B, D). Both sets of policies in XPD-RPG just play action E, the only rational action.
Player Y
0,0 O
1,
Player X
B
1,-1
10, -10
Figure 16: A mixed-motive game of "chicken',
Figure shows the mixed-motine game of 'chicken'. this case, AT and AT-RPG solutions align and always play action D minimize the victim's 's reward, forcing them to play the conservative action A.
Player Y
C
D
E
A
0,0
1,1
1, -1
Player X
B
1, -1
0,0
-1,1
C
-1, 1,
1, -1
0,0
Figure 17: A zero-sum game of
Figure 17 shows the zero-sum game of 'rock paper-scissors' In this case, AT and AT-RPG solutions have very similar behavior in which the adversary and victim oscillate between picking rock, paper, or scissors during training (similar to the oscillating behavior in [2]. We found that increasing the lookahead (e.g. greater than 8) AT-RPG led to this behavior stabilizing and converging on playing the strategy that randomizes across the three actions (in the same way lookahead stabilizes the oscillations in
20

---

G.2 STORM
1.0
SP
Reward
0.5
0.0
-0.5
20
60
100
Episode
Figure 18: CoMeDi learns to sabotage in modified version of STORM that allows easy sabotage.
0.2
PAIRED-RPG
Victim Reward
XPD-RPG
XPD
0.1
AT-RPG
PAIRED
SP
0.0
500
1000
1500
2000
Samples
Figure 19: Training curves for AP applied various victims in the unobserved STORM environment.
As expected, AP immediately learns to sabotage, revealing nothing useful about the robustness of each victim.
1.0
SP
Victim Reward
XPD-RPG
AT-RPG
0.5
PAIRED-RPG
PAIRED
XPD
0.0
2000
4000
8000
10000
Samples
Figure 20: Training curves for AP-RPG applied to various victims in the unobserved STORM environment. AP-RPG avoids the problem of sabotage but inconsistently finds weaknesses in victim policies.
21

---

1.0
Victim Reward
PAIRED-RPG
AT-RPG
PAIRED
0.5
XPD RPG
SP
XPD
0.0
2000
4000
8000
10000
Samples
Figure 21: Training curves for PAIRED-RPG attack applied to various victims in the unobserved STORM environment. PAIRED-RPG attack consistently finds weaknesses in victim policies while avoiding the problem of sabotage. These results also highlight that PAIRED-RPG and AT-RPG can produce more adversarially robust policies.
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Contributions can be found in the intro and the experiments subsections clearly outline supporting claims for each of the contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Discussed in Section B
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
23

---

Justification:
No theory contribution.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Psuedocode and full descriptions of the method (including hyperparamters Section and environments are provided as well as access to the code to replicate experiments.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
24

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
See Supplementary for Code with README instructions.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips //ipss.cc/pill wwtpcccpppppppppeeeeeeeeeeeeeeee httpa////giieee/eeeSebiiisssssss /Cc////oddiissssseeeeeee ssiiiiiiccyyyyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Environment details are given in Section 5.1 and hyperparamettes//etaill in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Error bars are reported and labeled where relevant.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
25

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
Given in Section E.3
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This work provides a new foundational method for applying adversarial optimization to general-sum settings. No immediate societal impact follows.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
26

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: NA
Guidelines:
The answer NA means that the paper does use existing assets. The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
27

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[Yes]
Justification:
See supplementary for code.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
28

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification:
Guidelines: NA
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy htte:pa//eerrippc.cc/o//ofeerees for what should or should not be described.
29


Reviews:
[{"review_id": "mf8TbDXuxy", "reviewer": "Reviewer_9KSf", "summary": "This paper presents a new formalism and method to enhance the adversarial training of methods in multi-agent reinforcement learning.\nIt targets the specific cooperative settings, where adversarial agents learning to find flaws in others' strategies can end up with a self-sabotage policy.\nThis implies that the adversarial agent ends up with a policy that prevents the victim from finding another policy that will improve its result.\nThis is also referred to as non-rational behavior.\nThe goal of RPO is to constrain the maximized adversarial objective by ensuring that the adversarial policy is the best response to one of the victim’s possible responses.\nFrom this constraint, RPG is derived to improve different adversarial training algorithms by ensuring that no self-sabotage policy is achieved.\nThis new paradigm enables the development of more robust learning strategies.\n\nThe introduction reminds us of the context of adversarial training and its success in zero-sum games, inherent to the objective of each agent that benefits from the loss of others.\nThe problem of these approaches in a cooperative scenario is that agents usually learn a « self-sabotage » behavior, where the attacker agent does not cooperate anymore, and the victims are not able to find any policy leading to better performance.\nThis is claimed to be a non-rational behavior and the whole goal of the paper is to enforce rationality.\nA clear example is provided showing the difference between a rational adversarial attack and a self-sabotage policy.\nThe introduction briefly describes how this rationality constraint is incorporated into the learning framework by introducing manipulators, copies of agents trained using opponent-shaping methods, thereby introducing an additional learning objective.\nContributions are: a new formalism (RPO), a new gradient-based method to train agents (RPG), five new adversarial algorithms extending existing ones with RPG, and an empirical demonstration of the benefits of the new paradigm.\n\nPreliminaries define the stochastic game framework, where the objective of agents is to maximize the expected sum of discounted rewards dependent on the joint policy of all agents, and what constitutes a best response policy.\n\nRPO is then defined.\nA reminder of the adversarial optimization, its application in a simple cooperative game, and what is an irrational self-sabotage policy, and what is a rational one.\nThe Rational-preserving Policy Optimization is defined: for each agent i, its adversarial objective is maximized by pi_i while there must exist a joint-policy of the others, pi_-i, for which pi_i is the best response.\nIts implication for a simple cooperative matrix game is provided to help build intuition.\nAdversarial training RPO is thus a generalization of adversarial training in zero-sum games.\n\nRPG is then defined and provides a solution to incorporate the RPO constraint into existing learning algorithms.\nFor each policy pi_i, there is a manipulator pi^M_i.\npi_i maximise the classical objective U(pi_i, pi^M_-i), enforcing pi_i to be the best response to pi^M_-i.\npi^M_-i, in parallel, are trained to maximise the adversarial objective.\nHow gradients are computed, and in which order, is then described, followed by a statement on partner-play regularization.\nA representation of the algorithm is presented and then described.\nFinally, the application of RPG to five existing adversarial algorithms is described.\n\nExperiments demonstrate four claims clearly identified: RPG allows learning 1) diverse policies, 2) robust policies, 3) non-trivial adversarial examples, and 4) non-self-sabotaging policies.\nExperiments are conducted in four environments: matrix games, Overcooked, STORM, and Hanabi, all of which are 2-player cooperative games (?).\nEnvironments and parameters are detailed in the appendix.\nThe four claims are demonstrated through dedicated experiments and metrics, showcasing the power of RPG against six other methods, while applying RPG to four of them.\n\nThe paper finishes with a related work section, reminding the need to find a non-sabotage adversarial training method, claiming that to the best of their knowledge, RPG is the first solution using opponent-shaping to achieve adversarial training.\nA future work direction is proposed: to use a line of work improving opponent-shaping with a more effective method in RPG.", "questions": "While RPO and RPG are defined for n-player stochastic games, experiments are two-player ones.\nLinked to the clarity issues of the three different joint-policies mathematical expression, wouldn’t it be clearer only to use i and -i. And maybe to define everything in a two-player game and state that it generalizes to n-player ones. Especially because pi_-i, the co-player policy, can represent a single or a joint policy without problem in the definition.\n\nWhat is the overhead of duplicating all agents and training several policies at the same time?\nI’d be happy to have an incentive for the computational cost of the approach and its scalability with more than two agents.\n\nAt the beginning of the paper, zero-sum games are not considered because self-sabotage has no meaning in that setting.\nHowever, this method could also improve the learning of policies in a competitive setting, especially when considering the first three claims empirically demonstrated.\nDid you try the algorithm in a competitive setting?", "limitations": "The paper lacks a proper conclusion that summarizes the findings and nuances the results that RPG and its application to the five algorithms provide.\n\nSuggestion: line 35, I would add the statement of line 99: preventing meaningful learning, because there exists no policy that will improve the returns of the attacked teammate.\n\nSuggestion: Implicitly, the Hanabi games tested are two-player, but it should be stated explicitly. \n\nTypo line 160 peusocode", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "The paper provides a high-quality explanation of the problem and the solution, and appears to be the first to address it, supported by rigorously conducted experiments. However, some clarity enhancement could benefit the reader, and a consideration of limitations and extensions beyond two-player cooperative settings could be added.\n\nQuality:\n\n(+) The submission is technically sound, and all claims are well supported with examples, references, and experiments.\n\n(+) The paper is standalone, and all details are provided in the appendix.\n\n(+) Experiments are rigorous, and many details are provided to analyse experiments, supporting the claims.\n\nClarity:\n\n(+) The paper is overall clearly written\n\n(-) I believe the three joint policies (pi_adversary, pi_victim), (pi_i, pi_-i) , (pi_1,…, pi_m) are sometimes used interchangeably, making Section 3.2, among others, hard to follow. While I believe it clarifies section 3.1, using all three of them is overwhelming.\n\n\nSignificance:\n\n(+) This paper has a significant impact on the community, as it presents the first method to address the self-sabotage problem in adversarial training.\n\n(+) Many existing methods are mentioned, tested, and surpassed by RPG.\n\nOriginality:\n\n(+) Along with possibly improving some readers’ understanding of self-sabotage, the approach is clearly original in solving it.", "was_revised": true, "final_justification": "I think this paper is a must for the community.\nIt is crystal clear that this new method can benefit the cooperative MARL community.\nThe number of environments tested is significant and the results benefit from a rigorous experimental protocol.\nThe authors addressed all my comments that will improve the paper.", "rebuttal": "Thank you for your review. We appreciate your noting that our “paper has a significant impact on the community” and that the experiments are “rigorously conducted.” We address each point individually:\n\n> “(-) I believe the three joint policies (pi_adversary, pi_victim), (pi_i, pi_-i) , (pi_1,…, pi_m) are sometimes used interchangeably, making Section 3.2, among others, hard to follow. While I believe it clarifies section 3.1, using all three of them is overwhelming.” \n\n> “While RPO and RPG are defined for n-player stochastic games, experiments are two-player ones. Linked to the clarity issues of the three different joint-policies mathematical expression, wouldn’t it be clearer only to use i and -i. And maybe to define everything in a two-player game and state that it generalizes to n-player ones.  Especially because pi_-i, the co-player policy, can represent a single or a joint policy without problem in the definition.”\n\nThank you for this feedback – we agree that the amount of notation through section 3 is overwhelming and we will improve this in the final version. We agree that it would be most clear to simply use i and -i in as many places as possible (potentially simply eliminating pi_adversary and pi_victim). However, the reason we also use the (pi_1,…, pi_m) is because even while the underlying game might only involve two players, the optimization problem on top of that game could involve more than two policies. For example, the adversarial diversity algorithm aims to find a set of N pairs of policies that are as diverse (i.e. incompatible) as possible. The choice of N can be larger than 2. We will do a better job of clarifying this distinction in Section 3 and simplify the notation where possible.\n\n> “What is the overhead of duplicating all agents and training several policies at the same time? I’d be happy to have an incentive for the computational cost of the approach and its scalability with more than two agents.”\n\nWe agree that this is an important detail to add to the paper and will add it to the final version for each of our experiments. On average, the original adversarial diversity algorithm with a population size of 2 is 2 times slower in wall-clock time (per step) than self-play while the AD-RPG (ours) is an additional 3 times slower than AD (due to the extra cost of manipulator policies and computing higher-order gradients). The computational overhead of the higher-order gradients is constant in the number of agents so the method should scale well beyond two agents. However, we will leave this investigation for future work. Moreover, recent works have demonstrated paths for further scaling higher-order gradients (Logan, et al. \"Optimizing ml training with metagradient descent\").\n\n> “At the beginning of the paper, zero-sum games are not considered because self-sabotage has no meaning in that setting. However, this method could also improve the learning of policies in a competitive setting, especially when considering the first three claims empirically demonstrated. Did you try the algorithm in a competitive setting?”\n\nThe main reason we did not test our algorithm in a zero-sum setting is because RPO is a strict-generalization of the associated algorithms from the zero-sum setting (explained on lines 121-124 of the submission) since adversarial incentives (minimizing your opponent’s reward) and individual incentives (maximize your own score) coincide. For example, AT-RPO simply reduces to the same problem as AT in the zero-sum setting. Because of this, we believed that using RPG in the zero-sum setting would just add unnecessary complexity to the algorithm. However, inspired by your question, we decided to empirically test our implementation of RPG in a zero-sum setting. We picked the matrix game of rock-paper-scissors: \n\n|       | **R**   | **P**   | **S**   |\n| ----- | ------- | ------- | ------- |\n| **R** | (0, 0)  | (-1, 1) | (1, -1) |\n| **P** | (1, -1) | (0, 0)  | (-1, 1) |\n| **S** | (-1, 1) | (1, -1) | (0, 0)  |\n\nand tested both AT and AT-RPG. Both methods have very similar behavior in which the adversary and victim oscillate between picking rock, paper, or scissors during training (similar to the oscillating behavior in Figure 13). We found that increasing the lookahead (e.g. greater than 8) in AT-RPG led to this behavior stabilizing and converging on playing the strategy that randomizes across the three actions (in the same way lookahead stabilizes the oscillations in Figure 13). Thank you for asking this question – we will include these results in the final-version and believe it will strengthen the paper. \n\n> “The paper lacks a proper conclusion …”\n\nWe will add a proper conclusion with the extra page allowed for the camera-ready version. Here is a draft:\n\nWe introduce the problem of *rationality-preserving policy optimization* and give a solution in the form of a gradient-based algorithm called *rational policy gradient (RPG)*. We demonstrate the diverse uses of RPG by constructing algorithms that robustify agents, expose rational adversarial examples, and learn genuinely diverse behaviors across general-sum settings. Our results highlight that the success of adversarial optimization in zero-sum training methods can be extended to the general-sum setting and we aim to inspire further research in this direction. Furthermore, we hope that the generality of RPG will enable the development of entirely new classes of adversarial optimization algorithms in general-sum settings.\n\n> “Implicitly, the Hanabi games tested are two‑player, but it should be stated explicitly.”\n\n We will state this in section 5.4 and the caption of Fig 7.\n\n> “Typo line 160 peusocode” and “Line 35”\n\n Thank you for the clarification suggestion – the typo will be fixed, and the line‑35 clarification added."}, {"review_id": "E9lCPoHa53", "reviewer": "Reviewer_XfB2", "summary": "The paper shows that naïve adversarial optimisation in cooperative/general-sum MARL provokes irrational self-sabotage. It formalises Rationality-preserving Policy Optimisation (RPO) and instantiates it with Rational Policy Gradient (RPG), which inserts a “manipulator” agent and higher-order opponent-shaping gradients so each learned policy remains a best-response to some partner. RPG variants of AP, AT, PAIRED and AD are evaluated on matrix games, STORM, Overcooked and Hanabi, demonstrating (i) avoidance of sabotage, (ii) higher cross-play robustness, and (iii) discovery of genuine policy diversity.", "questions": "What is “self-sabotage”, can you give a formal definition / theorem? It was first documented in Cui et al 2023 but I think it was just described instead of officially formalized. Moreover, the self-sabotage issue described in this paper and Cui et al 2023 can be easily patched by masking out invalid actions with large negative values?", "limitations": "1. The sample variance grows with look-ahead steps. \n2. Experiments do not cover >2 agents or partially-observable continuous control, so generalizability is uncertain.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strength**\n\n- This paper is well motivated to analyze and handle self-sabotage in diverse teammate generation problem, which has been a hard problem to define and tackle.\n- Provides a principled optimisation formalism (RPO) plus a general recipe (RPG) that retrofits several prior algorithms. Conceptually this algorithm is generalizable to any reinforcement learning algorithms.\n- Extensive experiments across four benchmarks; shows superior robustness/diversity versus strong baselines (SP, AD, CoMeDi, etc.). Ablations (look-ahead depth, partner-play coeff.) give insight into stabilizing dynamics.\n\n**Weakness**\n\n- Experiments remain in small/2-agent domains, although the formulation is pretty generalized.\n- Theoretical analysis is insufficient:\n    - L123, BR is a policy right? BR = \\arg\\max U(\\pi_{i}, \\pi_{-i}) for rigorousness\n    - No clear explanation on O_i and L_dice in this paper, even though it is referred to related work.\n    - Theoretically unclear why the method can prevent self-sabotage, and the paper does not define self-sabotage formally. Significant work should put into section 3 or 4 to explain the design of the method.\n    - The method looks pretty like a fixed-size population version of CoMeDi combined with DiCE update. So I am not sure how the authors compare a fixed-size population method with an iterative-expanding population method to make sure it is a fair comparison.\n- A few more related works on adversarial diversity & self-sabotage:\n    - (This paper has a study on how to design adversarial diversity to reduce self-sabotage) Charakorn, Rujikorn, Poramate Manoonpong, and Nat Dilokthanakul. \"Generating diverse cooperative agents by learning incompatible policies.\" *The Eleventh International Conference on Learning Representations*. 2023.\n    - (This paper uses Lagrange constraint satisfaction to anneal the self-sabotage) Rahman, M., Cui, J., & Stone, P. (2024, March). Minimum coverage sets for training robust ad hoc teamwork agents. In *Proceedings of the AAAI Conference on Artificial Intelligence* (Vol. 38, No. 16, pp. 17523-17530).", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. We are glad you found the work “well‑motivated” with “extensive experiments” and a “principled optimisation formalism.” We address each point individually:\n\n> “Experiments remain in small/2-agent domains, although the formulation is pretty generalized.”\n\nOur experiments use the same environments as many recent related research. Overcooked and Hanabi are regularly used to test various SOTA algorithms for cooperative environments. Indeed, Hanabi was identified as a challenge domain (Bard, Nolan, et al. \"The hanabi challenge: A new frontier for AI research.\") that still remains unsolved.\n\n\n> “L123, BR is a policy right? BR = \\arg\\max U(\\pi_{i}, \\pi_{-i}) for rigorousness?”\n\nThank you for catching this – this is a typo and will be fixed in the final version.\n\n\n> “No clear explanation on O_i and L_dice in this paper …”\n\nO_i is the optimization objective for agent i. For example, in the case of adversarial training, for agent adversary,  O_adversary = min_{π_adversary} U_victim (π_victim , π_adversary). It is first introduced in section 3.2 but we will add an explanation of the notation to the preliminary section for clarity.\n\nL_dice is formally defined in the appendix in section E.1 in equation (10), but we agree that this is an important detail to include in the main section and will include it in Section 4 with the extra page allowed in the final version.\n\n> “Theoretically unclear why the method can prevent self‑sabotage, and the paper does not define self‑sabotage formally.”\n> What is “self-sabotage”, can you give a formal definition / theorem? It was first documented in Cui et al 2023 but I think it was just described instead of officially formalized.”\n\nSimilar to existing work, we characterize self-sabotage as a general phenomenon when agents act against their own interests in order to minimize another player’s reward as part of an optimization problem. We do formally define Rationality-preserving Policy Optimization (Definition 3.1 on line 105) which we believe encapsulates the underlying issue behind self-sabotage by preventing agents from acting against their own interests. \n\n> “Moreover, the self-sabotage issue described in this paper and Cui et al 2023 can be easily patched by masking out invalid actions with large negative values?\n\nUnfortunately, the self-sabotage issue cannot simply be solved by masking out invalid actions. The sabotaging behavior is characterized as an entire state-action policy, involving a long sequence of actions and interactions that result in self-sabotage. It’s not clear how to precompute all of the policies that result in self-sabotaging behavior and it’s also not clear how to mask out this precomputed set of self-sabotaging policies during the optimization. \n\n> “The method looks pretty like a fixed-size population version of CoMeDi combined with DiCE update. So I am not sure how the authors compare a fixed-size population method with an iterative-expanding population method to make sure it is a fair comparison.”\n\nOur algorithm aims to solve the self-sabotage problem in a fundamentally different way than CoMeDi. While CoMeDi aims to limit the ability of agents to identify whether they are in self-play or cross-play during training, RPG entirely removes the incentive to self-sabotage. Appendix C includes a lengthy discussion of how RPG compares to existing approaches (including CoMeDi) and how our method overcomes the limitations of methods such as CoMeDi. Our paper demonstrates that CoMeDi fails to prevent self-sabotage in any of the environments and we don’t see how anything would improve for CoMeDi if it were changed to simultaneously learn the fixed-size population (as done with AT-RPG). Moreover, RPG is applicable to a wide range of adversarial optimization problems, whereas CoMeDi only applies to adversarial diversity.\n\n> “Sample variance grows with look‑ahead steps.”\n\nWe are not sure this is necessarily true and would appreciate any more intuition behind this claim. In the limit of look-ahead steps the base agent converges to some minima which could end up being very similar behavior (i.e., low variance). Some long-term meta-gradient approaches take advantage of this property using the implicit function theorem (which we leave to future work). Existing meta-gradient approaches (that use lookahead) have scaled to large settings with long lookahead (50+ step lookahead in Engstrom, Logan, et al. \"Optimizing ml training with metagradient descent.\"), so we believe there is a lot of potential for RPG to further scale.\n\n> “Related‑work omissions (Charakorn 2023, Rahman 2024).”\n\nThank you for bringing these to our attention. We will include a discussion of these related works in the final version."}, {"review_id": "dD0P6h7K7Q", "reviewer": "Reviewer_JDeS", "summary": "This paper introduces a new framework, Rationality-Preserving Policy Optimization (RPO), to address the problem of self-sabotage in adversarial optimization in multi-agent reinforcement learning (MARL). The authors propose the Rational Policy Gradient (RPG) algorithm to enforce rationality constraints via the introduction of manipulator agents. They demonstrate the framework’s ability to improve robustness, avoid sabotage, and produce diverse policies in cooperative and general-sum environments.", "questions": "Are manipulators optimized simultaneously with base agents, or is there an alternating schedule? Is there any stability concern when both influence each other via nested gradients?", "limitations": "No", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nThe paper addresses an important and understudied failure mode (self-sabotage) in adversarial MARL.\n\nThe introduction of manipulator agents and rationality constraints is novel and shows potential in expanding the utility of adversarial training beyond zero-sum games.\n\nExperimental results are comprehensive and support the claimed improvements in robustness, diversity, and stability.\n\nWeaknesses:\n\nWhile the paper provides strong motivation and intuitive examples (e.g., Figure 2), the mathematical formulation of the core problem is vague. For instance, the exact nature of the adversarial optimization problem being generalized is not clearly defined (e.g., is it a Markov game? Is the environment shared or decentralized?). The rationality constraint (Definition 3.1) is stated informally as “πi ∈ BR(π−i′)”, but the space of allowable π−i′ is not precisely defined. Are these stationary policies, stochastic, or deterministic?", "was_revised": true, "final_justification": "After reading the rebuttal and other reviewers' comments, I think this paper needs further improvement for an acceptance. Therefore, I will keep my current rating.", "rebuttal": "Thank you for your review. We appreciate your recognition that we “address an important and understudied failure mode” with “comprehensive experiments.” We address each point individually:\n\n> “the mathematical formulation of the core problem is vague. For instance, the exact nature of the adversarial optimization problem being generalized is not clearly defined (e.g., is it a Markov game? Is the environment shared or decentralized?). The rationality constraint (Definition 3.1) is stated informally as “πi ∈ BR(π−i′)”, but the space of allowable π−i′ is not precisely defined. Are these stationary policies, stochastic, or deterministic?”\n\nSection 2 contains some of these details: the problem is a general-sum partially-observable stochastic game (also known as a Markov game); the observations can be shared, decentralized, or partially shared according to the observation function O : S × A × Ω → [0, 1]; and the policies are stochastic.  We agree that these points are important and will be made more clear in the final version. The space of allowable π−i′ is the entire space Π−i and we will add this to the final version. Thank you for identifying these points as this will make the paper easier to understand.\n\n> “Are manipulators optimized simultaneously with base agents, or is there an alternating schedule? Is there any stability concern when both influence each other via nested gradients?”\n\nThe manipulators are indeed optimized simultaneously with the base agents. The updates in equations (3) and (4) happen together on each time step. Alternating or nested optimization schedules is a very interesting direction for future research. Stability concerns are often an issue in multi-agent learning settings and we did observe oscillating behaviors in certain matrix games that are known to have unstable equilibrium (Figure 13). However, increasing the number of lookahead steps in RPG actually leads to stabilizing this oscillating behavior (also in Figure 13). Further exploring the ability for manipulators to stabilize multi-agent learning dynamics would be another exciting direction for future research."}, {"review_id": "2pMZnOtM45", "reviewer": "Reviewer_gj2Y", "summary": "This paper introduces Rationality-preserving Policy Optimization (RPO), a new framework for adversarial optimization in multi-agent reinforcement learning that avoids the common failure mode of self-sabotage in cooperative or general-sum environments. Traditional adversarial training methods, successful in zero-sum games, perform poorly in cooperative settings because they incentivize agents to irrationally minimize each other's rewards, harming collective outcomes. To address this, the authors propose Rational Policy Gradient (RPG), a learning algorithm that trains agents to remain rational—optimal with respect to some teammate policy—while optimizing adversarial objectives through a novel “manipulator” agent structure. RPG enables robust, diverse, and self-consistent policy learning without explicit sabotage. Authors argue that RPG-based methods outperform existing baselines in robustness, diversity, and rationality across several empirical benchmark environments.", "questions": "1. Could the authors clarify what the variable n represents in Equation (4) and Algorithm 1?\n2. Are there any guidelines or intuitions for selecting the hyperparameters $\\alpha_1, \\alpha_2, \\alpha_3$ in Algorithm 1? For example, which one should be larger?\n3. There seems to be an inconsistency between the text and figure caption: Line 260 states that five policies are trained with different random seeds, while the caption of Figure 5 mentions two seeds. Please clarify the exact number of seeds used when evaluating robustness claims.\n4. In Figure 7, could you confirm whether the values under the adversarial attack columns refer to:\n(a) performance during adversarial training, or (b) performance after training without adversaries and testing with adversarial attacks? Additionally, as the content is tabular, it may be clearer to present it as a table rather than a figure.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n1.  Strengthen the contribution to explicitly connect the proposed unified framework to existing adversarial optimization algorithms and clarify how different methods can be seen as special cases.\n2. The toy example is a clear and well-constructed illustration of the problem, and the insights drawn from it are compelling.\n3. The paper gives an insightful theoretical framing that distinguishes rationality constraints from conventional adversarial training.\n\n## Weaknesses\n1. Regarding Claim 3: Finding rational adversarial examples, I am not fully convinced. For example, Line 306 states that PAIRED-A-RPG is more effective than AP-RPG at identifying policy weaknesses—but it is unclear how this conclusion is drawn. Could this be due to the added noise acting as an overly strong perturbation? Instead, I suggest evaluating each policy using a learned \"worst-case but rational\" adversary, and then reporting results under both (a) this learned attack and (b) a no-attack baseline. This would offer a more rigorous assessment of policy robustness.\n\n2. I understand the constraints imposed by the page limit, but including a conclusion section is important for completeness and helps reinforce the key takeaways of the paper.", "was_revised": true, "final_justification": "Most of my questions are for clarification and explanation. I understand more about this work after rebuttal, so I will keep my final score as shown.", "rebuttal": "Thank you for your review. We appreciate that “the paper gives an insightful theoretical framing” and the “insights drawn from [the examples] are compelling”. We address each point individually:\n\n> “Regarding Claim 3: Finding rational adversarial examples… it is unclear how this conclusion is drawn.”\n\nFigure 7 is quite nuanced and we will clarify each of the columns in the final version. If we understand your questions correctly, then your suggested (a) is in fact the AP-RPG and PAIRED-A-RPG columns: these are two different methods for constructing “worst-case but rational” adversaries against the different victims represented by the rows. The conclusion “PAIRED-A-RPG is more effective than AP-RPG” is drawn because between those two columns, PAIRED-A-RPG more consistently finds the worst-case (by achieving the lower score) than AP-RPG. Your suggestion for “(b) a no-attack baseline” is represented by the column “Training”: this shows the performance that the policy gets at the end of training (when paired with itself). Finally, the column “AP” simply demonstrates that the AP algorithm acts irrationally and always achieves zero reward. Thank you for raising these points since we believe this clarification will help the reader better understand the evidence supporting claim 3.\n\n> “Include a conclusion section …”\n\n We will add a proper conclusion with the extra page allowed for the camera-ready version. Here is a draft: \n\nWe introduce the problem of *rationality-preserving policy optimization* and give a solution in the form of a gradient-based algorithm called *rational policy gradient (RPG)*. We demonstrate the diverse uses of RPG by constructing algorithms that robustify agents, expose rational adversarial examples, and learn genuinely diverse behaviors across general-sum settings. Our results highlight that the success of adversarial optimization in zero-sum training methods can be extended to the general-sum setting and we aim to inspire further research in this direction. Furthermore, we hope that the generality of RPG will enable the development of entirely new classes of adversarial optimization algorithms in general-sum settings.\n\n> “Could the authors clarify what the variable n represents … ?”\n\nThe n in equation (4) should be an m, the number of policies in the optimization problem. The capital N variable in Algorithm 1 is the number of lookahead steps. We will add this clarification the caption of Figure 3. Thank you for improving the clarity of the paper with these comments.\n\n> “Are there any guidelines or intuitions for selecting the hyperparameters in Algorithm 1?”\n\nThis is a great question, a discussion of these hyperparameters can be found in appendix E.2 under section “Variable learning rates”, but will consider elevating it to the main text to make this point more visible. We often found it important to use a larger manipulator learning rate (alpha_2) than base agent learning rate (alpha_3) to magnify the manipulator’s ability to influence the base agent’s learning. A larger inner base agent learning rate (alpha_1) allows the manipulator to project the base agent’s update further into the future, similar to using additional lookahead steps.\n\n> “There seems to be an inconsistency between the text and figure caption: Line 260 states that five policies are trained with different random seeds, while the caption of Figure 5 mentions two seeds. Please clarify the exact number of seeds used when evaluating robustness claims.”\n\nWe agree that the wording is confusing and will be fixed in the final version. There are indeed 5 seeds per algorithm. The cross-play grids in Figure 5 depict the average score that a specific pair (i.e., “two seeds”) of policies get when paired as teammates in the two-player game.\n\n> “In Figure 7, could you confirm whether the values under the adversarial attack columns refer to: (a) performance during adversarial training, or (b) performance after training without adversaries and testing with adversarial attacks? \n\nIt is (b), the performance when testing with an adversarial attack (although some of the rows were also trained with adversarial methods). We agree that the explanation of this figure needs to be improved and will clarify this point along with your previous comments related to this figure.\n\n> Additionally, as the content is tabular, it may be clearer to present it as a table rather than a figure.”\n\nThank you for the suggestion, we will edit this in the final version."}, {"review_id": "9QLHRRnz4p", "reviewer": "Reviewer_6nz8", "summary": "In this paper, a method called RPO is proposed which is a formalism for adversarial optimization. The proposed method can avoid self-sabotage by ensuring agents remain rational. In detail, authors use opponent shaping techniques to optimize the adversarial objective. In summary， the contribution of this paper follows:\n(1) A novel algorithm, PRO, is proposed which can avoid self-sabotage by ensuring agents remain rational.\n(2) Authors empirically demonstrate the proposed algorithm and the proposed methos outperform existing baselines in popular cooperative environments.", "questions": "In this paper, a formalism called RPO is proposed which can overcomes the issue of self-sabotage. However, there are some problems:\n (1) The experiments are too toy. Four primary environment such as matrix games, Overcooked, STORM,Hanabi are performed. These experiments are too easy and only few agents play. More complex experiments should be performed,such as Markov Games with more agents, or \"StarCraft II\"\n(2) The details of neural networks for policy network should be given and study that how neural network parameters affect the results.\n(3)  \"Conclusion\" should be added at the last section of this paper.\n(4) Only pseudocode is available. How to obtain the open-source code and reproduce the experimental results?", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The strengths of this paper are:\n(1) The paper introduces a formalism called RPO that overcomes the issue of self-sabotage in any adversarial optimization algorithm.\n(2) The paper introduce a gradient-based deep learning algorithm called RPG.\n(3) The paper empirically demonstrates the method outperforms existing baselines.\nThe weaknesses of this paper are:\n(1) The experiments are too toy. Four primary environment such as matrix games, Overcooked, STORM,Hanabi are performed. These experiments are easy and only few agents. \n(2) Is it suitable for large number agents, or large scale Markov Game such as \"Go\" or \"MultiAgent System\",or StarCraft II?\n(3) There are some format error such as lacking \"conclusion\" at the last section of this paper.", "was_revised": true, "final_justification": "More complex experiments should be performed to validate your proposed method. For example more players or large-scale games. There are no novel results, thereby, I will keep my score.", "rebuttal": "Thank you for your review. We are grateful that you highlighted our “novel algorithm” and that the paper \"empirically demonstrates the method outperforms existing baselines\". We address each point individually:\n\n> “(1) The experiments are too toy. Four primary environment such as matrix games, Overcooked, STORM, Hanabi are performed. These experiments are too easy and only few agents play. More complex experiments should be performed,such as Markov Games with more agents, or \"StarCraft II\"\n\nOur experiments use the same environments as many recent related research. Overcooked and Hanabi are regularly used to test various SOTA algorithms for cooperative environments. Indeed, Hanabi was identified as a challenge domain (Bard, Nolan, et al. \"The hanabi challenge: A new frontier for AI research.\") that remains unsolved and in which existing algorithms (i.e., the algorithms used to solve Go) fail. Furthermore, the game is far from small considering how many possible combinations and sequences of cards each player can have and the fact that players need to communicate hidden information.\n\n> “Is it suitable for large number agents, or large scale Markov Game … ?”\n\nIn principle, all of our algorithms should scale to a large number of agents and larger Markov games. We observed a constant linear overhead (between 1.5-3 times) over existing algorithms across our experiments (a detail that we will include in the final version of the paper). We believe that further implementation and scaling compute resources will allow the algorithms to scale to these larger settings with the same benefits observed in our experiments.\n\n> “There are some format error such as lacking ‘conclusion’ …”\n\n We will add a proper conclusion with the extra page allowed for the camera-ready version. Here is a draft: \n\nWe introduce the problem of *rationality-preserving policy optimization* and give a solution in the form of a gradient-based algorithm called *rational policy gradient (RPG)*. We demonstrate the diverse uses of RPG by constructing algorithms that robustify agents, expose rational adversarial examples, and learn genuinely diverse behaviors across general-sum settings. Our results highlight that the success of adversarial optimization in zero-sum training methods can be extended to the general-sum setting and we aim to inspire further research in this direction. Furthermore, we hope that the generality of RPG will enable the development of entirely new classes of adversarial optimization algorithms in general-sum settings.\n\n> “Details of neural networks should be given...”\n\nThank you for catching this. Most hyperparameters for the experiments are given in appendix E.3 but we missed details about the neural network architecture: we used a network with two hidden layers of size 64 in Overcooked, STORM, and the matrix games, and two hidden layers with size 512 in Hanabi. The architecture is taken from a popular open-source implementation for self-play policies in these environments (Rutherford, Alexander, et al. \"Jaxmarl: Multi-agent rl environments and algorithms in jax.\"). These details will be added in the final version.\n\n> “‘Only pseudocode is available. How to obtain the open‑source code?’”\n\nWe will include a link to a full open-source implementation of the code in the final version of the paper along with scripts for reproducing the paper’s experiments."}]

Author rebuttal:

Meta-review: This paper introduces Rationality-preserving Policy Optimization (RPO), a framework that overcomes the self-sabotage problem that arises when applying adversarial optimization to cooperative multi-agent settings by ensuring that agents remain rational---i.e., that they adopt a best response policy to the policies of possible partner agents. The authors propose a gradient-based algorithm that solves RPO, called Rational Policy Gradient (RPG). The paper is well-motivated and addresses an important problem in multi-agent RL. Both the framework and the proposed technique appear novel, although they resemble some methods proposed in prior work. While some reviewers find the empirical evaluation comprehensive, others feel it could be strengthened with additional complex multi-agent environments. That said, the approach is empirically evaluated using standard benchmarks for cooperative multi-agent RL. Some aspects of the paper, including the formal setting, could be further clarified in the final version, and the authors are encouraged to do so.
Decision: Accept (poster)