Advantage Alignment Algorithms
Authors: Juan Agustin Duque
Milad Aghajohari
Tim Cooijmans
razvan ciuca
Tianyu Zhang
Gauthier Gidel
Aaron Courville
Venue: ICLR 2025 Oral (2025)
Keywords: Multi-agent Reinforcement Learning
Opponent Shaping
Social Dilemmas
General-Sum Games
Primary area: reinforcement learning
TL;DR: We introduce Advantage Alignment, a new family of algorithms for opponent shaping in general-sum games, designed to promote cooperation and avoid suboptimal outcomes.

Abstract: Artificially intelligent agents are increasingly being integrated into human decision-making: from large language model (LLM) assistants to autonomous vehicles. These systems often optimize their individual objective, leading to conflicts, particularly in general-sum games where naive reinforcement learning agents empirically converge to Pareto-suboptimal Nash equilibria. To address this issue, opponent shaping has emerged as a paradigm for finding socially beneficial equilibria in general-sum games. In this work, we introduce Advantage Alignment, a family of algorithms derived from first principles that perform opponent shaping efficiently and intuitively. We achieve this by aligning the advantages of interacting agents, increasing the probability of mutually beneficial actions when their interaction has been positive. We prove that existing opponent shaping methods implicitly perform Advantage Alignment. Compared to these methods, Advantage Alignment simplifies the mathematical formulation of opponent shaping, reduces the computational burden and extends to continuous action domains. We demonstrate the effectiveness of our algorithms across a range of social dilemmas, achieving state-of-the-art cooperation and robustness against exploitation.

Published as a conference paper at ICLR 2025
ADVANTAGE ALIGNMENT ALGORITHMS
Juan Duque Milad Aghajohari*, Tim Cooijmans, Razvan Ciuca, Tianyu Zhang, Gauthier Gidel, Aaron Courville University of Montreal & Mila
firstname last name umontreal.ca
ABSTRACT
Artificially intelligent agents are increasingly being integrated into human decision-making: from large language model (LLM) assistants to autonomous vehicles. These systems often optimize their individual objective, leading to conflicts, particularly in general-sum games where naive reinforcement learning agents empirically converge to Pareto-suboptimal Nash equilibria. To address this issue, opponent shaping has emerged as paradigm for finding socially beneficial equilibria in general-sum games. this work, we introduce Advantage Align- ment, a family of algorithms derived from first principles that perform opponent shaping efficiently and intuitively. We achieve this by aligning the advantages interacting agents, increasing the probability of mutually beneficial actions when their interaction has been positive. We prove that existing opponent shaping meth- ods implicitly perform Advantage Alignment. Compared to these methods, Ad- vantage Alignment simplifies the mathematical formulation of opponent shaping, reduces the computational burden and extends to continuous action domains. We demonstrate the effectiveness of our algorithms across range of social dilemmas, achieving state-of- -ottte--f-he--eee the-art cooperation and robustness against exploitation.
INTRODUCTION
Recent advancements in artificial intelligence, such as language models like GPT (Radford et al., 2018), image synthesis with diffusion models (Ho et al., 2020), and generalist agents like Gato (Reed et al., 2022), suggest future where Al systems seamlessly integrate into everyday human decision-making. While these systems often optimize for the goals of their individual users, this can lead to conflicts, especially in tasks that involve both cooperative and competitive elements. Social dilemmas, as introduced by Rapoport and Chammah (1965), describe scenarios where agents acting selfishly achieve worse outcomes than if they had cooperated. A global example is the climate change problem, where individual and national interests in economic growth often clash with the need for collective action to reduce carbon emissions and mitigate environmental degradation. The challenges we have faced in tackling this problem highlight the complexity of aligning individual interests with collective well-being.
As artificially intelligent systems become ubiquitous, there a pressing need to develop methods that enable agents autonomously align their interests with one another. Despite this, the deep re- inforcement learning community has traditionally focused on fully cooperative or fully competitive settings, often neglecting the nuances of social dilemmas. Sandholm and Crites (1996) empirically demonstrated that naive reinforcement leamning algorithms tend to converge to the worst Pareto sub- optimal Nash equilibria of Always Defect in social dilemmas like the Iterated Prisoner's Dilemma (IPD). Foerster et al. (2018b), demonstrated that the same is true for policy gradient methods, and introduced opponent shaping (LOOAA) to address this gap.
LOLA is an opponent shaping algorithm that influences the behavior of other agents by assuming they are naive learners and taking gradients with respect to simulated parameter updates. Following this approach, other opponent shaping algorithms that compute gradients with respect to simulated parameter updates have shown success in partially competitive tasks, including SOS (Letcher et al., 2021), COLA (Willi et al., 2022), and POLA (Zhao et al., 2022). More recently, LOQA (Aghajo- hari et al., 2024b) proposed an alternative form of opponent shaping by assuming control over the value function of other agents via REINFORCE estimators (Williams, 1992). This new approach to

---

Published as a conference paper at ICLR 2025
opponent shaping offers significant computational advantages over previous methods and lays the foundation for our work.
We introduce Advantage Alignment, family of algorithms designed to shape rational opponents by aligning their advantages when their historic interactions have been positive. We make two key as- sumptions about reinforcement learning agents: (1) they aim to maximize their own expected return, and (2) they take actions proportionally to this expected return. Under these assumptions, we demon- strate that opponent shaping reduces to aligning the advantages of different players and increasing the log probability of an action proportionally to their alignment. We show that this mechanism lies the heart of existing opponent shaping algorithms, including LOLA and LOQA. By distilling this objective, Advantage Alignment agents can shape opponents without relying on imagined parameter updates (as in LOLA and SOS) or stochastic gradient estimation that relies on automatic differen- tiation introduced in DiCE (Foerster et al., 201 (as in POLA, COLA, and LOQA). Furthermore, we demonstrate that Advantage Alignment preserves Nash Equilibria, ensuring that our algorithms maintain stable strategic outcomes.
We also introduce Proximal Advantage Alignment, which formulates Advantage Alignment as a modification to the advantage function used in policy gradient updates. By integrating this modified advantage into the Proximal Policy Optimization (PPO) (Schulman et al., 2017b) surrogate objec- tive, we develop scalable and efficient opponent shaping algorithm suitable for more complex environments. To identify and overcome challenges that arise from scale-which are often over- looked in simpler settings like the Iterated Prisoner's Dilemma (Rapoport and Chammah, 1965) and the Coin Game (Foerster et al., 201 2018b)-w apply Advantage Alignment to continuous variant of the Negotiation Game (Cao al., 2018) and Melting Pot's Commons Harvest Open (Agapiou al., 2023). In doing so, we aim to demonstrate the scalability of our methods and offer insights and solutions applicable to complex, real-world agent interactions.
Our key contributions are:
We introduce Advantage Alignment and Proximal Advantage Alignment (PAA), two opponent shaping algorithms derived from first principles and based on policy gradient estimators.
We prove that LOLA (and its variations) and LOQA implicitly perform Advantage Alignment through different mechanisms.
We extend REINFORCE- opponent shaping to continuous action environments and achieve state-of-fe -fethe-rrt results in continuous action variant of the Negotiation Game (Cao et al., 2018).
We apply PAA to the Commons Harvest Open environment in Melting Pot 2.0 (Agapiou et al., 2023), a high dimensional version of the tragedy of the commons social dilemma, achieving state- of-the-art results and showcasing the scalability and effectiveness of our methods.
2
BACKGROUND
2.1
SOCIAL DILEMMAS
Social dilemmas describe situations in which selfish behavior leads to sub-optimal collective out- comes. Such dilemmas are often formalized as normal form games and constitute a subset of general-sum games. A classical example of social dilemma is the Iterated Prisoner's Dilemma (IPD) (Rapoport and Chammah, 1965), in which two players can choose one of two actions: co- operate or defect. In the one-step version of the game, the dilemma occurs because defecting is dominant strategy, i.e., independently of what the opponent plays the agent is better off playing de- fect. However. by the reward structure the game, both the agent and the opponent would achieve a higher utility if they played cooperate simultancously. Beyond the IPD, other social dilemmas have been extensively studied the literature, including the Chicken Game and the Coin Game (Lerer and Peysakhovich, 201 8), the latter of which has a similar reward structure to IPD but takes place in grid world. In this paper we introduce a variation of the Negotiation Game (also known as the Exchange Game) Vault et al., 2015; Lewis et al., 2017), with a strong social dilemma compo- nent. Additionally, we evaluate our method on the Commons Harvest Open environment in Melting Pot 2.0 (Agapiou al., 2023), which exemplifies large-scale social dilemma. In this environment, agents must balance short-term personal gains from overharvesting common resources against the long-term collective benefit of sustainable use.
2

---

Published as a conference paper at ICLR 2025
2.2 MARKOV GAMES
In this work, we consider fully observable, general sum, n-player Markov Games (Shapley, 1953) which represented by tuple: M [NN,S, A, P, R.R)) 7). Here S is the state space, A A x x... x AA", is the joint action space for all players, P S x A + ((S) maps from every state and joint action a probability distribution over states, R {r ..... r"] is the of reward functions where each r' S x A + R maps every state and joint action to scalar return and y E [0, 1] the discount factor.
2.3 REINFORCEMENT LEARNING
Consider two agents playing a Markov Game, (agent) and 2 (opponent), with policies n² and n², parameterized by 0, and 02 respectively. We follow the notation of Agarwal et al. (2021), let denote trajectory with initial state distribution μ and (unconditional) distribution given by:
Pry (7) = (u(s)) (aol=u)= x ( P(s1 (o, o0, bu)
(1)
Where P(-ss, a, b), often referred as the transition dynamics, is a probability distribution over the next states conditioned on the current state being s, agent taking action a and opponent taking action b. Value- based methods like Q-learning ( (Watkins and Dayan, 1992) and SARSA (Rummery and Niranjan, 1994) learn an estimate of the discounted reward using the Bellman equation:
Q' ae, bt) r² (s at by) + 7 E..., [V*(s++)) a,, bi].
In policy optimization, both players aim to maximize their expected discounted return by performing gradient ascent with a REINFORCE estimator (Williams, 1992) of the form:
Ve V (w)
Pr ! in2
X' A (st, at Vo, log (a.ls(a
(3)
Here As,s, Q' (s. a, b) (s) denotes the advantage of the agent taking action a in state while the opponent takes action b.
3 OPPONENT SHAPING
Opponent shaping, first introduced in LOLA (Foerster et al., 2018b), is a paradigm that assumes the learning dynamics of other players can be controlled via some mechanism to incentivize desired behaviors. LOLA and its variants assume that the opponent is a naive learner, i.e. an agent that performs gradient ascent their value function, and differentiate through an imagined naive update of the opponent in order to shape it.
LOQA (Aghajohari al., 2024b) performs opponent shaping by controlling the Q-values of the opponent for different actions assuming that the opponent's policy is softmax over these Q-values:
# (belst) = exp Q² (se, b.)
(4)
Eo exp (2(ss,, b))
where Q" bt)
( (st, a, b.)]. The key idea is that these Q-values depend on n², and Banx' (Q2
hence the opponent policy #² can be differentiated w.r.t. 0,:!
(vo, Vo,.P(ssn
V 4 (b, st) # (b.lss)
Vo
(5)
This dependency of 2 on 0, leads to the emergence of an extra term in the policy gradient:
Ve, (²) = Puppp .n?
Σ 7 A' ar, b.) t=
Ve, log (arlst) Vo. log # (bel (st)
(6)
policy gradient term
upponent shaping term
Aghajohari et al. (2024b) demonstrate an effective way to account for this dependency using REIN- FORCE. The present work builds on the ideas of LOQA, but reduces opponent shaping to its bare components to derive Advantage Alignment from first principles.
See Appendix A.3 for derivation of this expression.
3

---

Published as a conference paper at ICLR 2025
Algorithm Advantage Alignment
Initialize: Discount factor " agent Q-value parameters ø' Q-value parameters of actor pa- rameters 0¹, opponent Q-value parameters o², Q-value parameters of actor parameters # for iteration= .....dd 2,.... do
Run policies r² and x² for T timesteps in environment and collect trajectory Compute agent critic loss LE using the TD error with r² and V Compute opponent critic loss L using the TD error with r² and V Optimize LG w.r.t. ø' and L3 w.r.t. o² with optimizer choice Compute generalized advantage estimates {A A/ {Ap..... A Compute agent actor loss, L summing equation 3 and equation Compute opponent actor loss, 12g summing equation 3 and equation 8 Optimize T. w.r.t 0 and 1 w.r.t. 8 with optimizer of choice
4 ADVANTAGE ALIGNMENT
4.1 МЕTHOО DESCRIPTION
Motivated by the goal of scaling opponent shaping algorithms more diverse and complex scenar- ios, we derive simple and intuitive objective for efficient opponent shaping. We begin from the assumptions that agents are learning to maximize their expected return, and will behave fashion that is proportional to this goal:
Assumption 1. Each agent learns to maximize their value function: max Vi()).
Assumption 2. Each opponent acts proportionally the exponent of their action-value function: r' (alss) & exp (3- Q'(s.a))
Using Equation 6 and substituting #² in place of n² (per Assumption 2), we obtain:
oo
Vo, (1) Pri
Σ A''A'(s ae, be) (V, log n (ar + Ve, log # (b. (bolss))
The first term is the usual policy gradient. The second term is the opponent shaping term and will be our focus. Approximating the opponent's policy (right hand side of equation 4) by ignoring the contribution due to the partition function, the opponent shaping term becomes:
2 Σ
3 EE Pej
A ac, b,) V.) Q
The gradient of the Q-value can be estimated by REINFORCE, estimator, which leads to a nested expectation. Aghajohari et al. (2024b) empirically showed that this nested expectation can be effi- ciently estimated from a single trajectory. We take the same approach (see Appendix A. 1) to obtain:
8
Σ A"' (sk, ak, (x, A ar by Vor n (a. (ar(st)
The expression above captures the essence of opponent shaping: agent should align its advantages with those of its opponent in order to steer towards trajectories that are mutually beneficial. More precisely, an agent increases the probability of actions that have high product between the sum of its past advantages and the advantages of the opponent at the current time step. For implementation details of the Advantage Alignment formula see Appendix A.6. Equation 8 depends only the log probabilities of the agent, which allows us to create a proximal surrogate objective that closely follows the PPO (Schulman et al., 2017b) formulation:
Topitt [min (rn((1))))) (0:) ( (st, as, bz), clip (r. e, 1 + (st, at, b,)]l,
(9)
where Tn denotes the ratio of the policy (new) after n updates of the original policy (old), and:
A (st, ac bb) = A5 ar, be) + By
Σ
yt t-k A' ax, bx)) A A2 an
(10)
4

---

Published as a conference paper at ICLR 2025
Advantage Alignment (AdAlign)
A
1.0
Probability of Cooperation
Σ Rxct
xd-
A
0.6
0.0
State
(h)
Figure 1: (a) The sign of the product of the gamma- discontdd past advantages for the agent, and the current advantage the opponent, indicates whether the probability of taking an action should increase decrease. (b) The empirical probability of cooperation of Advantage Alignment for each previous combination of actions in the one step history Iterated Prisoner's Dilemma, closely resembles tit-for-tat. Results are averaged over 10 random seeds, the black whiskers show one std.
This surrogate objective in equation is used to formulate the Proximal Advantage Alignment (Prox- imal AdAlign) algorithm (see appendix A.7 for implementation details).
Why is Assumption 1 necessary? Assumption 1 allows the agent to influence the learning dy- namics of the opponent, by controlling the values for different actions. After one iteration of the algorithm the agent changes the Q-values of the opponent for different actions and, since the oppo- nent aims to maximize their expected return, must change its behavior accordingly.
4.2 ANALYZING ADVANTAGE ALIGNMENT
Equation yields four possible different cases for controlling the direction of the gradient of the log probability of the policy. As with the usual policy gradient estimator, the sign multiplying the log probability indicates whether the probability of taking action should increase itively, when the interaction with the opponent has been positive (blue in figure la) the advantages of the agent align with that of the opponent: the advantage alignment term increases the log probability of taking an action if the advantage of the opponent is positive and decreases if it is negative. In contrast, the interaction has been negative (red figure la) the advantages are at odds with each other: the ad advantage alignment term decreases the log probability of taking an action if the advan- tage of the opponent positive and increases it if negative. We now relate existing opponent shaping algorithms to advantage alignment, and argue that these algorithms use the same underlying mechanisms. Theorem shows that LOLA update from Foerster et (2018b) can be written policy gradient method with opponent shaping term similar to equation 10. This shows the funda- mental relationship between opponent-shaping dynamics and advantage multiplications. Theorem 2 proves that LOQA's opponent shaping term has the same form that of Advantage Alignment, differing only by a scalar term.
decrease. Intu-
Theorem (LOLA as an advantage alignment estimator). Given a two-player game where players I and have respective policies x (als) and T² (b)s), where each policy is parametrised such that the set of gradients Vo, log n² (a)s) for all pairs (a, 8) form an orthonormal basis, the LOLA update for the first player correspond to a reinforce update with the following opponent shaping term
DO
2 d
Σ
dn
Vo- log x (arls
(11)
where Al. = A (sx, Cx by) and dy. is the occupancy measure of the tuple (ag, by 8x) and 3 is the step size the naive learner. See appendix A.2 for proof.
5

---

Published as a conference paper at ICLR 2025
Theorem 2 (LOQA as advantage alignment estimator). Under Assumption 2, the opponent shap- ing term in LOQA is equivalent to the opponent shaping term in Equation 8 up to (1 - # (bx (sk))
3- E
1+1
Σ kCt.
(1- (bb V or log r (arls))
(12)
(=0
#² (be (e) approximates the opponent policy as defined in LOQA. For a proof see appendix A.S.
Having established the connection between existing opponent shaping algorithms and Advantage Alignment, we now focus analyzing the theoretical properties of Advantage Alignment itself. We investigate the impact of the Advantage Alignment term on Nash equilibria. Theorem 3 demon- strates that Advantage Alignment preserves Nash equilibria, ensuring that if agents are already play- ing equilibrium strategies, the Advantage Alignment updates will not cause the policy gradient to deviate from them locally.
Theorem 3 (Advantage Alignment preserves Nash equilibria), Advantage Alignment preserves Nash equilibria. That is, if a joint policy (æi, constitutes a Nash equilibrium, then applying the Advantage Alignment formula will not change the policy, as the gradient contribution of the advantage alignment term is zero. The proof can be found in Appendix A.8.
EXPERIMENTS
We follow the evaluation protocol of LOQA (Aghajohari et al., 2024b), where the fixed policy that is generated by the algorithm is evaluated zero-shot against a distribution of policies.
5.1 ITERATED PRISONER'S 'S DILEMMA
We consider the full history version of IPD, where a gated recurrent unit (GRU) policy conditions on the full trajectory of observations before sampling action. In this experiment follow the architecture used in POLA (Zhao et al., 2022) (for details see appendix B.I). We also consider trajectories of length 16 with discount factor, 7, of 0.9. As shown in figure 1b, Advantage Align- ment agents consistently achieve a policy that resembles tit-for-tat (Rapoport and Chammah, 1965) empirically. Tit-for-tat consists of cooperating on the first move and then mimicking the opponent's previous move in subsequent rounds.
5.2
COIN GAME
The Coin Game is a 3x3 grid world environment where two agents, red and blue, take turns collecting coins. During each turn, coin of either red or blue color spawns at a random location on the grid. Agents receive a reward of +1 for collecting any coin but incur a penalty of if the opponent collects coin of their color. A Pareto-optimal strategy in the Coin Game is for each agent to collect only the coins matching their color, as this approach maximizes the total returns for both agents. Figure demonstrates that Advantage Alignment agents perform similarly to LOQA agents when evaluated against a league different policies: Advantage Alignment agents cooperate with themselves, cooperate with Always Cooperate (AC) and not exploited by Always Defect (AD).
5.3 NEGOTIATION GAME
In the original Negotiation Game, two agents bargain over n types of items over multiple rounds. In each round, both the quantity of items and the value each agent places on them are randomly set, but the agents only know their own values. They take turns proposing how to divide the items over a random number of turns. Agents can end the negotiation by agreeing to a proposal, and rewards are based on how well the agreement matches their private values. If they don't 't reach an agreement by the final turn, neither gets a reward. We modify the game first by making the values public, otherwise Advantage Alignment would have unfair edge over PPO agents by using the opponent's is value function. Secondly, we do one-shot, simultaneous negotiations instead of negotiation rounds lasting multiple iterations. Third, For a given item with agent value u. the reward of the agent ra depends on the proposal of the agent Pa and the proposal of the opponent po where Pa, Po E [0, 5]: ra Pa Va/ max(5. Pa po).
modify the reward function so that every negotiation yields reward.

---

Published as a conference paper at ICLR 2025
Kohigp
LOGA
POLA
MPO8
Raeodom
Return
AdAIgn
0.3
0.29
0.30
0.24
0.06
0.32
0.12
4.15
LOGA
0.28
0.30
0.17
0.16
0.35
40.05
0.28
0.2
0.16
0.17
0.19
0.01
0.29
0.77
4.11
POLA
0.24
0.24
0.19
0.19
0.38
.0.03
0.24
0.16
0.41
0.11
0.20
AC
4.27
0.1
0.08
0.05
0.03
0.42
0.27
0.00
0.53
AD
0.15
0.12
0.07
0.41
0.88
0.00
0.53
-02
0.25
0.26
0.24
0.20
0.05
0.53
0.01
Random
0.18
-0.15
-0.13
-0.11
0.11
4.53
-0.01
Figure 2: League Results of Advantage Alignment agents Coin Game: LOQA, POLA, MFOS, Always Cooperate (AC), Always Defect (AD), Random and Advantage Alignment (AdAlign). Each number in the plot is computed by running 10 random seeds of cach agent head to head with 10 seeds of another for 50 episodes of length 16 and averaging the rewards.
Note that the max operation at the denominator serves to break the invariance of the game dynamics to the scale of proposals. For example, without the max operation, there would no difference between Pa 1. = 1 and P. 5, P. 5. The social dilemma in this version of the negotiation game arises because both agents are incentivized to take as many items as possible, but by doing so, they end up with lower return compared the outcome they would achieve if they split the items based on their individual utilities. A Pareto-optimal strategy entails allowing the agent to take all the items that are more valuable them, and similarly for their opponent (this constitutes the Always Cooperate (AC) strategy in Figure 3a). We experiment with high nigh-contrast setting where the utilities of objects for the agents are orthogonal to each other: There are two possible combinations of values in this setup: = 5. 1 or C = 1, = 5.
As shown Figure 3a, PPO agents do not learn to solve the social dilemma. They learn the naive policy of bidding high for every item which means they get low return against themselves. PPO agents trained with shared rewards get high return against themselves, only exploited by PPO agents. They do not learn abandon cooperation and retaliate after they are defected against. Advantage Alignment agents solves the social dilemma. They cooperate with themselves while remaining non-exploitable against Always Defect.
5.4 MELTING Pot's COMMONS HARVEST OPEN
In Commons Harvest Open (Agapiou et al., 2023), a group of 7 agents interact environment in which there is 6 bushes with different amounts of apples. Agents receive reward of I for any apple consumed. Consumed apples regrow with probability dependent on the number of apples in their L2 neighborhood; specifically, if there This mechanism creates a tension between agents: they must exercise restraint to prevent extinction while also feeling compelled consume quickly out of fear that others may over-harvest.
apples nearby, consumed apples do not regrow.

---

Published as a conference paper at ICLR 2025
PPOUR
Return
0.50
0.39
0.44
2.52
0.45
0.37
UER
0.25
0.30
0.26
0.30
0.26
40
Agent
Agant (PPO)
0.25
0.34
0.30
(AdAlign)
Propooa
Adk gF
0.35
0.29
0.23
0.30
2.30
0.27
0.27
Usity
Agent
0.38
0.41
0.37
40
0.36
Prososa
Proplca
PPO 5R
0.46
0.35
(a)
(b)
Figure 3: (a) League Results of the Advantage Alignment agents in the Negotiation Game: Always Cooperate (AC), an agent which proposes 5 for items which are more valuable to it and 1 for items that are less valuable to it, Always Defect (AD), an agent that proposes regardless of the values, Advantage Alignment (AdAlign), PPO and PPO summing rewards (PPO-SR). Each number in the plot computed by running 10 random seeds of each agent head to head with 10 seeds another for 50 episodes of length 16 and averaging the rewards. Note that against Always Defect, Always Cooperate gets an average return of 0.25 while Always Defect gets 0.30. (b) Sample trajectories of AdAlign vs. AdAlign and PPO vs. PPO in the negotiation game. The numbers show the utilities and proposals, which have been rounded to integer values. AdAlign agents defect first (red) and progressively cooperate with each other (blue) while PPO agents Always Defect.
exploiter
adalign
random
ppo
ppo _
acb
vmpo
opre
acb bcb_p
opre_p
scenario 0
1.78
1.15
0.33
0.91
0.87
0.93
0.84
0.95
0.53
0.00
scenario_1 1.48
0.74
0.45
0.76
0.80
0.85
0.77
0.94
0.52
0.00
average
1.63
0.94
0.39
0.83
0.83
0.89
0.81
0.94
0.52
0.00
Figure 4: Comparison of different reinforcement learning algorithms Melting Pot's 2.0. Commons Harvest Open. The score is the focal return per capita, min-max normalized between random agent and an exploiter baseline (ACB agent with an LSTM policy/value network) trained for 10 steps. Following the protocol of the Melting Pot contest, select the best agent out of 10 seeds and evaluate it 100 times.
There are a number of complications that make the Melting Pot environments particularly challeng- ing. First, the environments are partially-observablee agents can only see a local window around themselves. Second, the partial observations are in the form of nigh-dimensional raw pixel data. Third, these environments often involve multiple agents- seven the case of Commons Harvest Open-which increases the complexity of interactions and coordination. Therefore, agents need to remember past interactions with other agents to infer their motives and policies. All these factors, combined with the inherent social dilemma reward structure of the game, make finding policies that are optimal with respect to social and individual objectives a non-trivial task.
We train GTrXL transformer (Parisotto al., 2019) for 34k steps, with context length of 30, and compare the normalized focal return per capita of our agents against the baselines in Melting Pot 2.0: Advantage-crine -rinti bascline (acb) (Espeholt et al., 2018), V-MPO (vmpo) (Song et al., 2019), options as responses (opre) /Vezhnevets et al., 2020), and prosocial versions of opre (opre p) and acb (acb p) that encourage cooperation. We also compare
implementations of PPO (ppo) and PPO
8

---

Published as a conference paper at ICLR 2025
Step
Step
Step
adalign
000
Figure 5: Frames evaluation trajectories for different algorithms. Qualitatively, we demonstrate that Proximal Advantage Alignment (adalign) also outperforms naive PPO (ppo) and PPO with summed rewards. The evaluation trajectories show how adalign agents are able to maintain a bigger number of apple bushes from extinction (2) for longer time that either ppo ppo.p. Note that in the Commons Harvest evaluation two exploiter agents, green and yellow, play against a focal population of 5 copies of the evaluated algorithm.
with summed rewards (ppo.p). Figure 4 shows that our best advantage alignment agent achieves on average 1.63 normalized per capita focal return in the Commons Harvest evaluation scenarios, significantly outperforming all baselines (see Appendix B.4). Figure 5, qualitatively shows the reason why Proximal Advantage Alignment outperforms PPO and PPO with summed rewards on one of the evaluation scenarios.
6 RELATED WORK
The Iterated Prisoner's Dilemma (IPD) was introduced by Rapoport and Chammah (1965). Tit-for- tat was discovered robust strategy against a population of opponents in IPD by Axelrod (1984), who organized multiple IPD tournaments. It was discovered only recently that IPD contains strate- gies that extort rational opponents exploitable cooperation (Press and Dyson, 2012). Sandholm and Crites (1996) were the first to demonstrate that two Q-learning agents playing IPD converge to mutual defection, which suboptimal. Later, Foerster et (2018b) demonstrated that the same true for policy gradient methods. Bertrand et al. (2023) were able to show that with optimistic initialization and self-play, Q-learning agents find a Pavlov strategy in IPD.
Opponent shaping was first introduced in LOLA Foerster et al. (2018b), as method for controlling the learning dynamics of opponents in a game. LOLA agent assumes the opponents are naive step look-ahead optimization update of the opponent. More
leamners and differentiates through formally, LOLA maximizes V (0) 8² + A62) where A is naive learning step in the direction that maximizes the opponent's value function 2 02)). Variations of LOLA have been introduced

---

Published as a conference paper at ICLR 2025
to have formal stability guarantees (Letcher et al., 2021), learn consistent update functions assuming mutual opponent shaping (Willi et al., 2022) and be invariant to policy parameterization (Zhao et al,, 2022). More recent work performs opponent shaping by having agent play against a best response approximation of their policy (A (Aghajohari et al., 2024a). LOQA (Aghajohari et al., 2024b), on which this work is based, performs opponent shaping by controlling the Q-values of the opponent using REINFORCE (Williams, 1992) estimators.
Another approach to finding socially beneficial equilibria in general sum games relies on modeling the problem as meta-game, where meta-rewards correspond to the returns on the inner game, meta- states correspond to joint policies of the players, and the meta-actions are updates to these policies. Al-Shedivat et al. (2018) introduce continuous adaptation framework for multi-task learning that uses meta-learning to deal with non-stationary environments. MFOS (Lu al., 2022) uses model- free optimization methods like PPO and genetic algorithms to optimize the meta-value of the meta- game. More recently Meta-Value Learning (Cooijmans et al., 2023) parameterizes the meta-valus value as neural network and applies Q-learning to capture the future effects of changes to the inner policies. Shaper (Khan et al., 2024), scales opponent shaping to high-dimensional general-sum games with temporally extended actions and long time horizons. It does so, by simplifying MFOS and effectively capturing both entra-episode and inter-episode information.
Melting Pot 2.0 (Agapiou et al., 2023) introduces comprehensive suite of multi-agent reinforce- ment learning environments that focus on social interactions and coordination challenges, providing valuable benchmark for evaluating the scalability and effectiveness of reinforcement learning algo- rithms in complex, cooperative-compttiii settings. The Negotiation Game, introduced by De Vault al. (2015); Lewis et al. (2017) and subsequently refined by Cao et al. (2018), has proven to be significant benchmark for studying general-sum games. It integrates elements of strategy and social dilemmas, necessitating that agents balance cooperation and competition to optimize their outcomes. Noukhovitch et al. (2021) analyze this complex benchmark, underscoring its importance in the field. Future investigations will turn towards an even more sophisticated simulation proposed by Zhang et al. (2022), which involves negotiations among countries and regions with diverse resource distri- butions and preferences in addressing climate change.
CONCLUSION
In this work, we introduced Advantage Aligument, novel family of algorithms designed to ad- dress the fundamental challenge of achieving self-interested cooperation multi-agent reinforce- ment learning, particularly in social dilemmas. By deriving our algorithms from first principles, we distilled opponent shaping to its core components, providing a simple yet powerful mechanism to align agents' advantages and foster mutually beneficial behaviors. Our approach unifies and gen- eralizes existing opponent shaping methods, such as LOLA and LOQA, demonstrating that they implicitly perform Advantage Alignment through different mechanisms. This unification not only simplifies the mathematical formulation of opponent shaping but also reduces computational com- plexity, enabling more efficient and scalable algorithms.
Our experiments across a range of social dilemmas, including the Iterated Prisoner's Dilemma, Coin Game, and continuous action variant of the Negotiation Game, demonstrate that Advantage Aligmment consistently achieves state-of-the-art cooperation and robustness against exploitation. Notably, we extended our methods complex, large-scale, general-sum environments like Melting Pot's Commons Harvest Open, addressing challenges that arise from partial observability, high- dimensional observations, and multi-agent interactions. In these settings, Advantage Aligmment agents learned sophisticated strategies that balance individual and collective interests, showcasing the potential of our algorithms to scale to real-world applications.
The significance of our work lies in providing a principled, efficient, and scalable solution to the longstanding problem of self-interested cooperation in general-sum games. By enabling agents to autonomously align their interests with one another, Advantage Alignment paves the way for more harmonious and socially beneficial interactions in artificial intelligence systems integrated into human decision-making processes. This has profound implications for the development of Al agents in diverse domains, from autonomous vehicles navigating shared environments to AI assistants collaborating with humans and other agents.
10

---

Published as a conference paper at ICLR 2025
8 ACKNOWLEDGEMENTS
J.D. would like to thank Michael Noukhovitch for the discussions related to the Negotiation Game We would also like acknowledge the technical support of Olexa Bilaniuk. This work was fi- nancially supported by A.C.'s CIFAR Canadian Al chair and Canada Research Chair in Learning Representations that Generalize Systematically.
REFERENCES
Agapiou, J. Vezhnevets, S., Dueñez-Guzzáán B. A., Matyas, J., Mao, Y., Sunehag, P., Köster, R., Madhushani, U., Kopparapu, K., Comanescu, R., Strouse, D., Johanson, M. B., Singh, S., Haas, J., Mordatch, I., Mobbs, D., and Leibo, Z. (2023). Melting 2.0. 2.0.
Agarwal, A., Jiang, N., Kakade, S., and Sun, W. (2021). Reinforcement Learning: Theory and Algorithms. Preprint.
Aghajohari, M., Cooijmans, T., Duque, J. A., Akatsuka, S., and Courville, A. (2024a). Best response shaping.
Aghajohari, M., Duque, J. A., Cooijmans, T., and Courville, A. (2024b). Loqa: Learning with opponent q-learning awareness.
Al-Shedivat, M., Bansal, T., Burda, Y., Sutskever, I., Mordatch, and Abbeel, P. (2018). Continu- ous adaptation via meta-leaming in nonstationary and competitive environments.
Axelrod, R. (1984). The Evolution of Cooperation. Basic, New York.
Bertrand, Q. Duque, J., Calvano, E., and Gidel, G. (2023). Q-learners can provably collude in the iterated prisoner's dilemma.
Cao, K., Lazaridou, A., Lanctot, M., Leibo, J. Z., Tuyls, K., and Clark, S. (2018). Emergent com- munication through negotiation.
Cooijmans, T., Aghajohari, M., and Courville, A. (2023). Meta-value learning: general framework for learning with learning awareness.
Vault, D., Mell, J., and Gratch, J. (2015). Toward natural turn-taking in virtual human negotia- tion agent. In AAAI Spring Symposia.
Espeholt, L., Soyer. H., Munos, R., Simonyan, K., Mnih, V.. Ward, T., Doron, Y., Firoiu, V., Harley, T., Dumning, I., Legg, S., and Kavukcuoglu, K. (2018). Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures.
Foerster, J., Farquhar, G., Al-Shedivat, M., Rocktäschel, T., Xing, E, P., and Whiteson, (2018a).
Dice: The infinitely differentiable monte-carlo estimator.
Foerster, J. N., Chen, R. Y., Al-Shedivat, M., Whiteson, S., Abbeel, P., and Mordatch, I. (2018b).
Learning with opponent-learning awareness.
Ho, J., Jain, A., and Abbeel, (2020). Denoising diffusion probabilistic models.
Khan, A., Willi, T., Kwan, N., Tacchetti, A., Lu, C., Grefenstette, E.. Rocktäschel, T., and Foerster, (2024). Scaling opponent shaping high dimensional games.
Lerer, and Peysakhovich, A. (2018). Maintaining cooperation in complex social dilemmas using deep reinforcement learning.
Letcher, A., Foerster, J., Balduzzi, D., Rocktäschel, T., and Whiteson, S. (2021). Stable opponent shaping differentiable games.
Lewis, M., Yarats, D., Dauphin, N. Parikh, D., and Batra, D. (2017). Deal or no deal? end-to-end learning for negotiation dialogues. arXiv preprint arXiv: 1706.05/25.
Lu, C., Willi, T., de Witt, C. S., and Foerster, J. (2022). Model-free opponent shaping-

---

Published as a conference paper at ICLR 2025
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I,, Wierstra, D., and Riedmiller, M. (2013). Playing atari with deep reinforcement learning.
Noukhovitch, M., LaCroix, T., Lazaridou, A., and Courville, A. C. (2021). Emergent communi- cation under competition. In Dignum, F., Lomuscio, A., Endriss, U., and Nowé, A., editors, AAMAS 21: 20th International Conference on Autonomous Agents and Multiagent Systems, Vir- tual Event, United Kingdom, May 3-7, 2021, pages 974-982. ACM.
Parisotto, E.,, Song, H. F., Rae, W., Pascanu, R., Gulcehre, C., Jayakumar, S. M., Jaderberg, M., Kaufman, R. L., Clark, A., Noury, S., Botvinick, M. M., Heess, N., and Hadsell, R. (2019). Stabilizing transformers for reinforcement learning.
Press, W. H. and Dyson, F. J. (2012). Iterated prisoner's dilemma contains strategies that dominate any evolutionary opponent. Proceedings of the National Academy of Sciences, 109(26): 10413.
Radford, A., Narasimhan, K., Salimans, T., and Sutskever, (2018). Improving language under- standing by generative pre-training.
Rapoport, A. and Chammah, A. (1965). Prisoner's Dilemma: A Study in Conflict and Cooperation. University of Michigan Press.
Reed, S., Zolna, K., Parisotto, E.,, Colmenarejo, S. G., Novikov, A., Barth-Maron, G., Gimenez, M., Sulsky, Y., Kay, J., Springenberg. J. T., Eccles, T., Bruce, J., Razavi, A., Edwards, A., Heess, N., Chen, Y., Hadsell, R., Vinyals, O., Bordbar, M., and de Freitas, N. (2022). A generalist agent.
Rummery, G. and Niranjan, M. (1994). On-line q-learning using connectionist systems. Technical Report CUED/F-INFENG/TR 166, Cambridge University.
Sandholm, T. and Crites, R. (1996). Multiagent reinforcement learning in the iterated prisoner's dilemma. Bio Systems, 37(1-2):147-16.
Schulman, J., Levine, S., Moritz, P., Jordan, M. I., and Abbeel, P. (2017a). Trust region policy optimization.
Schulman, J., Moritz, P., Levine, S., Jordan, M., and Abbeel, P. (2018). High-dimensional continu- control using generalized advantage estimation.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017b). Proximal policy optimization algorithms.
Shapley, L. (1953). Stochastic games. Proceedings of the national academy of sciences, 1995000095-
Song, H. F., Abdolmaleki, A., Springenberg. J. T., Clark, A., Soyer, H., Rac, J. W., Noury, S., Ahuja, A., Liu, S., Tirumala, D., Heess, N., Belov, D., Riedmiller, M., and Botvinick, M. M. (2019). V-mpo: On-policy maximum posteriori policy optimization for discrete and continuous control.
Vezhnevets, A., WU, Y., Eckstein, M., Leblond, R., and Leibo, Z. (2020). OPtions as REsponses: Grounding behavioural hierarchies in multi-agent reinforcement learning. In III, H. D. and Singh, A., editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 9733-9742. PMLR.
Watkins, J. C. H. and Dayan, P. (1992). Q-learning. Machine Learning, 8(3):279-292.
Willi, T., Letcher, A., Treutlein, J., and Foerster, J. (2022). Cola: Consistent learning with opponent- learning awareness.
Williams, R. (1992). Simple statistical gradient-following algorithms for connectionist reinforce- ment learning. Machine Learning, 8:229-256.
Zhang, T., Williams, A., Phade, S. R., Srinivasa, S., Zhang, Y., Gupta, P., Bengio, Y., and Zheng, S. (2022). Ai for global climate cooperation: Modeling global climate negotiations, agreements, and long-term cooperation in rice-n. Social Science Research Network.
12

---

Published as a conference paper at ICLR 2025
Zhao, S., Lu, C., Grosse, R. B., and Foerster, J. N. (2022). Proximal learning with apponent-learning awareness.
13

---

Published as a conference paper at ICLR 2025
Appendix
Table of Contents
A Mathematical Derivations
15
A.1 Deriving the Advantage Alignment Formula
15
A.2 Proof of Theorem
16
A.3 Gradient of LOQA
18
Gradient I.OQA in Continuous Action Spaces
A.5 Proof Theorem
A.6 Advantage Alignment Implementation
20
A.7 Proximal Advantage Alignment
20
A.8 Proof of Theorem 3
21
A.9 N-Player Advantage Alignment
21
B Experimental Details
22
Iterated Prisoner's Dilemma
22
Coin Game
22
B.3 Negotiation Game
23
B.4 Melting Pot's Commons Harvest Open
24
C
Additional Figures
25
C.I. Negotiation Game Training Curves
25
C.2 Coin Game Full League Results
25
Ablation Study Commons Harvest Open
26
14

---

Published as a conference paper at ICLR 2025
A MATHEMATICAL DERIVATIONS
A.1 DERIVING THE ADVANTAGE ALIGNMENT FORMULA
In this section derive the Advantage Alignment formula in equation equation from the opponent shaping expression in equation equation and assumption 2. Recall assumption 2:
x' (a(ss) exp ( (Q'e,) a, b)]
Note that if 1,3 2 and if i 2,3 1. Recall the opponent shaping policy gradient expression:
8
V8, V (n)
Σ
A' (sc,
Vei log x (ag1s)) + Ve, log n7 (be
(A)
(B)
We expand the term (B) above splitting the expectation, by assumption we can write:
Σ
A (st, az, bt) Vo, log x (belsa)
t==
=B.
7 A A'',,, be) Vo, log exp B Buwrr (1=,) (st, b.))]
P pe
3 Pr
Y'A' (sc, ae, Ve, Ve, ( (Q a, 66)]
1==0
where is the second line we used the fact that the expected advantage zero. For convenience of notation we define:
r Ct. be), A == A' (st, ax, by)
These are the reward and advantage of agent at time step t after taking action a and opponent taking action b,. From the Bellman equation equation 2 we expand as follows:
B E Twpppt
y A ar, be) Vol Bawn (1e0) 122 a b.)
(13)
-B-E TuPpi
7't VAAV 01
E. r? t,. 12 V2(s') s2, bt
(14)
=B BB.F E TwPri 2
1 V Si
(8())) s2, dp
(15)
=BEE
141 A Pinpppt
* A V a1 log T ('
84 by
(16)
T- ,
80
= -..E
A o log n (ak
8t: br
(17)
T-Pri
m m
= B E
1+++1 4 Vo (a'cl
81, br
(18)
T-Pr m m
k=111
= 3 1
E
1+++1 A V log ग. (aal>k)
(19)
T~
(=0 1=0k=tt
We use Bellman equation in line (14), and the policy gradient theorem in line (16). In line (17), we use the fact that 7' A' (St, at, bt.b.) measurable w.r.t. the natural filtration of the process up to time f, Fe, and the independence of the two terms conditioned on F, by the Markov property. In line (18) we use linearity of expectation. Reorganizing the summations in causal form we get the desired result:
line (19) we use the rule of the iterated expectations.
Dc
3 E T-PPTT
Σ
Σ 7 +
A7 ax bb) A2 A2 ar be Vo nt x²(a
(8)
t=
15

---

Published as a conference paper at ICLR 2025
A.2 PROOF OF THEOREM
Lemma (Policy changes under gradient ascent). Given a policy Te(a|s) parametrised such that the set of gradients V log To(als) ) for all pairs (a, 8) form an orthonormal basis, and value function V V(0), the following holds:
d da (otas) VoV. VoTo(als) = (d(
(20)
dy(s) = Σ Pr(ss 8) == A(a(s) = (alss) - V(8)
(21)
(22)
Proof. The policy gradient theorem gives us:
VoV(0)= = Enan
Σ
A(a 8t Vo Voggoo
(23)
t=
( (a)
(24)
(s,a)
Taking the dot product of this expression with VoTo(a''s') = no (a'la'')Ve log (a'a's') and invok- ing the assumed orthonormality of gradients:
Vono(a'|s') VoV(O)
(25)
(4(ssmaaala)aaa (Vo Volog =a(als) Vo log To (a'ls') (x.a)
Σ
(26)
(27)
Theorem 1. (LOLA policy gradient) Given a two-player game where players and have respec- live policies x (a]s) and (b]s), where each policy is parametrised such that the set of gradients Ve, log n² (a|s) for all pairs (a, 8) form an orthonormal basis, the LOLA update for the first player correspond to reinforce update with advantage
ALOLA a, b.) A' aq bb) + Σ h ALA
(1I)
where A = A'ss, ax by) and d is the occupancy measure of the tuple (ax, by, sk)
Proof. LOLA (Foerster et al., 2018b) optimizes the return of the agent under an imagined opti- mization step of the opponent (assuming the opponent naive learning algorithm). Under their notation, a LOLA agent optimizes V (B2, 02 + A62) where Ad, is gradient ascent step on the parameters of the opponent 62. Note that along this proof because we consider the method proposed by (Foerster et al., 2018b) we use their way to compute gradients. Particularly, one does not use Assumption 2, and consequently assume that Vo, log Te, (and respectively Vo, log To, = 0.)
Since computing this value function explicitly is difficult, LOLA uses the first-order Taylor expan- sion surrogate objective:
(' 0 A02) v' (z), (A02)" Vo, (0 02)
(28)
The gradient of the expression above w.r.t. the parameters 8 of the agent given by
Va, (10 8 + VV = Va, (8, + 3 (Va, Vo, ( 82)) Vo, (2, (2)
(29)
16

---

Published as a conference paper at ICLR 2025
The first-order terms above is computed using the Advantage form of the REINFORCE estimator, which is given by equation equation 3. Foerster et al. (2018b) derive the following REINFORCE estimator for the second-order term:
Ve, Vo, (0), (22)
(30)
= Prp n,2
Σ
20
r'rl Σ
Vo, log x (arls))
Σ va log (brlse)
(31)
Now, we use the following fact
Σ
bi)
Σ
axbs-k
Σ
ax hcc
Σ>
akbs-k
Σ
Ct
1=0
1=0
1=0
8=0 k=0
8=0 k=
S=0k==
S=
to expand the second order term beginning from equation 29, to bring out the advantage A!:
Vo, Va2
(0) 0²)
(32)
Vo, log xt ((a/)1))
Σ Voz
Vo, log x² (belot))
T-Pr m
(33)
Va, log π (ax (xx Vo, log 2 xk)s
(34)
E Thuli
V., log x (ak Vo, log x² (b:-
(35)
E PP m
Ve, log (aclsk) Vog log (b:-n/0c-k)
7/144
(36)
1=0
E
Vo, log मt (anlss) log 2 (b:
(37)
E Prp 2
Σ Vo, log x (aclsk) log (bg > A
(38)
where we reorder the terms of the summation to sum over future rewards instead of past gradient terms in line 34, use the law of iterated expectation in line 36 and a baseline subtraction in line 38.
Per Equation equation 29, we multiply this Hessian with the gradient of the value function
Vo, V2 (0,, 02) E
? A aa,, bels:) log x (b.lst)
(39)
Σ (a)
s) bls) Vo, log n (bls)
(40)
(a,a,))
where where (a, b, 8) is the occupancy measure of the state actions tuple (a, b, s), and use the assumption that the gradients (Ve, log (als)) form an orthonormal basis obtain
E
Σ
Vo, log x " sa dd, (ar ko by 81 R) Ai 7" 7'A7
To completes the proof, we finally need switch the summations to get
Σ
* AAtd (ax bk sx-))) Ve, log x (arls))
T
t=
17

---

Published as a conference paper at ICLR 2025
A.3 GRADIENT OF LOQA
Recall the opponent policy approximation used in LOQA, which takes a softmax over the Q-values of the opponent. We assume exact estimates of these Q-values:
exp Q (8t oe, be)
# (bbllse))
exp Q² as,
exp
Note that (b, differentiable w.r.t. the parameters 01 of the policy of the agent because the Q-values depend on T1. Therefore, we can use the policy gradient theorem (see Aghajohari et al. (2024b)) to differentiate the value function the opponent w.r.f. the parameters of the agent. For convenience of notation we define:
1b(b) = Q'(s:: ac
Computing the gradient of the approximated opponent's policy we get:
(xp (1,
V 0, 22, (belss) Vo,
(41)
(st. ex Q1,
Vo, exp (H(b,)
exp (b) Vo. exp
(42)
E. exp (b(b)
Expep (b))
exp (b,) Vo. (b(b.)
expQ.(1) En exp (b( (1)
(43)
E, ex
(2, exp (b))
exp (i(b))
exp (1( Vo, 0i(b)
(o,0)(())
Σ
(44)
exp
E. exp((())
(belss))
Vo, Qi(b))
Σ
(bbsss
(45)
where we used the quotient rule in line (42) and equation equation 4 in line(45). By the chain rule, the gradient of the log probability is:
= Vo, a (b, st)
Ve, log # (bolse)
Vo, Qb(b))) - Σ (bbls Q.))
(bbb(()))
This concludes the derivation.
A.4 GRADIENT OF LOQA IN CONTINUOUS ACTION SPACES
We derive the gradient of the opponent's policy 7b(ss) with respect to the agent's parameters O1, assuming a continuous action space.
The opponent's policy is defined as:
exp((2(s,b)) n (b|s) IA exp((2 bD) db
(46)
where Q2 b) is the Q-value of the opponent for action b, and the continuous action space.
Our goal is to compute the gradient of πt (b(ss) with respect to 01, the parameters of agent 1, which affect (s, b) through interactions.
Taking the log of (b)ss, we get:
log n" (bls) = (x(,))))))))))))))))))())) - -lo (S, (xx)((((s))))
(47)
The gradient of log x (b)s) with respect to 01 is:
Ve, log " (b|s)) = Ve, (²) - Ve, log ( (/exp((((3,0))00
(48)
18

---

Published as a conference paper at ICLR 2025
Next, we compute the gradient of the log partition function ((s) = SA exp(Q?
Ve, log ((s) = Ve, (s)
LA exp((P(ss db /
db'
(49)
Z(s)
which simplifies to:
Vo, log Z(x) = vb (x, db'
(50)
Now, applying the chain rule to compute the gradient of π (b|s), we get:
/1 क (, (2))
T 7 (b|s) Vo, ('..b))
(51)
We are allowed to interchange the gradient and the integral by applying Leibniz's rule, which holds under the following conditions: exp(Q2 (s. f/) and its gradient Vo, exp(Q2(s.B)) B)) are continu- ous, as both the exponential function and the Q-value function (* 6') are smooth. 2. The integral SA exp(Q"(s, b') db' converges due to the boundedness of (s, b') or a rapid decay over the action space. 3. We assume Va. Q² 6') is bounded, ensuring the interchange of the gradient and integral well-defined. Thus, the final expression for the gradient of the opponent's policy is:
V (b|s) (b|s) (Ve, (s..b) /
(b(ss) Q (s)b)) aK)
(52)
The Integral above is intractable, which makes continuous action LOQA hard to scale.
A.5 PROOF OF THEOREM 2
Proof. In practice, LOQA deviates from the approach discussed in Appendix A.3. Specifically, does not differentiate through all of the Q-values, but only through that of the action b, actually observed in the sampled trajectory:
exp (sc, ae, bt)
# (belst) = cxp ( az b.) + Eozb, exp Q² (st, ag, b). b)
(53)
differentiale
This choice is made because the trajectory provides an estimate of the Q-value of each opponent action b,. This estimate statistically depends the agent's actions aett and therefore can be stochas- tically differentiated w.r.t 01 using REINFORCE. The other Q-values will be estimated by function approximators, which depend only implicitly on and cannot be differentiated.
Differentiating equation 53 leads to a simplified gradient:
exp (3 ac, bc)
Ve, V0, (belo)) = Vo.
(54)
exp Q
(st, at, b.) Eoeb, exp (st,
(exp Qb(b.) + Eosr, exp Q7 (b) exp Q2(b.)
Vo, exp (b.)
(55)
expp (b,) Edes,, exp (i(b)
Σ Eiger. Logbb exp (b( + exp Q7 exp exp (+(b))
(56)
exp exp (bb)) + Dotb, @x)
#bbblls - (b.ls.))
Qi(b.)
(57)
By the chain rule, the gradient of the log probability
(bels))
Ve, log = (belst)
(bxs)((()s))) (((b).
(58)
#2 82)
The difference between LOQA and Advantage Alignment lies in the extra scaling factor (1 #² (b)())))) which accounts for the partition function. Plugging equation 58 into the generalized policy gradient equation equation 6 proves the theorem.
19

---

Published as a conference paper at ICLR 2025
Algorithm 2 Proximal Advantage Alignment
Initialize: Discount factor " agent Q-value parameters ø' Q-value parameters of actor pa- rameters 0¹, opponent Q-value parameters o², Q-value parameters of actor parameters # for iteration= 1.... do
Run policies r² and x² for T timesteps in environment and collect trajectory Compute agent critic loss LE using the TD error with r² and V Compute opponent critic loss L using the TD error with r² and V Optimize LG w.r.t. o¹ and L w.r.t. o² with optimizer choice Optimize L w.r.t. o² and L w.r.t. Ø² with optimizer of choice Compute generalized advantage estimates {Ai..... A4), Compute agent actor loss, 1.2, using equation 9
.7}}
Compute opponent actor loss, 1.2, using equation 9
Optimize Lg w.r.t. 0t and L w.r.t. 8² with optimizer of choice
A.6 ADVANTAGE ALIGNMENT IMPLEMENTATION
To more clearly see the Advantage Alignment formula as an influence over each individual log probability term recall the formulation:
Σ r+1 Σn A'-hAA'(sns ak. bu) (se, ar V a (acls:)
(8)
The 7f term helps regularize the linear scaling of the sum of the advantages of the agent. Alterna- tively could regularize using 1/(1 instead:
1 I+t
E T-
Σ
Σ
As au ax.bbe) be) A2 at, b,) n (aelse) AeE
(59)
Which accounts to increasing the probability of the actions that align the sum the past advantages of the agent up the current time-step i and the advantage of the opponent at the current time- step, f. In our implementation we use equation 59, as it considers more terms in the future and works better in practice.
A.7 PROXIMAL ADVANTAGE ALIGNMENT
We can combine the two policy gradient terms into single one to come up with a proximal Advan- tage Alignment formulation:
E
Σ V log T (arlst) + B 1=0
Σ
V log v (aclse)
(60)
1=0
ket
Where 3 is the weight put into the Advantage Alignment loss (the negative inverse of the Boltzmann constant times the temperature). Then we have:
(
Σ 1=
rd
A1 [A+
By
Σ A3
a log x (aelsz)
(61)
Pr
Rect
This is just the normal advantage policy gradient with a modified advantage A
xe
T- he
Σ
7' V'A T' where Aj Al BY
Σ
Σ K
- Ab
AP.
(62)
Recall the Trust Region Policy Optimization (TRPO) (Schulman et al., 2017a) objective, want to maximize the value function while maintaining the updated policy close policy space:
max (1)
(63)
s.t. sup |+|s) (1s) S
20

---

Published as a conference paper at ICLR 2025
We can use the PPO (Schulman et al., 2017b) surrogate objective:
E
min (r),()))4), clip (r,, ()
(64)
Now we apply it to the Advantage Alignment formulation that uses the modified advantage on the policy gradient equation 62:
E
Tu 12 imin (r), clip (r.((1):11 ATH1,
where we denote ,, (0r]s)) to the updated policy and r, (0,) = nTh (a, (s))/ss (ar]æ,) ) the ratio
between the updated policy and the old policy. We used generalized advantage estimation (GAB) (Schulman et al., 2018) to compute the advantages in this expression. Algorithm summarizes the implementation of Proximal Advantage Alignment.
A.8 PROOF OF THEOREM 3
Let 01. ...,. On be the parameter each agent, Te, (als) be the policies represented by those parameters, and Bx) be the value function of agent as a function of all the other agents.
Lemma (Zero Advantages At Nash). For all Nash Equilibria of the game, if there exist parameters of ..., 0% such that noz ni. where πद is the policy of agent al the Nash, then for all action-state pairs with non-zero probability under the Nash policies, we have As(als)
Proof. By the Bellman Optimality Equation, at an optimal policy the value of agent becomes (*)) = arg max, (*(a, 8), hence all actions with non-zero probability under गद have the same (" 8), and since A(a, 8) = Q(a s) (s)), the advantage will vanish.
We now use lemma 2 to prove that the Advantage Alignment term is zero at a Nash equilibrium.
Proof. Under Advantage Alignment, the updates we take can be represented by
8 0 + TuPti
B Vo, log To, (at (st)
(65)
B = A 3
A
(66)
ipi
But by lemma 2, (b.ls)) = 0 for all actions at a Nash, hence the second term vanishes, as does the first term for the same reason.
A.9 N-PLAYER ADVANTAGE ALIGNMENT
Consider the n-player setup, we can use the policy gradient theorem and assumption 2 to derive the following expression:
Vo, log (atlst) Σ Va, log n5 ( (af(or) )]
Do
Va, (p) E
Σ
(67)
P
Which naturally leads to the following modification to the PPO advantage following the derivation used Proximal Advantage Alignment:
1 = A
By
Σ
Σ
A
(68)
j
Here we use the standard game theory notation of to refer to the current player and -i to refer to all other players. Similarly at denotes the action of player at time
21

---

Published as a conference paper at ICLR 2025
B EXPERIMENTAL DETAILS
B.1 ITERATED PRISONER'S 'S DILEMMA
We use an MLP layer connected to GRU followed by another MLP head for both the actor and critic networks, similar the architecture used in POLA (Zhao et al., 2022). We also use replay buffer of agents collected during training, following Aghajohari et al. (2024b). All of our IPD
experiments run in 50 minutes nvidia A100 gpu.
Table I: IPD Hyperparameters
Parameter
Value
Actor Training Optimizer
Adam
Actor Training Entropy Beta
0.15
Actor Training Learning Rate (Actor Loss)
0.0001
Advantage Alignment Weight
0.3
Actor Hidden Size
64
Layers Before GRU
Q- Value Training Optimizer
Adam
Q- Value Training Learning Rate
0.001
Q- Value Training Target EMA Gamma
0.99
Valuee Hidden Size
64
Batch Size
2048
Self-Play
True
Reward Discount Factor
0.9
Agent Replay Buffer Capacity
10000
Agent Replay Buffer Update Frequency
Agent Replay Buffer Current Agent Fraction
Advantage Alignment Discount Factor
0.9
B.2 COIN GAME
We use the same architecture used for IPD with an MLP connected to a GRU unit, followed by another MLP. We experimented with both Advantage Alignment (Equation equation 8) and Proximal Advantage Alignment (Equation equation 9), with Advantage Alignment performing better (this is the one used). All of Coin Game experiments run in 30 minutes in a nvidia A100 gpu.
Table 2: Coin Game Hyperparameters
Parameter
Value
Actor Training Optimizer
Adam
Actor Training Entropy Beta
0.1
Actor Training Learning Rate (Actor Loss)
0.002
Advantage Alignment Weight
0.25
Actor Hidden Size
64
Layers Before GRU
1
Q- V-Vuuee Training Optimizer
Adam
Valuue Training Learning Rate
0.005
Q- Value Training Target EMA Gamma
0.99
Q- Value Hidden Size
64
Batch Size
512
Self-Play
True
Reward Discount Factor
0.96
Agent Replay Buffer Capacity
10000
Agent Replay Buffer Update Frequency
10
Agent Replay Buffer Current Agent Fraction
0
Advantage Alignment Discount Factor
0.9
22

---

Published as a conference paper at ICLR 2025
B.3 NEGOTIATION GAME
We experimented with both Advantage Alignment (Equation equation 8) and Proximal Advantage Aligmment (Equation equation 9), with the original Advantage Aligmment performing better.
The game observations are concatenation of the availability of the items, agent's s value for each item, opponent's 's value for each item, and previous round proposals. This makes up for input vector of length 15. The previous round proposals are especially important as the agents need to examine whether the opponent defected against them by proposing high proposals for item in which the value of the item is higher for the agent compared to the value of the item to the opponent. In other words, if the opponent wanted to gain a little return in exchange of huge loss to the agent, defecting.
Agent's Architecture:
observation is then processed by an encoder. The encoder is a GRU network. The GRU network consists of first two Linear Layers with relu non-linearity in between. Then it is passed to a GRU unit.
Encoder:The
The output of the GRU is then fed a two-layer MLP with relu non-linearities for the critic module of the agent. Additionally, we concatenate the output of the encoder with the time, the index of the step of the game, for the value function as otherwise would be hard estimate the value of the state without knowing how long the game is going to go on for.
Critic:
Actor: The actor is the most complex component as deals with continuous actions. The output of the encoder is passed to an MLP with relu non- con-linearities and the output of the MLP is passed to a tanh activation and scaled by 2.5, the output this MLP used as the mean of normal distribution. The logarithm of the standard deviation is modeled by single global parameter in the actor. Next, a sample of this normal distribution is passed through tanh activation and scaled and shifted back to (0, 5). Computing the log probability of this transformations requires careful implementation. Especially if the atanh operation that is used numerically unstable. Please refer to the code released with this paper for the exact implementation.
Hyperparameters: Please refer to for the hyperparameters used in our negotiation game experi- ments. We use replay buffer our gather trajectories although the rate that it is mixed with fresh trajectories small.
Table Negotiation Game Hyperparameters
Parameter
Value
Actor Training Optimizer
Adam
Trajectory Length
Encoder Layers
2
MLP Model Layers
Replay Buffer Size
100000
Replay Buffer Update Size
500
Replay Buffer Off- Off-policy Ratio
0.05
Q Value Training Optimizer
Adam
Optimizer (Actor) Learning Rate
0.001
Optimizer (Critic) Learning Rate
0.001
Entropy Beta
0.005
Advantage Alignment Weight
3.0
Self-Play
True
Batch Size
16384
Gradient Clipping Norm
1.0
Note that the optimization of the agents in the negotiation game is unstable, preventing from taking the last checkpoint. our experiments in Fig 3a we select the checkpoint that corresponds to the best achieved return for the agent during the optimization of the agent and the opponent. While we are not completely certain, we observe the instability happens when the policy distribution concentrates around the maximum possible proposal which is 5. We clipped the atanh operation in our implementation for more numerical stability. All of our Negotiation Game experiments run in hour nvidia A 1100 gpu.
23

---

Published as a conference paper at ICLR 2025
B.4 MELTING PoT's COMMONS HARVEST OPEN
We experimented with both Advantage Alignment (Equation equation 8) and Proximal Advantage Alignment (Bquation equation 9), with Proximal Advantage Alignment performing better.
Agent's Architecture: In the Commons Harvest Open environment, agents receive observations consisting of local view of the environment in the form of raw pixel data. Each observation is an image frame capturing the agent's 's immediate surroundings. We use a 3 layer convolutional neural network, following (Mnih al., 2013), encode the observations, which are then passed GTrXL tranformer (Parisotto et al., 2019).
The observation frames are processed by an encoder. The encoder a GTrXL transformer network (Parisotto et al., 2019). The GTrXL network consists of transformer layers, each with model dimension 192 and a feedforward dimension of 192. The transformer capable of handling sequences up to maximum length of 1000 steps, capturing temporal dependencies in the agents' observations. In practice, we use a context length of 15.
Encoder:
The output of the encoder is then fed to two-layer Multi-Layer Perceptron (MLP) with ReLU non-linearities for the critic module of the agent. To provide temporal context, we concatenate the current time step to the encoder's output before feeding it to the critic. This helps the critic estimate the value the state more accurately, as the remaining time an episode can affect the expected return.
Critic:
The actor network shares the encoder with the critic. The output of the encoder is passed through another MLP with ReLU non-linearities to produce logits over discrete action choices. The policy is modeled as a categorical distribution over these actions, which include turning around, moving in different directions, and zapping other agents.
Actor:
Hyperparameters: Please refer to Table for the hyperparameters used in our Commons Harvest Open experiments.
Table Commons Harvest Open Hyperparameters
Parameter
Value
Self-Play
True
Batch Size
512
Optimizer (Actor) Learning Rate
I x 10-5
Optimizer (Critic) Learning Rate
1 x 10-5
Entropy Beta
0.1
Advantage Alignment Weight
1.0
Clip Gradient Norm
10.0
Transformer Layers
3
Transformer Model Dimension
192
Transformer Feedforward Dimension
192
Discount Factor (?)
0.99
PPO Clip Range
0.1
PPO Updates per Batch
2
Normalize Advantages
True
Context Length
15
We use parallelized environment with 6 copies of Commons Harvest Open to make training more efficient. Following LOQA (Aghajohari et al., 2024b), we keep replay buffer of past agent param- eters to ensure robustness against a distribution of policies. From this replay buffer we sample agents at each iteration and play against 5 self-play agents with the current version of the policy. For each environment, we use the on-policy trajectories to compute losses for the actor and critic. In total, our Commons Harvest Open experiments last 24 hours on nvidia L40s gpu.
24

---

Published as a conference paper at ICLR 2025
C ADDITIONAL FIGURES
C.I NEGOTIATION GAME TRAINING CURVES
Figure 6 shows the training curves of Advantage Alignment on 10 seeds.
Average Reward
Agent vs AC Rewards
Agent AD Rewards
6.40
0.50
0:35
0.45
Reward
0.25
0:30
0.24
0.2
0.35
0.22
0.20
0.30
0.20
0.18
1002
Stops
Stops
Stops
Figure 6: Training curves of Advantage Alignment averaged over 10 seeds.
C.2 COIN GAME FULL LEAGUE RESULTS
Figure shows the head-to-head results of all agents we experimented with in league.
Kobkign
Planston
Ashiigoli
AdhigonB
Adbiggoll
Adsigniig
Return
0.28
0.28
0.24
0.01
0.31
0.15
0.10
0.13
0.14
0.18
0.20
0.20
AdAlgn
0.29
0.30
024
0,06
0.32
0.12
0.15
0226
0.7
0.28
0.28
0.29
LOGA
0.18
0.17
0,19
0.01
0.28
0.07
0,13
2
0,18
0.1
0.18
0.16
POLA
0.18
0.16
0.19
0.21
0.20
0.41
0.21
0.40
0.37
0.25
0.24
0.24
MFGS
0.01
0.01
0.35
0.36
0.28
0.32
0.36
0.63
0.11
0.62
0.58
0.47
0.43
0:48
AC
0.00
0.06
0.50
0.42
0.27
0.00
0.53
0.01
0.32
0.38
0.08
0.08
AD
0.16
0.26
0.26
0.24
0.20
0.26
0.53
0.01
0.22
0.48
0.31
0.30
0.27
Random
0.07
-0.1
47000
0.30
-0.20
0.01
0.53
0.00
4.002
40.07
0.07
-0
AdAign-CE
0.00
0.00
0.00
40.2
-0.1 18
0.04
-0A7
0.64
0.63
40.00
-0 00
-0
AdAlgr-F
0.10
0.08
0.11
0.15
0.26
0.18
0.22
0.16
0.14
0.10
0.11
0.11
AdAlggs-8
0.31
0.13
0.13
0.14
0.10
0.13
0.16
0.23
0.14
0.14
0.12
0.12
0.13
AdAAgss/
0.12
0.11
0.14
0.36
0.28
0.19
0.18
0.17
0.16
0.13
0.12
0.13
AdAAgn-VS
Figure 7: The head-to-head results of all variants of the coin game agents experimented with in this paper. All numbers are an average of 10 seeds of one type of agent with 10 seeds of another type of agent, where each pair play 32 games. We ablate Advantage Alignment masking different components of the gradient. Cooperative (C), masks when both advantages (E), masks when the advantage of the agent is positive and the advantage of the opponent is negative; Vengeful (V), masks when the advantage of the agent is negative and the advantage of the opponent is positive; Spiteful (S), masks when both ad advantages are negative.
positive; Empathetic
25

---

Published as a conference paper at ICLR 2025
C.3 ABLATION STUDY COMMONS HARVEST OPEN
Figure 8: Sample trajectories for Proximal Advantage Alignment agents with different 3 weight. We select the best of seeds for each value of f.. On the first row: == 0.5, agents reach a policy where they try to consume the apples as fast as possible. On the second row: 1, agents reach a "bush guarding" policy, zapping any other agents coming into the same bush. On the third row: = agents reach policy where they rotate around specific paths, preventing the extinction of the bushes.
Interestingly the value of 3, which is used to control the weight of the advantage aligmment term in equation 10, leads agents to converge to different policies. With a low value of the weight (3 = 0.5), empirically observed that most runs converge to greedy policy that attempts to consume apples as soon as possible. With a value of B = 1, we find policies that show "bush guarding" behavior preventing other agents from approaching their bush, and consuming apples within that bush with moderate restraint. This is the policy that shows the best evaluation performance in figure 4. With high values of the weight (3 2), most runs find rotating strategy which agents stick eating only subset of the apples on each bush. This policy has the highest pro-social return out of all of them. However, the rotating strategy also vulnerable to exploitation from greedy agents and does poorly in the evaluation scenarios. Figure showcases what these policies look like in practice.
26


Reviews:
[{"review_id": "s25T19Z2HM", "reviewer": "Reviewer_GHM1", "summary": "This work aims to encourage cooperation in normal form games through opponent shaping. This is done by proposing Advantage Alignment, where the product of the advantages of the player's value-network and the opponent's value-network are used to update the policy. This means that, when advantages are aligned (i.e., they have the same sign), the log probabilities increase or decrease accordingly. The authors show theoretically that LOLA and LOQA, 2 other opponent shaping algorithms, also follow the advantage alignment principles. They then experimentally validate advantage alignment on Coin Game, Negotiation Game, and the high-dimensional Commons Harvest Open environment.", "questions": "Additional clarification questions:\n- Fig6 seems to indicate that Advantage Alignment is not really stable (increasingly noisy rewards, partial collapse towards the end of training against AC and AD). Is it sensible to hyperparameters? Or is there another reason for this behavior?\n- Fig4 selects the best agent out of 10 seeds. Given the variance observed for the Negotiation Game in Fig6 (which granted is not the same game as Fig4), how representative are the results in Fig4?\n- I am curious how Advantage Alignment can play against AD or AC if it can't change its policy (AD and AC are fixed policies). Doesn't this break Assumption 1?\n\n\nOverall, I recommend for acceptance, for the strengths listed above (theoretically sound, well motivated, experimentally justified).", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "Strengths:\n- This paper follows a long line of work on opponent shaping (LOLA, POLA, COLA, LOQA) and builds on LOQA to propose a new opponent shaping algorithm. While it feels a bit incremental, I find the work very well motivated, and theoretically justified. Extensive experiments on different domains and with relevant baselines confirm its practical performance.", "weaknesses": "Concerns:\n- Having access to the opponent's value function results in a specific setting where all the player's preferences are public. The negotiation game completely changes if we know the preferences of the opponent, and we could devise a simple strategy that maximizes the average utility of both players. Since the utilities in this experiment are orthogonal to each other, there does not seem to be any real dilemma. For these reasons, I believe the insights gained from the Negotiation Game experiments to be limited.\n- Advantage Alignment uses the product of advantages to align the policy. When there are more than 2 players involved (as in Commons Harvest Open), does the alignment depend on the product of the advantages of all players? If so, if a single agent does not cooperate, the whole alignment product fails. This would limit the scaling of Advantage Alignment to $n$ players.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4D0zSSKYSp", "reviewer": "Reviewer_FsFC", "summary": "This paper focuses on opponent shaping.  They propose an opponent shaping formulation that aligns the advantages of interacting agents, thereby simplifying the mathematical formulation of opponent shaping and reducing the associated computational complexity. In this formulation, the probability of future mutually beneficial actions is increased when their interaction has been positive. They demonstrate their methodologies' effectiveness on several social dilemmas and connect opponent shaping's past successes with this new advantage alignment formulation.", "questions": "- For the coin game, in the always defect case, LOQA receives slightly higher return than Advantage Alignment. Do you have any ideas for why this is?\n-  In figure 3, there is mention of green values to show cooperation, but I do not see the green values in the figure.\n- In Appendix 6, you mention that equation 59 uses the sum of past advantages of the agent up to the current time step and the advantage of the opponent at the current time step. Did you experiment with much tuning for number of terms to include in the sum of past advantages or number of future terms?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "This paper is very well-written, and I appreciated how the authors helped the reader build intuition and understanding of the significance of their technique in section 4. The experiments served to reiterate the strength of their algorithm's performance, and the authors gave solid context for why each environment was selected.", "weaknesses": "The main concern I have with the paper is its individuality from the LOQA work, which concerns a similar technique, applied on similar problems, that achieves similar results. Both the more complex experiments (negociation and and harvest open) and the attached proofs in the appendix helped differentiate some of the beneficial aspects of Advantage Alignment in terms of scalability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "27z1CoWWS1", "reviewer": "Reviewer_Niaw", "summary": "This paper investigated a long-standing research problem in multi-agent systems and game theory, named social dilemma. To address this problem, this paper stood on the side of opponent shaping and proposed a method that attempted to align advantages of various agents, to achieve a compromise between social welfare and self-interests. The main assumption lies in the connection between the opponent's policy gradient and the agent's policy parameters. Building on this, this paper proposed a paradigm called Advantage Alignment, and a corresponding RL algorithm based on PPO. This paper also unified other approaches of opponent shaping, such as LOLA and LOQA, in the paradigm of Advantage Alignment. The experimental results on various tasks showed the effectiveness of the proposed Advantage Alignment.", "questions": "See concerns in weaknesses. If the these concerns can be resolved, I will consider to raise the score. However, for the moment I can only give a reject due to those concerns.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This paper proposed an original idea that derives the advantage alignment to implement the policy gradient with respect to the opponent (the opponent shaping term), which can reduce the computational complexity. This paradigm has been shown to have connections to previous opponent shaping approaches, which is a progress in this research direction.\n2. The general quality of this paper is good. Although there are some technical points that I require some furtehr clarifications, most of proofs are correct to my best knowledge. The theorem proofs are actually dependent on assumptions raised in this paper, and assumptions are provided with explanations. The experiments were conducted on sufficient banchmarks, in comparison with multiple baselines. Furthermore, the results not only include the numeric results, but also involve some demonstrations of test case, which make the paper more comprehensive. \n3. This paper is well written. It has a clear motivation in Introduction and a thorough introduction of social delimma and opponent shaping. For those people who are not doing research in this direction, it is still friendly enough for them to catch up. \n4. As for the significance of the research problem, social delimma, it is surely a significant problem. The main reason from my own perspective is that it can simulate a class of social problems. About the opponent shaping direction, the significance to me is sceptical, since the requirement of opponent's knowledge seems like a bit strong assumption. However, I do not deny that this is a necessary step towards the weaker and more realistic assumptions. As a result, this paper is significant within the resesrch community of opponent shaping.", "weaknesses": "I have some technical concerns about this paper, which are specifically listed as follows:\n1. In line 720, could you explain why the $\\beta$ term is missing?\n2. In line 739, could you give more details about how equation (16) is derived from equation (15)?\n3. In line 755, could you give more details about how to transform from (19) to (8), step by step?\n4. In line 773, could you give more details about how equation (24) is derived from equation (23)?\n5. In line 783, even with the assumption of orthonormal gradients, it is still not clear why equation (27) can be derived from equation (26). Could you please clarify this step by step?\n6. In line 792, is $A^{*}_{\\text{LOLA}}$ equal to $V^{1}$?\n7. In line 809, could you explain why $\\alpha \\nabla_{\\theta_{2}} V^{2}(\\theta_{1}, \\theta_{2})$ is equal to $\\Delta \\theta_{2}$?\n8. In line 1001, there is a typo: equation equation -> equation.\n\nAdditionally, I also have some concerns about the experimental analysis:\n\n9. In the results shown in Figure 1b, could you give some intuitive interpretation on the asymmetry between the results of CD and DC? For example, is it related to your theoretical claims?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8luVuFYZWz", "reviewer": "Reviewer_V6em", "summary": "The paper proposes a novel method for opponent shaping based on the advantage functions of the interacting agents. The derivation unifies prior works through the lens of advantage alignment. Empirically, the proposed method is effective in learning complex coordination behaviors under various multi-agent environments.", "questions": "- Could the authors explain how LOLA and LOQA and AdAlign are related? I see the derivation in Theorem 1 and 2 but do not understand the implication and how they differ in terms of their training objectives.\n- It is not clear why increasing the log prob when both agents' advantages are negative is desirable (Eq. 8 and Fig. 1a). Can the authors elaborate on this?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The proposed derivation is novel, and provides novel insight on opponent shaping methods\n- The paper is theoretically grounded and well motivated.\n- The resulting derivation unifies prior works nicely (Theorem 1 and 2).\n- Broad suite of baselines and evaluation environments.", "weaknesses": "- I wish Section 2 and 3 are more detailed, and cover more technical background on prior works. The additional information would allow the reader appreciate the contribution much better.\n- The evaluation protocol is not provided. For readers that have little background in opponent shaping, it is not clear if the agent is trained prior to the evaluation or being trained (and thus adapted) during the evaluation episodes.\n- It is not clear why increasing the log prob when both agents' advantages are negative is desirable (Eq. 8 and Fig. 1a).\n- To my understanding, the proposed method does not outperform the baselines LOQA algorithm. The authors should discuss the differences and why it is desirable to use the proposed method over LOQA.\n- The paper claims that the proposed method is more efficient than the baseline but does not elaborate on this aspect.\n\nMinor comments:\n- The advantage function is not defined mathematically.\n- I believe Eq.2 is not the right expression, though I think this does not affect the main derivation of the paper.\n- The connection between Eq.5 and Eq.6 is not explained.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)