Aligning Individual and Collective Objectives in Multi-Agent Cooperation
Authors: Yang Li
Wenhao Zhang
Jianhong Wang
Shao Zhang
Yali Du
Ying Wen
Wei Pan
Venue: NeurIPS 2024 poster (2024)
Keywords: Mixed-motive cooperation
Mixed-motive game
cooperative AI
Primary area: reinforcement_learning
TL;DR: We propose a novel optimization method, AgA, that employs gradient adjustments to progressively align individual and collective objectives in the mixed-motive setting.

Abstract: Among the research topics in multi-agent learning, mixed-motive cooperation is one of the most prominent challenges, primarily due to the mismatch between individual and collective goals. The cutting-edge research is focused on incorporating domain knowledge into rewards and introducing additional mechanisms to incentivize cooperation. However, these approaches often face shortcomings such as the effort on manual design and the absence of theoretical groundings. To close this gap, we model the mixed-motive game as a differentiable game for the ease of illuminating the learning dynamics towards cooperation. More detailed, we introduce a novel optimization method named \textbf{\textit{A}}ltruistic \textbf{\textit{G}}radient \textbf{\textit{A}}djustment (\textbf{\textit{AgA}}) that employs gradient adjustments to progressively align individual and collective objectives. Furthermore, we theoretically prove that AgA effectively attracts gradients to stable fixed points of the collective objective while considering individual interests, and we validate these claims with empirical evidence. We evaluate the effectiveness of our algorithm AgA through benchmark environments for testing mixed-motive collaboration with small-scale agents such as the two-player public good game and the sequential social dilemma games, Cleanup and Harvest, as well as our self-developed large-scale environment in the game StarCraft II.

Aligning Individual and Collective Objectives in Multi-Agent Cooperation
Yang Li The University of Manchester yang. 1---4maaaceester..mm ac.
Wenhao Zhang Shanghai Jiao Tong University venhao_zhangee baaaaao_zhanaasjee tu. en
Jianhong Wang! INFORMED-AI Hub University of Bristol
Shao Zhang
Yali Du King's ' College London yali du@kcl ac. uk
Shanghai Jiao Tong University
shaozhang@sjtu. edu. cn
f jianhong. wang@bristol.com ac.uk
Ying Wen"
Wei Pan"
Shanghai Jiao Tong University ying. wends ju edu.cn
The University of Manchester wei pan@mancheheeeeer.....mm ac uk
Abstract
Among the research topics in multi-agent learning, mixed- -ixteemmtive cooperation one of the most prominent challenges, primarily due to the mismatch between individual and collective goals. The cutting-edge research is focused on incorporating domain knowledge into rewards and introducing additional mechanisms to incentivize cooperation. However, these approaches often face shortcomings such as the effort manual design and the absence of theoretical groundings. To close this gap, we model the mixed-motive game as differentiable game for the ease of illuminating the learning dynamics towards cooperation. More detailed, introduce a novel optimization method named A Atruistic Gradient Adjustment (AgA) that employs gradient adjustments progressively align individual and collective objectives. Furthermore, we theoretically prove that AgA effectively attracts gradients to stable fixed points the collective objective while considering individual interests, and we validate these claims with empirical evidence. We evaluate the effectiveness of our algorithm AgA through benchmark environments for testing mixed-motive collaboration with small-scale agents such as the two-player public good game and the sequential social dilemma games, Cleanup and Harvest, as well as our self-developed large-scale environment in the game StarCraft II.
Introduction
Multi-agent cooperation primarily focuses on learning how to promote collaborative behavior in shared environments. In general, multi-agent cooperation research is categorized into two prominent areas: pure-motive cooperation and mixed-motive cooperation [Du et al. 2023 McKee et al. 2020). Recent progress in cooperative Multi-Agent Reinforeement Learning (MARL) has been primarily focusing on pure-motive cooperation, also known as common payoff games. This game models situations where each agent's individual goal fully aligns with the collective objectives (Yu al. 2022
Wang et al. 2020 Zhong et al. 2024 Li et al. 2024 Nevertheless, mixed-motive cooperation
Cooresponding authors. Jianhong Wang is a visiting researcher at the University of Manchester.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

is more widespread across real-world situations. It is usually defined by imperfect alignment between
individual and collective rationalities [Rapoport 1974 McKee et al. 2020].
Recent studies in mixed-motive cooperative MARI have largely employed hand-crafted designs to promote collaboratio.. One popular approach is to align objectives as per existing mechanisms in cooperative games, such as reputation [Anastassacos et al. 2021], norms [Vinitsky et al. 2023]. and contracts (Hughes et al., 2020 Another prevalent method leverages intrinsic motivation to align individual and collective objectives, enhancing altruistic collaboration by integrating heuristic knowledge into the incentive function. Conventionally, some works accumulate individual rewards with the group to promote altruistic conduct [Hostallero et al., 2020, Peysakhovich and Lerer 2018
Apt and Schäfer, 2014, Roesch et al. [024]. Furthermore, some studies derive more sophisticated preference signals from the rewards of other agents (Hughes et al. 2018 [ccKee et al. 2020].
Additionally, several approaches aim to learn the potential influences of an agent's actions on others [Yang et al.) [020 Daques et al., 2019 Lu et al., [022]. Most of these algorithms rely heavily on carefully crafted designs, necessitating significant human expertise and detailed domain knowledge. On the other hand, several studies leverage Nash equilibria and related game theory concepts, such as the price of anarchy, to automatically modify rewards by learning additional weights that adjust the original objectives [Gemp et al., 2022 Kwon et al. 2023]. However, finding Nash equilibria in nonconvex games presents a greater challenge compared to identifying minima in neural networks [Letcher et al. [019a]].
In this study, we introduce the differentiable mixed-motive game (DMG), an effective framework for analyzing learning dynamics at both individual and collective levels. Furthermore, we derive the Altruistic Gradient Adjustment (AgA) algorithm, which aligns individual and collective objectives by modifying the gradient. We theoretically prove that AgA, with an appropriately chosen sign for the adjustment term, can successfully guide the gradient towards stable fixed points of the collective objective while considering individual interests. Additionally, empirical evidence from optimization trajectory visualizations and ablation studies validates our claims.
We also conduct comprehensive experiments to verify the effectiveness of the proposed AgA algo- rithm. First, optimization trajectory analysis and ablation studies validate our theoretical conclusions. Next, a series of experiments conducted in various environments demonstrate that the AgA algorithm outperforms related baselines in both gradient adjustment and mixed-motive cooperation areas. In addition to commonly used testbeds like the public goods matrix game and sequential social dilemma games (Cleanup and Harvest) [Leibo et al. 2017]. which are limited in terms of agent scale, action space, and task complexity, we introduce more complex mixed-motive environment called Selfish- MMM2, an adaptation of the MMM2 map from the StarCraft II game [Samvelyan et al., 2019). Selfish-MMM2 offers the following significant improvements over other mixed-motive games: it supports large-scale scenarios with 10 heterogeneous controlled agents facing enemies, features large action space with size of 18'0 which vastly exceeds the action spaces in Cleanup 9 and Harvest 8 that are limited to homogeneous agents. Delfish-MMM2 also has a larger action space and greater task complexity than the 10-player PD testbed (with a size of 2 used in D3C [Gemp et al. 2022]. which claims to solve the large-scale problem in mixed-motive games.
In summary, the contributions of this paper are as follows: (1) We are the first work to model the mixed-motive game as a differentiable game (to the best of our knowledge) and propose the AgA algorithm to align individual and collective objectives from gradient perspective. (2) We theoretically prove that, in the neighborhood of fixed points, AgA could pull the gradient toward stable fixed points of the collective objectives and push the gradient away from unstable fixed points. (3) We introduce Selfish-MMie -M1i, a novel large-scale conduct comprehensive experiments across multiple settings that verify our theoretical claims and demonstrate the superior performance of the AgA algorithm.
-oxive cooperation environment, and
2 Related Work
Mixed-motive cooperation. Mixed-motive cooperation refers to scenarios where the group's 's
objectives are sometimes aligned and at other times conflicted [Philip S. Gallo and McClintock 1965
Recently, there has been surge in academic interest in the Sequential Social Dilemma (SSD) Leibo
jt al. 2017). which expands the concept from its roots matrix games (Macy and Flache| 2002
to Markov games. Prosocial (Peysakhovich and Lerer 2018 improves collective performance by
2

---

blending individual rewards to redraft the agent's overall utility. Inequity aversion further integrates the concept into Markov games by adding envy (disadvantageous inequality) and guilt (advantageous inequality) rewards to the original individual rewards (Hughes et al. 2018). To further promote cooperation, the gifting mechanism- crucial strategy in mixed-motive cooperation [Lupu and Precup 2020] allows agents to influence each other's ss reward functions through peer rewarding. Besides, PED-DQN [Hostallero et al. 20200 introduces an automatic reward-shaping MARL method that gradually adjusts rewards to shift agents' actions from their perceived equilibrium towards more cooperative outcomes. Social Value Orientation (SVO) strategy introduces a unique shared rewards- based compensation approach, encouraging behavior modifications in line with interdependence theory [McKee et al., 2020]. The LIO strategy bypasses the need to modify extrinsic rewards by empowering agent to directly influence its partner's 's actions (anng et al., 2020]. Meanwhile, other studies introduce new cooperative mechanisms such as incorporating a reputation model [McKee Recently,,Roesch al. 2021], J. social norms (Vinitsky et al. 2023 Recently, |Roesch et al. 2024) proposed selfishness
level method that incorporates social welfare into individual rewards to enhance altruistic cooperation.
Additionally, several works have explored automatically modifying rewards online. D3C [Gemp)
ct al. 2022] learns to mix rewards to improve efficiency in a Nash equilibrium, Kwon et al. [2023] extended D3C by addressing the problem of automatically modifying individual agent objectives to optimize a desired global objective. Despite these advances, many current studies lack both cost efficiency in design and theoretical analysis of alignment and convergence. To address this, our study applies gradient perspective to align goals and further investigates the learning dynamics of our proposed method.
Gradient- based Methods. Our proposed AgA is fundamentally a gradient adjustment methodology, making gradient-bascl -ased optimization methods highly relevant to the context of this paper. Methods based on gradient have been developed to find stationary points, such as the Nash equilibrium or stable fixed points. Optimistic Mirror Descent leverages historical data to extrapolate subsequent gradients Daskalakis et al.] (2188) while Gidel et al. (2019) extends this concept by advocating averaging methodologies and variants of extrapolation techniques. Consensus gradient adjustment, consensus optimization, is a technique that embeds a consensus agreement term within the gradient to ensure its convergence [Mescheder al., -017]- Learning with Opponent-L_earning Awareness (LOLA) utilizes information from other players to compute one player's anticipated learning steps (Foerster) al. 20188. Subsequently, Stable Opponent Shaping (SOS) [Letcher al., 2019b] and Consistent Opponent-Learning Awareness (COLA) (Willi et al. 2022 methods have enhanced the LOLA algorithm, targeting convergence assurance and inconsistency elimination, respectively. Symplectic Gradient Adjustment (SGA) alters the update direction towards the stable fixed points based on a novel decomposition of game dynamics [Balduzzi al., 2018 Letcher et al. 2019a]. Recently, Learning to Play Games (L2PG) ensures convergence towards a stable fixed point by predicting updates to players' parameters derived from historical trajectories (Chen et al. 2023). However, these methods, while focusing on zero-sum or general sum games, could potentially act counter to their individual interests, as they may prioritize stability over minimizing personal loss. Our research specifically targets mixed-motie motive setting with the aim of reconciling individual and collective objectives.
3 Preliminaries
3.1 Differential Game
The theory of differential games was initially proposed by Isaacs [965), aiming to expand the scope of sequential game theory to encompass continuous-time scenarios. Through the lens of machine learning, we formalize the differential game, as shown in Definition 3.1
Definition 3.1 (Differential Game [Balduzii et al. 2018 Letcher et al. (2019a): ). A differential game ......,... is the set of players. The parameter set
could be defined as a tuple {N, w. L). where N w w " E Fd is defined, each with w, Rdd: and d En= dz. Here, {= (l, Rd - R) represents the corresponding losses. These losses are assumed to be at least twice differential. Each player i E N is equipped with a policy, parameterized by wi aiming minimize its loss f,-
We write the simultaneous gradient {(w) of differential game {(w) (V wi fi...., Vw, (n)
Rd, which is the gradient of the losses with respect to the parameters of the respective players. Furthermore, Hessian matrix H mentioned in a differential game is the Jacobian matrix of the simultaneous gradient.
3

---

AgA (Durs) Simulice
Ayá (Ouns Simul-Ce
Action Player
Action of Praper
Start Point
Player
Action Player
Action Player
(a) Collective Reward Landscape
(b) Reward Contour of Player 1
(c) Reward Contour of Player'
Figure 1: Trajectories of optimization in a two-player DMG (as delineated in Example 4.1). Fig displays the trajectories over the collective reward landscape deeeper orange equates higher rewards. Remarkably, only Simul-Co and AgA make successful strides towards the social optimum. Fig Tbd Fig. [c trajectories on the individual reward contour, underscoring Simul-Co's neglect for Player 's interests as it navigates through the crests and troughs of its reward. Conversely, our AgA optimizes along the summit of Player I's reward while also maximizing the collective reward, demonstrating successful alignment.
The learning dynamics of differential game often refers to the process of sequential updates over uc. The learning rule for each player is defined as the operator, (learning rate) to determine the distance to move for each update.
- -E, where y is step size
3.2 Gradient Adjustment Optimization
The stable fixed point, criterion initiated from stability theory, exhibits its stability (robustness) to minor perturbations of environments, making it applicable to many real-world scenarios.
Definition 3.2. point w" is a fixed point if (()") = O. If H (w=)= 2 and H Hww") is invertible, the fixed point w" is called stable fixed point. If H (w=) x O, the point is called unstable.
naive idea to steer the dynamic towards convergence at fixed points involves minimizing [[()))] Assuming the Hessian H (w) is invertible, then ( [[)](()) H = 0 holds true if and only if However, it could converge unstable fixed points Mescheder et al.) [017]. Hence, the consensus optimization method has been proposed, incorporating gradient adjustment [Mescheder et al. ]. shown as follows: & = E X V 11 = E + A H For simplicity, we will reter to the consensus gradient adjustment as CGA. While CGA proves effective in certain specific scenarios, such as two-player zero-sum games, it unfortunately falls short in general games [Balduzzi al. 2018]. To address this shortage, symplectic gradient adjustment (SGA) [Balduzzi et al., 2018 Letcher et al. [2019a] is introduced to find the stable fixed point in general sum games, such that - A'C. Herein, A represents the antisymmetrie component of the generalized Helmholtz decomposition H, (w)) Sww) + Aww), where S(w) denotes the symmetric component.
Method
In this section, we first define differentiable mixed-motive games (DMG) and analyzes the alignment dilemma faced by existing methods, which is described in Section 4.1
4.1 We then propose the Altruistic Gradient Adjustment (AgA) algorithm in Section 4.2 to align the individual and collective objectives by modifying the gradient. Finally, case study demonstrating AgA's effectiveness in toy two-player DMG is presented in Section 4.3
4.1 Differentiable Mixed-motive Game
We first formulate the mixed-motive game differentiable game. Specifically, the differentiable mixed-motive game (DMG) defined as a tuple (N, w. where E, E l is least twice the differentiable loss function for the agent i. Differentiable losses exhibit the mixed motivation property: minimization of individual losses can result in a conflict between individuals or between individual and collective objectives (c.g., maximizing individual stats and winning the game are often

---

Sc
H. -00
H. =
Unitable Fixed Peint
Stable Flxed Point
Case I:
Case 2:
Pask out Unstable Fised Point
Pill Joward Stable Fised Point
De Action Player
(b)
Figure 2: Left figure Illustration of Corollary 4.3 In case 1, within an unstable fixed point's neighborhoods, an appropriate selection of the À sign push AgA evade the unstable fixed point and pull towards stable fixed point in its neighborhoods shown in case 2. Right figure b: Alignment Effectiveness of AgA. The comparison between AgA (shown in red) and AgA without sign alignment (AgA-Sign, in purple) trajectories spans 40 steps, marked at every tenth step. Norm gradients represented with blue arrows. Starting from the 14th step, sign alignment pulls the gradient toward the steepest direction, resulting in AgA reducing the number of steps by approximately 15% compared to AgA-Sign by the end of the trajectory.
conflict in basketball matches). In addition to the simultancous gradient {(w) of individual losses with respect to the parameters of the respective players, we use fe(w) refer to the gradient of the collective loss: (w()) = (V le..... Vw,, fe)
Alignment Dilemma in DMG. Direct optimization of individual losses and collective loss straightforward ideas to solving DMG problems. However, minimizing only the individual loss of each agent is unlikely to achieve collective optimum [Leibo et al., 2017 McKee al., 2020). Conversely, optimizing the collective loss might produce better overall outcomes but risks neglecting individual interests [Roesch et al.) 2024]. Additionally, the local convergence of gradient descent in singular collective functions is not always guaranteed [Balduzzi et al. 2018).
Example 4.1 gives two-player differentiable mixed-motinn game. We provide visual representation of optimization trajectories using a series methods to investigate the learning dynamics involved in resolving the example, shown in Fig. [
Example 4.1. Consider a two-player DMG with E (a1, a2) = -in 02 + az) and (2(1), (2) - [coss[1 (1 + a1 - +1+ 22)²) + a d2)) where a, represents the action of the player i (i = 1, 2), and a, E R. The rewards for the two players are the negation of their respective losses.
We first implement simultaneous optimization of individual losses (Simul-Ind) with respect to each parameter, defined by the learning rule w; = w; EE for E (1,2), where & = V L,- Simultaneous optimization of collective loss (Simul-Co) replaces the individual gradient with the collective gradient &. of collective loss le = ε + fz. Furthermore, in order to investigate the learning dynamics of prevalent gradient modification optimization approaches such SGA and CGA for the two-player differentiable mixed-motive game, we modify the learning rule by integrating the respective adjusted gradient delineated in Section 3.2
Fig. Ia shows the optimization paths on the collective reward landscapes, with every path starting from the front-bottom of the landscape. The collective reward is defined as social welfare, i.c., the sum of individual rewards. As shown in the figure, Simul-Ind, CGA and SGA converge to unstable points local maxima. Simul-Co effectively navigates towards the apex of the collective reward landscape as depicted in Fig. Ta However, Simul-Co is ineffective in aligning individual and collective objectives, leading to the neglect of individual interests. As depieted in Fig. [bb the update trajectory navigates through the crests and troughs of Player I's reward contour. The trajectory suggests disregard for Player I's preferences, signifying that the updates are predominantly driven by the overarching collective goal.
4.2 Altruistic Gradient Adjustment
To tackle the challenge of aligning objectives in DMG, propose the Altruistic Gradient Adjustment (AgA) method. Unlike existing gradient adjustment techniques [Mescheder al., 2017 Balduzzi]
al. 2018
Letcher et al. 2019a Chen et al. 2023), which primarily aim at achieving stable fixed
5

---

points for individual objectives, our AgA method simultaneously considers both individual and collective objectives when seeking stable fixed points for the collective objective. Specifically, AgA is defined as follows.
Definition 4.2 (Altruistic Gradient Adjustment). Altruistic gradient adjustment (AgA) extends the gradient term in the learning dynamic as
+ - +
(1)
where À E R is alignment parameter, >Lndj is called adjustment term. In Eadj, En and H. is the gradient vector and Hessian matrix of the game about collective loss.
Note that the Hessian matrix H. is symmetric. For brevity, we denote VH, as V 11)) which simplifies to H.S In our AgA method, the formula & + X(ς + H. includes the Hessian matrix, but we do not compute it directly. Instead, we calculate Hessian-vector products H Se. which suffice for determining the adjustment gradient and the sign of X. This approach reduces computational complexity, enhancing the effectiveness of the AgA method. The cost for computing Hessian-vector products H &. is ((n) for weights [Pearlmutter 1994. Refer to Section 6 for detailed analysis.
While AgA introduces additional complexity, we theoretically demonstrate that an appropriate choice of the sign of ensures that AgA pulls the gradient towards a stable fixed point through the adjustment term in the neighborhood of fixed points. Conversely, when dealing with an unstable fixed point, AgA will push the gradient away from these unstable points.
Before introducing the corollary, we first provide some basic notations. The inner product of vectors and b are denoted by (a, b). If the Hessian matrix H is non-negative-defiiie. (Ec, VH) 2 0 for a non-zero &.. Analogously, if H is negative-definite (Ec. VH) < 0 for a non-zero &.. Lastly, the angle between the vectors and b is denoted by 0(a, b). Then, we state the corollary as follows:
Corollary 4.3. In the neighborhood of fixed points of the collective objective, AgA will pull the gradient toward stable fixed points, which means O(E, VHe)) S ((E, VHH.), and push away from unstable ones, indicated by o(g, VHa)
VHL), if X satisfies X (Ec, VH.)(E VHe)
I|V 13) 2.
The proof is provided in Appendix B Fig. 2a presents an illustrative visualization of Corollary 4.3 The figure depicts two scenarios: the neighborhoods of stable and unstable fixed point (denoted by star), respectively. In the neighborhood of an unstable fixed point of the collective objective, as depicted in Case 1, AaA gradient & is pushed out of the region, resulting in faster escape compared to the original collective gradient &c. This behavior is exemplified by the green arrow, Conversely, Case 2 illustrates the scenario of stable fixed point, wherein our AaA gradient È is pulled towards the stable equilibrium, exemplified by the red arrow.
AgA could be casily integrated into any centralized training with decentralized execution (CTDE) framework within MARL. Detailed statements on the implementation of AgA, including the pseu- docode, are provided Appendix C
4.3 Alignment Effectiveness of AgA: Toy Experiment
To address the two-player DMG as outlined in Example we incorporate the AgA method into the fundamental gradient descent algorithm. The optimization trajectories in both the collective reward landscape and the individual player reward contour are represented in Fig. [a| Fig. [b] and Fig. [c In contrast to the objective misalignment exhibited by Simul-Co, AgA successfully aligns the agents' interests, evidenced in Fig. [b and Fig Ic. Fig. lb depicts the trajectory of AgA meticulously carving its course along the summit of the individual reward contour. Furthermore, AgA is also shown to improve Player 2 reward (as shown in Fig. Ic), despite a slower convergence rate than Simul-Co.
In addition, Fig. 26 presents a critical comparison between the trajectories of AgA (in red) and AgA without sign alignment (AgA-Sign, depicted purple), as outlined in Corollary 4.3 This side-by-side comparison covers 40 steps and features highlighted points every tenth step. It provides visual illustration of the effectiveness introduced by sign alignment starting from the 14th step the trajectories of AgA and AgA without sign alignment. Remarkably, norm gradients are represented by

---

Table 1: The comparison of the average individual rewards (denoted as r,, r2), social welfare (denoted as SW), and equality metric (denoted as E) on two- player public goods game. We show the mean of value and 95% confidence interval utilizing 50 random runs.
Metrics
Simul-Ind
CGA
SGA
SVO
Simul-Co
S1.
AgA
T1
1.133 0.063
1,156
0.060
1.175 + 0.062
1.104
0.054
1.433 + 0.056
1.314 +: 0.062
1.443:t
0.042
1.184 +: 0.065
1.150 4: 0.057
1.137 + 0.063
1.060 = 0.051
1.381 + 0.065
1.371 +: 0.057
1.459
0.041
SW
2,316+ 0.039
2,306
0.039
2.312+ 0,044
2.164 0.026
2.814 + 0.033
2,684 + 0.049
2,903+ 0.023
E
0.923+ 0.014
0.929 0.012
0.924# 0.013
0,930 = 0.011
0.941 + 0.014
0.940 + 0.011
0.960 0.008
blue arrows, indicating the direction of the fastest updates. As depicted in the figure, AgA with sign selection is aligned toward the steepest update direction, resulting reduction of approximately 15% in the number of steps compared to AgA-Sign by the end of the trajectory.
5
Experiments
To assess the effectiveness of the proposed AgA algorithm, we perform comprehensive experiments across various environments, ranging from simple to complex, and from small to large scales. The initial enviromment involves a two-player public goods game (see Section P.I. followed by sequential social dilemma environments (see Section [52]]: Cleanup and Harvest, involving 5 homogeneous players. The final test is conducted in our specially developed Selfish-MMM2 environment (see Section (.3). which is both more complex and larger in scale, featuring approximately 10 controlled agents of distinct types, competing against 11 adversaries. Comparative experiments employ commonly used baseline approaches, such as simultaneous optimization with individual and collective losses (termed Simul-Ind and Siml-Co), CGA [Mescheder al., 2017], SGA [Balduzzi et al. 2018 SVO McKKee et al. 2020]. and the selfishness level approach (termed SL) [Roesch et al. 2024
5.1 Two-Player Public Goods Game
A two-player public goods matrix game G is widely utilized to study cooperation in social dilemmas.
The game involves players [1 with parameters {w, W22 and payoffs (Pi.P2), The social welfare, denoted as SW = P1 + P2. Each player, i.e., i E {1,2). contributes an amount ag within a budgeted range [0, bj, and the host evenly distributes these contributions
{(a, + a2), where < c S 2.
Consequently, each player's payoff (a a2) is estimated as b-a. (a) + 02) - In our experiments, we set the budget b to 1 and weight e to 1.5, with the social optimum of the game at 11.
Results. Table Dollenpreentts a comparison of individual rewards and the collective outcome, derived from 50 random trials, each limited to 100 steps. The rows T1, T2, SW, and E' represent the individual rewards for each player, the group's total welfare (SW), and the equality metric (E), respectively. The equality metric is based on the Gini coefficient (G) [David 1968). with E defined as E = 1 G = nip Σ i(p; p). where p is the mean of the ranked payoff vector p. and n is the total number of players. A higher value indicates greater equality among the players.
The social optimum of the game occurs when all players allocate their entire budget, achieving the highest possible social welfare score of 3. Among all algorithms, AgA achieves the closest social welfare score to this optimum, with value of 2.90. Furthermore, AgA exhibits the greatest degree of fairness, with two players receiving nearly identical rewards and the highest equality value, in contrast to the baseline algorithms. Furthermore, Fig. i in Appendix D illustrates the action distributions of these algorithms, showing that AgA actions are more tightly clustered around the social optimum actions (1, 1). This further indicates that AgA approaches the social optimum more effectively.
5.2 Sequential Social Dilemma: Cleanup and Harvest
We then verify the AgA algorithm in two widely used sequential social dilemma games with 5 homogeneous agents: Harvest and Cleanup [Hughes et al. 2018). In the Cleanup scenario, agents predominantly gather rewards by harvesting apples in an orchard, where the yield is affected by the river's pollution levels. Neglecting rising pollution levels ultimately ceases apple production, thus setting up trade-off between individual gains and communal welfare. In contrast, the Harvest
7

---

Table 2: The comparison of the average equality metric (denoted as E) on Harvest and Cleanup. We show the mean of equality value and standard deviation utilizing three random runs.
AgA
Envs
Simul-Ind
Simul-Co
SVO
CGA
SL.
À = 0.1
A
A 100
A = 1000
0.973
0.975
0.974
0.950
0.972
0.981
0.988
0.982
0,980
Harvest
+: 0.005
+ 0.006
+ 0.007
+ 0.051
1 0.005
0.006
0.003
0.012
0.006
0.841
0.948
0,902
0.903
0.946
0.940
0.956
0.959
0.905
Cleanup
1 0.071
0.013
1 0.019
0.034
1 0.016
0.017
0.007
0.011
0.022
scenario compensates agents for apple collection, with the regeneration of apples being ideally dependent on the proximity to other apples. Here, the communal challenge manifests itself as over-harvesting apples, which diminishes their regrowth rate and, consequently, the aggregate rewards for the group. Our rescarch assessed AgA against various benchmarks, including Simul-Ind Simul-Co, CGA, SVO, and SL. Simul-Ind was implemented using the IPPO algorithm de Witl et al. 2020]. while the remaining benchmarks employed the common parameters of the IPPO framework. The formula for calculating the collective loss for CGA and AgAs, which aims to improve both the performance of the group and the equitable distribution between agents, is expressed
as Σ: r; a(1 - arctan Exaset
where is a constant. Further details on the experimental setups and the algorithms used are provided in Appendix E.I
Varying À values. Fig. 3a and Fig. 3b explore the social welfare outcomes from AgA under three distinct À values: 0.1, 100, and 1000. Among these, À 100 demonstrates superior performance in both the Cleanup and Harvest scenarios. Therefore, in the subsequent experiments involving sign alignment and comparing primary results with baseline models, we will employ X 100 as the default parameter.
The effectiveness of sign alignment. In the second row of Fig. Fig. 3d and Fig. 3e illustrate the social welfare comparisons between AgA and AgA without sign alignment (AgA-Sigh during
their training phases. Referencing Corollary 1.. near fixed point, sign alignment helps direct the gradient towards stable fixed points and away from unstable ones. During the latter half of the training
sessions shown in Fig. 3d and Fig. 3e it is observed that the social welfare (SW) for AgA-Sign does not improve to the same extent as it does for AgA, highlighting the effectiveness of sign alignment.
Baseline Comparison. As illustrated in Fig. 3g and Fig. 3h AgA demonstrates notable improve- ment in social welfare over Simul-Ind, Simul Co, CGA, SVO, and SL. Particularly, within the Cleanup scenario, AgA recorded average social welfare of 105.15, marking approximately 56% increase over the runner-up method SL.. In the Harvest scenario, AgA shows an average improvement of 33.55 in social welfare over the next best method SVO. Table deviation of the equality metric achieved by different methods in the Harvest and Cleanup environ- ments. A value closer signifies more equal rewards among agents. As shown in the table, AgA outperforms the baseline methods, demonstrating its ability to effectively balance the interests of all team members.
the mean and standard
5.3 Belfish-MMM2
While sequential social dilemma games are frequently used to study mixed-motive problems, they relatively simplistic compared to other experimental environments in MARL,, particularly when considering real-world applications. To address this gap, we introduce a sophisticated mixed- motive testbed called Selfish-MMM2. which adapted from the MMM2 map in the StarCraft II game (Samvelyan et al., 2019). Unlike the standard SMAC framework, which employs a shared reward structure, our methodology assigns individual objectives each agent, thereby enhancing the complexity of mixed motives. Specifically, we have redesigned the reward systems so that each agent's 's personal gain is directly tied to the damage they inflict on adversaries, diverging from traditional approaches where collective damage is evenly distributed among all agents. Additionally, penalties for agent elimination emphasize the importance of self seff-preservation and highlight agents' tendencies towards self-interested behavior. In comparison to widely utilized multi-agent environments such as Cleanup [Hughes et al. 2018), Harvest [Hughes et al. 2018), and the 10-player Prisoner's Dilemma (PD) for scrge-scale evaluations [Gemp al. 2022 Belfish-MMMR is distinguished by two notable
8

---

800
700
100
90
SW
50
400
Win Rate
83
ApA (2
300
AgA (A.)
89
AgA (1 0.1)
AgA 100)
AzA (2
100
AgA (A
AgA (A=100)
20
AgA (A
AgA (1-00)
(a- 1000)
(10000))
(1-1000)
-10
0.25 0.59 0.75 1,00 1.25 1.50 1.76 2,00
100
0.0
0.2
9.4
0.6
1.0
0.2
0.4
0.6
0.8
100
Stepa (1x7)
Total Stapa (1w7)
Total Btspa (1w7)
(a) Cleanup
(b) Harvest
(c) Selfish-MMME
700
100
800
50
400
Win Rate
300
50
AgA
AgA
20
ApA
Agh-S5g Sign
AgA-Sign
ApA-Bign
-100
0.00 0.25 0.50 0.75 1,00 1,25 1.50 1.75 2
-100
0.0
0.2
8.4
0.6
0.8
1.8
0.0
0.2
0.4
0.6
0.5
1.0
Total Steps (107)
Total Steps (107)
Total Stops (107)
(d) Cleanup
(c) Harvest
(f) MMMish-MMM2
140
120
7.0
650
Winh Rate
80
Stmudhidd
Staus-Co
CG4
avo
Aga
300
Simmukin
Stmut-Co
CG4
Byo
Agh
Slrmsd-and
Simul-co
CG4
8vo
s1
Agh
Cleanup
(h) Harvest
Selfish-MMM2
Figure 3: The first row displays results comparing different values of the alignment parameter across three environments: Cleanup and Harvest (measurement of social welfare, SW) and Selfish- MMM2 (focusing on win rate). The second row examines the performance differences between the proposed AgA and AgA without sign alignment (AgA-Sign) lines indicate the mean social welfare calculated in three seeds, while the surrounding shaded areas represent the 95% confidence interval. The third compares the AgA method with baseline approaches on these testheds. Each bar represents the mean collective results of each method and the error bars indicate the 95% confidence interval.
the three testbeds. The bold
features: 1) Large-Scale Agents: Selfish-MMM2 manages the interactions of 10 heterogencous
agents, categorized into three types: seven Marines, two Marauders, and one Medivac, confronting 11 enemy units. This configuration is in stark contrast Cleanup and Harvest, which each utilize five homogeneous agents. 2) Extensive Action Space: The action space in Selfish-MMM2, sized at 18 vastly larger than that of Cleanup and Harvest, which are limited to 9 and 8 respectively. It also surpasses the action space of the 10-player PD, which is 210
In evaluating the Selfish-MM22 environment, we use the average win rate against the built-in AI bot in SMAC to measure the collective performance. The experimental setup employs the IPPO algorithm dd Witt al. 2020 for the Simul-Ind algorithm, while the MAPPO Yuu et al. 2022 is utilized for the Simul-Co, Co. SVO, CGA, SL and AgA algorithms. More details on the Mclish-MMM2 environment and implementation specifics are available in the Appendix E.2
Varying À Values and the effectiveness of Sign Alignment: Fig. 3c presents the average win rates in various settings of the align parameter À in the AgA scenario. The findings indicate that increased values are associated with faster convergence in the training period. Among these, À value of 100 demonstrates a higher overall effectiveness compared to other values tested. Based on these findings, we have set default value of 100 for subsequent experiments. Furthermore, Fig. 3f in the second row illustrates comparable conclusion in sequential social dilemma games. During the

---

Table 3: Comparison of the average running time between baseline methods and AgA in the two- player public goods game, including total duration, timesteps, time per step, and the time per step ratio relative to Simul-Ind.
Metrics
Simul-Ind
Simul-Co
SL
CGA
SGA
AgA
Duration (ms)
1165.79
910.15
1149.97
3041.84
3007.77
1034.69
Steps
4272
3252
3887
4478
4179
1389
Step Time (ms)
0.27
0.28
0.30
0.68
0.72
0.74
Ratio
1.00
1.04
1.11
2.52
2.67
2.74
initial training phase, the convergence rates of AgA and AgA-Sign are nearly identical. However, as training progresses, AgA exhibits convergence points of superior performance, highlighting the critical role of sign alignment achieving convergence near fixed points.
Baseline Comparison. As depicted in Fig. 3i the bar chart displays a comparison of the average win rates along with 95% confidence intervals for AgA versus other baseline approaches. Our AgA approach attains the highest team performance, securing a 92.64% win rate with the built-in bot, which is 23.61 percent more than the next best win rate achieved by the SL method.
Discussion
Computational Cost of AgA. The altruistic gradient adjustment is expressed as & & X(g H.Te),, {c), where the key computational cost comes from the Hessian- vector product. H" E... The cost for computing Hessian-vect vector products, H Ec, is (n) for n weights (Pearlmutter 1994). This introduces added complexity compared to standard methods in mixed-motive MARL, making AgA's running time generally more than twice as long as gradient-based methods. Table provides comparison of the average running time between AgA and baseline methods in a two-player public goods game, including total duration, timesteps, time per step, and the time ratio relative Simul-Ind Simul-Ind, Simul-Co, and SL are standard gradient-based methods, while CGA and SGA modify gradients. Our results show that AgA takes 2-3 times longer per step but remains the most efficient, requiring only 1389 steps over 50 runs, compared to around 4000 steps for the baselines.
Limitations. Despite conducting series of experiments to verify the proposed AgA algorithm, cooperation testbed, these environments remain
including using a new large-scale somewhat removed from real-world applications. Although strive align the objectives of both individuals and the collective, our primary focus in this paper is on converging towards stable fixed points of the collective objective. For future research, plan to delve deeper into the interaction of individual objectives in mixed-motive games, with an increased emphasis on understanding their dynamics in real-world mixed-motive cooperation scenarios.
7 Conclusion
In this paper, we propose AgA, optimization method specifically designed to align individual and collective objectives through gradient adjustments mixed-motive cooperation scenarios. To achieve this, we first model the mixed-motive game as differentiable game, offering powerful tool for analyzing learning dynamics both individual and collective levels. Furthermore, we theoretically demonstrate that AgA can effectively attract gradients to stable fixed points and support our claims with empirical evidence. To evaluate the effectiveness of AgA in complex and large-scale scenarios, we introduce new mixed-motive environment called Selfish-MMM2,, which features heterogeneous large-scale agents, a larger action space, and increased task complexity. Comprehensive experiments, ranging from simple public goods games to Harvest, Cleanup, and Selfish-MMM2, show AgA's superior performance, consistently outperforming existing baselines across multiple evaluation metrics. These results validate our theoretical claims and highlight the effectiveness of AgA in achieving cooperative behavior in aulti- systems.
10

---

Acknowledgement
This work is partially supported by National Key R&D Program of China (2022ZD01 14804) and National Natural Science Foundation of China (62106141). Jianhong Wang fully supported by the Engineering and Physical Sciences Research Council [Grant Ref: EP/Y028732/1). The authors also thank Shuqing Shi for his kind assistance and advice.
References
N. Anastassacos, J. García, Hailes, and M. Musolesi. Cooperation and reputation dynamics with reinforcement learning. In F. Dignum, A. Lomuscio, U. Endriss, and A. Nowé, editors, AAMAS '21: 20th International Conference on Autonomous Agents and Multiagent Systems, Virtual Event, United Kingdom, May 3-7, 2021, pages 115-123, ACM, 2021. doi: 10.5555/3463955.4433777 URL https: //op::://www... ifaamas...... org/ wtt Pwr./eeeeeeenss////eeeeeee /ataa/021/ppfs//p/a/aa påf
K. R. Apt and G. Schäfer. Selfishness level of strategic games. Artif. Int. Res., 49(1):007-240, 2014. ISSN 1076-9757.
D. Balduzzi, Racaniere, J. Martens, J. Foerster, K. Tuyls, and T. Graepel. The mechanics of n-player differentiable games. In International Conference on Machine Learning, pages 354 363.. PMLR, 2018.
X. Chen, N. Vadori, T. Chen, and Z. Wang. Learning to optimize differentiable games. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 5036-5051, PMLR, 23-29 Jul 2023. URL BtLLhtttss:/proceeeiiii mmr..mm......... .pres// v202/ chan23ab............ html
C. Daskalakis, A. Ilyas, V. Syrgkamis, and H. Zeng. Training gans with optimism. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 2018, Conference Track Proceedings. OpenReview.net, 2018. URL httpp:///peenreview net/ Forum?id-SJJysbbAZ,
H. A. David. Gini's mean difference rediscovered. Biometrika, 55(3):773-575, 1968. ISSN 00063444. URL http://www. jtwp::jsstoooo jstor wtggstable//233/e2
S. de Witt, T. Gupta, D. Makoviichuk, V. Makoviychuk, H. S. Torr, M. Sun, and S. Whiteson. Is independent learning all you need in the stareraft multi-agent challenge? CoRR, abs/2011.095333 2020. URLpttpps: // arxiv.xivvvvvaa org/ abs 20111 09533
Y. Du, J. Leibo, U. Islam, R. Willis, and Sunehag. review of cooperation in multi-agent learning. CoRR, abs/2312.00116 2023. doi: 10-8850/AARXI22312.5112 URL https 7/doi. ordoi.org//... 48550/ 1/850//axiv.... 231.
J. N. Foerster, R. Y. Chen, M. Al-Shedivat, S. Whiteson, P. Abbeel, and I. Mordatch. Learning with opponent-learning awareness. In E. André, S. Koenig, M. Dastani, and G. Sukthankar, editors, Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS 2018, Stockholm, Sweden, July 10-15, 2018, pages 122-130. International Foundation for Autonomous Agents and Multiagent Systems Richland, SC, USA ACM, 2018. URL http://al acm. org/ httg//iiaiionn.cm cfm7id=3237408
I. Gemp, K. R. McKee, Everett, E. A. Duéñiez-Guzmán, Y. Bachrach, D. Balduzzi, and A. Tac- chetti. D3C: reducing the price of anarchy in multi-agent learning. In P. Faliszewski, V. Mascardi, C. Pelachhad, and M. E. Taylor, editors, 21 International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2022, Auckland, New Zealand, May 9-13, 2022, pages 498- 506. International Foundation for Autonomous Agents and Multiagent Systems (IFAAMAS). 2022. doi: 10.5555/353555035555555 aamas2022/ (alaas2222pddss/ppee http://ffsppp....../ htt..//...p.... pdf
URL https: ptttps://ww.... ifaamas.aa.mmmmm org prg/Proceedings/
G. Gidel, H. Berard, Vignoud, P. Vincent, and S. Lacoste-Julien. variational inequality perspective on generative adversarial networks. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL.
httts://opperrvviet net fow.mooddrr1aaaiee....

---

D. E. Hostallero, D. Kim, S. Moon, K. Son, W. J. Kang, and Y. Yi. Inducing cooperation through reward reshaping based on peer evaluations in deep multi-agent reinforcement learning, In A. E. F. Seghrouchni, G. Sukthankar, B. An, and N. Yorke-Smith, editors, Proceedings of the 19th International Conference on Autonomous Agents and Multiagent Systems, AAMAS '20, Auckland, New Zealand, May 9-13, 2020, pages 520-528. International Foundation for Autonomous Agents and Multiagent Systems, 2020, doi: 10.555533957611.339833 URL https://dl. acm org/ doi 555/3333555533887611111
E. Hughes, Z. Leibo, M. Phillips, K. Tuyls, E. A. Dučñizz-uzzmán, G. Castañeda, I. Dunning. T. Zhu, K. R. McKee, R. Koster, H. Roff, and TT Graepel. Inequity aversion improves cooperation in intertemporal social dilemmas. In Bengio, H. M. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Gamett, editors, Advances in Neural Information Processing Systems 31: Anmual Conference on Neural Information Processing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréul, Camada, pages 3330-3340, 2018. URLhttps: / 2018/ /atppe//20/////aa/aee6/6666//000 /aaaa///eaa////a00ffffffffffff/
negurrppp wwtees-bbsttttt.eeeee html
cc/
E. Hughes, T. W. Anthony, T. Eccles, J. Z. Leibo, D. Balduzzi, and Y. Bachrach. Learning to resolve alliance dilemmas in many-player zero-sum games. CoRR, abs/2003.00799, 2020. URL
https: hoopss///axxvv.oo org/ abs/
/bt/aaass.....//////////// 2003 00799
R. Isaacs. Differential Games: A Mathematical Theory with Applications to Warfare and Pursuit, Control and Optimization. Dover books on mathematics. Wiley, 1965. ISBN 142866222 URL httpt::tttbosks....tt.o// google. co. wtt...//bbbook...tt.gtAAAAA...
N. Jaques, A. Lazaridou, Hughes, C. Gulcehre, Ortega, D. Strouse, J. Z. Leibo, and N. De Fre- itas. Social influence as intrinsic motivation for multi-agent deep reinforcement learning. In International conference on machine learning, pages 4040-3049. PMLR, 2019.
D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In Y. Bengio and Y. LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. bttp:///: org abs/1412 1412 6980
M. Kwon, J. P. Agapiou, E. A. Dužfiez-Guzmán, R. Elie, G. Piliouras, K. Bullard, and I. Gemp. Auto-aligning multiagent incentives with global objectives. In ICML Workshop on Localized Learning (LLW), 2023.
J. Z. Leibo, V. F. Zambaldi, M. Lanctot, J. Marecki, and T. Graepel. Multi-agent reinforcement learning in sequential social dilemmas. CoRR, abs/1702.03037,, 2017. URL http://arxiv org/ abs/1702.032
A. Letcher, D. Balduzzi, S.
Racanière, J. Martens, Foerster, K. Tuyls, and T. Graepel. Differentiable game mechanics. Journal of Machine Learning Research, 20(84):1-40 2019a. h
oww.lrr.rrppapprrs.ee ht/////0/1110000 html
A. Letcher, J. N. Foerster, D. Balduzzi, T. Rocktäschel, and S. Whiteson. Stable opponent shaping in differentiable games. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019b. URL httpss///oppereeiiiee net net/forumPid-SS(jjaSS5
Y. Li, S. Zhang, J. Sun, W. Zhang, Y. Du Y. Wen, X. Wang, and W. Pan. Tackling cooperative incompatibility for zero-shot human-ai coordination. J. Artif. Intell. Res., 80: 139111885...1 2024. doi: 10.1613/JAIR.I 15884. URL tattps:///oi: orw/oogg//11 1613 /aarr//jar.... 1. 15884
C. Lu, T. Willi, C. A. S. de Witt, and N. Foerster. Model-free opponent shaping. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvári, G. Niu, and S. Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 1-398-14411 PMLR, 2022. URL https //proceedings. .mlr. ppwss/vvs22 1w2dd. html
the A. I Lupu and D. Precup. Gifting in multi-agent reinforcement learning. In Proceedings the 19th
International Conference autonomous agents and multiagent systems, pages 789-797, 2020.
12

---

D. Macy and A. Flache. Learning dynamics in social dilemmas. Proceedings of the National Academy of Sciences of the United States of America, 99 Suppl 3:7229-36, 06 2002. doi: 10.1073/pnas. 092080099.
K. R. McKee, I. Gemp, B. McWilliams, E. A.
E. Hughes, and J. Z. Leibo. Social diversity and social preferences in mixed-motive reinforcement learnin.. In A. E. F. Seghrouchni, G. Sukthankar, B. An, and N. Yorke-Smith, editors, Proceedings of the 19th International Con- ference on Autonomous Agents and Multiagent Systems, AAMAS '20 Auckland, New Zealand, May 9-13, 2020, pages 869-877. International Foundation for Autonomous Agents and Multiagent Systems, 2020. doi: 10.5555/3398766.33988888 URL https: //dl. acm. org 5555/ 3398761
K. R. McKee, E. Hughes, T. O. Zhu, M. J. Chadwick, R. Koster, A. G. Castaneda, C. Beattie, T. Graepel, M. Botvinick, and J. Z. Leibo. A multi-agent reinforcement learning model of reputation and cooperation in human groups. arXiv preprint 0rXiv:2103.04982 2021.
L. M. Mescheder, S. Nowozin, and A. Geiger. The numeries of gans. In Guyon, U. von Luxburg, S. Bengio, H. M. Wallach, R. Fergus, S. V. N. Vishwanathan, and R. Garnett, ed- itors, Advances in Neural Information Processing Systems 30: Annual Conference on Neu- ral Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA.
htw.rooeeednngnnuurrrrrsssp.ssss neurips wwwparips.ccppppeee.eeaaaaaa htcepppper2211/haas// pages 1825- 2017. URL Abstract.bttt........... html html
B. A. Pearlmutter. Fast Exact Multiplication by the Hessian. Neural Computation, 6(1):147-160. 01 1994. ISSN 0899-7667.
doi: 10. 10.1162/neco. 19w.eeec....99.... 1994.6.1.147 URL httth:://p/::i wtw.:/...rr/11..1....111.
neco 191.........1.177
A. Peysakhovich and A. Lerer. Prosocial learning agents solve generalized stag hunts better than selfish ones. In E. André, S. Koenig, M. Dastani, and G. Sukthankar, editors, Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS 2018, Stockholm, Sweden, July 10-15, 2018, pages 2043-2044. International Foundation for Autonomous Agents and Multiagent Systems Richland, SC, USA / ACM, 2018. //d1 acm. org/ citation. wff:iddmmmnmmnnc
URL http'
J. Philip S. Gallo and C. G. McClintock. Cooperative and competitive behavior in mixed-motive games. Journal of Conflict Resolution, 9(1):68-78, 1965. doi: 10.1 10.117/0022022775000066 URL https://dol org/10. /00. 1177 11777000220226609000000
A. Raffin, A. Hill, A. Gleave, A. Kanervisto, M. Ernestus, and N. Dormann. Stable-baselines: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22 (268):1-8, 2021. http:
wttp/pppprs/v////// 20- 1314/ 1364.html
A. Rapoport.
Prisoner's 's dilemma- recollections and observations. In Game Theory as a Theory of a Conflict Resolution, pages 17-34. Springer, 1974.
S. Roesch, S. Leonardos, and Y. Du. Selfishnesss Level Induces Cooperation in Sequential Social Dilenmas. International Foundation for Autonomous Agents and Multiagent Systems (IFAAMAS), 2024.
M. Samvelyan. T. Rashid, C. de Witt, G. Farquhar, N. Nardelli, T. G. J. Rudner, C. Hung, H. S. Torr, J. N. Foerster, and S. Whiteson. The starcraft multi-agent challenge. In E. Elkind, M. Veloso, N. Agmon, and M. E. Taylor, editors, Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS '19, Montreal, QC, Canada, May 13-17, 2019, pages 2186 2188 International Foundation for Autonomous Agents and Multiagent Systems, 2019. URL http::/:/// acm. org/
E. Vinitsky, N. Jaques, J. Leibo, A. Castenada, and E. Hughes. An open source implementation of sequential social dilemma games. bttpps//ggt thub. httm///ggeeeiintts//sseeeseeeeee social, dwwemmammmmmmmmmmmmmmmmaaaaaaaea wwteeleaaaaaaaaaeeeeeeeeaaaee httss/ssssss/eeeee 182 2019. GitHHub repository.
E. Vinitsky, R. Köster, P. Agapiou, E. A. Guzmnzz-Gzzmán A. S. Vezhnevets, and J. z. Leibo. Fugene2023eollectivvin multi-agent settings. Collective Intelligence, 2(2): 16339137231160000 2023. doi: 10. 10.1177/96339133221 1177/268391372311168
httpst h/tp://ooooo//..////
13

---

J. Wang, Y. Zhang, T. Kim, and Y. Gu. Shapley vavalue: A local reward approach to solve global reward games. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty- Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances Artificial Intelligence, EAAI 2020, New York, NY,
USA, February 7-12, 2020, pages 7285-7292. AA Press, 2020. doi: 10.1609//AAAAL.3441.05.00 322000000....
URL https: //wii. org/10. 10..9669aaaaia v34i05. 6220
T. Willi, A. H. Letcher, Treutlein, and J. Foerster. COLA: Consistent learning with opponent- learning awareness. In K. Chaudhuri, S. Jegelka, L. Song. C. Szepesvari, G. Niu, and S. Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 23804-2388 23831. PMLR, 17-23 Jul 2022. URL https::/pprcceee
mlr press/ http/vv12////// willi22a html
J. Yang. A. Li. M. Farajtabar, P. Sunehag. E. Hughes, and H. Zha. Learning to incentivize other learn- ing agents. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems 33; Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 12, 2020, virtual, 2020. URL BtRpstts::/ httpttt::/procceddingge neurips cc/ cc/paper///pee/pppppppppppp.. 2tt///2220//haa/ 1000 Abstractt html
C. Yu, A. Velu, E. Vinitsky. J. Gao, Y. Wang, A. M. Bayen, and Y. Wu. The sur- prising effectiveness of PPO in cooperative multi-agent games. In S. Koyejo, S. Mo- hamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neu- ral Information Processing Systems 35: Annual Conference on Neural Information Pro- cessing Systems 2022, NeurIPS 2022, New Orleans. LA, USA, November 28 Decem- ber 9, 2022, 2022. URL http: httpp/paprssse//////ee nips cc/ paperpppe.... httppppppp__feeee fiwes/paper////eeeeeeeeaaaaaeeee 2atp//222//aaaaaa / 9c155500100007793334114 -aspaaaaaaootttttttttattettteeee Dbstract-Dattsets_aaddd Bew.hadannnnaaksssssaaaaaaasasee
html
Y. Zhong, G. Kuba, X. Feng. S. Hu, Ji, and Y. Yang. Heceoogenncoou-gggeeetnneennonno reinaoooem learning.
J. Mach. Learn. Res., 25:32:1-32:67, 2024. URL httpppp://iili.roogrpppppssss//p
html
14

---

A
Broader Impacts
This paper presents a novel approach in the field multi-agent learning, aimed at enhancing collab- oration among agents with mixed motivations. The proposed framework addresses fundamental challenge in machine learning: enabling agents with diverse objectives to cooperate effectively to achieve collective goals. The work described in this paper holds significant potential for various societal applications, By facilitating effective cooperation among agents with different motivations, our approach can be applied to domains such video games, autonomous systems, smart cities, and decentralized resource allocation. These advancements can lead to numerous societal benefits, including improved efficiency in transportation systems, optimized resource allocation energy grids, and enhanced makinng in complex multi-agent environments. Furthermore, our findings con- tribute the broader field of machine learning by offering insights into managing mixed-motie motivvitiionnn scenarios. This work paves the way for future research and innovation, driving progress in multi- agent learning and its applications across various domains. However, while our method was initially developed to address fundamental challenges in decision decision-making making within machine learning contexts, it conceivable that our proposed objective alignment techniques could be repurposed for harmful purposes, such attacking existing cooperative systems.
B Proof of Corollary 4.3
Corollary 4.3. In the neighborhood of fixed points of the collective objective, AgA will pull the gradient toward stable fixed points, which means O VHe) s A(Ec, VHe), and push away from unstable ones, indicated by O(E, VHe) 8(Ec, VHe), if X satisfies X (Ec, VH.)( VHe)
1²)
Proof. Prior to embarking on the elucidation of the corollary's proof, it essential first introduce relevant notations and the foundational concepts. Take note of the notation ((a, b), which symbolizes the angular measure between two vectors. Furthermore, measure between AgA gradient & and a reference update direction w. For simplicity, we Xo denote the AgA gradient = E. X (5 H Eo) ,as defined in Eq. Specifically, denotes collective gradient & and v denotes (E E.)
write Ox w) to represent the angular
We then extend the definition of infinitesimal alignment (Balduzzi et al. 2018 to our proposed AgA gradient. Given a third reference vector w. the infinitesimal alignment for AgA gradient with w is defined as
align(É, w) = dA
(cos200x1xx-
(2)
Intuitively, may presume that vectors u and w are directionally aligned, given that u w > 0. In this scenario, align > 0 implies that vector v is drawing u closer towards the reference vector Conversely, vector is propelling away from pushes u away from the reference vector w when u and w share the same directional orientation. Conversely, pulls towards the reference vector w when the vectors and not oriented in the same direction.
Similar arguments hold for align 0:
The ensuing lemma provides simplified method for determining the sign of align, circumventing the need for generating an exact solution.
Lemma B.1 (Sign of align.). Given AgA gradient and reference vector VHo the sign of infinitesi- mal alignment align could be calculated by
sign (align(E: VHo)) sign ((S, (HE. VHo) +[VHH(1)))
15

---

Proof. By the definition of 02, we could directly get:
co
(3a)
(E. VH.)
(3b)
|111 VHell
(E: + A(E + VHo), VHo)
(3c)
|K I/Hell
(Ec. VHL)² 2 + 2X/E. VHL) + VHe. VH.) + 0APP)
(3d)
(Ec, VH.)² + 2X VEL) VHe) 2X/ & VH.) + 0(22)
(3e)
(IEIP-
EE., VHL)² + 2X(Ec. VH.ll + (E. (He))+ 0(19)
(3f)
[|É|12- |VH. 2
In accordance with the established definition of infinitesimal alignment, it becomes feasible to further compute the sign of align by taking the derivative of with respect to cos?0x:
sign (align(É. VHo))
(4a)
d
sign
cass02)
(4b)
dd
ssggn ((Eer VHe) ((E, VH.) + ([VH2(l)))
(4c)
Lemma B.I empowers to calculate the sign of infinitesimal alignment relying on components that are readily computable.
Let us observe (Ee, VHe) = E.. - H.C. for symmetric Hessian matrix H Under the assumption of &. * O, we could derive:
if H. ž O, then (Ec. VHe) 0;
if H. < O, then (Ec. VH2) < 0.
As demonstrated by Eq. 5 when situated in the neighborhood of a stable fixed point (that is to say, H. = 0), we observe (Ec, Hee 2 O. In converse circumstances, (Ec, VHo) < O occurs. Following this, we shall proceed delve into two specific scenarios: stability and instability of the fixed point.
1) Case 1: In a neighborhood of a stable fixed point. If we are in a neighborhood of a stable fixed point then (Ec, VHe) 0. This indicates that both vectors & and VHe point in the same direction, i..., O(Ee, VH.)) < t/2. Referring to Lemma [.. the sign of align(E, VHe) is the same as the sign of (E, VH.) + I 1 if (E., VHo) 2 O. Consequently, we have sign(A) (ign((E, VHe) /HH 112) sign(align) matching the claim of our corollary.
Let us now conjecture the scenarios of sign(align). namely, sign(align) 2 0 or sagn(align) < 0. When sign(align) 0, it follows that sign(A) 0. Consequently, vector v will draw towards w. Subsequently, when sign(align. < 0, we observe that sign(A) < 0. A negative À reverses the mechanism such that v pushes away from w, aligning with the discussion following Eq. 2 that assumes a positive X. Thus, regardless of the sign of align, if we set sign(A) = sign (E. VHe) |[VH 12), vector ensures is drawn towards w.
From now, we prove that in the neighborhood of stable fixed point, if we let sign satisfy A (E.. VH.)((E. VH.) + |VH,.) 2 0, the AgA gradient & will be pulled more closer to VH. than E..
2) Case 2: In a neighborhood of a unstable fixed point. The proofing approach for the unstable case exhibits similarity to Case 1.
16

---

Therefore, we prove that if we let the sign of À follows the condition X (Ec. VS VH.) [|VH.,13 - the optimization process using the AgA method exhibit following attributes. In areas close to fixed points, 1) if the point is stable, the AgA gradient will be pulled toward this point, which means that B(5, VHe)= I ((Cc. VHc);; 2) if the point represents unstable equilibria, the AgA gradient will be pushed out of the point, indicating that 0(E, VH.) 2 ((E., VHe). An illustrative example of the corollary is provided in Fig. 2a
C Implementation of MARL Algorithms with AgA
The concrete implementation of altruistic gradient adjustment is elaborated Algorithm T AgA can be easily incorporated into any centralized training decentralized execution (CTDE) framework of MARL. In a typical CTDE framework, each agent, denoted as i, strives to learn policy that employs local observations to prompt a distribution over personal actions. The unique characteristic during the centralized learning phase that agents gain supplementary information from their peers that is inaccessible during the execution stage. Furthermore, centralized critic often operates in tandem in CTDE settings, leveraging shared information and evaluating individual agent policies' potency. To incorporate
proposed AgA algorithm into CTDE frameworks, supplementary reward exchanges between agents are required, offering the essential inputs for Algorithm [ The individual losses calculated from individual rewards conforming to established MARL algorithm framework. such as IPPO dd Witt et al., 2020) or MAPPO Yu et al. 2022). Here, the collective loss la is calculated according individual rewards; for example, it could be a sum of individual rewards an elementary transformation thereof. Furthermore, the AgA algorithm requires a magnitude value of the alignment parameter λ, which should be positive entity. The sign of À is then calculated in line 5 of Algorithm|] and is based on each gradient component calculated in lines 2-4. Subsequently, the algorithm produces the adjusted gradient É. È is then distributed across each corresponding parameter, and optimization is carried out using a suitable optimizer such Adam optimizer [Kingma and Ba 2015).
D Additional Experiment Results
D.1 Result Visualization of Two-player Public Goods Game
Fig. 4 depiccs the actions produced by various methods in a two-player public game. The action gen- erated by each method represented by circle, which is color-coded according to the corresponding method. To ensure fairness, each method's updates are constrained to a maximum of 100 steps. These methods include the individual loss-driven simultaneous optimization (Simul-Ind) Ind), two variants of gradient adjustment methods termed as CGA and SGA, and the simultaneous optimization leveraging collective loss (Simul-Co) alongside our proposed AgA, where X fundamentally underpinned by the gradient ascent algorithm. The illustration consolidates data from 50 randomized runs initialized from diverse starting points. The 'X' demarcated circles indicate the average actions mapped to each method. From the figure, it's apparent that Simul-Ind, CGA, SGA, SVO tend to converge towards scenarios where at least one player abstains from contributing, an occurrence often tagged as 'free-riding.. In contrast, Simul-Co (represented by yellow circle) and our AgA method (illustrated as red circle) tend to converge towards scenarios characterized by
Each of these methods
Algorithm Altruistic Gradient Adjustment (AgA)
1: Input: individual losses f. = [f,,... En), collective loss lc, parameters w = [w), ... ,,,, magnitude of alignment parameter X
2: g + [gr,d(L,, w.) for (L,, w.) E (f, w)]
3: &. Igrad(Le, w.) for w; E w]
4: VH. t Igrad( 111 w.) for w; E w
5: A x sign (2 VHe) ((S, VH.) + I(HH))
6: Output: = E X(E VHe)
(Plug into any gradient descent optimizer)
17

---

1.0
(1,1) Social Optimum
Action of Player
Simul-Ind
Simul-Co
CGA
SGA
0.4
AgA
Mean Action
0.6 Action of Player 1
Figure 4: The scatter of actions in two-player public goods game achieved through different optimization methods, Each circle represents the position attained within a maximum of 100 steps, with the color indicating the corresponding method. The 'X' mark represents the mean actions of 50 random runs. With the exception of Simul-Co, the baseline methods converge towards the Nash equilibrium (0,0). Notably, while both AgA and Simul-Co display altruistic behavior, the actions of AgA are more tightly clustered around the (1,1) point compared to Simul-Co.
enhanced contributions or 'altruism'. The observed results underline that players employing the AgA method exhibit heightened altruistic tendencies relative to those operating under Simul-Co. Further, AgA exhibits superior alignment towards socially optimal outcome point 1).
E Experiment Details
E.1 Sequential Social Dilemma Games: Cleanup and Harvest
SSD games [Vimitsky et al. 2019] implements the Harvest and Cleanup as grid world games. The agents use partially observed graphics observation, which contains grid of 15 x 15 centered on themselves. Compared to the original Cleanup and Harvest games, SSD games add a fire beam mechanic, where agents fire the grid to hit other partners. An agent will gain reward if harvesting an apple and -1 reward firing a beam. Besides, being hit by a beam will result in 50 individual reward loss. Under this mechanic, selfish agents easily use fire prevent others from harvesting apples to gain more individual rewards but harm social welfare.
We utilized PPO algorithm in hable-baselines!? [Raffin et al. 202T] to implement the baselines and our methods, with all the agents using separated policy parameters for Simul-Ind and sharing same policy parameters for other experiments. For SVO, we modify the individual reward to be
- a(1 - actan sum,, To employ CGA and AgAs to PPO training process, we compute both individual and collective PPO-Clip policy losses and subsequently utilize them to calculate the adjusted gradient through automatic differentiation. We don't make changes to the critic loss nor the critic net optimization process.
Most experiments were conducted on a node with a Tesla V100 GPU (32GB memory) and 40 CPU cores. The byper-parameters for PPO training are as follows.
The learning rate is le-4
The PPO clipping factor is 0.2.
The value loss coefficient is 1.
The entropy coefficient is 0.001.
The is 0.99.
The total environment step is 1e7 for Harvest and 2e7 for Cleanup. The environment episode length is 1000.
The grad clip is 40.
18

---

Teammates
Enemies
Medivac
2 Marauders
Marines
Figure Semantic Diagram of MMM2 map in SMAC
E.2 Selfsh-MMMM2
SMAC, often employed as a testbed within the sphere of cooperative MARL, is characterized by the shared reward system among players. Within this paper, we present the MMM2 map transformed as a versatile testbed designed for mixed-motive cooperation. Specifically, shown in Fig. MMM2 map encompasses 10 agents and adversaries enumerated below:
Controlled Agents: Comprised of Medivac, 2 Marauders, and 7 Marines.
Adversaries: Incorporates Medivac, 3 Marauders, and 8 Marines.
Moreover, this setting embodies both heterogeneity and asymmetry, where agents within the same team exhibit diverse gaming skills. Additionally, the team compositions, for instance, variation in agent number and type, differ in both opposing factions. Consequently, the intricate gaming mechanism, the presence of heterogeneous players, and the discernibee asymmetry conspire to establish an exemplary milieu conducive the exploration of mixed-motive issues.
In order to create a mixed-motive testbed, we adjusted the reward system hailing from the original SMAC environment. With this, we proposed a revamped reward function that features distinet components: reward for imposing damage and penalty associated specifically with agent fatalities. In move to potentially intensify internal conflicts amongst agents, the reward for imposing damage is solely allocated to the agent responsible for executing the attack on the enemy. Additionally, the death an agent results the imposition of extra penalty. Consequently, this mechanism fosters an environment where agents are predisposed towards individual protection and separate reward acquisition, opposed collective cooperation.
Formally, the agent i's reward at step is defined as follows:
delta-enemy; [previous- health current-health) (previous- shield current-shield))
BBooyy
The individual reward for Player i is determined as follows: the player dies, then r; = delta-enemy, penalty; otherwise. the player survives, then ry delta-enemy,-
Experiment settings. Simul-Ind leverages IPPO with recurrent policies and distinct policy networks. Conversely, Simul-Co employs MAPPO with recurrent policies and consolidated policy networks.
The implementation of the IPPO and MAPPO algorithms presented in this paper is founded on the methodology detailed in (Yu al. 2022]. We utilize gradient adjustment optimization methods, such as CGA and AgAs, derived from MAPPO, implementing gradient adjustments as outlined in Algorithm
Most experiments were conducted on a node with two NVIDIA GeForce RTX 3090 GPUs and 32 CPU cores. The hyper- hyper-parameters for paO-based training follows.
19

---

The learning rate is 5e-4
The PPO clipping factor is 0.2.
The value loss coefficient
The entropy coefficient is 0.01.
The is 0.99.
The total environment step is le7.
The factor /3 in reward function is
The environment episode length is 400.
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main contributions and scopes of our paper are include in the Abstract
and Introduction. the last paragraph of Introduction, we list our main contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
In the Conclusion section, we discuss the limitations of the work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
21

---

Justification:
We provide full proof of our theoretical results in Appendix B
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide the details to reproduce the main experimental results in Ap- pendix and the main code supplemental material (we will release them when the paper is accepted).
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
22

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We provide the main code in the supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips //ipss.cc/pill wwtpcccpppppppppeeeeeeeeeeeeeeee httaa//gguudeee/eeeeabiiiisssss /Cc////oddiiisssseeeeeee ssiiiiiiscyyyyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
We provide all the training and test details in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: interval.
We report error bars for all experimental results using 95% confidence
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
23

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes)
Justification:
The computer resources are included in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
The research conforms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the potential societal impacts in Appendix A
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The paper explicitly mentioned and cited the original codes and environments used.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
25

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The new assets well introduced.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "gqdIT9GniP", "reviewer": "Reviewer_a1aR", "summary": "The paper investigates the topic of cooperation in a mixed-motive multi-agent setting. They first propose the formulation of a mixed motive game as a differentiable game. Leveraging the structure of the latter, they propose a gradient-based optimization algorithm, AgA. The paper both discusses the theoretical guarantees of AgA, and its effectiveness in an empirical setting. Finally, the authors introduce a modification of the MMM2 map in the StarCraft II game, to further test the empirical performances of AgA.", "questions": "- How much slower is AgA compared to the other algorithms? Do you have any empirical experiments which can shows this in practice?\n\n- Do you have any results which show the individual metrics achieved by the agents when using AgA vs other algorithms? It could be valuable to see some plot which shows both the cooperative and individual rewards, and how these vary depending on the value the hyperparameter $\\lambda$ is set to\n\n- Did you run the experiments for Figure 3 for more seeds? Do the results still hold?", "limitations": "No limitations", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "- The paper is theoretically sound. The AgA algorithm is simple, but at the same time well justified.\n- The experiment are broad and well designed. The Selfish-MMM2 environment addresses some of the limitations of the other ones, and I think is a principled way of showing the performance of the algorithm in larger and more complex environments", "weaknesses": "- There are some typos in the paper, and I feel the presentation can be improved. For example, in the Related Work section in page 2, there are some sentences which are incomplete or non-sensical. I am specifically referring to ' While PED-DQN enables agents 79 to incrementally adjust their reward functions for enhanced collaborative action through inter-agent 80 evaluative signal exchanges [Hostallero et al., 2020], Gifting directly reward other agents as part of 81 action space [Lupu and Precup, 2020].' \n- I think the author should discuss more in depth the complexity of this method. How does it scale with respect to the other algorithms proposed? There is no experiment that highlights this in the paper\n- There is no discussion of how AgA satisfied individual incentives", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Weakness 1: ... I am specifically referring to ' While PED-DQN enables age ...**\n\n**Response:** Thank you for your thorough and detailed reviews of my paper. We apologize for the unclear sentence and have rewritten it to improve clarity:\n*To further promote cooperation, the gifting mechanism—a crucial strategy in mixed-motive cooperation[2]—allows agents to influence each other’s reward functions through peer rewarding. \nBesides, PED-DQN[3] introduces an automatic reward-shaping MARL method that gradually adjusts rewards to shift agents’ actions from their perceived equilibrium towards more cooperative outcomes.* \n\n**Weakness 2 & Question 1: ...the complexity of this method... & How much slower is AgA compared to the other algorithm...**\n\n**Response:** Please see Common Response.\n\n**Weakness 3 & Question 2:There is no discussion of how AgA satisfied individual incentives & Do you have any results which show the individual metrics**\n\n**Response:**  Thank you for your valuable advice regarding individual metrics. Your suggestions are helpful and will contribute to improving the quality of our experiments and the overall paper. To evaluate individual incentives, we introduce the **Gini coefficient** [1], a commonly used measure of income equality, to assess reward equality in cooperative AI settings [4]. We utilized an expedited method for calculating the Gini coefficient and derived an associated equality metric $E$, defined as $E := 1 - G$. The Gini coefficient $G$ is computed from the ranked payoff vector $p$, which arranges each individual's rewards in ascending order, as $G = \\frac{2}{n^2 \\bar{p}} \\sum_{i=1}^n i(p_i - \\bar{p})$, where $\\bar{p}$ is the mean of the ranked payoff vector $p$ and $n$ is the total number of players. *A higher $E$ value indicates greater equality.*\n**Table 1** presents the comparison of the mean and standard deviation of the equality metric achieved by different methods in the Harvest and Cleanup environments (note that the selfish-MMM2 environment involves heterogeneous agents, making it challenging to directly compare rewards achieved by different types of agents). A value closer to 1 indicates more equal rewards among agents. **As shown in Table 1, our proposed AgA method outperforms the baselines, demonstrating that AgA can effectively consider all interests within the team. Furthermore, as illustrated in Figure 3 of our manuscript, AgA achieves the highest collective rewards. Therefore, our AgA methods adequately address both individual and collective interests.**\n\n*Table 1: The comparison of the equality metric.*\n| **Envs**    | **Simul-Ind**             | **Simul-Co**              | **SVO**                   | **CGA**                   | **SL**                   | **AgA ($\\lambda = 0.1$)**   | **AgA ($\\lambda = 1$)** | **AgA ($\\lambda = 100$)** | **AgA ($\\lambda = 1000$)** |\n|-------------|---------------------------|---------------------------|---------------------------|---------------------------|---------------------------|------------------------------|-------------------------|---------------------------|-----------------------------|\n| **Harvest** | 0.973 ± 0.005             | 0.975 ± 0.006             | 0.974 ± 0.007             | 0.950 ± 0.051             | 0.972 ± 0.005             | 0.981 ± 0.006                | **0.988 ± 0.003**       | 0.982 ± 0.012             | 0.980 ± 0.006               |\n| **Cleanup** | 0.841 ± 0.071             | 0.948 ± 0.013             | 0.902 ± 0.019             | 0.903 ± 0.034             | 0.946 ± 0.016             | 0.940 ± 0.017                | 0.956 ± 0.007           | **0.959 ± 0.011**         | 0.905 ± 0.022               |\n\n**Question 3:** Did you run the experiments for Figure 3 for more seeds? Do the results still hold?\n\n**Response:** We run all algorithms with three seeds and report the mean and variance with a 95\\% confidence interval. The results across different environments, including a toy game, a two-player public goods game, Harvest, Cleanup, and our developed selfish-MMM2 show that our method, AgA, consistently outperforms the baselines. Therefore, we believe that the results would remain consistent with more seeds.\n\n**Reference:**\n\n[1] David, H. A. Gini’s mean difference rediscovered. Biometrika, 55(3):573–575, 1968. ISSN 00063444. URL http://www.jstor.org/stable/2334264.\n\n[2] Lupu, A. and Precup, D. Gifting in multi-agent reinforcement learning. In Proceedings of the 19th International Conference on autonomous agents and multiagent systems, pp. 789–797, 2020.\n\n[3] D. E. Hostallero, D. Kim, S. Moon, K. Son, W. J. Kang, and Y. Yi. Inducing cooperation through reward reshaping based on peer evaluations in deep multi-agent reinforcement learning. In A. E. F. Seghrouchni, G. Sukthankar, B. An, and N. Yorke-Smith, editors, Proceedings of the 19th International Conference on Autonomous Agents and Multiagent Systems, AAMAS ’20, Auckland, New Zealand, May 9-13, 2020, pages 520–528. International Foundation for Autonomous Agents and Multiagent Systems, 2020. doi: 10.5555/3398761.3398825. URL https://dl.acm.org/doi/10.5555/3398761.3398825.\n\n[4] Du, Yali, Joel Z. Leibo, Usman Islam, Richard Willis, and Peter Sunehag. \"A review of cooperation in multi-agent learning.\" arXiv preprint arXiv:2312.05162 (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5bTtTJEN5z", "reviewer": "Reviewer_sisd", "summary": "This paper introduces a novel optimization method called AGA that employs gradient adjustments to progressively align individual and collective objectives. They prove that this method attracts gradients to stable fixed points of the collective objective while considering individual interests. Their method is empirically validated on sequential social dilemmas games, Cleanup and Harvest, and a high dimensional environment, StarCraft II.", "questions": "- Figure 1. It’s not clear how to interpret the reward contours graphs. Can you clarify what the x-axis and y-axis? How are the contours formed? They appear to be integers corresponding to the Player’s actions but where is the action space defined? \n- What is the significance of AGA moving along the summit? \n- In Proposition 4.1, what is the gradient without the subscript referring to? \n- L292, how is the collective loss equation determined?", "limitations": "Yes.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- Preliminaries are clear \n- Proofs are given graphs to explain the intuition \n- Good comparison to baselines\n- Varied experimental environments (grid game + modified SMAC) \n- Strong experimental results", "weaknesses": "- No clear weaknesses", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Question 1:** Figure 1. It’s not clear how to interpret the reward contours graphs...\n\n**Response:** Thank you for your advice to improve the clarity of the graph description. The x-axis and y-axis represent the actions of the two players, i.e., for player $i$ =\\{1, 2\\}, $a_i \\in \\mathbb{R}$, where $a_i$ is the action of player $i$. In our future version, we will include the definition of the action space in Example 4.1.  The contours in the graph represent the same rewards that can be achieved by the players based on different actions. The contours are formed by sampling the actions of the two players at regular intervals and calculating the corresponding rewards for each action pair. Points that yield the same reward are then connected to form the contour lines, representing levels of equal reward that can be achieved by the players based on their actions. Thank you again for your valuable advice. We will incorporate this explanation into the manuscript to make the graph clearer.\n\n**Question 2:** What is the significance of AGA moving along the summit?\n\n**Response:** According to the definition of Example 4.1, the reward function for Player 1 has a maximum value of 1. Thus, moving along the summit represents the algorithm's ability to maximize the player's reward during updates, ensuring that the player's interests are not overlooked. On the other hand, the Simul-Co method focuses solely on improving collective rewards and neglect individual player interests.\n\n**Question 3:** In Proposition 4.1, what is the gradient without the subscript referring to?\n\n**Response:** The gradient without the subscript is initially defined in Section 3.1 as follows: \"We write the simultaneous gradient $\\xi(w)$ of a differential game as $\\xi(w) = (\\nabla_{w_1}\\ell_1, \\dots, \\nabla_{w_n}\\ell_n) \\in \\mathbb{R}^d,$ which represents the gradient of the losses with respect to the parameters of the respective players.\" Since this definition is located far from the Proposition, we will reiterate it in our future version to provide clarity.\n\n**Question 4:** L292, how is the collective loss equation determined?\n\n**Response:** The collective loss function is determined experimentally within the context of mixed-motive problems and is inspired by the SVO method. Our motivation is to achieve equality while maximizing social welfare, ensuring that the loss function guides a fair contributions  among participants.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oPkdIwXyJ5", "reviewer": "Reviewer_x8HM", "summary": "The paper proposes Altruistic Gradient Adjustment (AgA) which adjusts the gradients of individual and collective losses to align individual and collective objectives. Besides, the authors prove that AgA effectively attracts gradients to stable fixed points of the collective objective while less sacrificing individual interests. Experiments and example illustrations demonstrate the effectiveness of AgA.", "questions": "1. Could the authors compare the running time of AgA with other baselines?\n2. Could the authors provide more discussion about AgA and Simul-Co? For example, if Simul-Co could achieve a higher collective reward than AgA like in Figure 1, one can redistribute the collective reward for each agent so that the redistributed individual rewards for each agent can be better than AgA.\n3. In Algorithm 1, are the parameters w=[w_1,…,w_n] configured manually or determined by the task itself?", "limitations": "NA.", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1.\tThe paper is well-written and well-organized. \n2.\tThis topic is important with less attention in the MARL community.\n3.\tThe experiments are performed on several environments and some example illustrations are given to help understand the proposed technique.\n4.\tThe most important contribution of this paper is Corollary 4.3. This algorithm tries to seek stable fixed points for the collective objective while previous works aim to seek stable fixed points for individual objectives. This will possibly benefit the collective welfare as well as the individual interest at the same time, which would be useful as demonstrated in the experiments.", "weaknesses": "1.\tAgA introduces an additional adjustment term, which needs to be tuned case by case.\n2.\tAgA introduces additional computation complexity to compute Hessian-vector products.\n3.\tIt is not clear whether AgA also has the ability to always keep all individual objectives best. For example, in Figure 1(c), Simul-Co could achieve a better reward for Player 2 than AgA while Simul-Co also achieves a higher collective reward than AgA.\n4.\tThere is only one map of SMAC to be tested.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Weakness 1:** AgA introduces an additional adjustment term, which needs to be tuned case by case.\n\n**Response:**  Thank you for your feedback. Our AgA method indeed includes an gradient adjustment component, however, **the additional adjustment term is derived automatically based on our theoretical framework**, as outlined in Proposition 4.2 and Corollary 4.3, and is implemented in our derivation code. While the adjustment term itself does not require manual intervention, the parameter  $\\lambda$  does need to be tuned for different tasks to ensure optimal performance. \n\n**Weakness 3:** It is not clear whether AgA also has the ability to always keep all individual objectives best. For example, in Figure 1(c), Simul-Co could achieve a better reward for Player 2 than AgA while Simul-Co also achieves a higher collective reward than AgA.\n\n**Response:** In mixed-motive settings, the group’s incentives can sometimes align and sometimes conflict [1][2]. Therefore, **it is almost impossible to consistently optimize all individual objectives simultaneously. If all individual objectives could always be optimal, the problem would reduce to a fully cooperative scenario where all incentives are perfectly aligned.** In Section 4.1, we emphasize the mixed-motive nature of our proposed differentiable mixed-motive game: \"minimization of individual losses can result in a conflict between individuals or between individual and collective objectives (e.g., maximizing individual stats and winning the game are often conflict in basketball matches).\" \n\n[1] McKee, Kevin R., Ian Gemp, Brian McWilliams, Edgar A. Duéñez-Guzmán, Edward Hughes, and Joel Z. Leibo. \"Social diversity and social preferences in mixed-motive reinforcement learning.\" arXiv preprint arXiv:2002.02325 (2020).\n[2]Du, Yali, Joel Z. Leibo, Usman Islam, Richard Willis, and Peter Sunehag. \"A review of cooperation in multi-agent learning.\" arXiv preprint arXiv:2312.05162 (2023).\n\n**Question 1:** Could the authors compare the running time of AgA with other baselines?\n\n**Response:** Please see common response.\n\n**Question 2:** Could the authors provide more discussion about AgA and Simul-Co? For example, if Simul-Co could achieve a higher collective reward than AgA like in Figure 1, one can redistribute the collective reward for each agent so that the redistributed individual rewards for each agent can be better than AgA.\n\n**Response:** Thank you for the insightful question regarding the comparison between AgA and Simul-Co. While Simul-Co focuses on maximizing collective rewards, it often overlooks individual agent incentives, potentially leading to dissatisfaction among agents with lower individual rewards. Redistributing collective rewards is also a potential method to address the problem. However, redistribution often encounters the credit assignment problem, one of the most challenging issues in the MARL field. It complicates the redistribution process, typically requiring expert manual design and substantial engineering effort.\nIn this work, our goal is to propose an automatic method that modifies the gradient to seamlessly align individual and collective interests. Unlike traditional reward-shaping techniques, our approach doesn’t rely on manual intervention but instead automatically adjusts the gradient to ensure a balance between individual incentives and overall social welfare. \n\n**Question 3:** In Algorithm 1, are the parameters w=[w_1,…,w_n] configured manually or determined by the task itself?\n\n**Response:**  The parameters $w = [w_1, \\dots, w_n]$ are initially defined in Definition 3 as follows: \"The parameter set $w=[w_i]^n\\in \\mathbb{R}^d$ is defined, each with $w_i\\in \\mathbb{R}^{d_i}$ and $d = \\sum_{i=1}^n d_i$. ... Each player $i\\in N$ is equipped with a policy, parameterized by $w_i$, aiming to minimize its loss $\\ell_i$.\" In practice, the parameters $w = [w_1, \\dots, w_n]$ in Algorithm 1 refer to the neural network parameters, and our method is based on the popular PPO architecture. These parameters are learned and optimized during the training process rather than being manually configured.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0dE2ouy3o1", "reviewer": "Reviewer_1xxK", "summary": "The paper lies in the intersection of multi-agent and game theory. In particular, dealing with \"mixed-motive cooperative games\". This is, games in which the maximization of individual rewards hampers the maximization of collective rewards. This is, when agents seek to maximize their individual reward/utility, and disregard the collective utility they are worst off as a collective. These type of problems are usually called \"social dilemmas\" in game theory. The objective is to achieve the 'social optimum', this is, the maximum collective reward a group of agents can obtain, beyond the individual reward.\n\nThe methodology proposed in the paper is to align the individual rewards towards the collective maximization rewards via \"gradient shaping\". This is, move the gradients of the individual reward maximization policy towards the collective reward. In order to shift the individual rewards, the algorithm includes an \"alignment\" parameter that aligns two optimization problems (individual maximization vs collective optimization) that otherwise, will pull against each other.\n\nSince the collective optimization point is an equilibrium point, the algorithm shows convergence.", "questions": "I don't think this is fair or true. The mentioned papers do provide substantial theoretical analysis. This is not the way to place your paper.\n\n\"On the other hand, several studies focus on auto-matically modifying rewards by learning additional weights to adjust the original objectives [Gemp et al., 2020, Kwon et al., 2023]. However, these studies often suffer from a lack of interpretability and insufficient theoretical analysis regarding the alignment of individual and collective objectives.\"", "limitations": "Authors have addressed the limitations of the paper on the Conclusions section.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.Simplicity: The algorithm is simple and intuitive. It can be easily implemented in the current major multi-agent algorithms.\n\n2.  Testing in well selected benchmark environments: The environments selected to test the algorithm are very standard in the field. Results show superiority against the selected benchmarks.\n\n3. Ablation studies. The paper presents ablation studies for the relevant parameters. In particular, the \"lambda\" term controlling the alignment. \n\n4. Presentation: the narrative is well written and makes the reading very straightforward. The paper provides intuition as well as rigor on the methodology proposed.", "weaknesses": "See below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Question 1:** I don't think this is fair or true. The mentioned papers do provide substantial theoretical analysis. This is not the way to place your paper...\n\n**Response:** We apologize for the misunderstanding and appreciate your careful review. We have revised the statement as follows to make it more clear: \n“On the other hand, several studies focus on automatically modifying rewards by learning additional weights to adjust the original objectives [1,2]. Most of these methods leverage Nash equilibria or related concepts from game theory, such as the price of anarchy. The challenge of finding Nash equilibria in nonconvex games is more difficult than finding minima in neural networks [3], and it is not intuitive to explain how individual incentives align with the collective incentives during the optimization process.\"\n\n[1] I. Gemp, K. R.McKee, R. Everett, E. A. Duéñez-Guzmán, Y. Bachrach, D. Balduzzi, and A. Tacchetti. D3c: Reducing the price of anarchy in multi-agent learning. arXiv preprint arXiv:2010.00575, 2020.\n[2] M. Kwon, J. P. Agapiou, E. A. Duéñez-Guzmán, R. Elie, G. Piliouras, K. Bullard, and I. Gemp. Auto-aligning multiagent incentives with global objectives. In ICML Workshop on Localized Learning (LLW), 2023.\n[3] Letcher, A., Balduzzi, D., Racaniere, S., Martens, J., Foerster, J., Tuyls, K. and Graepel, T., 2019. Differentiable game mechanics. Journal of Machine Learning Research, 20(84), pp.1-40.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Thank you to all the reviewers for your diligent efforts and invaluable feedback. Your comments will greatly contribute to enhancing the quality of our paper. We hope that we have addressed your concerns in our response. If you have any further questions, please don't hesitate to engage in discussion with us, and we will respond promptly.

**Common Response Regarding the Running Time of AgA:**

Thank you for the valuable comments regarding the running time and complexity of our proposed AgA algorithm. In this response, we first address the additional complexity introduced by AgA and then provide an analysis of its practical running time. First, the AgA method is a type of gradient adjustment method, where the modified gradient is given by $\xi_{c} + \lambda \left( \xi + H_c^T\xi_c \right)$ (see Proposition 4.2 in the paper). As discussed in the paper, it is not necessary to compute the Hessian matrix $H_c$ directly. Instead, we compute Hessian-vector products $H_c^T\xi_c$ for the modified gradient, which have a computational cost of $\mathcal{O}(n)$ for $n$ weights [1]. Thus, compared to most current methods in mixed-motive MARL, the additional complexity mainly arises from the Hessian-vector products. **As a result, the running time of AgA is generally expected to be more than twice as long as that of standard gradient-based methods.
The practical running time analysis also demonstrate the conclusion.** Table 1 presents the average running time of baseline methods and AgA in the two-player public goods game (see Sec. 5.1 in the paper). The table includes the total duration, total number of timesteps over 50 runs, the time per step, and the time per step ratio compared to our AgA method.
The experiments are conducted in Macbook Pro with Apple M1 Pro Chip.
The values in the ratio row are calculated by dividing the time per step of each method by the time per step of the AgA method, which facilitates an easy comparison of running times.
The Simul-Ind, Simul-Co, and SL are standard gradient-based methods, while CGA and SGA are gradient-modification methods. **Our findings indicate that the AgA method takes approximately 2-3 times longer per step compared to standard gradient-based methods. Additionally, AgA is slightly slower than other gradient-modification methods due to the more complex operations involved in sign judgment.** Despite having the highest running time, AgA is the most efficient method, requiring only 1389 steps for 50 runs, compared to around 4000 steps for the baselines.
Our experiments in Harvest, Cleanup, and Selfish-MMM2 were conducted on different servers, such as A100, V100, and 3090. Therefore, we cannot directly compare the running times across these setups. However, we have selected some experiments run on the same servers (A100 with 80 GB) to provide a fair comparison of the running times. In the Harvest environment, AgA takes approximately 100.33 minutes on average for training, compared to 47.33 minutes for Simul-Co. This difference, which is about a factor of two, is due to the acceleration provided by the A100 and PyTorch. 

*We will incorporate the discussion into the future version. Thank you again to all reviewers for your valuable advice.*

*Table 1: Comparison of the running time between AgA and baseline methods.*
| Metrics | **Simul-Ind**             | **Simul-Co**              | **SL**                   | **CGA**                   | **SGA**                   | **AgA** |
|-------------|---------------------------|---------------------------|---------------------------|---------------------------|---------------------------|------------------------------|
Total Duration (ms)| 1165.79 | 910.15 | 1149.97 | 3041.84 | 3007.77 | 1034.69 |
Total Steps | 4272 | 3252 | 3887 | 4478 | 4179 | 1389
Time Per Step (ms)| 0.27 | 0.28 | 0.30 | 0.68 | 0.72 | 0.74 |
Ratio| 0.37 | 0.38 | 0.40 | 0.91 | 0.97 | 1.00 |

[1] B. A. Pearlmutter. Fast Exact Multiplication by the Hessian. Neural Computation, 6(1):147–160, 01 1994. ISSN 0899-7667. doi:10.1162/neco.1994.6.1.147.

Meta-review: This work proposes a gradient adjustment technique for aligning individual and collective objectives in cooperative multiagent learning. The authors propose modifying the vanilla collective objective with a term that incorporates individual objectives as well as a consensus term that seeks fixed points of the collective objective. In experiments, this weight on the latter terms is set quite high such that the original collective objective plays less of a role.

The reviewers find the “The paper is theoretically sound. The AgA algorithm is simple, but at the same time well justified” with “Strong experimental results”. The reviewers were initially concerned about runtime, but the authors have argued hessian-vector products are efficient and reported only 2-3x slower runtimes empirically (please include these runtime results somewhere). The Gini coefficient is a nice addition to capture individual agent performance as requested by reviewers x8HM and a1aR. “The most important contribution of this paper is Corollary 4.3”, drawing inspiration from the analysis of SGA.

To improve the paper, it would be helpful to see more ablations on the three terms that make up the update. To align with individual objectives, it seems reasonable to assume the $\xi$ term is required. The question is then how to incorporate $\xi_c$ and $\nabla \mathcal{H}_c$.

Given the reviewers generally felt positive about the paper, we are recommending to accept this submission.

Regarding the [Gemp, Kwon] citations, I would remove the statement “it is not intuitive to explain how individual incentives align with the collective incentives during the optimization process.” In that work, individual vs collective alignment is precisely captured by a reward-sharing matrix during the learning process. One can intuitively measure alignment by measuring the norm of the difference between whatever the current sharing matrix is versus the collective sharing matrix (the ones matrix divided by number of agents).
Decision: Accept (poster)