Aligning Individual and Collective Objectives in Multi-Agent Cooperation

Authors: yang  li
wenhao  zhang
jianhong  wang
shao  zhang
yali  du
ying  wen
wei  pan
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 13:42:54

Abstract: Among the research topics in multi-agent learning, mixed-motive cooperation is one of the most prominent challenges, primarily due to the mismatch between individual and collective goals. The cutting-edge research is focused on incorporating domain knowledge into rewards and introducing additional mechanisms to incentivize cooperation. However, these approaches often face shortcomings such as the effort on manual design and the absence of theoretical groundings. To close this gap, we model the mixed-motive game as a differentiable game for the ease of illuminating the learning dynamics towards cooperation. More detailed, we introduce a novel optimization method named \textbf{\textit{A}}ltruistic \textbf{\textit{G}}radient \textbf{\textit{A}}djustment (\textbf{\textit{AgA}}) that employs gradient adjustments to progressively align individual and collective objectives. Furthermore, we theoretically prove that AgA effectively attracts gradients to stable fixed points of the collective objective while considering individual interests, and we validate these claims with empirical evidence. We evaluate the effectiveness of our algorithm AgA through benchmark environments for testing mixed-motive collaboration with small-scale agents such as the two-player public good game and the sequential social dilemma games, Cleanup and Harvest, as well as our self-developed large-scale environment in the game StarCraft II.

Main paper: 
 introduction: 
 Multi-agent cooperation primarily focuses on learning how to promote collaborative behavior in shared environments. In general, multi-agent cooperation research is categorized into two prominent areas: pure-motive cooperation and mixed-motive cooperation [Du et al., 2023 , McKee et al., 2020] . Recent progress in cooperative Multi-Agent Reinforcement Learning (MARL) has been primarily focusing on pure-motive cooperation, also known as common payoff games. This game models situations where each agent's individual goal fully aligns with the collective objectives [Yu et al., 2022 , Wang et al., 2020 , Zhong et al., 2024 , Li et al., 2024] . Nevertheless, mixed-motive cooperation is more widespread across real-world situations. It is usually defined by imperfect alignment between individual and collective rationalities [Rapoport, 1974 , McKee et al., 2020] .Recent studies in mixed-motive cooperative MARL have largely employed hand-crafted designs to promote collaboration. One popular approach is to align objectives as per existing mechanisms in cooperative games, such as reputation [Anastassacos et al., 2021] , norms [Vinitsky et al., 2023] , and contracts [Hughes et al., 2020] . Another prevalent method leverages intrinsic motivation to align individual and collective objectives, enhancing altruistic collaboration by integrating heuristic knowledge into the incentive function. Conventionally, some works accumulate individual rewards with the group to promote altruistic conduct [Hostallero et al., 2020 , Peysakhovich and Lerer, 2018 , Apt and Schäfer, 2014 , Roesch et al., 2024] . Furthermore, some studies derive more sophisticated preference signals from the rewards of other agents [Hughes et al., 2018 , McKee et al., 2020] . Additionally, several approaches aim to learn the potential influences of an agent's actions on others [Yang et al., 2020 , Jaques et al., 2019 , Lu et al., 2022] . Most of these algorithms rely heavily on carefully crafted designs, necessitating significant human expertise and detailed domain knowledge. On the other hand, several studies leverage Nash equilibria and related game theory concepts, such as the price of anarchy, to automatically modify rewards by learning additional weights that adjust the original objectives [Gemp et al., 2022 , Kwon et al., 2023] . However, finding Nash equilibria in nonconvex games presents a greater challenge compared to identifying minima in neural networks [Letcher et al., 2019a] .In this study, we introduce the differentiable mixed-motive game (DMG), an effective framework for analyzing learning dynamics at both individual and collective levels. Furthermore, we derive the Altruistic Gradient Adjustment (AgA) algorithm, which aligns individual and collective objectives by modifying the gradient. We theoretically prove that AgA, with an appropriately chosen sign for the adjustment term, can successfully guide the gradient towards stable fixed points of the collective objective while considering individual interests. Additionally, empirical evidence from optimization trajectory visualizations and ablation studies validates our claims.We also conduct comprehensive experiments to verify the effectiveness of the proposed AgA algorithm. First, optimization trajectory analysis and ablation studies validate our theoretical conclusions. Next, a series of experiments conducted in various environments demonstrate that the AgA algorithm outperforms related baselines in both gradient adjustment and mixed-motive cooperation areas. In addition to commonly used testbeds like the public goods matrix game and sequential social dilemma games (Cleanup and Harvest) [Leibo et al., 2017] , which are limited in terms of agent scale, action space, and task complexity, we introduce a more complex mixed-motive environment called Selfish-MMM2, an adaptation of the MMM2 map from the StarCraft II game [Samvelyan et al., 2019] . Selfish-MMM2 offers the following significant improvements over other mixed-motive games: it supports large-scale scenarios with 10 heterogeneous controlled agents facing 11 enemies, features a large action space with a size of 18 10 , which vastly exceeds the action spaces in Cleanup 9 5 and Harvest 8 5 that are limited to 5 homogeneous agents. Selfish-MMM2 also has a larger action space and greater task complexity than the 10-player PD testbed (with a size of 2 10 ) used in D3C [Gemp et al., 2022] , which claims to solve the large-scale problem in mixed-motive games.In summary, the contributions of this paper are as follows: (1) We are the first work to model the mixed-motive game as a differentiable game (to the best of our knowledge) and propose the AgA algorithm to align individual and collective objectives from a gradient perspective. (2) We theoretically prove that, in the neighborhood of fixed points, AgA could pull the gradient toward stable fixed points of the collective objectives and push the gradient away from unstable fixed points.(3) We introduce Selfish-MMM2, a novel large-scale mixed-motive cooperation environment, and conduct comprehensive experiments across multiple settings that verify our theoretical claims and demonstrate the superior performance of the AgA algorithm.blending individual rewards to redraft the agent's overall utility. Inequity aversion further integrates the concept into Markov games by adding envy (disadvantageous inequality) and guilt (advantageous inequality) rewards to the original individual rewards [Hughes et al., 2018] . To further promote cooperation, the gifting mechanism-a crucial strategy in mixed-motive cooperation [Lupu and Precup, 2020] -allows agents to influence each other's reward functions through peer rewarding. Besides, PED-DQN [Hostallero et al., 2020] introduces an automatic reward-shaping MARL method that gradually adjusts rewards to shift agents' actions from their perceived equilibrium towards more cooperative outcomes. Social Value Orientation (SVO) strategy introduces a unique shared rewardsbased compensation approach, encouraging behavior modifications in line with interdependence theory [McKee et al., 2020] . The LIO strategy bypasses the need to modify extrinsic rewards by empowering an agent to directly influence its partner's actions [Yang et al., 2020] . Meanwhile, other studies introduce new cooperative mechanisms such as incorporating a reputation model [McKee et al., 2021] , social norms [Vinitsky et al., 2023] . Recently, Roesch et al. [2024] proposed a selfishness level method that incorporates social welfare into individual rewards to enhance altruistic cooperation. Additionally, several works have explored automatically modifying rewards online. D3C [Gemp et al., 2022] learns to mix rewards to improve efficiency in a Nash equilibrium. Kwon et al. [2023] extended D3C by addressing the problem of automatically modifying individual agent objectives to optimize a desired global objective. Despite these advances, many current studies lack both cost efficiency in design and theoretical analysis of alignment and convergence. To address this, our study applies a gradient perspective to align goals and further investigates the learning dynamics of our proposed method.Gradient-based Methods. Our proposed AgA is fundamentally a gradient adjustment methodology, making gradient-based optimization methods highly relevant to the context of this paper. Methods based on gradient have been developed to find stationary points, such as the Nash equilibrium or stable fixed points. Optimistic Mirror Descent leverages historical data to extrapolate subsequent gradients Daskalakis et al. [2018] , while Gidel et al. [2019] extends this concept by advocating averaging methodologies and variants of extrapolation techniques. Consensus gradient adjustment, or consensus optimization, is a technique that embeds a consensus agreement term within the gradient to ensure its convergence [Mescheder et al., 2017] . Learning with Opponent-Learning Awareness (LOLA) utilizes information from other players to compute one player's anticipated learning steps [Foerster et al., 2018] . Subsequently, Stable Opponent Shaping (SOS) [Letcher et al., 2019b] and Consistent Opponent-Learning Awareness (COLA) [Willi et al., 2022] methods have enhanced the LOLA algorithm, targeting convergence assurance and inconsistency elimination, respectively. Symplectic Gradient Adjustment (SGA) alters the update direction towards the stable fixed points based on a novel decomposition of game dynamics [Balduzzi et al., 2018 , Letcher et al., 2019a] . Recently, Learning to Play Games (L2PG) ensures convergence towards a stable fixed point by predicting updates to players' parameters derived from historical trajectories [Chen et al., 2023] . However, these methods, while focusing on zero-sum or general sum games, could potentially act counter to their individual interests, as they may prioritize stability over minimizing personal loss. Our research specifically targets mixed-motive setting with the aim of reconciling individual and collective objectives. 
differential game: 
 The theory of differential games was initially proposed by Isaacs [1965] , aiming to expand the scope of sequential game theory to encompass continuous-time scenarios. Through the lens of machine learning, we formalize the differential game, as shown in Definition 3.1. Definition 3.1 (Differential Game [Balduzzi et al., 2018 , Letcher et al., 2019a] ). A differential game could be defined as a tuple {N , w, ℓ}, where N = {1, . . . , n} is the set of players. The parameter set w= [w i ] n ∈ R d is defined, each with w i ∈ R di and d = n i=1 d i . Here, ℓ = {ℓ i : R d → R} n i=1represents the corresponding losses. These losses are assumed to be at least twice differential. Each player i ∈ N is equipped with a policy, parameterized by w i , aiming to minimize its loss ℓ i .We write the simultaneous gradient ξ(w) of a differential game as ξ(w) = (∇ w1 ℓ 1 , . . . , ∇ wn ℓ n ) ∈ R d , which is the gradient of the losses with respect to the parameters of the respective players. Furthermore, Hessian matrix H mentioned in a differential game is the Jacobian matrix of the simultaneous gradient. The learning dynamics of differential game often refers to the process of sequential updates over w.The learning rule for each player is defined as the operator, w ← wγξ, where γ is a step size (learning rate) to determine the distance to move for each update.The stable fixed point, a criterion initiated from stability theory, exhibits its stability (robustness) to minor perturbations of environments, making it applicable to many real-world scenarios.Definition 3.2. A point w ⋆ is a fixed point if ξ(w ⋆ ) = 0. If H(w ⋆ ) ⪰ 0 and H(w ⋆ ) is invertible, the fixed point w ⋆ is called stable fixed point. If H(w ⋆ ) ≺ 0, the point is called unstable.A naive idea to steer the dynamic towards convergence at fixed points involves minimizing 1 2 ∥ξ(w)∥ 2 . Assuming the Hessian H(w) is invertible, then ∇( 1 2 ∥ξ(w)∥ 2 ) = H T ξ = 0 holds true if and only if ξ = 0. However, it could converge to unstable fixed points [Mescheder et al., 2017] . Hence, the consensus optimization method has been proposed, incorporating gradient adjustment [Mescheder et al., 2017] , shown as follows:ξ = ξ + λ • ∇ 1 2 ∥ξ(w)∥ 2 = ξ + λ • H T ξ.For simplicity, we will refer to the consensus gradient adjustment as CGA. While CGA proves effective in certain specific scenarios, such as two-player zero-sum games, it unfortunately falls short in general games [Balduzzi et al., 2018] . To address this shortage, symplectic gradient adjustment (SGA) [Balduzzi et al., 2018 , Letcher et al., 2019a] is introduced to find the stable fixed point in general sum games, such that ξ = ξ + λ • A T ξ. Herein, A represents the antisymmetric component of the generalized Helmholtz decomposition of H, H(w) = S(w) + A(w), where S(w) denotes the symmetric component. 
method: 
 In this section, we first define differentiable mixed-motive games (DMG) and analyzes the alignment dilemma faced by existing methods, which is described in Section 4.1. We then propose the Altruistic Gradient Adjustment (AgA) algorithm in Section 4.2 to align the individual and collective objectives by modifying the gradient. Finally, a case study demonstrating AgA's effectiveness in a toy two-player DMG is presented in Section 4.3.We first formulate the mixed-motive game as a differentiable game. Specifically, the differentiable mixed-motive game (DMG) is defined as a tuple (N , w, ℓ), where ℓ i ∈ ℓ is at least twice the differentiable loss function for the agent i. Differentiable losses exhibit the mixed motivation property: minimization of individual losses can result in a conflict between individuals or between individual and collective objectives (e.g., maximizing individual stats and winning the game are often < l a t e x i t s h a 1 _ b a s e 6 4 = " H q i 9 i e X I g I z w k 5 p wE v 5 h W Y X d 7 l 8 = " > A A A B / X i c b V D L S s N A F L 2 p r 1 p f 8 b F z E y y C q 5 J I U Z d F N y 4 r 9 A V N K J P p p B 0 6 m Y S Z i V h D 8 F f c u F D E r f / h z r 9 x 0 m a h r Q c G D u f c y z 1 z / J h R q W z 7 2 y i t r K 6 t b 5 Q 3 K 1 v b O 7 t 7 5 v 5 B R 0 a J w K S N I x a J n o 8 k Y Z S T t q K K k V 4 s C A p 9 R r r + 5 C b 3 u / d E S B r x l p r G x A v R i N O A Y q S 0 N D C P 3 B Z l Q 5 K 6 I V J j P 0 j d B 5 p l A 7 N q 1 + w Z r G X i F K Q K B Z o D 8 8 s d R j g J C V e Y I S n 7 j h 0 r L 0 V C U c x I V n E T S W K E J 2 h E + p p y F B L p p b P 0 m X W q l a E V R E I / r q y Z + n s j R a G U 0 9 D X k 3 l I u e j l 4 n 9 e P 1 H B l Z d S H i e K c D w / F C T M U p G V V 2 E N q S B Y s a k m C A u q s 1 p 4 j A T C S h d W 0 S U 4 i 1 9 e J p 3 z m n N R q 9 / V q 4 3 r o o 4 y H M M J n I E D l 9 C A W 2 h C G z A 8 w j O 8 w p v x Z L w Y 7 8 b H f L R k F D u H 8 A f G 5 w 9 F Q J X F < / l a t e x i t > ⇠ < l a t e x i t s h a 1 _ b a s e 6 4 = " t e I F f j 7 K l Q S 0 a 5 X J N x i i K l I 9 G S U = " > A A A B + X i c b V D L S s N A F L 2 p r 1 p f U Z d u B o v g q i Q i 6 r L o x m U F + 4 A m h M l 0 0 g 6 d T M L M p F h C / 8 S N C 0 X c + i f u / B s n b R b a e m D g c M 6 9 3 D M n T D l T 2 n G + r c r a + s b m V n W 7 t r O 7 t 3 9 g H x 5 1 V J J J Q t s k 4 Y n s h V h R z g R t a 6 Y 5 7 a W S 4 j j k t B u O 7 w q / O 6 F S s U Q 8 6 m l K / R g P B Y s Y w d p I g W 1 7 M d a j M M q 9 J x b k Z D Y L 7 L r T c O Z A q 8 Q t S R 1 K t A L7 y x s k J I u p 0 I R j p f q u k 2 o / x 1 I z w u m s 5 m W K p p i M 8 Z D 2 D R U 4 p s r P 5 8 l n 6 M w o A x Q l 0 j y h 0 V z 9 v Z H j W K l p H J r J I q d a 9 g r x P 6 + f 6 e j G z 5 l I M 0 0F W R y K M o 5 0 g o o a 0 I B J S j S f G o K J Z C Y r I i M s M d G m r J o p w V 3 + 8 i r p X D T c q 8 b l w 2 W 9 e V v W U Y U T O I V z c O E a m n A P L W g D g Q k 8 w y u 8 W b n 1 Y r 1 b H 4 v R i l X u H M M f W J 8 / P Y K U E Q = = < / l a t e x i t > ⇠ c< l a t e x i t s h a 1 _ b a s e 6 4 = " H q i 9 i e X I g I z w k 5 p w conflict in basketball matches). In addition to the simultaneous gradient ξ(w) of individual losses with respect to the parameters of the respective players, we use ξ c (w) to refer to the gradient of the collective loss: ξ c (w) = (∇ w1 ℓ c , . . . , ∇ wn ℓ c ) .E v 5 h W Y X d 7 l 8 = " > A A A B / X i c b V D L S s N A F L 2 p r 1 p f 8 b F z E y y C q 5 J I U Z d F N y 4 r 9 A V N K J P p p B 0 6 m Y S Z i V h D 8 F f c u F D E r f / h z r 9 x 0 m a h r Q c G D u f c y z 1 z / J h R q W z 7 2 y i t r K 6 t b 5 Q 3 K 1 v b O 7 t 7 5 v 5 B R 0 a J w K S N I x a J n o 8 k Y Z S T t q K K k V 4 s C A p 9 R r r + 5 C b 3 u / d E S B r x l p r G x A v R i N O A Y q S 0 N D C P 3 B Z l Q 5 K 6 I V J j P 0 j d B 5 p l A 7 N q 1 + w Z r G X i F K Q K B Z o D 8 8 s d R j g J C V e Y I S n 7 j h 0 r L 0 V C U c x I V n E T S W K E J 2 h E + p p y F B L p p b P 0 m X W q l a E V R E I / r q y Z + n s j R a G U 0 9 D X k 3 l I u e j l 4 n 9 e P 1 H B l Z d S H i e K c D w / F C T M U p G V V 2 E N q S B Y s a k m C A u q s 1 p 4 j A T C S h d W 0 S U 4 i 1 9 e J p 3 z m n N R q 9 / V q 4 3 r o o 4 y H M M J n I E D l 9 C A W 2 h C G z A 8 w j O 8 w p v x Z L w Y 7 8 b H f L R k F D u H 8 A f G 5 w 9 F Q J X F < / l a t e x i t > ⇠ < l a t e x i t s h a 1 _ b a s e 6 4 = " t e I F f j 7 K l Q S 0 a 5 X J N x i i K l I 9 G S U = " > A A A B + X i c b V D L S s N A F L 2 p r 1 p f U Z d u B o v g q i Q i 6 r L o x m U F + 4 A m h M l 0 0 g 6 d T M L M p F h C / 8 S N C 0 X c + i f u / B s n b R b a e m D g c M 6 9 3 D M n T D l T 2 n G + r c r a + s b m V n W 7 t r O 7 t 3 9 g H x 5 1 V J J J Q t s k 4 Y n s h V h R z g R t a 6 Y 5 7 a W S 4 j j k t B u O 7 w q / O 6 F S s U Q 8 6 m l K / R g P B Y s Y w d p I g W 1 7 M d a j M M q 9 J x b k Z D Y L 7 L r T c O Z A q 8 Q t S R 1 K t A L 7 y x s k J I u p 0 I R j p f q u k 2 o / x 1 I z w u m s 5 m W K p p i M 8 Z D 2 D R U 4 p s r P 5 8 l n 6 M w o A x Q l 0 j y h 0 V z 9 v Z H j W K l p H J r J I q d a 9 g r x P 6 + f 6 e j G z 5 l I M 0 0 F W R y K M o 5 0 g o o a 0 I B J S j S f G o K J Z C Y r I i M s M d G m r J o p w V 3 + 8 i r p X D T c q 8 b l w 2 W 9 e V v W U Y U T O I V z c O E a m n A P L W g D g Q k 8 w y u 8 W b n 1 Y r 1 b H 4 v R i l X u H M M f W J 8 / P Y K U E Q = = < / l a t e x i t > ⇠ c Unstable Fixed Point < l a t e x i t s h a 1 _ b a s e 6 4 = " Z A G s d A y C A I w L F X b 7 j G n W e l w t L 5 s = " > A A A B / X i c b V D L S s N A F L 2 p r 1 p f 8 b F z M 1 g E V y W R o i 6 L b r q s Y G u h C W E y n b R D J 5 M w M x F q K P 6 K G x e K u P U / 3 P k 3 T t s s t P X A w O G c e 5 h 7 T 5 h y p r T j f F u l l d W 1 9 Y 3 y Z m V r e 2 d 3 z 9 4 / 6 K g k k 4 S 2 S c I T 2 Q 2 x o p w J 2 t Z M c 9 p N J c V x y O l 9 O L q Z + v c P V C q W i D s 9 T q k f 4 4 F g E S N Y G y m w j 7 w Y 6 2 E Y 5 c 1 J Q J B n s g Q 5 g V 1 1 a s 4 M a J m 4 B a l C g V Z g f 3 n 9 h G Q x F Z p w r F T P d V L t 5 1 h q R j i d V L x M 0 R S T E R 7 Q n q E C x 1 T 5 + W z 7 C T o 1 S h 9 F i T R P a D R T f y d y H C s 1 j k M z O d 1 V L X p T 8 T + v l + n o y s + Z S D N N B Z l / F G U c 6 Q R N q 0 B 9 Z q 7 V f G w I J p K Z X R E Z Y o m J N o V V T A n u 4 s n L p H N e c y 9 q 9 d t 6 t X F d 1 F G G Y z i B M 3 D h E h r Q h B a 0 g c A j P M M r v F l P 1 o v 1 b n 3 M R 0 t W k T m E P 7 A + f w C 6 1 p T C < / l a t e x i t > Hc 0 < l a t e x i t s h a 1 _ b a s e 6 4 = " u j y R Y H l k Y o J n E 8 Q m 5 C A b Z F V q H 0 s = " > A A A B / 3 i c b V D L S s N A F J 3 U V 6 2 v q O D G z W A R X J V E i r o s u u m y g n 1 A E 8 L k d t I O n T y c m Q g l d u G v u H G h i F t / w 5 1 / 4 6 T N Q l s P D B z O u Z d 7 5 v g J Z 1 J Z 1 r d R W l l d W 9 8 o b 1 a 2 t n d 2 9 8 z 9 g 4 6 M U w G 0 D T G P R c 8 n k n I W 0 b Z i i t N e I i g J f U 6 7 / v g m 9 7 s P V E g W R 3 d q k l A 3 J M O I B Q y I 0 p J n H j k h U S M / y J p T D 7 A j U w B 6 j y 3 P r F o 1 a w a 8 T O y C V F G B l m d + O Y M Y 0 p B G C j i R s m 9 b i X I z I h Q D T q c V J 5 U 0 I T A m Q 9 r X N C I h l W 4 2 y z / F p 1 o Z 4 C A W + k U K z 9 T f G x k J p Z y E v p 7 M 0 8 p F L x f / 8 / q p C q 7 c j E V J q m g E 8 0 N B y r G K c V 4 G H j B B Q f G J J g Q E 0 1 k x j I g g o H R l F V 2 C v f j l Z d I 5 r 9 k X t f p t v d q 4 L u o o o 2 N 0 g s 6 Q j S 5 R A z V R C 7 U R o E f 0 j F 7 R m / F k v B j v x s d 8 t G Q U O 4 f o D 4 z P H 1 / K l b A = < / l a t e x i t > Hc ⌫ 0 (a)Alignment Dilemma in DMG. Direct optimization of individual losses and collective loss are straightforward ideas to solving DMG problems. However, minimizing only the individual loss of each agent is unlikely to achieve a collective optimum [Leibo et al., 2017 , McKee et al., 2020] . Conversely, optimizing the collective loss might produce better overall outcomes but risks neglecting individual interests [Roesch et al., 2024] . Additionally, the local convergence of gradient descent in singular collective functions is not always guaranteed [Balduzzi et al., 2018] .Example 4.1 gives a two-player differentiable mixed-motive game. We provide a visual representation of optimization trajectories using a series of methods to investigate the learning dynamics involved in resolving the example, as shown in Fig. 1 .Example 4.1. Consider a two-player DMG with ℓ 1 (a 1 , a 2 ) = -sin(a 1 a 2 + a 2 2 ) and ℓ 2 (a 1 , a 2 ) = -[cos(1 + a 1 -(1 + a 2 ) 2 ) + a 1 a 2 2 ], where a i represents the action of the player i (i = 1, 2), and a i ∈ R. The rewards for the two players are the negation of their respective losses.We first implement simultaneous optimization of individual losses (Simul-Ind) with respect to each parameter, defined by the learning rule w i = w iγξ i , for i ∈ {1, 2}, where ξ i = ∇ wi ℓ i . Simultaneous optimization of collective loss (Simul-Co) replaces the individual gradient with the collective gradient ξ c of collective loss ℓ c = ℓ 1 + ℓ 2 . Furthermore, in order to investigate the learning dynamics of prevalent gradient modification optimization approaches such as SGA and CGA for the two-player differentiable mixed-motive game, we modify the learning rule by integrating the respective adjusted gradient as delineated in Section 3.2. Fig. 1a shows the optimization paths on the collective reward landscapes, with every path starting from the front-bottom of the landscape. The collective reward is defined as social welfare, i.e., the sum of individual rewards. As shown in the figure, Simul-Ind, CGA , and SGA converge to unstable points or local maxima. Simul-Co effectively navigates towards the apex of the collective reward landscape as depicted in Fig. 1a . However, Simul-Co is ineffective in aligning individual and collective objectives, leading to the neglect of individual interests. As depicted in Fig. 1b , the update trajectory navigates through the crests and troughs of Player 1's reward contour. The trajectory suggests a disregard for Player 1's preferences, signifying that the updates are predominantly driven by the overarching collective goal.To tackle the challenge of aligning objectives in DMG, we propose the Altruistic Gradient Adjustment (AgA) method. Unlike existing gradient adjustment techniques [Mescheder et al., 2017 , Balduzzi et al., 2018 , Letcher et al., 2019a , Chen et al., 2023] , which primarily aim at achieving stable fixed points for individual objectives, our AgA method simultaneously considers both individual and collective objectives when seeking stable fixed points for the collective objective. Specifically, AgA is defined as follows. Definition 4.2 (Altruistic Gradient Adjustment). Altruistic gradient adjustment (AgA) extends the gradient term in the learning dynamic asEQUATIONwhere λ ∈ R is alignment parameter, λξ adj is called adjustment term. In ξ adj , ξ c and H c is the gradient vector and Hessian matrix of the game about collective loss.Note that the Hessian matrix H c is symmetric. For brevity, we denote ∇H c as ∇ w 1 2 ∥ξ c ∥ 2 , which simplifies to H T c ξ c . In our AgA method, the formula ξ c + λ(ξ + H T c ξ c ) includes the Hessian matrix, but we do not compute it directly. Instead, we calculate Hessian-vector products H T c ξ c , which suffice for determining the adjustment gradient and the sign of λ. This approach reduces computational complexity, enhancing the effectiveness of the AgA method. The cost for computing Hessian-vector products H T c ξ c is O(n) for n weights [Pearlmutter, 1994] . Refer to Section 6 for a detailed analysis.While AgA introduces additional complexity, we theoretically demonstrate that an appropriate choice of the sign of λ ensures that AgA pulls the gradient towards a stable fixed point through the adjustment term in the neighborhood of fixed points. Conversely, when dealing with an unstable fixed point, AgA will push the gradient away from these unstable points.Before introducing the corollary, we first provide some basic notations. The inner product of vectors a and b are denoted by ⟨a, b⟩. If the Hessian matrix H is non-negative-definite, ⟨ξ c , ∇H⟩ ≥ 0 for a non-zero ξ c . Analogously, if H is negative-definite, ⟨ξ c , ∇H⟩ < 0 for a non-zero ξ c . Lastly, the angle between the two vectors a and b is denoted by θ(a, b). Then, we state the corollary as follows: Corollary 4.3. In the neighborhood of fixed points of the collective objective, AgA will pull the gradient toward stable fixed points, which means θ( ξ, ∇H c ) ≤ θ(ξ c , ∇H c ), and push away from unstable ones, indicated by θ( ξ,∇H c ) ≥ θ(ξ c , ∇H c ), if λ satisfies λ • ⟨ξ c , ∇H c ⟩(⟨ξ, ∇H c ⟩ + ∥∇H c ∥ 2 ) ≥ 0.The proof is provided in Appendix B. Fig. 2a presents an illustrative visualization of Corollary 4.3. The figure depicts two scenarios: the neighborhoods of stable and unstable fixed point (denoted by a star), respectively. In the neighborhood of an unstable fixed point of the collective objective, as depicted in Case 1, our AaA gradient ξ is pushed out of the region, resulting in a faster escape compared to the original collective gradient ξ c . This behavior is exemplified by the green arrow. Conversely, Case 2 illustrates the scenario of a stable fixed point, wherein our AaA gradient ξ is pulled towards the stable equilibrium, exemplified by the red arrow.AgA could be easily integrated into any centralized training with decentralized execution (CTDE) framework within MARL. Detailed statements on the implementation of AgA, including the pseudocode, are provided in Appendix C.To address the two-player DMG as outlined in Example 4.1, we incorporate the AgA method into the fundamental gradient descent algorithm. The optimization trajectories in both the collective reward landscape and the individual player reward contour are represented in Fig. 1a , Fig. 1b , and Fig. 1c . In contrast to the objective misalignment exhibited by Simul-Co, AgA successfully aligns the agents' interests, as evidenced in Fig. 1b  
experiments: 
 To assess the effectiveness of the proposed AgA algorithm, we perform comprehensive experiments across various environments, ranging from simple to complex, and from small to large scales. The initial environment involves a two-player public goods game (see Section 5.1), followed by sequential social dilemma environments (see Section 5.2): Cleanup and Harvest, involving 5 homogeneous players. The final test is conducted in our specially developed selfish-MMM2 environment (see Section 5.3), which is both more complex and larger in scale, featuring approximately 10 controlled agents of 3 distinct types, competing against 11 adversaries. Comparative experiments employ commonly used baseline approaches, such as simultaneous optimization with individual and collective losses (termed Simul-Ind and Siml-Co), CGA [Mescheder et al., 2017] , SGA [Balduzzi et al., 2018] , SVO [McKee et al., 2020] , and the selfishness level approach (termed SL) [Roesch et al., 2024] .A two-player public goods matrix game G is widely utilized to study cooperation in social dilemmas.The game involves players {1, 2} with parameters {w 1 , w 2 } and payoffs {p 1 , p 2 }. The social welfare, denoted as SW = p 1 + p 2 . Each player, i.e., i ∈ {1, 2}, contributes an amount a i within a budgeted range [0, b], and the host evenly distributes these contributions as c 2 (a 1 + a 2 ), where 1 < c ≤ 2. Consequently, each player's payoff p i (a 1 , a 2 ) is estimated as ba i + c 2 (a 1 + a 2 ). In our experiments, we set the budget b to 1 and weight c to 1.5, with the social optimum of the game at (1, 1).. Table 1 presents a comparison of individual rewards and the collective outcome, derived from 50 random trials, each limited to 100 steps. The rows r 1 , r 2 , SW , and E represent the individual rewards for each player, the group's total welfare (SW), and the equality metric (E), respectively. The equality metric is based on the Gini coefficient (G) [David, 1968] , with E defined asE = 1 -G = 1 -2 n 2 p n i=1 i(p i -p), where p is the mean of the ranked payoff vector p, and n is the total number of players. A higher E value indicates greater equality among the players.The social optimum of the game occurs when all players allocate their entire budget, achieving the highest possible social welfare score of 3. Among all algorithms, AgA achieves the closest social welfare score to this optimum, with a value of 2.90. Furthermore, AgA exhibits the greatest degree of fairness, with two players receiving nearly identical rewards and the highest equality value, in contrast to the baseline algorithms. Furthermore, Fig. 4 in Appendix D illustrates the action distributions of these algorithms, showing that AgA actions are more tightly clustered around the social optimum actions (1, 1). This further indicates that AgA approaches the social optimum more effectively.We then verify the AgA algorithm in two widely used sequential social dilemma games with 5 homogeneous agents: Harvest and Cleanup [Hughes et al., 2018] . In the Cleanup scenario, agents predominantly gather rewards by harvesting apples in an orchard, where the yield is affected by the river's pollution levels. Neglecting rising pollution levels ultimately ceases apple production, thus setting up a trade-off between individual gains and communal welfare. In contrast, the Harvest , where α is a constant. Further details on the experimental setups and the algorithms used are provided in Appendix E.1.Varying λ values. Fig. 3a and Fig. 3b explore the social welfare outcomes from AgA under three distinct λ values: 0.1, 1, 100, and 1000. Among these, λ = 100 demonstrates superior performance in both the Cleanup and Harvest scenarios. Therefore, in the subsequent experiments involving sign alignment and comparing primary results with baseline models, we will employ λ = 100 as the default parameter.The effectiveness of sign alignment. In the second row of Fig. 3 , Fig. 3d and Fig. 3e illustrate the social welfare comparisons between AgA and AgA without sign alignment (AgA-Sign) during their training phases. Referencing Corollary 4.3, near a fixed point, sign alignment helps direct the gradient towards stable fixed points and away from unstable ones. During the latter half of the training sessions shown in Fig. 3d and Fig. 3e , it is observed that the social welfare (SW ) for AgA-Sign does not improve to the same extent as it does for AgA, highlighting the effectiveness of sign alignment.Baseline Comparison. As illustrated in Fig. 3g and Fig. 3h , AgA demonstrates a notable improvement in social welfare over Simul-Ind, Simul-Co, CGA, SVO, and SL. Particularly, within the Cleanup scenario, AgA recorded an average social welfare of 105.15, marking approximately a 56% increase over the runner-up method SL. In the Harvest scenario, AgA shows an average improvement of 33.55 in social welfare over the next best method SVO. Table 2 compares the mean and standard deviation of the equality metric achieved by different methods in the Harvest and Cleanup environments. A value closer to 1 signifies more equal rewards among agents. As shown in the table, AgA outperforms the baseline methods, demonstrating its ability to effectively balance the interests of all team members.While sequential social dilemma games are frequently used to study mixed-motive problems, they are relatively simplistic compared to other experimental environments in MARL, particularly when considering real-world applications. To address this gap, we introduce a sophisticated mixedmotive testbed called Selfish-MMM2, which is adapted from the MMM2 map in the StarCraft II game [Samvelyan et al., 2019] . Unlike the standard SMAC framework, which employs a shared reward structure, our methodology assigns individual objectives to each agent, thereby enhancing the complexity of mixed motives. Specifically, we have redesigned the reward systems so that each agent's personal gain is directly tied to the damage they inflict on adversaries, diverging from traditional approaches where collective damage is evenly distributed among all agents. Additionally, penalties for agent elimination emphasize the importance of self-preservation and highlight agents' tendencies towards self-interested behavior. In comparison to widely utilized multi-agent environments such as Cleanup [Hughes et al., 2018] , Harvest [Hughes et al., 2018] , and the 10-player Prisoner's Dilemma (PD) for large-scale evaluations [Gemp et al., 2022] , Selfish-MMM2 is distinguished by two notable features: 1) Large-Scale Agents: Selfish-MMM2 manages the interactions of 10 heterogeneous agents, categorized into three types: seven Marines, two Marauders, and one Medivac, confronting 11 enemy units. This configuration is in stark contrast to Cleanup and Harvest, which each utilize five homogeneous agents. 2) Extensive Action Space: The action space in Selfish-MMM2, sized at 18 10 , is vastly larger than that of Cleanup and Harvest, which are limited to 9 5 and 8 5 , respectively. It also surpasses the action space of the 10-player PD, which is 2 10 .In evaluating the Selfish-MMM2 environment, we use the average win rate against the built-in AI bot in SMAC to measure the collective performance. The experimental setup employs the IPPO algorithm [de Witt et al., 2020] for the Simul-Ind algorithm, while the MAPPO [Yu et al., 2022] is utilized for the Simul-Co, SVO, CGA, SL and AgA algorithms. More details on the Selfish-MMM2 environment and implementation specifics are available in the Appendix E.2.Varying λ Values and the effectiveness of Sign Alignment: Fig. 3c presents the average win rates in various settings of the align parameter λ in the AgA scenario. The findings indicate that increased λ values are associated with faster convergence in the training period. Among these, a λ value of 100 demonstrates a higher overall effectiveness compared to other values tested. Based on these findings, we have set a default value λ of 100 for subsequent experiments. Furthermore, Fig. 3f in the second row illustrates a comparable conclusion in sequential social dilemma games. During the Baseline Comparison. As depicted in Fig. 3i , the bar chart displays a comparison of the average win rates along with 95% confidence intervals for AgA versus other baseline approaches. Our AgA approach attains the highest team performance, securing a 92.64% win rate with the built-in bot, which is 23.61 percent more than the next best win rate achieved by the SL method. 
discussion: 
 Computational Cost of AgA. The altruistic gradient adjustment is expressed asξ = ξ c + λ(ξ + H T c ξ c ),where the key computational cost comes from the Hessian-vector product, H T c ξ c . The cost for computing Hessian-vector products, H T c ξ c , is O(n) for n weights [Pearlmutter, 1994] . This introduces added complexity compared to standard methods in mixed-motive MARL, making AgA's running time generally more than twice as long as gradient-based methods. Table 3 provides a comparison of the average running time between AgA and baseline methods in a two-player public goods game, including total duration, timesteps, time per step, and the time ratio relative to Simul-Ind. Simul-Ind, Simul-Co, and SL are standard gradient-based methods, while CGA and SGA modify gradients. Our results show that AgA takes 2-3 times longer per step but remains the most efficient, requiring only 1389 steps over 50 runs, compared to around 4000 steps for the baselines.Limitations. Despite conducting a series of experiments to verify the proposed AgA algorithm, including using a new large-scale mixed-motive cooperation testbed, these environments remain somewhat removed from real-world applications. Although we strive to align the objectives of both individuals and the collective, our primary focus in this paper is on converging towards stable fixed points of the collective objective. For future research, we plan to delve deeper into the interaction of individual objectives in mixed-motive games, with an increased emphasis on understanding their dynamics in real-world mixed-motive cooperation scenarios.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: We provide all the training and test details in the Appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
conclusion: 
 In this paper, we propose AgA, an optimization method specifically designed to align individual and collective objectives through gradient adjustments in mixed-motive cooperation scenarios. To achieve this, we first model the mixed-motive game as a differentiable game, offering a powerful tool for analyzing learning dynamics at both individual and collective levels. Furthermore, we theoretically demonstrate that AgA can effectively attract gradients to stable fixed points and support our claims with empirical evidence. To evaluate the effectiveness of AgA in complex and large-scale scenarios, we introduce a new mixed-motive environment called Selfish-MMM2, which features heterogeneous large-scale agents, a larger action space, and increased task complexity. Comprehensive experiments, ranging from simple public goods games to Harvest, Cleanup, and Selfish-MMM2, show AgA's superior performance, consistently outperforming existing baselines across multiple evaluation metrics. These results validate our theoretical claims and highlight the effectiveness of AgA in achieving cooperative behavior in multi-agent systems.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: We report error bars for all experimental results using a 95% confidence interval.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: We discuss the potential societal impacts in Appendix A. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The paper explicitly mentioned and cited the original codes and environments we used.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: This paper presents a novel approach in the field of multi-agent learning, aimed at enhancing collaboration among agents with mixed motivations. The proposed framework addresses a fundamental challenge in machine learning: enabling agents with diverse objectives to cooperate effectively to achieve collective goals. The work described in this paper holds significant potential for various societal applications. By facilitating effective cooperation among agents with different motivations, our approach can be applied to domains such as video games, autonomous systems, smart cities, and decentralized resource allocation. These advancements can lead to numerous societal benefits, including improved efficiency in transportation systems, optimized resource allocation in energy grids, and enhanced decision-making in complex multi-agent environments. Furthermore, our findings contribute to the broader field of machine learning by offering insights into managing mixed-motivation scenarios. This work paves the way for future research and innovation, driving progress in multiagent learning and its applications across various domains. However, while our method was initially developed to address fundamental challenges in decision-making within machine learning contexts, it is conceivable that our proposed objective alignment techniques could be repurposed for harmful purposes, such as attacking existing cooperative systems.Corollary 4.3. In the neighborhood of fixed points of the collective objective, AgA will pull the gradient toward stable fixed points, which means θ( ξ, ∇H c ) ≤ θ(ξ c , ∇H c ), and push away from unstable ones, indicated by θ( ξ,∇H c ) ≥ θ(ξ c , ∇H c ), if λ satisfies λ • ⟨ξ c , ∇H c ⟩(⟨ξ, ∇H c ⟩ + ∥∇H c ∥ 2 ) ≥ 0.Proof. Prior to embarking on the elucidation of the corollary's proof, it is essential to first introduce relevant notations and the foundational concepts. Take note of the notation θ(a, b), which symbolizes the angular measure between two vectors. Furthermore, we write θ λ ( ξ, w) to represent the angular measure between AgA gradient ξ and a reference update direction w. For simplicity, we use ξ = u + λv to denote the AgA gradient ξ = ξ c + λ ξ + H T c ξ c ,as defined in Eq. 1. Specifically, u denotes collective gradient ξ c and v denotes ξ + H T c ξ c . We then extend the definition of infinitesimal alignment [Balduzzi et al., 2018] to our proposed AgA gradient. Given a third reference vector w, the infinitesimal alignment for AgA gradient ξ with w is defined asEQUATIONIntuitively, we may presume that vectors u and w are directionally aligned, given that u T w > 0.In this scenario, align > 0 implies that vector v is drawing u closer towards the reference vector w. Conversely, vector v is propelling u away from w. Similar arguments hold for align < 0: v pushes u away from the reference vector w when u and w share the same directional orientation. Conversely, v pulls u towards the reference vector w when the vectors u and w are not oriented in the same direction.The ensuing lemma provides a simplified method for determining the sign of align, circumventing the need for generating an exact solution.Lemma B.1 (Sign of align.). Given AgA gradient ξ and reference vector ∇H c , the sign of infinitesimal alignment align could be calculated bysign align( ξ, ∇H c ) = sign ⟨ξ, ∇H c ⟩(⟨ξ c , ∇H c ⟩ + ∥∇H c ∥ 2 ) .Therefore, we prove that if we let the sign of λ follows the condition λ • ⟨ξ c , ∇H c ⟩(⟨ξ, ∇H c ⟩ + ∥∇H c ∥ 2 ) ≥ 0, the optimization process using the AgA method exhibit following attributes. In areas close to fixed points, 1) if the point is stable, the AgA gradient will be pulled toward this point, which means that θ( ξ, ∇H c ) ≤ θ(ξ c , ∇H c ); 2) if the point represents unstable equilibria, the AgA gradient will be pushed out of the point, indicating that θ( ξ, ∇H c ) ≥ θ(ξ c , ∇H c ). An illustrative example of the corollary is provided in Fig. 2a .The concrete implementation of altruistic gradient adjustment is elaborated in Algorithm 1. AgA can be easily incorporated into any centralized training decentralized execution (CTDE) framework of MARL. In a typical CTDE framework, each agent, denoted as i, strives to learn a policy that employs local observations to prompt a distribution over personal actions. The unique characteristic during the centralized learning phase is that agents gain supplementary information from their peers that is inaccessible during the execution stage. Furthermore, a centralized critic often operates in tandem in CTDE settings, leveraging shared information and evaluating individual agent policies' potency. To incorporate our proposed AgA algorithm into CTDE frameworks, supplementary reward exchanges between agents are required, offering the essential inputs for Algorithm 1. The individual losses are calculated from individual rewards conforming to established MARL algorithm framework, such as IPPO [de Witt et al., 2020] or MAPPO [Yu et al., 2022] . Here, the collective loss ℓ c is calculated according to individual rewards; for example, it could be a sum of individual rewards or an elementary transformation thereof. Furthermore, the AgA algorithm requires a magnitude value of the alignment parameter λ, which should be a positive entity. The sign of λ is then calculated in line 5 of Algorithm 1 and is based on each gradient component calculated in lines 2-4. Subsequently, the algorithm produces the adjusted gradient ξ. ξ is then distributed across each corresponding parameter, and optimization is carried out using a suitable optimizer such as Adam optimizer [Kingma and Ba, 2015] .Fig. 4 depicts the actions produced by various methods in a two-player public game. The action generated by each method is represented by a circle, which is color-coded according to the corresponding method. To ensure fairness, each method's updates are constrained to a maximum of 100 steps. These methods include the individual loss-driven simultaneous optimization (Simul-Ind), two variants of gradient adjustment methods termed as CGA and SGA, and the simultaneous optimization leveraging collective loss (Simul-Co) alongside our proposed AgA, where λ = 1. Each of these methods is fundamentally underpinned by the gradient ascent algorithm. The illustration consolidates data from 50 randomized runs initialized from diverse starting points. The 'X' demarcated circles indicate the average actions mapped to each method. From the figure, it's apparent that Simul-Ind, CGA, SGA, SVO tend to converge towards scenarios where at least one player abstains from contributing, an occurrence often tagged as 'free-riding'. In contrast, Simul-Co (represented by a yellow circle) and our AgA method (illustrated as a red circle) tend to converge towards scenarios characterized by Algorithm 1 Altruistic Gradient Adjustment (AgA)1: Input: individual losses ℓ = [ℓ 1 , • • • , ℓ n ], collective loss ℓ c , parameters w = [w 1 , • • • , w n ], magnitude of alignment parameter λ 2: ξ ← [grad(ℓ i , w i ) f or (ℓ i , w i ) ∈ (ℓ, w)] 3: ξ c ← [grad(ℓ c , w i ) f or w i ∈ w] 4: ∇H c ← [grad( 1 2 ∥ξ c ∥ 2 , w i ) f or w i ∈ w] 5: λ ← λ × sign 1 d ⟨ξ c , ∇H c ⟩ ⟨ξ, ∇H c ⟩ + ∥∇H c ∥ 2 6: Output: ξ = ξ c + λ(ξ + ∇H c ){Plug into any gradient descent optimizer} enhanced contributions or 'altruism'. The observed results underline that players employing the AgA method exhibit heightened altruistic tendencies relative to those operating under Simul-Co. Further, AgA exhibits a superior alignment towards socially optimal outcome point (1, 1).E.1 Sequential Social Dilemma Games: Cleanup and Harvest SSD games [Vinitsky et al., 2019] implements the Harvest and Cleanup as grid world games.The agents use partially observed graphics observation, which contains a grid of 15 × 15 centered on themselves. Compared to the original Cleanup and Harvest games, SSD games add a fire beam mechanic, where agents can fire on the grid to hit other partners. An agent will gain 1 reward if harvesting an apple and -1 reward if firing a beam. Besides, being hit by a beam will result in a 50 individual reward loss. Under this mechanic, selfish agents can easily use fire to prevent others from harvesting apples to gain more individual rewards but harm social welfare.We utilized PPO algorithm in stable-baselines3 [Raffin et al., 2021] to implement the baselines and our methods, with all the agents using separated policy parameters for Simul-Ind and sharing same policy parameters for other experiments. For SVO, we modify the individual reward to be r iα(1actan sum j,j̸ =i rj ri. To employ CGA and AgAs to PPO training process, we compute both individual and collective PPO-Clip policy losses and subsequently utilize them to calculate the adjusted gradient through automatic differentiation. We don't make changes to the critic loss nor the critic net optimization process.Most experiments were conducted on a node with a Tesla V100 GPU (32GB memory) and 40 CPU cores. The hyper-parameters for PPO training are as follows.• The learning rate is 1e-4• The PPO clipping factor is 0.2.• The value loss coefficient is 1.• The entropy coefficient is 0.001.• The γ is 0.99.• The total environment step is 1e7 for Harvest and 2e7 for Cleanup.• The environment episode length is 1000.• The grad clip is 40. SMAC, often employed as a testbed within the sphere of cooperative MARL, is characterized by the shared reward system among players. Within this paper, we present the MMM2 map transformed as a versatile testbed designed for mixed-motive cooperation. Specifically, as shown in Fig. 5 , MMM2 map encompasses 10 agents and 12 adversaries as enumerated below:• Controlled Agents: Comprised of 1 Medivac, 2 Marauders, and 7 Marines.• Adversaries: Incorporates 1 Medivac, 3 Marauders, and 8 Marines.Moreover, this setting embodies both heterogeneity and asymmetry, where agents within the same team exhibit diverse gaming skills. Additionally, the team compositions, for instance, variation in agent number and type, differ in both opposing factions. Consequently, the intricate gaming mechanism, the presence of heterogeneous players, and the discernible asymmetry conspire to establish an exemplary milieu conducive for the exploration of mixed-motive issues.In order to create a mixed-motive testbed, we adjusted the reward system hailing from the original SMAC environment. With this, we proposed a revamped reward function that features two distinct components: a reward for imposing damage and a penalty associated specifically with agent fatalities. In a move to potentially intensify internal conflicts amongst agents, the reward for imposing damage is solely allocated to the agent responsible for executing the attack on the enemy. Additionally, the death of an agent results in the imposition of an extra penalty. Consequently, this mechanism fosters an environment where agents are predisposed towards individual protection and separate reward acquisition, as opposed to collective cooperation.Formally, the agent i's reward at step t is defined as follows:delta-enemy i = j∈Enemy i hit j at t [(previous-health -current-health) + (previous-shield -current-shield)]The individual reward for Player i is determined as follows: If the player dies, then r i = delta-enemy iβ × penalty; otherwise, if the player survives, then r i = delta-enemy i .Experiment settings. Simul-Ind leverages IPPO with recurrent policies and distinct policy networks. Conversely, Simul-Co employs MAPPO with recurrent policies and consolidated policy networks. The implementation of the IPPO and MAPPO algorithms presented in this paper is founded on the methodology detailed in [Yu et al., 2022] . We utilize gradient adjustment optimization methods, such as CGA and AgAs, derived from MAPPO, implementing gradient adjustments as outlined in Algorithm 1.Most experiments were conducted on a node with two NVIDIA GeForce RTX 3090 GPUs and 32 CPU cores. The hyper-parameters for PPO-based training are as follows.• The learning rate is 5e-4 • The PPO clipping factor is 0.2.• The value loss coefficient is 1.• The entropy coefficient is 0.01.• The γ is 0.99.• The total environment step is 1e7.• The factor β in reward function is 1.• The environment episode length is 400.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: We provide the main code in the supplemental material. Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 


Decision: This work proposes a gradient adjustment technique for aligning individual and collective objectives in cooperative multiagent learning. The authors propose modifying the vanilla collective objective with a term that incorporates individual objectives as well as a consensus term that seeks fixed points of the collective objective. In experiments, this weight on the latter terms is set quite high such that the original collective objective plays less of a role.

The reviewers find the “The paper is theoretically sound. The AgA algorithm is simple, but at the same time well justified” with “Strong experimental results”. The reviewers were initially concerned about runtime, but the authors have argued hessian-vector products are efficient and reported only 2-3x slower runtimes empirically (please include these runtime results somewhere). The Gini coefficient is a nice addition to capture individual agent performance as requested by reviewers x8HM and a1aR. “The most important contribution of this paper is Corollary 4.3”, drawing inspiration from the analysis of SGA.

To improve the paper, it would be helpful to see more ablations on the three terms that make up the update. To align with individual objectives, it seems reasonable to assume the $\xi$ term is required. The question is then how to incorporate $\xi_c$ and $\nabla \mathcal{H}_c$.

Given the reviewers generally felt positive about the paper, we are recommending to accept this submission.

Regarding the [Gemp, Kwon] citations, I would remove the statement “it is not intuitive to explain how individual incentives align with the collective incentives during the optimization process.” In that work, individual vs collective alignment is precisely captured by a reward-sharing matrix during the learning process. One can intuitively measure alignment by measuring the norm of the difference between whatever the current sharing matrix is versus the collective sharing matrix (the ones matrix divided by number of agents).

Reviews: review: paper_summary: paper_summary: The paper investigates the topic of cooperation in a mixed-motive multi-agent setting. They first propose the formulation of a mixed motive game as a differentiable game. Leveraging the structure of the latter, they propose a gradient-based optimization algorithm, AgA. The paper both discusses the theoretical guarantees of AgA, and its effectiveness in an empirical setting. Finally, the authors introduce a modification of the MMM2 map in the StarCraft II game, to further test the empirical performances of AgA.
strength_weakness: strength_weakness: - There are some typos in the paper, and I feel the presentation can be improved. For example, in the Related Work section in page 2, there are some sentences which are incomplete or non-sensical. I am specifically referring to ' While PED-DQN enables agents 79 to incrementally adjust their reward functions for enhanced collaborative action through inter-agent 80 evaluative signal exchanges [Hostallero et al., 2020], Gifting directly reward other agents as part of 81 action space [Lupu and Precup, 2020].' 
- I think the author should discuss more in depth the complexity of this method. How does it scale with respect to the other algorithms proposed? There is no experiment that highlights this in the paper
- There is no discussion of how AgA satisfied individual incentives
questions: questions: - How much slower is AgA compared to the other algorithms? Do you have any empirical experiments which can shows this in practice?

- Do you have any results which show the individual metrics achieved by the agents when using AgA vs other algorithms? It could be valuable to see some plot which shows both the cooperative and individual rewards, and how these vary depending on the value the hyperparameter $\lambda$ is set to

- Did you run the experiments for Figure 3 for more seeds? Do the results still hold?
limitations: limitations: No limitations
paper_summary: paper_summary: This paper introduces a novel optimization method called AGA that employs gradient adjustments to progressively align individual and collective objectives. They prove that this method attracts gradients to stable fixed points of the collective objective while considering individual interests. Their method is empirically validated on sequential social dilemmas games, Cleanup and Harvest, and a high dimensional environment, StarCraft II.
strength_weakness: strength_weakness: - No clear weaknesses
questions: questions: - Figure 1. It’s not clear how to interpret the reward contours graphs. Can you clarify what the x-axis and y-axis? How are the contours formed? They appear to be integers corresponding to the Player’s actions but where is the action space defined? 
- What is the significance of AGA moving along the summit? 
- In Proposition 4.1, what is the gradient without the subscript referring to? 
- L292, how is the collective loss equation determined?
limitations: limitations: Yes.
paper_summary: paper_summary: The paper proposes Altruistic Gradient Adjustment (AgA) which adjusts the gradients of individual and collective losses to align individual and collective objectives. Besides, the authors prove that AgA effectively attracts gradients to stable fixed points of the collective objective while less sacrificing individual interests. Experiments and example illustrations demonstrate the effectiveness of AgA.
strength_weakness: strength_weakness: 1.	AgA introduces an additional adjustment term, which needs to be tuned case by case.
2.	AgA introduces additional computation complexity to compute Hessian-vector products.
3.	It is not clear whether AgA also has the ability to always keep all individual objectives best. For example, in Figure 1(c), Simul-Co could achieve a better reward for Player 2 than AgA while Simul-Co also achieves a higher collective reward than AgA.
4.	There is only one map of SMAC to be tested.
questions: questions: 1. Could the authors compare the running time of AgA with other baselines?
2. Could the authors provide more discussion about AgA and Simul-Co? For example, if Simul-Co could achieve a higher collective reward than AgA like in Figure 1, one can redistribute the collective reward for each agent so that the redistributed individual rewards for each agent can be better than AgA.
3. In Algorithm 1, are the parameters w=[w_1,…,w_n] configured manually or determined by the task itself?
limitations: limitations: NA.
paper_summary: paper_summary: The paper lies in the intersection of multi-agent and game theory. In particular, dealing with "mixed-motive cooperative games". This is, games in which the maximization of individual rewards hampers the maximization of collective rewards. This is, when agents seek to maximize their individual reward/utility, and disregard the collective utility they are worst off as a collective. These type of problems are usually called "social dilemmas" in game theory. The objective is to achieve the 'social optimum', this is, the maximum collective reward a group of agents can obtain, beyond the individual reward.

The methodology proposed in the paper is to align the individual rewards towards the collective maximization rewards via "gradient shaping". This is, move the gradients of the individual reward maximization policy towards the collective reward. In order to shift the individual rewards, the algorithm includes an "alignment" parameter that aligns two optimization problems (individual maximization vs collective optimization) that otherwise, will pull against each other.

Since the collective optimization point is an equilibrium point, the algorithm shows convergence.
strength_weakness: strength_weakness: See below.
questions: questions: I don't think this is fair or true. The mentioned papers do provide substantial theoretical analysis. This is not the way to place your paper.

"On the other hand, several studies focus on auto-matically modifying rewards by learning additional weights to adjust the original objectives [Gemp et al., 2020, Kwon et al., 2023]. However, these studies often suffer from a lack of interpretability and insufficient theoretical analysis regarding the alignment of individual and collective objectives."
limitations: limitations: Authors have addressed the limitations of the paper on the Conclusions section.

Comments: title: Thanks
Thanks
Thanks
Thanks
Main concerns addressed
Thank you to the authors for their response.
comment: I have no further comments and would like to maintain my score. Best of luck with the paper.
Dear Reviewer,

Thank you for your encouraging message and for your valuable suggestions, which have significantly strengthened our work. Thanks~

Best,

All Authors
Dear Reviewer,

Thank you for your feedback and for carefully reviewing our detailed reply. 

Best,
All authors
Dear Reviewer,

Thanks for taking the time to review our response and your valuable input. We appreciate your continued engagement with our work.

Best,

All Authors
Dear Reviewer,

Thank you very much for your positive feedback and for the time you invested in reviewing our manuscript. We are pleased to hear that the additional experiments and the revisions we provided addressed your main concerns. **We will make sure to incorporate these elements to further enhance the clarity and completeness of our work.**

Best,

All Authors
Dear Authors, 
Thanks for the answer and for the additional experiments provided. I suggest the authors to both include the discussion and results provided here on the individual incentive and the computational complexity of AgA in the final version of the manuscript.

Since the main concerns I raised were addressed, I'll increase my score accordingly.
Thank you for the detailed reply. Most of my questions and concerns are addressed. I would like to maintain my score.
Thank you for replying to the rebuttal and including my suggestion into your work. My rating stands.good luck with your paper!
Thank you for your response to my comments. I have read your rebuttal and will provide further comments soon, as needed.