Related Work
Mixed-motive cooperation. Mixed-motive cooperation refers to scenarios where the group’s objectives are sometimes aligned and at other times conflicted (Philip S. Gallo & McClintock, 1965 ) .
Recently, there has been a surge in academic interest in the Sequential Social Dilemma (SSD) (Leibo et al., 2017 ) , which expands the concept from its roots in matrix games (Macy & Flache, 2002 ) to Markov games.
Prosocial (Peysakhovich & Lerer, 2018 ) improves collective performance by blending individual rewards to redraft the agent’s overall utility.
While PED-DQN enables agents to incrementally adjust their reward functions for enhanced collaborative action through inter-agent evaluative signal exchanges (Hostallero et al., 2020 ) ,
Gifting directly reward other agents as part of action space (Lupu & Precup, 2020 ) .
Inequity aversion further integrates the concept into Markov games by adding envy (disadvantageous inequality) and guilt (advantageous inequality) rewards to the original individual rewards (Hughes et al., 2018 ) .
Simultaneously, Social Value Orientation (SVO) strategy introduces a unique shared rewards-based compensation approach, encouraging behavior modifications in line with interdependence theory (McKee et al., 2020b ) .
The LIO strategy bypasses the need to modify extrinsic rewards by empowering an agent to directly influence its partner’s actions (Yang et al., 2020 ) .
Meanwhile, other studies introduce new cooperative mechanisms such as incorporating a reputation model as an intrinsic reward (McKee et al., 2023 ) . The CNM strategy delineates acceptable and unacceptable behaviors using classifiers as social norms, aiding agents to learn advantageous norms for improved returns per agent (Vinitsky et al., 2023 ) .
Despite these advances, many current studies lack both cost-efficiency in design and theoretical analysis of alignment and convergence. To address this, our study applies a gradient perspective to align goals and further investigates the learning dynamics of our proposed method.

Gradient-based Methods. Our proposed AgA is fundamentally a gradient adjustment methodology, making gradient-based optimization methods highly relevant to the context of this paper.
Methods based on gradient have been developed to find stationary points, such as the Nash equilibrium or stable fixed points.
Optimistic Mirror Descent leverages historical data to extrapolate subsequent gradients (Daskalakis et al., 2018 ) , while Gidel et al. ( 2020 ) extends this concept by advocating averaging methodologies and variants of extrapolation techniques.
Consensus gradient adjustment, or consensus optimization, is a technique that embeds a consensus agreement term within the gradient to ensure its convergence (Mescheder et al., 2017 ) .
Learning with Opponent-Learning Awareness (LOLA) utilizes information from other players to compute one player’s anticipated learning steps (Foerster et al., 2018 ) .
Subsequently, Stable Opponent Shaping (SOS) (Letcher et al., 2021 ) and Consistent Opponent-Learning Awareness (COLA) (Willi et al., 2022 ) methods have enhanced the LOLA algorithm, targeting convergence assurance and inconsistency elimination, respectively.
Symplectic Gradient Adjustment (SGA) alters the update direction towards the stable fixed points based on a novel decomposition of game dynamics (Balduzzi et al., 2018 ; Letcher et al., 2019 ) .
Recently, Learning to Play Games (L2PG) ensures convergence towards a stable fixed point by predicting updates to players’ parameters derived from historical trajectories (Chen et al., 2023 ) .
However, these methods, while focusing on zero-sum or general sum games, could potentially act counter to their individual interests, as they may prioritize stability over minimizing personal loss.
Our research specifically targets the mixed-motive setting with the aim of reconciling individual and collective objectives.