Introduction
Multi-agent cooperation primarily focuses on learning how to promote collaborative behavior in shared environments.
Typically, research divides multi-agent cooperation into two prominent areas, pure-motive cooperation and mixed-motive cooperation (Du et al., 2023 ; McKee et al., 2020a ) .
Recent progress in cooperative Multi-Agent Reinforcement Learning (MARL) has primarily focused on addressing pure-motive cooperation, which also known as a common-payoff game, represents situations where individual goals fully align with collective objectives (Yu et al., 2022 ; Wang et al., 2020 ; Zhong et al., 2023 ; Li et al., 2024 ) .
Despite pure-motive cooperation, mixed-motive cooperation, which is typically defined by the imperfect alignment between individual and collective rationalities, is more common in real-world scenarios (Rapoport, 1974 ; McKee et al., 2020a ) .

Recent mixed-motive cooperative studies in MARL have largely employed hand-crafted designs to promote collaboration.
Incorporating additional mechanisms to align objectives appears to be a popular avenue, including aspects such as reputation (Anastassacos et al., 2021 ) , norms (Vinitsky et al., 2023 ) , and contracts (Hughes et al., 2020 ) .
Another prevalent method draws on intrinsic motivation to align individual and collective objectives, augmenting altruistic collaboration by integrating heuristic knowledge into the incentive function.
Conventionally, some works accumulate individual rewards with the group to cultivate altruistic conduct (Hostallero et al., 2020 ; Peysakhovich & Lerer, 2018 ; Apt & Schäfer, 2014 ) .
Furthermore, some studies derive more sophisticated preference signals from the rewards of other agents (Hughes et al., 2018 ; McKee et al., 2020a ) . Additionally, several approaches aim to learn the potential influences of an agent’s actions on others (Yang et al., 2020 ; Jaques et al., 2019 ; Lu et al., 2022 ) .
Most of these algorithms are highly dependent on carefully crafted designs, necessitating significant human expertise and detailed domain knowledge. However, they lack in-depth theoretical analysis concerning alignment and convergence.

In this work, we first present the differentiable mixed-motive game (DMG), premised on the assumption that the players’ loss functions are at least twice differentiable.
The proposed formulation offers an efficient tool for analyzing the dynamics of gradient-based methods when optimizing mixed-motive objectives.
Furthermore, we explore the optimization trajectories of simultaneous optimization methods and gradient adjustment methods ( consensus optimization (Mescheder et al., 2017 ) and symplectic gradient adjustment (Letcher et al., 2019 ) ) in the context of a two-player DMG, as shown in Fig. 1 .
First, we observe that gradient adjustment methods are ineffective in addressing mixed-motive problems.
These methods tend to prioritize stability at the expense of mitigating individual loss, which conflicts with the interests of individual participants (Chen et al., 2023 ) .
While simultaneous optimization with collective loss successfully enhances collective interests, it seems to neglect individual interests.

To address the issue, we propose an A ltruistic G radient A djustment ( AgA ) optimization algorithm, designed to align individual and collective objectives from a gradient perspective.
We provide a theoretical proof to suggest that appropriately choosing the sign of the alignment weight of the gradient terms could speed up the convergence of the AgA optimization strategy towards stable fixed point of collective objective while effectively bypassing unstable ones.
The trajectory visualizations in Fig. 1 , along with the sign selection ablation study trajectories presented in Fig. 3 , serve to demonstrate successful alignment of objectives.

To further assess the effectiveness of our proposed AgA approach, we perform an assortment of experiments on a two-player public goods game and two sequential social dilemma games (Cleanup and Harvest) (Leibo et al., 2017 ) . Given the relatively simple nature of these standard test scenarios used in mixed-motive cooperation contexts, we chose to modify the widely used SMAC environment (Samvelyan et al., 2019 ) to create a more complex mixed-motive cooperative setting. Evaluation based on various performance metrics including social welfare, equality, and win rate, suggests that the AgA approach surpasses the performance of baseline methods.

The contribution of the paper can be summarized as follows:
(1) We introduce the differentiable mixed-motive game (DMG) and propose Altruistic Gradient Adjustment (AgA) algorithm to align individual and collective objectives from a gradient perspective.
(2) We theoretically prove that careful selection of the alignment weight’s sign can expedite convergence in AgA and effectively avoid unstable fixed points.
(3) Visualizations based on a two-player DMG show that our method efficiently aligns individual and collective objectives. Further experiments conducted on mixed-motive testbeds and modified SMAC environments underscore the superior performance of our AgA method in several measurement metrics compared to baselines.