Altruistic Gradient Adjustment
In this section, we propose the alignment of individual and collective interests towards achieving stable fixed points of collective objectives within the context of mixed-motive cooperation.
Revisit that the learning dynamics of simultaneous optimization on collective loss is given by {\bm{w}}={\bm{w}}-\gamma{\bm{\xi}}_{c}, where {\bm{\xi}}_{c} represents the gradients of collective loss \ell_{c} respectively, with respect to the parameters of each player {\bm{w}}_{i}\in{\bm{w}} .
Although Simul-Co efficiently enhances collective performance, its shortcomings lie in not guaranteeing stable fixed points (Balduzzi et al., 2018 ) and overlooking individual interests as discussed in Example 4.1 .

To tackle this challenge, we introduce the Altruistic Gradient Adjustment (AgA) method with the goal of aligning individual and collective objectives while seeking stable fixed points.
Unlike existing gradient adjustment methods (Mescheder et al., 2017 ; Balduzzi et al., 2018 ; Letcher et al., 2019 ; Chen et al., 2023 ) that primarily focus on identifying the stable fixed points of individual losses, our proposed AgA method integrates the individual gradient into the collective gradient and ensures convergence to stable fixed points of the collective loss.
Specifically, AgA is defined as follows.

Altruistic gradient adjustment (AgA) extends the gradient term in the learning dynamic as

\tilde{{\bm{\xi}}}:={\bm{\xi}}_{c}+\lambda{\bm{\xi}}_{adj}={\bm{\xi}}_{c}+\lambda\left({\bm{\xi}}+{\bm{H}}_{c}^{T}{\bm{\xi}}_{c}\right), (2)

where \lambda\in\mathbb{R} is alignment parameter, \lambda{\bm{\xi}}_{adj} is called adjustment term. In {\bm{\xi}}_{adj} , {\bm{\xi}}_{c} and {\bm{H}}_{c} is the gradient vector and Hessian matrix of the game about collective loss.

Note that the Hessian matrix {\bm{H}}_{c} is a symmetric matrix.
To simplify the notation, we use \nabla{\mathcal{H}}_{c} to denote the term \nabla_{\bm{w}}\left(\frac{1}{2}\|{\bm{\xi}}_{c}\|^{2}\right) , i.e., {\bm{H}}_{c}^{T}{\bm{\xi}}_{c} .

Intuitively, AgA is designed to involve both collective and individual objectives, in order to harmonize the interests of individuals and the team.
The alignment parameter \lambda adjusts the balance between the individual and the collective objective.
Furthermore, we minimize {\mathcal{H}}_{c}=\frac{1}{2}\|{\bm{\xi}}_{c}\|^{2} to converge to a fixed point.
This objective is reflected in the gradient term {\bm{H}}_{c}^{T}{\bm{\xi}}_{c} in \tilde{{\bm{\xi}}} .
Assuming that {\bm{H}}_{c} is invertible, we can deduce that \nabla{\mathcal{H}}_{c}={\bm{H}}_{c}^{T}{\bm{\xi}}_{c}=0 , if and only if {\bm{\xi}}_{c}=0 .
Therefore, minimizing {\mathcal{H}}_{c} will guarantee the convergence to a fixed point (Letcher et al., 2019 ) .

Impact of the Sign of \lambda . While the modified AgA gradient introduces additional complexity, we theoretically demonstrate that an appropriate choice of the sign of \lambda ensures that the AgA gradient \tilde{{\bm{\xi}}} is attracted towards a stable fixed point. In contrast, when dealing with an unstable fixed point, AgA is pushed away the point.

Before we delve into the corollary of the sign selection for \lambda , we first establish some foundational definitions. The inner product of vectors {\bm{a}} and {\bm{b}} are denoted by \langle{\bm{a}},{\bm{b}}\rangle . If the Hessian matrix {\bm{H}} is non-negative-definite, \langle{\bm{\xi}}_{c},\nabla{\mathcal{H}}\rangle\geq 0 for a non-zero {\bm{\xi}}_{c} . Analogously, if {\bm{H}} is negative-definite, \langle{\bm{\xi}}_{c},\nabla{\mathcal{H}}\rangle<0 for a non-zero {\bm{\xi}}_{c} . Lastly, the angle between the two vectors {\bm{a}} and {\bm{b}} is denoted by \theta({\bm{a}},{\bm{b}}) .

The following corollary states that AgA with a suitable sign of \lambda will bring the adjusted gradient \tilde{{\bm{\xi}}} point closer to a stable fixed point and escape from unstable fixed points.

If we let the sign of \lambda satisfy \lambda\cdot\langle{\bm{\xi}}_{c},\nabla{\mathcal{H}}_{c}\rangle(\langle{\bm{\xi}},\nabla{\mathcal{H}}_{c}\rangle+\|\nabla{\mathcal{H}}_{c}\|^{2})\geq 0 , the optimization process using the AgA method exhibit following attributes.
Primarily, in neighborhood of fixed point of collective objective, if the point is stable ( {\bm{H}}_{c}\succeq 0 ), the AgA gradient will be pulled toward this point, which means that \theta(\tilde{{\bm{\xi}}},\nabla{\mathcal{H}}_{c})\leq\theta({\bm{\xi}}_{c},\nabla{\mathcal{H}}_{c}) . On the other hand, if the point represents unstable equilibria where {\bm{H}}_{c}\prec 0 , the AgA gradient will be pushed out of the point, indicating that \theta(\tilde{{\bm{\xi}}},\nabla{\mathcal{H}}_{c})\geq\theta({\bm{\xi}}_{c},\nabla{\mathcal{H}}_{c}) .

The proof is provided in Appendix A . Fig. 2 presents a illustrative visualization of Corollary 4.3 .
The figure depicts two scenarios: the neighborhoods of stable and unstable fixed point (denoted by a star), respectively.
In the neighborhood of an unstable fixed point of the collective objective, as depicted in Case 1, our AaA gradient \tilde{{\bm{\xi}}} is pushed out of the region, resulting in a faster escape compared to the original collective gradient {\bm{\xi}}_{c} . This behavior is exemplified by the green arrow.
Conversely, Case 2 illustrates the scenario of a stable fixed point, wherein our AaA gradient \tilde{{\bm{\xi}}} is pulled towards the stable equilibrium, exemplified by the red arrow.

Implementation of MARL Algorithms with AgA. The concrete implementation of altruistic gradient adjustment is elaborated in Algorithm 1 .
AgA can be easily incorporated into any centralized training decentralized execution (CTDE) framework of MARL.
In a typical CTDE framework, each agent, denoted as i , strives to learn a policy that employs local observations to prompt a distribution over personal actions.
The unique characteristic during the centralized learning phase is that agents gain supplementary information from their peers that is inaccessible during the execution stage.
Furthermore, a centralized critic often operates in tandem in CTDE settings, leveraging shared information and evaluating individual agent policies’ potency.
To incorporate our proposed AgA algorithm into CTDE frameworks, supplementary reward exchanges between agents are required, offering the essential inputs for Algorithm 1 .
The individual losses are calculated from individual rewards conforming to established MARL algorithm framework, such as IPPO (de Witt et al., 2020 ) or MAPPO (Yu et al., 2022 ) .
Here, the collective loss \ell_{c} is calculated according to individual rewards; for example, it could be a sum of individual rewards or an elementary transformation thereof.
Furthermore, the AgA algorithm requires a magnitude value of the alignment parameter \lambda , which should be a positive entity.
The sign of \lambda is then calculated in line 5 of Algorithm 1 and is based on each gradient component calculated in lines 2-4.
Subsequently, the algorithm produces the adjusted gradient \tilde{{\bm{\xi}}} . \tilde{{\bm{\xi}}} is then distributed across each corresponding parameter, and optimization is carried out using a suitable optimizer such as Adam optimizer (Kingma & Ba, 2015 ) .