Differentiable Mixed-motive Game
We first formulate the mixed-motive game as a differentiable game.
Specifically, the differentiable mixed-motive game (DMG) is defined as a tuple ({\mathcal{N}},{\bm{w}},{\bm{\ell}}) , where \ell_{i}\in{\bm{\ell}} is at least twice the differentiable loss function for the agent i .
Differentiable losses exhibit the mixed motivation property: minimization of individual losses can result in a conflict between individuals or between individual and collective objectives (e.g., maximizing social welfare or beating opponents in basketball matches).
The primary objective of DMG is to optimize the collective objective by minimizing collective loss, denoted as \ell_{c} , while simultaneously taking into account the need to minimize individual objectives, represented by {\bm{\ell}} .
Before delving into our interest alignment methods, let us first introduce some notations.
We use {\bm{\xi}}_{c}({\bm{w}}) to refer the gradient of collective loss: {\bm{\xi}}_{c}({\bm{w}})=\left(\nabla_{\mathbf{w}_{1}}\ell_{c},\ldots,\nabla_{\mathbf{w}_{n}}\ell_{c}\right).

Nonetheless, direct optimization of individual or collective losses can lead to convergence issues and a mismatch between individual and collective interests.
It is improbable that by minimizing only the individual loss of each agent, a collective minimum will be achieved (Leibo et al., 2017 ; McKee et al., 2020b ) .
On the other hand, optimizing collective loss might produce improved collective outcomes, but at the risk of overlooking individual interests.
This phenomenon is also demonstrated in Example 4.1 .
Furthermore, the local convergence of the gradient descent on singular collective functions is not always guaranteed (Balduzzi et al., 2018 ) . This also typifies the problem of credit assignment in MARL.

Alignment Dilemma in DMG. Here, we provide an example of a two-player differentiable mixed-motive game and the optimization results of the aforementioned methods.

Consider a two-player DMG with \ell_{1}(a_{1},a_{2})=-sin(a_{1}a_{2}+a_{2}^{2}) and \ell_{2}(a_{1},a_{2})=-[cos(1+a_{1}-(1+a_{2})^{2})+a_{1}a_{2}^{2}] .
The rewards for the two players are the negation of their respective losses.

We provide a visual representation of optimization trajectories using a series of methods to investigate the learning dynamics involved in resolving the toy two-player differentiable mixed-motive game.
In particular, we first implement simultaneous optimization of individual losses (Simul-Ind) with respect to each parameter, defined by the learning rule {\bm{w}}_{i}={\bm{w}}_{i}-\gamma{\bm{\xi}}_{i} , for i\in\{1,2\} , where {\bm{\xi}}_{i}=\nabla_{{\bm{w}}_{i}}\ell_{i} .
Simultaneous optimization of collective loss (Simul-Co) replaces the individual gradient with the collective gradient {\bm{\xi}}_{c} of collective loss \ell_{c}=\ell_{1}+\ell_{2} .
Furthermore, in order to investigate the learning dynamics of prevalent gradient modification optimization approaches such as SGA and CGA for the two-player differentiable mixed-motive game, we modify the learning rule by integrating the respective adjusted gradient as delineated in Section 3.1 .

Fig. 1(a) shows the optimization paths on the collective reward landscapes, with every path starting from the front-bottom of the landscape.
The collective reward is defined as social welfare, i.e., the sum of individual rewards.
As shown in the figure, Simul-Ind, CGA , and SGA converge to unstable points or local maxima.
Simul-Co effectively navigates towards the apex of the collective reward landscape as depicted in Fig. 1(a) .
However, Simul-Co is ineffective in aligning individual and collective objectives, leading to the neglect of individual interests.
As depicted in Fig. 1(b) , the update trajectory navigates through the crests and troughs of Player 1’s reward contour. The trajectory suggests a disregard for Player 1’s preferences, signifying that the updates are predominantly driven by the overarching collective goal.