RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Alignment Efficiency of AgA: A Toy Experiment
To address the two-player DMG as outlined in Example 4.1 , we incorporate the AgA method into the fundamental gradient descent algorithm.
The findings reveal that the AgA method is effective in facilitating the alignment of objectives.
The optimization trajectories in both the collective reward landscape and the individual player reward contour are represented in Fig. 1(a) , Fig. 1(b) , and Fig. 1(c) , with brighter colors reflecting higher collective rewards.
In contrast to the objective misalignment exhibited by Simul-Co, AgA successfully aligns the agents’ interests, as evidenced in Fig. 1(b) and Fig 1(c) . Fig. 1(b) depicts the trajectory of AgA meticulously carving its course along the summit of the individual reward contour.
Furthermore, AgA is also shown to improve the reward of Player 2 (as shown in Fig. 1(c) ), despite a convergence rate slower than that of Simul-Co. Evidently, the trajectory of AgA (as depicted in Fig. 1(b) ) takes 100 steps, whereas Simul-Co takes 65 steps.

In addition, Fig. 3 presents a critical comparison between the trajectories of AgA (in red) and AgA without sign alignment (AgA w/o Sign, depicted in purple), as outlined in Corollary 4.3 .
This side-by-side comparison covers 40 steps and features highlighted points every tenth step.
It provides a visual illustration of the efficiency introduced by sign alignment starting from the 14th step in the trajectories of AgA and AgA without sign alignment.
Remarkably, norm gradients are represented by blue arrows, indicating the direction of the fastest updates.
As depicted in the figure, AgA with sign selection is aligned towards the fastest update direction, resulting in AgA progressing approximately 6 steps ahead of AgA without Sign at the end of the trajectory.
Experiments
In this section, we conduct a series of experiments to verify the AgA algorithm.
First, we provide an overview of the experimental settings and clarify the evaluation metrics employed.

Baselines and Evaluation Metrics. A comparison of experiments involves specific baselines such as simultaneous optimization with individual and collective losses (abbreviated as Simul-Ind and Siml-Co), CGA (Mescheder et al., 2017 ) , SGA (Balduzzi et al., 2018 ) and SVO (McKee et al., 2020a ) .
In the two-player public goods game experiment, these techniques are simply integrated into the basic gradient ascent algorithm, whereas, for more intricate settings like Cleanup, Harvest (Leibo et al., 2017 ) , and SMAC (Samvelyan et al., 2019 ) , these baselines are infused within various RL algorithms. Detailed operations for implementing these RL algorithms are further elucidated in their respective sections.

We assess collaborative performance in mixed-motive games using a three-tier approach. Primarily, we employ social welfare to gauge overall team performance. Next, an equality metric is introduced to scrutinize fairness in environments such as Cleanup and Harvest. Lastly, the probability of victory against opposition in an SMAC framework determines the probability of the team’s success. Social Welfare is a commonly accepted metric for evaluating the performance of mixed motivation cooperation, representing the total episodic rewards earned by all agents as SW=\sum_{i=1}^{n}r_{i} .
Social welfare lacks the ability to analyze teamwork in mixed cooperative-competitive contexts. Here, we use the average Win Rate against the built-in computer AI bot in SMAC to assess algorithmic performance.
We also involve the Equality due to the measures like social welfare and win rate are limited to a general performance evaluation, ignoring individual interest.
Therefore, the Gini coefficient, originally intended as an income equality measure (David, 1968 ) , has been modified to assess reward equality in cooperative AI settings. This study utilizes an expedited Gini coefficient calculation method, with an associated equality metric E , formulated as E:=1-G . Each individual’s rewards are arranged in ascending order to create a new payoff vector {\bm{p}} , and the Gini coefficient is computed as G=\frac{2}{n^{2}\bar{{\bm{p}}}}\sum_{i=1}^{n}i\left(p_{i}-\bar{{\bm{p}}}\right) , followed by E=1-G . Here, \bar{{\bm{p}}} denotes the mean of the ranked payoff vector {\bm{p}} and n signifies the total number of players. A larger E value implies greater equality.
Two-Player Public Goods Game
A two-player Public Goods matrix game {\mathcal{G}} is widely utilized to study cooperation in social dilemmas. The game involves players \{1,2\} with parameters \{{\bm{w}}_{1},{\bm{w}}_{2}\} and payoffs \{p_{1},p_{2}\} . The social welfare, denoted as SW=p_{1}+p_{2} . Each player, i.e., i\in\{1,2\} , contributes an amount a_{i} within a budgeted range [0,b] , and the host evenly distributes these contributions as \frac{c}{2}(a_{1}+a_{2}) , where 1<c\leq 2 . Consequently, each player’s payoff p_{i}(a_{1},a_{2}) is estimated as b-a_{i}+\frac{c}{2}(a_{1}+a_{2}) .
In our experiments, we set the budget b to 1 and weight c to 1.5 , with the Nash equilibrium of game at (0,0) .

Results. Table 1 compares averindividual rewards, social welfare, and the equality metric sourced from 50 random runs, each capped at 100 steps. The rows r_{1},r_{2} and SW represent the rewards accrued by each individual player and the collective group respectively. Row E denotes the equality metric.
The table reveals that AgA surpasses benchmark methods in individual rewards and social welfare, and equality metric.
The social optimum of the game is at coordinates (1,1) , which corresponds to a maximum achievable social welfare of 3 . Among all the algorithms, AgA achieves the closest social welfare score to this optimum, with a value of 2.90.
In addition, AgA demonstrates the highest level of equality, with a score of 0.96, when compared to the baseline algorithms.
Additionally, Fig. 6 in Appendix B visualizes the action distributions of these methods, demonstrating convergence points, further supporting that AgA converges more efficiently towards the social optimum.
Sequential Social Dilemma: Cleanup and Harvest
We conduct experiments in sequential social dilemma environments with a group size of five namely, Harvest and Cleanup (Hughes et al., 2018 ) .
In Cleanup, the agents primarily derive rewards from harvesting apples in a nearby orchard, which is contingent upon river pollution levels.
If negligence towards the escalating pollution levels surpasses a threshold, apple growth ceases, hence leading to a conflict of interest between obtaining individual rewards and safeguarding joint welfare.
Harvest’s premise, on the other hand, involves agents receiving rewards by gathering apples, whereby optimal apple regrowth is dependent on the presence of adjacent apples.
A collective dilemma appears when rampant harvesting reduces the apple regeneration rate, thus decrementing overall team rewards.
Simul-Ind, Simul-Co, CGA, and SVO are benchmarked against our AgA methodology in our experiments. The implementation of the Simul-Ind algorithm is based on the IPPO algorithm (de Witt et al., 2020 ) , with all other strategies leveraging the shared parameters of the IPPO algorithm as well.
The joint reward formula used for the calculation of collective loss for Simul-Co, CGA, and AgAs is designed to promote both improved collective performance and equitable behavior among agents, i.e, \sum_{i}\left(r_{i}-\alpha(1-actan\left(\frac{sum_{j,j\neq i}r_{j}}{r_{i}}\right)\right) with constant \alpha . More detailed information on these environments and algorithm execution can be found in Appendix C.1 .

Results. Fig. 4(a) and Fig. 4(c) present the average episodic social welfare comparison of the AgA algorithm and AgA/Sign, excluding the sign selection (Corollary 4.3 ).
These figures also highlight the effect of varying the alignment parameter \lambda .
The prescribed \lambda values for AgA/Sign is 100 for Cleanup and 0.1 for Harvest, resulting in the best performance for each. Failing to select signs in AgA/Sign leads to poorer collective results. Fig. 4(b) and Fig. 4(d) compare the social welfare achieved by AgA, with the optimal \lambda value, with baseline methods. Here, AgA (\lambda=100) in Cleanup and AgA (\lambda=0.1) in Harvest surpass the benchmark methods for the majority of the training process.
Table 2 presents the peak social welfare achieved during training alongside the equality metric. For simplicity, the AgA column denotes the respective optimal \lambda for each scenario. In Cleanup, AgA surpasses the second-best method, Simul-Co, by over 10 points. However, although Simul-Ind shows superior equality performance, it delivers the lowest social welfare. Contrarily, AgA achieves a satisfactory balance between equity and team performance. Analogous conclusions apply to the Harvest, where AgA outperforms baselines in social welfare and equality.
Mixed-motive SMAC
Despite their widespread use in the study of mixed-motive games, sequential social dilemma games can be relatively simplistic compared to other testbeds in the field of MARL, particularly when considering real-world scenarios.
Cooperative Multi-Agent Reinforcement Learning often adopts more intricate environments such as SMAC (Samvelyan et al., 2019 ) . In this game environment, agents must collaborate to defeat their opponents.
We modify the MMM2 map from SMAC as a mixed-motive testbed.
Here, agents command various game entities confronting opposing forces. Unlike the original StarCraft II game’s team-oriented architecture, we instill individual interests in each agent to enhance mixed-motive characteristics. Reward mechanisms have been modified such that an agent’s individual reward aligns with the damage it inflicts on the enemy, replacing the standard practice of distributing total damage across all agents. Penalties for agent removal also reinforce self-preservation, accentuating inherent agent selfishness.
The experimental implementation utilizes the IPPO algorithm (de Witt et al., 2020 ) for Simul-Ind, while employing the MAPPO framework (Yu et al., 2022 ) for Simul-Co, SVO, CGA, and AgA. More details can be found in Appendix C.2 .

Results. Fig. 5(a) compares average win rates for various \lambda parameters in AgA, also encompassing AgA without sign selection, as stated in Corollary 4.3 . The results suggest comparable performance for \lambda=1 and 100 , with a lower value of 0.1 trailing noticeably. Notably, AgA with \lambda=0.1 underperforms, even when contrasted with AgA’s ablation study. Fig. 5(b) benchmarks the win rate for AgA (\lambda=100) against Simul-Ind, Simul-Co, CGA, and SVO. Within this context, Simul-Ind, implemented via IPPO, struggles with the mixed-motive MMM2 task. Impressively, our AgA (\lambda=100) method achieves a win rate peak of 0.85 (standard deviation of 0.09 ), significantly outperforming the SVO method that only nets 44.55 .
Conclusion
In this paper, we propose AgA, an gradient adjustment method, specifically designed to align individual and collective objectives within mixed-motive cooperation scenarios.
To materialize this, we first model the mixed-motive game as a differentiable game, thereby facilitating the analysis of learning dynamics.
Thereafter, we analyze the optimization paths in a toy differentiable mixed-motive game.
We reveal that optimization techniques that heavily rely on collective loss suffer from inherent limitations. These approaches often overlook individual preferences, and existing gradient adjustment methods are inadequate for addressing mixed-motive cooperation challenges.
Our AgA strategy efficiently bridges the gap between individual and collective objectives.
The optimization paths further support our theoretical claims by demonstrating that careful selection of the sign of the alignment weight enables convergence towards stable fixed points while escaping unstable ones. Comprehensive experiments conducted on mixed-motive testbeds and modified SMAC environments provide further evidence of the superior performance of AgA. It consistently outperforms existing baselines across multiple evaluation metrics.

Limitations and Future Work. The experiments conducted in this study were confined to a single map of the SMAC environment. Our future objectives include investigating the mixed-motive cooperative mechanism in real-world scenarios and developing a more complex mixed-motive testbed.
In this paper, while we strive to align the objectives of both individuals and the collective, our primary objective is to converge towards a stable equilibrium of the collective objective.
For future research, we intend to broaden our comprehension of how individual objectives interact in mixed-motive games, placing a greater emphasis on understanding their dynamics in mixed-motive cooperation.
Impact Statements
This paper presents a novel approach in the field of multi-agent learning, aimed at advancing collaborations with mixed motivations.
The proposed framework addresses a fundamental challenge in machine learning, wherein agents with diverse objectives are required to cooperate to achieve collective goals.
The work described in this paper holds significant potential for various societal applications.
By enabling effective cooperation among agents with different motivations, our approach can be leveraged in domains such as games, autonomous systems, smart cities, and decentralized resource allocation.
Such advancements have numerous societal consequences, including improved efficiency in transportation systems, optimized resource allocation in energy grids, and enhanced decision-making in complex multi-agent environments.
Furthermore, our findings contribute to the broader field of machine learning by offering insights into managing mixed-motivation scenarios.
Consequently, our work paves the way for future research and innovation, driving progress in multi-agent learning and its applications across various domains.
B.1 Result Visualization of Two-player Public Goods Game
Fig. 6 depicts the actions produced by various methods in a two-player public game.
The action generated by each method is represented by a circle, which is color-coded according to the corresponding method.
To ensure fairness, each method’s updates are constrained to a maximum of 100 steps.
These methods include the individual loss-driven simultaneous optimization (Simul-Ind), two variants of gradient adjustment methods termed as CGA and SGA, and the simultaneous optimization leveraging collective loss (Simul-Co) alongside our proposed AgA, where \lambda=1 .
Each of these methods is fundamentally underpinned by the gradient ascent algorithm.
The illustration consolidates data from 50 randomized runs initialized from diverse starting points.
The ‘X’ demarcated circles indicate the average actions mapped to each method.
From the figure, it’s apparent that Simul-Ind, CGA, SGA, SVO tend to converge towards scenarios where at least one player abstains from contributing, an occurrence often tagged as ‘free-riding’.
In contrast, Simul-Co (represented by a yellow circle) and our AgA method (illustrated as a red circle) tend to converge towards scenarios characterized by enhanced contributions or ’altruism’.
The observed results underline that players employing the AgA method exhibit heightened altruistic tendencies relative to those operating under Simul-Co. Further, AgA exhibits a superior alignment towards socially optimal outcome point (1,1) .
C.1 Social Dilemma Games: Cleanup and Harvest
SSD games (Vinitsky et al., 2019 ) implements the Harvest and Cleanup as grid world games.The agents use partially observed graphics observation, which contains a grid of 15\times 15 centered on themselves. Compared to the original Cleanup and Harvest games, SSD games add a fire beam mechanic, where agents can fire on the grid to hit other partners. An agent will gain 1 reward if harvesting an apple and -1 reward if firing a beam. Besides, being hit by a beam will result in a 50 individual reward loss. Under this mechanic, selfish agents can easily use fire to prevent others from harvesting apples to gain more individual rewards but harm social welfare.

We utilized PPO algorithm in stable-baselines3 (Raffin et al., 2021 ) to implement the baselines and our methods, with all the agents using separated policy parameters for Simul-Ind and sharing same policy parameters for other experiments. For SVO, we modify the individual reward to be r_{i}-\alpha(1-actan\left(\frac{sum_{j,j\neq i}r_{j}}{r_{i}}\right) . To employ CGA and AgAs to PPO training process, we compute both individual and collective PPO-Clip policy losses and subsequently utilize them to calculate the adjusted gradient through automatic differentiation. We don’t make changes to the critic loss nor the critic net optimization process.

The hyper-parameters for PPO training are as follows.

The learning rate is 1e-4

The PPO clipping factor is 0.2.

The value loss coefficient is 1.

The entropy coefficient is 0.001.

The \gamma is 0.99.

The total environment step is 1e7 for Harvest and 2e7 for Cleanup.

The environment episode length is 1000.

The grad clip is 40.
C.2 Mixed-motive StarCraft II
SMAC, often employed as a testbed within the sphere of cooperative MARL, is characterized by the shared reward system among players. Within this paper, we present the MMM2 map transformed as a versatile testbed designed for mixed-motive cooperation.
Specifically, MMM2 map encompasses 10 agents and 12 adversaries as enumerated below:

Controlled Agents: Comprised of 1 Medivac, 2 Marauders, and 7 Marines.

Adversaries: Incorporates 1 Medivac, 3 Marauders, and 8 Marines.

Moreover, this setting embodies both heterogeneity and asymmetry, where agents within the same team exhibit diverse gaming skills. Additionally, the team compositions, for instance, variation in agent number and type, differ in both opposing factions.
Consequently, the intricate gaming mechanism, the presence of heterogeneous players, and the discernible asymmetry conspire to establish an exemplary milieu conducive for the exploration of mixed-motive issues.

In order to create a mixed-motive testbed, we adjusted the reward system hailing from the original SMAC environment. With this, we proposed a revamped reward function that features two distinct components: a reward for imposing damage and a penalty associated specifically with agent fatalities.
In a move to potentially intensify internal conflicts amongst agents, the reward for imposing damage is solely allocated to the agent responsible for executing the attack on the enemy. Additionally, the death of an agent results in the imposition of an extra penalty.
Consequently, this mechanism fosters an environment where agents are predisposed towards individual protection and separate reward acquisition, as opposed to collective cooperation.

Formally, the agent i ’s reward at step t is defined as follows:

delta-enemy_{i}=\sum_{j\inEnemy\atop i hit j at t}[(previous-health-current-health)+(previous-shield-current-shield)]

The individual reward for Player i is determined as follows: If the player dies, then r_{i}=\text{delta-enemy}_{i}-\beta\times\text{penalty} ; otherwise, if the player survives, then r_{i}=\text{delta-enemy}_{i} .

Experiment settings. Simul-Ind leverages IPPO with recurrent policies and distinct policy networks. Conversely, Simul-Co employs MAPPO with recurrent policies and consolidated policy networks.
The implementation of the IPPO and MAPPO algorithms presented in this paper is founded on the methodology detailed in (Yu et al., 2022 ) .
We utilize gradient adjustment optimization methods, such as CGA and AgAs, derived from MAPPO, implementing gradient adjustments as outlined in Algorithm 1 .

The hyper-parameters for PPO-based training are as follows.

The learning rate is 5e-4

The PPO clipping factor is 0.2.

The value loss coefficient is 1.

The entropy coefficient is 0.01.

The \gamma is 0.99.

The total environment step is 1e7.

The factor \beta in reward function is 1.

The environment episode length is 400.
(a) Collective Reward Landscape; (b) Reward Contour of Player 1; (c) Reward Contour of Player 2; Trajectories of optimization in a two-player DMG (as delineated in Example 4.1 ).
Fig. 1(a) displays the trajectories over the collective reward landscape - deeper orange equates to higher rewards.
Remarkably, only Simul-Co and AgA make successful strides towards the social optimum. Fig. 1(b) and Fig. 1(c) delineate trajectories on the individual reward contour, underscoring Simul-Co’s neglect for Player 1’s interests even though it achieves higher rewards with fewer steps (65 steps versus 100 steps of AgA). Conversely, our AgA optimizes along the summit of Player 1’s reward while also maximizing the collective reward, demonstrating a successful alignment .
Figure 2 : Illustration of Corollary 4.3 . In case 1, within an unstable fixed point’s neighborhoods, an appropriate selection of the \lambda sign push AgA to evade the unstable fixed point and pull towards a stable fixed point in its neighborhoods as shown in case 2.; Illustration of Corollary 4.3 . In case 1, within an unstable fixed point’s neighborhoods, an appropriate selection of the \lambda sign push AgA to evade the unstable fixed point and pull towards a stable fixed point in its neighborhoods as shown in case 2.
(a) Reward Contour of Player 1; (b) Reward Contour of Player 2; The comparison between AgA (shown in red) and AgA without sign alignment (AgA w/o Sign, in purple) trajectories spans 40 steps, marked at every tenth step. Norm gradients are represented with blue arrows. Starting from the 14th step, sign alignment pull the update direction. Consequently, AgA is about 6 steps ahead of AgA w/o Sign at the trajectory’s end.
(a) Cleanup: AgA with different \lambda and AgA/Sign; (b) Cleanup: comparison between AgA and baselines; (c) Harvest: AgA with different \lambda and AgA/Sign; (d) Cleanup: comparison between AgA and baselines; Comparison of Episodic Social Welfare: our proposed AgA methods along with baselines on Cleanup (figure a and b) and Harvest (figure c and d) tasks. Figure (a) and (c) provide an illustration of varying alignment parameter \lambda values in AgA, as well as AgA sans sign selection (AgA/Sign), as applied to Cleanup and Harvest tasks respectively. Subfigures (b) and (d) exhibit the social welfare obtained by AgA utilizing the optimal \lambda value, in comparison to those secured by Simul-Ind, Simal-Co, CGA, SGA and SVO. Bold lines represent the average of the social welfare across three seeds, with the encompassing shaded areas denoting the standard deviation.
(a) MMM2: AgA with various \lambda and AgA/Sign; (b) MMM2: comparison between AgA and baselines; The comparison of win rates on the mixed-motive MMM2 map of SMAC.
The left figure shows results of varying alignment parameter \lambda values in AgA, as well as AgA sans sign selection (AgA/Sign),
The right figure compare the win rate of AgA (\lambda=100) with baselines.
The bold lines illustrate the mean of the win rate over 3 seeds, while the encompassing shaded regions depict the standard deviation.
Figure 6 : The scatter of actions in a two-player public goods game achieved through different optimization methods. Each circle represents the position attained within a maximum of 100 steps, with the color indicating the corresponding method.
The ’X’ mark represents the mean actions of 50 random runs. With the exception of Simul-Co, the baseline methods converge towards the Nash equilibrium (0,0). Notably, while both AgA and Simul-Co display altruistic behavior, the actions of AgA are more tightly clustered around the (1,1) point compared to Simul-Co.; The scatter of actions in a two-player public goods game achieved through different optimization methods. Each circle represents the position attained within a maximum of 100 steps, with the color indicating the corresponding method.
The ’X’ mark represents the mean actions of 50 random runs. With the exception of Simul-Co, the baseline methods converge towards the Nash equilibrium (0,0). Notably, while both AgA and Simul-Co display altruistic behavior, the actions of AgA are more tightly clustered around the (1,1) point compared to Simul-Co.