Fair Cooperation in Mixed-Motive Games via Conflict-Aware Gradient Adjustment
Authors: Woojun Kim
Katia P. Sycara
Venue: NeurIPS 2025 spotlight ()
Keywords: Fairness
Cooperation
Mixed-Motive Games
Multi-agent Reinforcement Learning
Gradient adjustment
Primary area: social_and_economic_aspects_of_machine_learning
TL;DR: We propose FCGrad, a conflict-aware gradient adjustment method that improves social welfare while ensuring fairness in multi-agent mixed-motive settings by resolving conflicts between individual and collective objectives.

Abstract: Multi-agent reinforcement learning in mixed-motive settings presents a fundamental challenge: agents must balance individual interests with collective goals, which are neither fully aligned nor strictly opposed. To address this, reward restructuring methods such as gifting and intrinsic motivation have been proposed. However, these approaches primarily focus on promoting cooperation by managing the trade-off between individual and collective returns, without explicitly addressing fairness with respect to agents’ task-specific rewards. In this paper, we propose an adaptive conflict-aware gradient adjustment method that promotes cooperation while ensuring fairness in individual rewards. The proposed method dynamically balances policy gradients derived from individual and collective objectives in situations where the two objectives are in conflict. By explicitly resolving such conflicts, our method improves collective performance while preserving fairness across agents. We provide theoretical results that guarantee monotonic non-decreasing improvement in both the collective and individual objectives and ensure fairness. Empirical results in sequential social dilemma environments demonstrate that our approach outperforms baselines in terms of social welfare, while maintaining fairness.

Fair Cooperation in Mixed-Motive Games via Conflict-Aware Gradient Adjustment
Woojun Kim Robotics Institute
Katia Sycara Robotics Institute
Carnegie Mellon University Pittsburgh, PA 15213 woojunkGandrew. cmu. edu
Carnegie Mellon University
Pittsburgh, PA 15213 sycaradandrew, cmu edu
Abstract
Multi-agent reinforcement learning in mixed-motive settings presents a fundamen- tal challenge: agents must balance individual interests with collective goals, which are neither fully aligned nor strictly opposed. To address this, reward restructuring methods such gifting and intrinsic motivation have been proposed. However, these approaches primarily focus on promoting cooperation by managing the trade- off between individual and collective returns, without explicitly addressing fairesss with respect to agents' task-specific rewards. In this paper, we propose an adaptive conflict-aware gradient adjustment method that promotes cooperation while ensur- ing fairness in individual rewards. The proposed method dynamically balances policy gradients derived from individual and collective objectives in situations where the two objectives are in conflict. By explicitly resolving such conflicts, our method improves collective performance while preserving fairness across agents. We provide theoretical results that guarantee monotonic non con-deeresiing improve- ment both the collective and individual objectives and ensure fairness. Empirical results in sequential social dilemma environments demonstrate that our approach outperforms baselines terms of social welfare, while maintaining fairness.
Introduction
agget reinforcement learning (MARL) aims train multiple agents maximize cumulative rewards in a given task. Depending on the reward structure, MARL is typically categorized into three settings: cooperative, adversarial, and mixed-motive. In the mixed-motive setting, agents' rewards are neither fully aligned (as in cooperative settings) nor entirely opposed (as in adversarial settings), necessitating that each agent balances self-interest with the collective interest. This mixed- motive setting is frequently encountered in real-world applications. For example, in traffic control systems, each agent (c.g., a local intersection controller) may aim to minimize local congestion, which can conffict with global traffic flow optimization if not coordinated. A similar tension happens in sequential social dilemmas (SSDs) such as Cleanup or Harvest [18), where agents must invest in public goods (e.g., cleaning waste harvesting resources judiciously) that benefit the group but do not yield immediate individual rewards.
However, achieving such balance in mixed- mooede settings is inherently challenging. Excessively selfish behavior by agents can deteriorate collective welfare, which, in turn, negatively impacts each agent's 's own return- cerennnnn vicious cycle that ultimately harms all participants. Additionally, in some scenarios, certain agents must sacrifice their own returns improve the collective outcome, potentially leading to unfairness. Conversely, tasks that require cooperation. Therefore, is crucial to enhance collective outcome while ensuring faireess by appropriately balancing individual and collective interests.
excessive focus faimesss can hinder learning in
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

In mixed-motive settings, many approaches adopt reward restructuring by incorporating intrinsic rewards such as social influence [12]. formal contracts (9) gifting [3 and inequity aversion [TO] These methods primarily aim to maximize the collective return by mediating the trade-off between self-interest and collective interests. For example, gifting mechanisms promote cooperation by enabling agents to share portion of their rewards with others. However, despite their effectiveness in inducing cooperation, such reward restructuring may raise fairness concerns, for example, the gifted reward is intrinsic and not part of the task-defined reward that agents are fundamentally trained to maximize. Consider the Cleanup environment: agents only receive extrinsic rewards for collecting apples, yet apples will only regrow if waste is cleaned. It is often observed that some agents specialize in cleaning waste while others collect apples and subsequently gift a portion of their reward to those who sacrificed their own gain. Although this leads to improved collective performance, the agents engaged in waste cleaning never directly receive task rewards from apple collection. This becomes trained with the collective return, since some agents encouraged to
even worse if the agents clean the waste all the time. Aside from reward restructuring, an approach has been proposed to align individual and collective objectives by adjusting policy gradients toward stable fixed points of the collective return, while still considering individual interests (20). However, this method does not adequately consider fairness, as it primarily focuses on stability without explicitly addressing the conflict between individual and collective objectives.
In order to enhance cooperation while ensuring fairness, we propose fair and conffict- gradient adjustment method (FCGrad) that dynamically balances gradients derived from individual and collective objectives by explicitly handling conflicts between them. FCGrad first detects the presence of conflicts, and when conflicts are found, it projects one gradient onto the normal plane of the other- preserving one objective's 's direction while avoiding interference with the other. Notably, FCGrad prioritizes the gradient associated with the lower objective value. For example, if the individual objective is lower than the collective objective, indicating that the agent is an unfair situation, we project the individual gradient onto the normal plane of the collective gradient and use the result as the final update. This enables cooperation be enhanced while maintaining fairness by resolving conflicts. We provide theoretical results showing that, under certain assumptions, the proposed gradient method guarantees monotonic non-decreasing improvement in both collective and individual objectives. We further show that the two objectives converge to the same value, leading to all agents' objectives aligning- thus ensuring individual fairness. In addition, we empirically demonstrate the effectiveness of FCGrad in terms of a-fairness (25). which captures both performance and fairness, in the Unfair Coin Game and two sequential social dilemma environments: Cleanup and Harvest.
2 Background and Related Works
2.1 Partially Observable Stochastic Game
A Partially Observable Markov Game (POMG) models multi-agent decision-making under un- certainty 21 4. A POMG is defined as tuple (N, S, {A,) T, (0,)
[R. where N is the number of agents, S is the set of states, A; is the action set of agent i, T : S x A x AN + A(S) is the transition function, O, S + AOO,) is the observation function, R, S x Ai x... X AN - IR is the reward function for agent i, and 7 E [0.1] is the discount factor. Here, depending on the reward structure, a POMG can represent various types of multi-agent settings: cooperative settings 13
16), where all agents share an identical reward function (i.e., where agents have directly opposing objectives, often modeled as zero-sum (i.e., Em" r" 0); or mixed-motive settings 24 17. where agents' rewards are neither fully aligned nor strictly opposed, creating simultaneous incentives for both cooperation and competition.
rl _ p ~'); adversarial settings 8
2.2 Mixed-motive Coordination in Multi-Agent RL
We consider mixed-motive settings, where agents' self-interest often conflicts with collective outcomes. Let us define the collective return as the average of individual returns: Root = 1 >i (s(ss)) a), where R' a) is the individual return of Agent i. In the context of gradient- based learning, a conflict occurs when the local and collective return gradients are misaligned, that is, when E R*] V [Roa] < 0, where 0 denotes the parameters of Agent i's policy.
N

---

To enhance cooperation (i.e. maximize collective reward) while avoiding conflicts, a variety of
approaches have been proposed, including inequity aversion [TO] 30), social influence [ reciprocal
reward shaping [3, formal contract mechanisms [9], and gifting-based cooperation 23 17 Many
of these approaches are studied in the context of Sequential Social Dilemmas (SSDs) 18 a prominent
class of mixed-motee motive settings in which agents repeatedly arbitrate between short-term selfish actions
and long-term collective returns. For example, [17] proposed a gift-based method that balances altrusim and self-interest based based social relationships between agents. [22 proposed an intrinsic motivation method that rewards agents for exerting causal influence over others' actions, thereby improving coordination in SSDs. [0O] introduced inequity aveesse agents that learn to cooperate by assigning temporal credit to prosocial behavior and penalizing inequitable outcomes. The aforementioned methods can be broadly viewed forms of reward shaping, wherein additional intrinsic or socially-informed rewards guide agents toward cooperative behavior.
In contrast to reward shaping approaches, recent work 200 has explored direct optimization in the gradient space to reconcile individual and collective objectives. Specifically, the Altruistic Gradient Adjustment (AgA) method [20] modifies the policy gradients of both the collective and individual objectives, pulling agents toward stable fixed points of the collective objective and pushing them away from unstable ones. The adjusted gradient for Agent is defined as Paga = Geol + M( + -Haneeeos)
where gool and gind are the gradients of the collective and individual objectives for Agent H is the Hessian of the collective return with respect to the policy parameters, and À is the adjustment - (oot) + || Geol | coefficient and its sign is determined by Ageul H gool) (gind H
This adjustment steers the update direction according to the local stability of the collective objective. Despite its effectiveness, AgA incurs additional computational complexity, focuses on the stability of the collective objective rather than directly resolving gradient conflicts, and provides no guarantees of monotonic improvement or fairness.
2.3 Gradient Adjustment
Gradient adjustment approaches have been actively investigated in multi-task learning 32
22
26 283. For example, CAGrad (22) formulates a quadratic program to compute conflict-averse convex combination of gradients, achieving better trade-off's at the cost of increased complexity, and Nash-MTL (26) frames the task-weighting problem as a bargaining game, using the Nash bargaining solution to promote fairness and efficiency across tasks. Another method that inspires our work is PCGrad (32), which mitigates conflicts by projecting each conflicting gradient onto the normal plane of the other, offering simple yet effective solution with low computational overhead. Specifically, when two gradients 91 and 92 are conflicted, PCGrad adjusts them by projecting one onto the normal 91
plane of the other, i.e., 91 PCorad as the final update.
PCGrad PCGrad and 92 g2, and then uses the average of g1 Teal
91
2.4 Fairness in Multi-agent RL
Fairness concerns how returns are distributed among agents rather than how large the total return is, making it complementary, but often orthogonal cooperation and efficiency. Fairness has been considered in multi-agent RL literature in both cooperative and mixed-motin settings [34 6 29]. For example, in cooperative settings, (34 formulates fairness as the optimization of fair social welfare function and [6) proposes a method for achieving team fairness by enforcing permutation- equivariant policies, which mitigate emergent unfairness caused by asymmetric role assignment. In mixed-motive settings, [[7] shows enhanced fairness when measuring the sum of individual rewards and gifts, whereas in this paper we evaluate fairness using individual rewards only. [10], inspired by the literature on inequality in economics [5), explicitly leverages fairness by adding both dastvvantage and advantage inequality terms to the reward of each agent to improve cooperation in SSD. Specifically, the shaped reward for Agent is r' p aIA/(N 1) E max(r, r., 0)
BIA/(N- Ejæ; mar(r, rr: 0), where A and 3IA weight disadvantage and advantage inequity, respectively.
Note that throughout this paper, we define fairness in terms of task-defined extrinsic individual rewards, the quantities that agents are fundamentally trained to maximize, and do not consider intrinsic rewards such as gifting, as they do not directly reflect actual participation in the underlying task. A more detailed discussion on this assumption is provided in Appendix A
3

---

(Vet < Vina)
Algorithm 1: FCGrad
9pcGnd
9col
Input: Policy parameters 0, learning rate y/
(Vet 2 Vus)
weighting factor 3
Jool
9 FCGml
Compute gind := VnVina (0),
gcol := Vo Vool (0)
Find
Find
2 (Gind, Icol) 2 0 then
FFCrrad
(1 gind + Ageoti
(a) Conflict
(b) Non-conflict
4 else
if Von((0) 2 Vin(()) then
Figure 1: FCGrad illustration: (a) When con- flicts occur, the gradient corresponding to the lower objective-eithee
ggol 9ind) 1| | |
9FCCrad f
gind
Gools
individual or collective- is projected onto the normal plane of the gradient of the higher objective; (b) When no conflctt is detected, a task-dependent weighted sum of the two gradients is applied.
7 else
(gind: 9col) Findi
9 FCGrrd
Ycol
11gind
Return 0 + + 719 "Ppcrrad
3 Methodology
In mixed-motive settings, the individual and collective objectives may be either aligned or in conflict. When they are aligned, optimizing both objectives is sufficient, as neither interferes with the other. In such cases, an appropriately weighted combination of the two can be effective. However, when the objectives are in conflict, it becomes essential to explicitly address the interference between them, as prioritizing one may hinder the other. This is because focusing solely on the individual objective may hinder learning in tasks where cooperative behavior is essential for maximizing individual returns, while focusing solely on the collective objective may compromise fairness among agents. Therefore, it is important to (1) recognize when such conflicts arise and (2) correspondingly adjust the individual and collective objectives, appropriately considering both fairesss and cooperation.
To this end, we propose a fair and conflictaaware gradient adjustment method, called FCGrad, which guarantees the monotonic non-decrease of both individual and collective objectives, while preserving fairness across individual objectives. Specifically, when the individual and collective gradients are in conflict, FCGrad projects the gradient associated with the lower expected return onto the normal plane of the other. This projected gradient remains a valid ascent direction for its own objective while avoiding interference with the other, and is then used the final update. For example, if the individual expected return is lower than the collective expected return, indicating that the agent is disadvantaged in terms of fairness, we project the gradient of the individual objective onto the normal plane of the collective gradient and use as the update direction. The detailed procedure and a visual illustration of FCGrad are provided in Algorithm| 1 and Fig. respectively. In the following, present the detailed method along with its theoretical analysis.
3.1 FCGrad: Fair and Conflict-aware Gradient Adjustment
We now describe how FCGrad operates from the perspective of Agent i. Let E Rd denote the parameters of the policy my for Agent i. Let us define V0)a(0) and (0) as the expected individual and collective returns, respectively, computed under the initial state distribution. Note that Vind (0) and (0) are the individual and collective objectives, respectively. Let gind V Vina (0) and Gcol := Voa denote the gradients of the individual and collective objectives, respectively. These represent ascent directions for (0) and (O), meaning that for a sufficiently small 17 > 0, the
following holds: (m + 710ma) > (f) and (ia + 79ca)) > (o)
FCGrad proceeds as follows: (1) check whether 9ind and gool are in conflict by examining the sign of their inner product, where a negative inner product indicates the presence of conflict. (2) if (find: Fool) 2 0 (i.e., non-conflict), FCGrad uses the weighted sum of two gradients: = (1 - 3)mm + Sgoot (3) (gind, Jeat) < 0 (i.e., conflict), FCGrad places more weight on the individual (collective) gradient when the collective (individual) objective is greater, in order to ensure fairness.

---

The corresponding gradient is given by
givd Deot
if (Vool 2 Vindd if (Vonl < Vind)
9pCGrad
where Deal and Find are the projections of Gcol and Find, respectively, onto the normal plane of another gradient vector, given by
Gind gcol)
(Ccol Plnd)
Jcol Gool
1iin ||2
Finds
Find
Find
Ponl
Gcol
(4) update the policy parameter with the step size 0 + 0 + my. Note that feol projects fooa onto the normal plane Ind- Thus, Jool is still a valid ascent direction for the collective objective while preserving the individual reward. This indicates that FCGrad prioritizes the individual objective without compromising the collective one when the agent is in an unfair situation, i.c., when the individual objective is lower. Conversely, when the collective objective is lower, FCGrad prioritizes the collective objective without compromising the individual one.
3.2 Theoretical Analysis
In this section, we prove that FCGrad guarantees monotonically non-decreasing improvements in both the collective and individual objectives, and that both objectives converge same value. This ensures that the expected individual returns across agents also converge to the same value.
Theorem 3.1 Assume Vina(0 and (0) are differentiable and L-smooth. Let the update direction Then, for sufficiently small step size 1j > 0, the update 0 t- 0 + 11
g be defined in Equation 7 yields monotonically non-decreasing improvements in both (i) and (m)
Proof. See Appendix
Theorem 3.1 states that FCGrad ensures monotonic non-decreasing improvements in both (0,) and Voa under certain assumptions. Note that all agents are updated using FCGrad, both the individual objectives of all agents and the collective objective, defined as the expected return averaged across agents, improved accordingly. However, monotonic improvement alone does not guarantee fairness. To establish fairness, it is necessary to further show that the individual and collective values converge the same value over time, which in turn implies that all agents' individual values also become equal. The next theorem formalizes this result by proving that the gap hetween (ina();) and (0,) vanishes under mild conditions.
Theorem 3.2 Assume (0) and (0) be L-smooth, and let 8, := V.))(e)) (O))(O)) denote the value gap iteration t. Assume the step size satisfies the Robbins-Monro conditions: mhe S Joel/ L with Σ, > O and any f, if |o, 2 c, then there exists t' 2 such that (gmnd,y Dcot,t) ) Then, the value gap converges to zero:
00 and Σ, ni < oc. Also assume conflict recurrence, meaning that for any
lim (eea(B)) Veur(B:)) 0.
Proof. See Appendix
Theorem 3.2 states that the gap between the collective and individual objectives converges to zero under certain assumptions, including conflict recurrence, where conflicts occur continuously. This assumption reasonable in mixed-motive settings, especially near equilibrium, because agents face inherent tensions between cooperation and self-interest, and they approach equilibrium, misalignments in their objectives can continue to induce conflicts, even with small policy updates. Under the assumption that all agents use FCGrad, the individual objectives of all agents converge the collective objective, and thus all individual objectives converge to the same value. This, in turn, implies that individual fairness is achieved.
3.3 Practical Algorithm
We now introduce a practical FCGrad-based multi-agent RI. algorithm for mixed-motive settings. We consider decentralized training and execution, where each agent does not have access to other

---

(a) Unfair coins
(b) Cleanup
(c) Harvest
Figure 2: The environments considered in our experiments: (a) Unfair Coins-green coins appear more frequently than red coins, inducing fairness challenges; (b) Cleanup with distinct spawn positions- two agents (cyan and pink) spawn near waste areas, while the rest (blue and purple) spawn farther away: and (c) Harvest with distinct spawn positions- agents (blue and green) spawn near apple (red) regions, while the rest spawn farther away.
agents' information but shares rewards, as commonly assumed in gifting mechanisms 23 agent trains its policy and value functions for both individual and collective returns solely based on its own local observations and the shared rewards. For this, we construct two separate value networks for the individual and collective objectives, while sharing a common encoder between them. Each value function is trained using generalized advantage estimation [27] to compute the corresponding advantage estimates. Using the value functions, and collective objectives. denoted as Gind and Jcol, respectively, via the PPO policy gradient. These gradients are then combined using FCGrad to determine the final update direction.
17
Fach
compute the policy gradients of the individual
Experimental Results
4.1 Experimental Setup
Environments We conduct our experiments using the JAX-base codebase and environments provided by the SocialJAX suite 77. We modify the existing environments Coins, Cleanup, and Harvest-to incorporate fairness perspective. Specifically, since Cleanup and Harvest already involve inherent fairness dilemmas, we introduce only minor changes by assigning distinct respawn positions to the agents. For the Coin Game, which originally focuses on the conflict between individual and collective objectives, we introduce asymmetry in the potential rewards that agents can obtain, creating disparity in individual incentives. Fig illussrate the considered environments. We provide detailed descriptions the following sections.
Metric As our goal is to maximize returns while ensuring fairness, both performance and fairness metrics should be jointly considered for evaluation. We use a-fairness (25] as the evaluation metric, where, given individual returns (r1, 1. ,NN), the fairness utility is defined as
if + log(r.), o
Ur(rn ,TN)=
.NN
(4)
Notably, the fairness utility recovers several well-known objectives for specific values of o: it corresponds to the collective return when = 0, the geometric mean individual rewards-also known as Nash Social Welfare- whenn 1, and the minimum individual reward when - co. Thus, a = reflects no consideration of fairness, and as a increases, the evaluation increasingly prioritizes fairness over aggregate performance. In summary, consider the following three representative instances of a- a-fairness return our evaluation: (i) average return (Mean, a = 0), (ii) geometric mean return (GeoMean, that a-fairness return considers both performance and fairness, where a determines the trade-off between them. The reported results averaged over four random seeds.
1), and (iii minimum individual return (Min, oo). Note
Baselines We evaluate FCGrad with six baselines: (a) collective reward optimization (Col), individual reward optimization (Ind), (c) inequity aversion reward restructuring (IA) 100 (d)
weighted gradient combination of gind and Jool (denoted as Weighted), which corresponds to FCGrad
without conflict handling, (c) PCGrad 32). and (f) Altruistic Gradient Adjustment (AgA) 20)) Note

---

that baselines (f)-(f) use the same architecture as FCGrad, where two separate value functions are trained for individual and collective objectives; they differ only in the policy update rule based on Gind and Boot All methods are implemented on top of the IPPO 3
Hyperparameter We introduce hyperparameter / for FCGrad, which determines the weight between the collective and individual objectives when there is no conflict. 3 plays particularly important role tasks that require high-level cooperation. We set 3 to 0.5, 0.7, and 0.8 for the Unfair Coin Game, Cleanup, and Harvest, respectively. The same values of / are used for the baseline method, Weighted. Additional hyperparameters for IPPO are provided in Appendix
4.2 Unfair Coins
The Coins environment
19 consists of two agents (green and red) and two types of coins, each associated with one of the agents. When a coin appears, it assigned a color with probabilities Peren and Pred- An agent receives reward of 1 for collecting any coin, regardless of its color. However, collecting a coin of the opposite color imposes a penalty of -2 2 on the other agent, creating a conflict between individual gain and cooperative behavior. In contrast to the original setting [9), where Pureen and Pred are both set to 0.5-so that collecting coins matching each agent's 's color naturally aligns with fairness and also maximizes the collective reward- consider an unfair variant where Pareen = 15/16 16 and Prod 1/16, introducing an inherent asymmetry in coin appearances. Although optimal collective performance still requires agents to collect coins matching their own color. this setup raises a fairness concern: the green agent receives substantially more rewards due to the higher frequency of green coins. To mitigate this imbalance and achieve fairer outcome, the green agent must occasionally yield coins to the red agent, sacrificing some collective reward in favor of equity.
Results. In the Unfair Coin environment,
Col
Inc
IN
Weighted
PCGrad
AgA
FCGrad
achieving fairness requires the green agent to yield some of its coins the red agent, thereby reducing its own reward. In other words, there
Unfair coins
120
100
exists strong trade-off between collective per-
formance and fairness. Therefore, we particu- larly focus on the performance trend with re-
40
spect to as well as the Min performance, which places greater emphasis on fairness the
20
return of the most disadvantaged agent.
Mean (a=0)
GeoMean (a=1)
Min (a==)
Fig. 3 presents the a-fairness returns in the fair coin environment (top) and the individual return of the green and red agents (bottom).. The performance of Col, Ind, and AgA is observed to decrease more dramatically as a increases compared to FCGrad and PCGrad, which are conflict-aware methods. Interestingly, both the collective and individual approaches result in ex- tremely unfair outcomes, but in opposite direc- tions. Col, which maximizes collective reward, trains both agents to collect their own coins. As result, the green agent, with more coin opportu- nities, gains higher returns, leading to unfairness. In contrast, with the individual objective, the red agent outperforms the green agent, possibly because the green agent is more frequently penalized by negative rewards due to the abundance of green coins. Mcanwhile, the red agent learns without such penalties, accelerating its progress. However, FCGrad shows little variation across agents as a changes, indicating achieved fairness. Notably, FCGrad outperforms the baselines in terms of Min performance. As shown in Fig. 3 both the green and red agents converge to nearly identical returns, showing that fairness effectively achieved.
120
Red agent 18 8:
Greed agent
8. as
102
52
Step 109
Step 1x 109
Figure 3: Top: Performance across agents in terms of mean, geometric mean, and minimum. Note that higher value of a places more emphasis on fairness. Bottom: Agent-wise returns Red and Green Agents.
4.3 Cleanup
The Cleanup environment consists of N = agents, apples, and waste. Fach agent receives a reward of 1 for collecting an apple. Apples grow in orchard, but their growth depends on the
7

---

Col
Ind
IA
Weighted
PCGrad
AgA
FCGrad
Cleanup
120
100
100
Agent
8
Agent
80
80
60
10
40
10%
200
Agent 2
100
Agent 3
200
Mean (a-0)
GeoMean
Min (ct-oe)
350
(a-1)
150
50
e-fairnsss returns (Cleanup)
Step 109
Step Ix
(b) Individual returns (Cleanup)
Harvest
Agent O
16
Agent
60
40
50op
104
20
70
Agent: 22
Agent 3 26
40
Mean (0=0)
GeoMeen (a=1)
Min
(c) a-fairness returns (Harvest)
Step
159
(d) Individual returns (Harvest)
Figure 4: a-fairness returns and individual returns in the cleanup and harvest environments.
amount of waste present in the environment. Waste accumulates at a constant rate, and beyond certain threshold, apple growth ceases entirely. Therefore, in order sustain apple regrowth, some agents must sacrifice their immediate reward by cleaning the waste. This creates a social dilemma, as the necessary act of cleaning benefits the group but does not provide direct individual reward, thereby generating a tension between self-interest and cooperative behavior. In contrast to the original configuration, where agents are randomly spawned across the map, we fix the spawn positions of agents: some (Agents 2 and in our case) are placed near the apple orchard, while others (Agents ( and 1) are positioned closer the waste area. This spatial asymmetry further amplifies the conflict between fairness and efficiency. Note that, unlike the Unfair Coin, Cleanup introduces an intertemporal perspective, involving a trade-off between short-term individual interest and long-term collective interest [10].
Results. Fig. (a) and (b) show the a-fairness performance and individual rewards during training in the Cleanup environment. FCGrad outperforms the baselines in terms of both GeoMean and Min, which reflect not only total return but also fairness. In addition, FCGrad achieves comparable performance to Col in terıs of Mean, which is the optimization target of Col. As shown in Fig. 4 (b), under Col, Agent 3 learns to monopolize apple collection, while Agent by primarily cleaning waste. In contrast, FCGrad leads all four agents to obtain reasonably similar returns demonstrating more fair behavior and achieving the best result in terms of Min. Since using the collective reward essential in this environment, methods that rely heavily on individual rewards, such as and IA, fail to learn effectively. addition, AgA fails to properly balance between individual and collective objectives, also struggle to learn successfully.
trained sacrifice
4.4 Harvest
The Harvest environment features N 4 agents and apples distributed across orchard patches. Each agent receives a reward of 1 per apple, but regrowth is stochastic and depends nearby apples within a fixed radius. Over-harvesting depletes resources, risking environmental collapse, and thus agents must coordinate implicitly sustain long-term returns. This creates social dilemma between

---

short-term individual gain and long-term collective benefit. We also introduce spatial asymmetry: Agents and spawn near apples, while Agents 2 and spawn farther away, making collection easier for the former. Similar to the Cleanup, Harvest also poses intertemporal challenges for both cooperation and fairness.
Results. Fig. (C) and (d) show the a-fairness returns and individual agent returns during training. FCGrad outper- forms the baselines across the considered a values. With the Col, Agents and 1 achieve higher returns than Agents 2 and 3, indicating that they focus solely on collecting ap- ples while accounting for the intertemporal dilemma, but not addressing the resulting unfairness toward Agents 2 and 3. contrast, FCGrad leads all four agents achieve similar returns, implying that Agents O and take into account the outcomes of Agents 2 and Similar to the results in Cleanup, methods that rely heavily on individual rewards, such as Ind and IA, perform poorly, though they achieve marginal learning. AgA performs better than the -rwward based methods, but still underperforms compared to FCGrad.
Harvest
60
50
40
20
0.5
0.6
0.7
0.8
0.9
1.0
Figure 5: GeoMean of FCGrad with re-
spect to | in the harvest environment.
4.5 Additional Analysis: Ablation and Fairness Metrics
3 determines the balance be- tween the collective and individual objectives when no conflict is detected. It plays partic- ularly important role in tasks that require high- level cooperation. For example, in Cleanup, ig- noring the collective objective makes difficult for agents to discover how to improve their indi- vidual rewards. We observed this phenomenon maximizing in-
Weighting factor:
Coin
Cleanup
Harvest
Alg
Gini
Jain
Gini
Jain
Gini
Jain
Col
0.474
0.526 0.558 0.432 0.182 0.882
Ind
0,509 0.498 0,522 0.515 0.136 0.936
IA
0.122
0.942 0.536 0.497 0.087 0.973
Weighted
0.048
0.991 0.266 0,801 0.146 0.928
PCGrad
0.039 0.994 0.469 0.572 0.101 0.965
in the previous seccionn-slle dividual rewards does not perform well. We present the GeoMean performance of FCGrad in the Harvest environment in Fig. [ which shows that 3 value between 0.7 and 0.8 yields the best performance. Thus, 33 reflects the re- quired degree of cooperation over self-interest.
AgA
0.238
0.749 0.331 0.723
0.123 0.948
FCGrad
0.010 0.999 0.223 0.835 0.093 0.959
Table 1: Addtional faimesss evaluation using Gini coefficient and Jain's 's index. Lower Gini and higher Jain indicate greater fairness. Top-2 most fair scores each column are highlighted in bold.
Additional Fairness metrics: We additionally evaluate faimess using the Gini coefficient [2] and
Jain's index 11]. The Gini coefficient is defined as Gini(ri,".. ,rN) ΣN ΣN Insrryl and Jain's
2N E
index is defined as Jain(r1, ... /NN = (>: r22
NE r where both metrics range between 0 and I and lower Gini and higher Jain values indicate better fairness. Table presents the results, showing that FCGrad generally achieves superior fairness.
Conclusion
In this work, we address the long-standing challenge of achieving both cooperation and fairness in mixed-motive multi-agent RL. We propose FCGrad, a conflict-aware gradient adjustment method that explicitly resolves gradient-level conflicts between individual and collective objectives. FCGrad dynamically adjusts the update direction based on which objective is more disadvantaged by projecting one gradient onto the normal plane of the other. We theoretically prove that this mechanism guarantees monotonic improvement and convergence of both objectives to the same value. Consequently, individual objectives across agents also converge, ensuring fairness. Extensive experiments in Unfair Coin environment and sequential social dilemma settings, Cleanup and Harvest, demonstrate that FCGrad not only improves overall performance but also achieves superior fairness, as measured by a-faimess return metrics.

---

Limitation In practice, the recurrence of gradient conflicts, required for our theoretical guarantee, may not hold, it can be influenced by the weighting factor non-conflict cases. Understanding this interplay is promising direction for future work.
Broader Impact Our work promotes fairness in learned behaviors, potentially preventing emergent inequalities in decentralized systems. We believe it has a positive societal impact.
6
Acknowledgement
This work was supported by the ONR MURI grant N00014-25-1-21166
References
[1] Jasmine Jerry Aloor, Siddharth Nagar Nayak, Sydney Dolan, and Hamsa Balakrishnan. Cooper- ation and fairness in multi-agent reinforcement learning. Journal on Autonomous Transportation Systems, 2(2):1-25, 2024.
[2] Herbert A David. Miscellanea: Gini's mean difference rediscovered. Biometrika, 55(5))5k3k,,55)577335--5557 1968.
[3] Christian Schroeder De Witt, Tarun Gupta, Denys Makoviichuk, Viktor Makoviychuk, Philip HS Torr, Mingfei Sun, and Shimon Whiteson, Is independent learning all you need in the starcraft multi-agen challenge? arXiv preprint 20.iv:2011.095333 2020.
[4] Rosemary Montry-Montemerlo, Geoff Gordon, Jeff Schneider, and Sebastian Thrun. Approxi- mate solutions for partially observable stochastic games with common payoffs. In Proceedings of the Third International Joint Conference on Autonomous Agents and Multiagent Systems, 2004. AAMAS 2004., pages 136 143. IEEE, 2004.
[5] Dirk Engelmann and Martin Strobel. Inequality aversion, efficiency, and maximin preferences in simple distribution experiments. American economic review, 94(4):857-869, 2004.
[6] Niko A Grupen, Bart Selman, and Daniel D Lee. Cooperative multi-agent fairness and equiv- ariant policies. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 9350 9359, 2022.
[7] Zihao Guo, Richard Willis, Shuqing Shi, Tristan Tomilin, Joel Z Leibo, and Yali Du. Socialjax: An evaluation suite for multi-agent reinforcement learning in sequential social dilemmas. arXiv preprint 21Xiv:2503.14570 2025.
[8] Songyang Han, Sanbao Su, Sihong He, Shuo Han, Haizhao Yang. Shaofeng Zou, and Fei Miao. What is the solution for state-adversarial multi-agent reinforcement learning? Transactions on Machine Learning Research.
[9] Andreas Haupt, Phillip Christoffersen, Mehul Damani, and Dylan Hadfield-Menell. Formal contracts mitigate social dilemmas in multi- multi-agent agent reinforcement learning. Autonomous Agents and Mulli-Agent Systems, 38(2):1-38, 2024.
[10] Edward Hughes, Joel Z Leibo, Matthew Phillips, Karl Tuyls, Edgar Duefiez-Gzmman, Antonio García Castañeda, lain Dunning, Tina Zhu, Kevin McKee, Raphael Koster, et al. Inequity aversion improves cooperation in intertemporal social dilemmas. Advances in neural information processing systems, 31, 2018.
[11] Rajendra K Jain, Dah-Ming W Chiu, William R Hawe, et al. A quantitative measure of faimess and discrimination. Eastern Research Laboratory, Digital Equipment Corporation, Hudson, MA, 21(1), 1984.
[12] Natasha Jaques, Angeliki Lazaridou, Edward Hughes, Caglar Gulcehre, Pedro Ortega, DJ Strouse, Joel Z Leibo, and Nando De Freitas. Social influence as intrinsic motivation for multi-agent deep reinforcement learning. In International conference on machine learning, pages 3040-3049, PMLR, 2019.
10

---

[13] Jeewon Jeon, Woojun Kim, Whiyoung Jung, and Youngchul Sung. Maser: Multi- Multi-agent rein- forcement learning with subgoals generated from experience replay buffer. In International conference on machine learning, pages 10041-10052. PMLR, 2022.
[14] Woojun Kim, Whiyoung Jung, Myungsik Cho, and Youngchul Sung. A variational approach to mutual information- coordination for multi-agent reinforcement learning. Proceedings of the 2023 International Conference Aulonomous Agents and Multiagent Systems, pages
40 40-48,27 2023.
[15] Woojun Kim and Youngchul Sung. An adaptive rearopyyaggguliiiattio framework for multi- agent reinforcement learning. In International Conference on Machine Learning, pages 16829. 16852. PMLR, 2023.
[16] Woojun Kim and Youngchul Sung. Parameter sharing with network pruning for scalable multi- agent deep reinforcement learning. In Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems, pages 1942-1950, 2023.
[17] Fanqi Kong, Yizhe Huang, Song- Cang Zhu, Siyuan Qi, and Xue Feng. Learning to balance altruism and self-interest based on empathy in mixed-motive games. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.
[18] JZ Leibo, VF 7ambaldi, M Lanctot, J Marecki, and T Graepel. Multi-agent reinforcement learning in sequential social dilemmas. In AAMAS, volume 16, pages 464 473. ACM, 2017.
[19] Adam Lerer and Alexander Peysakhovich. Maintaining cooperation complex social dilemmas using deep reinforcement learning. arXiv preprini arXiv: 1707.000770006 2017.
[20] Yang Li, Wenhao Zhang. Jianhong Wang, Shao Zhang, Yali Du, Ying Wen, and Wei Pan. Aligning individual and collective objectives in agett cooperation. In The Thirty-eighth Annual Conference Neural Information Processing Systems.
[21] Michael L Littman. Markov games as framework for multi-agent reinforcement learning. In Machine learning proceedings 1994, pages 157-163. Elsevier, 1994.
[22] Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu. Conflict-averse gradient descent for multi-task learning. Advances in Neural Information Processing Systems, 34:18878-18890, 2021.
[23] Andrei Lupu and Doina Precup. Gifting in multi-agent reinforcement learning, In Proceedings of the 19th International Conference on autonomous agents and multiagent systems, pages 789-797, 2020.
[24] Kevin R McKee, Ian Gemp, Brian McWilliams, Edgar A Duéñez-Guzmán, Edward Hughes, and Joel Z Leibo. Social diversity and social preferences mixed-motive reinforcement learning. arXiv preprini arXi::2002.002325 2020.
[25] Jeonghoon Mo and Jean Walrand. Fair end-to-end window- bindew congestion control. JEEE/ACM Transactions on networking, 8(5):556-56) 567,557777770 2000.
[26] Aviv Navon, Aviv Shamsian, Idan Achituve, Haggai Maron, Kenji Kawaguchi, Gal Chechik, and Ethan Fetaya. Multi-task learning bargaining game. In International Conference Machine Learning, pages 16428-16446. PMLR, 2022.
[27] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.024333 024388 2015.
[28] Dmitry Senushkin, Nikolay Patakin, Arseny Kuznetsov, and Anton Konushin. Independent component alignment for multi-task learning. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages
2023.
[29] Martin Smit and Fernando P Santos. Learning fair cooperation in mixed-motive games with indirect reciprocity. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, pages 220-228, 2024.

---

[30] Jane X Wang, Edward Hughes, Chrisantha Fernando, Wojciech M Czarnecki, Edgar A Duéñezz Guzmán, and Joel Z Leibo. Evolving intrinsic motivations for altruistic behavior, In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgen Systems, pages 683-692,22 2019.
[31] Lantao Yu, Jiaming Song. and Stefano Ermon. Multi-agent adversarial inverse reinforcement learning. In International Conference Machine Learning, pages 7194-72011 PMLR, 2019,
[32] Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Advances in neural information processing systems, 33:5824-5836, 2020.
[33] John L Zhou, Weizhe Hong, and Jonathan Kao. Reciprocal reward influence encourages cooperation from self-interester agents. Advances in Neural Information Processing Systems, 37:59491-5995 2024.
[34] Matthieu Zimmer, Claire Glanois, Umer Siddique, and Paul Weng. Learning fair policies in decentralized cooperative multi-agent reinforcement learning. In International conference machine learning, pages 12967-12978. PMLR, 2021.
12

---

A Discussion on Fairness
In this appendix, we clarify the modeling assumption underlying our notion of fairness and contrast it with alternative perspectives such as tionally focuses on different regime: fairness grounded purely task-defined extrinsic rewards that directly reflect each agent's actual behavior, rather than assuming the availability of contract or currency-like mechanisms (e.g., reward exchanges or gifting) for compensating agents.
-based faimess. Our formulation inten-
A.1 Extrinsic-Reward
Fairness vs. Reward Redistribution
Prior approaches such as gifting or incentive mechanisms (e.g., 17 (23) allow agents to redistribute rewards among one another-often interpreted as a currency-like signal contract that enables division of labor. Under such assumptions, reward transfers are considered real, tangible returns and can be used to compensate agents for sacrificial roles (e.g., pollution cleaning without harvesting apples).
In contrast, work adopts more primitive perspective of fairness: we consider only lask-defined extrinsic rewards that arise directly from environment state-action outcomes (e.g.. rewards from collecting apples in the Cleanup environment). We intentionally do not assume the existence of an auxiliary payment mechanism, such as money or transferable reward tokens, that is external to the environment dynamies. From this viewpoint, one agent continuously cleans while others only harvest apples, such an outcome is deemed unfair unless the cleaner also receives direct extrinsic returns. This modeling choice focuses on fairness that reflects actual participation the task, rather than contractual compensation.
A.2 Pareto Optimality and the Role of a- a-Fairness
We emphasize that our objective is not compute approximate theortheo equilibrium (e.g.. Nash or correlated equilibrium), but rather to learn Pareto-optimal outcomes. In particular, a-fairness used only an evaluation metric, not training objective. By varying a, one can evaluate different trade-offs between pure efficiency (a = 0), multiplicative balance (a = 1), and max-min fairness (a - oc). FCGrad yields outcomes that lie on the Pareto frontier across these trade-offs: in terms of collective return it performs comparably to the best baselines, while in terms of a = 1 or it significantly improves fairness without degrading performance.
13

---

B Theoretical Results
Lemma B.1 Let R > T. a continuously differentiable and L-smooth function. Let 91 (oJ be the gradient of J at point 0, and let 92 E R be any vector satisfying (91, 92) > 0. Then, 2(91.92) the update 0 192 yields a strict improvement:
for small step size r L
j(B + n92) J(O).
Proof. Since J is L-soooth, for any 0 E RR, update direction 92 E Rd, and step size >> 0, the following inequality holds:
L 5 11921
+ 2 J(0) (0)))))))))))
Let 91 = VQJ( Then:
L
J(0 n92) 2 ((0) + 7(91 92)
2 11
Define the right-hand side as a function
L r 1921
A n(n, 92)
Since g2) > this a concave quadratic function that is positive for small enough ry. Specifically, the inequality (nr) >> 0 holds when:
2(g1, 92) L1|92|1 ||
Therefore, for any
(0,
2(0) T.||sal|
have:
J(O + (gan)) J(0).
Theorem B.2 Assume Vinal and (o) are differentiable and L-smooth. Let the update direction g be defined as in Equation 7 Then, for sufficiently small step size 1j >> O, the update O <- 0 + 199 yields monotonically non-decreasing improvements both (f) and Vie(f).
We consider three cases:
Case 1: (Non-conflict) Gind 2 0. Then = Fidd + (1 3oot Since Finds Gcol are ascent
directions for Vind, Veol, respectively, their convex combination also satisfies:
Gind ginal 1| + (1 3)m Ieal 0
(5)
Gool g Bgool Glnd + (1 (3) | || > 0
(6)
Since Iind 9 and 9ind are positive, according to Lemma 3.1, yields a strict improvement in both Vind and V Vell-
Case 2: (Conflict) Find feol < O and (in)()) < Veoal (0). We then use: = Find
Real T[gcot
| Geot- Now,
(gind Deol)
(gind 9col) ||9tal (etmal?21 11 (Aind Icol) >0 0
Gind
Tind
Find
|| 1|2
|19col
(gind |[
Icol) (gool, Acol)
Gool Gcol
find
0
Since Ind g is positive, according to Lemma 3.1, g yields strict improvement in Vina- In addition, since Jeol is zero, g does not decrease Voot-
Case 3: (Conflict) find Sool 0 and (i) > (O) Symmetric to Case 2: g yields striet improvement both Vont and does not decrease Vina-
Thus, in all cases, g induces monotonically non-decreasing improvements in Vin and Vool-
14

---

Lemma B.3 (Single conflict step) When the conftict happens (i.e., (gind 9col) < 0), then for suffi-
ciently small step size, 0 2 nt S //6el//L. L. we have
Tt 16 ||de1
L4+1 Le
(8)
-2
Proof. When & < (i.e. Van > Vina), we use 9 = Find
Jcol- Since L is L-smooth function, ||
the following holds
L411 Le S (V 9) + 2 Thil99 1991
(9)
Here,
(VL 9) = 6e(Aind (col) Se(nind g - Gool g) = See g) = del1g11?
(10)
Thus, we have
Lx41- - L S mb
L 2 11lll1111 S nolellll +
|6 1T
Th =
11 |11e11
(11)
2
2
Lemma B.4 (Single non-conftict step) When the conftict does not happen, (i.e., (gind gcol) 2 0), the proposed gradient used. We assume that the step size meets the Robbins-Mone conditions (iee.
ot= lh -0, -cc nhe < oo.) Then, the following holds:
Σ 11 - Lell < 00
(12)
IEN
where is the set of all non-conflict indices.
Proof. g 3gmd + (1 3) Let us define G susu (Igr,ell 1192,11)) (< oo).
Since L-smooth, we have
L ni 1|
T++1 La S VoLa, g) +
(13)
2
Since Vol, = (eind Bcal),
( g)1l ||ō, (gind Icot, 39ind + (1 3) 3c
(14)
S |8e1 Al]gnnal Hainallllgen Ill Scoll + B)ll I1l - ]sall]
(Cauchy-Schwarl
(15)
- |6e| Bllllll + (1 - [1 || 2G S 2G
(16)
Based on the assumption of the step size r7 (n) S //6ll/ L), we have
( VaLLe de)l < 2G 60 n
20
(C)
Since || = [],8gmd + Scolll I 3 |gnaal + (1 - 3) lgcoll I G. the following holds.
L n 11llo11
L G2
(17)
Combined above, we have
[1+1 L.ll < / L2 G
5 c2 L
(18)
2
Define Co = 5 G2 to obtain
[411 Lel s Co Con..
Because Eee n < oc (Robbins-Mool Monro assumption),
D
Σ L 4d < Co Σ
< Co
1
2
< xc.
KEN
15

---

Theorem B.5 Let Vind and Cot be L-smooth. Assume the step size satisfies the Robbins-Monp conditions: 0 Tr S /6.l L with Ee T) and Σ. bh se. Also assume conflict recurrence, meaning that for any 0 and any I, if [6 2 then there exists t!' 2 L such that (gin, 9eot,e 0. Then, the value gap converges to zero:
lim [Via(Oe) (o)+ O.
(19)
Proof. Denote conflict indices by C and non-conflict by N. Lemma A.3 and Lemma A.4 give for every horizon T
LT Lu
1 Σ
me lell
Co
Σ
(20)
2
LEC,KT
BEN,
According to the assumption of the Robbins-Monro, the following holds:
Σ TEC
ne 16 d+ < cc.
(21)
For any conflict step the projection property and bounded gradients imply || 2 > 0 with = minn(1gaaa. [l Il). Thus, we have
Σ AEC
nr 0e S
1: 119:111 < co.
(22)
IEC
Here, we use contradiction. Assume lim sup, 16 = Eu > 0. Set :: Eu/2. By the assumption,
there exists infinite set C. = {t C 16,1 2 =}. Then for every E C.. The 16,1 T. E. Because
L 11 xc Etec, Th & = oc, contradicting the finiteness of Eq. 22 Therefore, limsup; |6 = 0.
16

---

C Implementation Details
All experiments were run local server equipped with an AMD EPYC 7713 64-Core CPU and five NVIDIA RTX 6000 Ada Generation GPUs. Each rollout consisted of 64-256 parallel environments depending on the task, and training time per run ranged from 8 hours. The official implementation of FCGrad is available at: btttps://githh com/w] wwi.kiimmmmmmggg..
C.1
Unfair Coin
Each agent has CNN-based actor-critic network. The observation is processed through three convolutional layers with kernel sizes of 5 x 5, 3 x 3, and x 3, each with 32 channels and ReLU activations, followed by a fully connected layer with 64 units. The actor head outputs categorical distribution over discrete actions, while the critic consists of two separate heads estimating the individual and collective value functions.
We train the networks using the Adam optimizer with a learning rate x 10-4, linearly annealed time. PPO used with clipping threshold of 0.2 and two update epochs per iteration, using 500 minibatches. We collect trajectories from 256 parallel environments, each running for 1000 steps per rollout. The discount factor set to 7 0.99 and the GAE parameter À 0.95. The entropy and value loss coefficients are set to 0. respectively. Gradients are clipped to maximum global norm of 0.5.
C.2 Cleanup
Each agent is equipped with a convolutional actor-critical network. The observation is processed through three convolutional layers with kernel sizes of x 5, 3 and 3 each with 32 channels and ReLU activations, followed by a fully connected layer with 64 units. The actor outputs a discrete actions, and the critie consists of two heads that estimate the individual and collective value functions, respectively.
categorical distribution
Training is performed using PPO with a clipping threshold of 0.2 and two update epochs per iteration. total 500 minibatches are used per update, with data collected from 64 parallel environments running 1000 steps per rollout. The discount factor is set to y 0.99, and the GAE parameter is set to À = 0.95. We use the Adam optimizer with an initial learning rate of 5 x 10 44 which is linearly annealed during training. The value loss coefficient and entropy coefficient are both set to 0.01, and the value function loss is weighted by 0.5. Gradients are clipped with a maximum global norm of 0.5.
C.3 Harvest
Each agent is equipped with convolutional actor-critical network. The observation is processed through three convolutional layers with kernel sizes of x 5, 3 3, and 3, each with 32 channels and ReLU activations, followed by fully connected layer with 64 units. The actor outputs categorical distribution over discrete actions, and the critic consists of two heads that estimate the individual and collective value functions, respectively.
Training is performed using PPO with a clipping threshold 0.2 and two update epochs per iteration. A total of 500 minibatches are used per update, with data collected from 64 parallel environments running 1000 steps per rollout. The discount factor set to y = 0.99, and the GAE parameter is set to X 0.95. We use the Adam optimizer with an initial learning rate of 5 x 10-4 which is linearly annealed during training. The entropy and value function loss coefficients are set to 0.01 and 0.5, respectively, Gradients are clipped to a maximum global norm of 0.5.
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state the core contributions and are consistent with both theoretical and empirical results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We stated the limitation in the conclusion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
18

---

Justification:
We included the proof in the Appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provided the details in the Appendix and the main paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
19

---

Answer:
[Yes]
Justification:
We cited the corresponding paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm
cessinnnnccyyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ccpppp.ccppulli//guuii wwwiiccgguie///eeeeeee CoodeSubeiiieeee ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provided the implementation details in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We provided the mean and variance of individual returns.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
20

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We stated this in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics ptctps://neur// cctpriissccccccccccccooo http://ss..............ccccccaaa
Answer:
[Yes]
Justification:
We follow the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We stated the broader impacts in the conclusion.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
21

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We have cited the paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppswpptseeee www.ree.wthhhe....c... com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
22

---

Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
23

---

Justification: [NA]
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy htteas////eerrpppoccc///o/rreees for what should or should not be described.
24


Reviews:
[{"review_id": "dRGORlpB8W", "reviewer": "Reviewer_ZK8G", "summary": "In summary, this paper introduces the idea of fair and conflict-aware gradient adjustments for the updates of policies which can achieve cooperation and fairness. It outperforms current baselines in terms of fairness in the social welfare in mixed-motive multi-agent sequential social dilemmas.\n\nThey propose a fair and conflict-aware gradient adjustment method (FCGrad) which guarantees a monotonic non-decrease of both individual and collective objectives, while preserving fairness across each individuals objective.\nFCGrad, which works in a decentralized model, uses two value functions, for the individual and for the collective. This imposes the assumption that there is a shared global reward. They then find the gradients of the individual and the collective objectives respectively. They then check the sign inner product of these gradients. If positive, they use a weighted sum of the two gradients, and if it is negative, there is a conflict. It will then place more weight on the gradient that is lower to ensure fairness. They do this by projecting the individual or collective gradients onto the normal plane of another gradient vector, the exact form is described in equation (2).\n\nThis is shown both theoretically and empirically.\n\nThe empirical results are shown in the unfair coins, cleanup and harvest environments, which are slightly tweaked versions of previously used versions of these environments.\nThey compare with 6 baselines: Collective reward optimization (Col), Individual reward optimization (Ind), inequity aversion reward restructuring (IA), FCGrad without conflict handling, PCGrad, and Altruistic Gradient Adjustment. They also include a hyperparameter for the weighting of FCGrad, they show the choice of this value is significant to the method working, and is environment dependent.\n\nOverall they show that FCGrad (Their method) results in being in the top two fairest outcomes across the methods over multiple fairness metrics, Gini and Jain. (Except Jain in Harvest, but it still close to the top of the performance)", "questions": "1.\tWhile it is not the focus of this study. What do you think the outcome of agents trained with this type of reward being mixed into an environment with other training regimes? \n\n2.\tWould they be exploited?\n\n3.\tWould they be able to coordinate with different behaviours that are off policy from their own?", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1.\tThis is an interesting take on how to achieve fairness in these mixed-motive environments.\n\n2.\tIt is a decentralized training approach which gives it scalability.\n\n3.\tIt compares to a number of known algorithms for fairness over multiple well known environments.\n\nWeaknesses:\n\n1.\tIt does describe how the model is set up, however, I believe a diagram would help a lot, even it were in the appendix.\n\n2.\tThe reported results only being over 4 random seeds makes it seem like its still possible that these are specific ones which it works particularly well for. Increasing the number of runs could help.", "was_revised": true, "final_justification": "I believe that the authors have addressed my concerns.\nTaking into account this and the other reviews, I will be keeping my score for possible acceptance.", "rebuttal": "We appreciate the reviewer’s valuable comments and questions. \n\nBelow, we provide our responses to the reviewer’s concerns and questions.\n\n\n## [1-1] Regarding \"a diagram for the model\"\nWe will include a diagram describing our model in the appendix of the final paper. Thank you for the helpful suggestion.\n\n---\n\n## [1-2] Regarding \"Increasing the number of runs\"\nWe have added two more seeds (6 in total) for Cleanup and Harvest.\n\nThe following table shows the performance of FCGrad and the baselines over 6 seeds:\n\n| Task                          | col      | ind      | iar      | wei       | pcg      | aga      | fcgrad (ours) |\n|------------------------------|----------|----------|----------|-----------|----------|----------|----------------|\n| Harvest (Geomean, α=1)       | 45.3±20.7 | 20.4±2.3 | 21.2±2.3 | 47.1±11.9 | 20.2±2.5 | 35.4±6.9 | **64.9±5.4**   |\n| Harvest (Min, α=∞)           | 23.3±20.2 | 12.6±1.4 | 14.3±0.2 | 29.3±13.2 | 13.8±0.7 | 24.8±1.4 | **49.5±12.4**  |\n| Cleanup (Geomean, α=1)       | 49.6±21.4 | 2.5±2.5  | 3.0±2.4  | 57.9±6.9  | 24.4±26.2| 3.0±2.5  | **76.0±15.6**  |\n| Cleanup (Min, α=∞)           | 12.0±10.3 | 0.01±0.00| 0.02±0.02| 31.1±11.5 | 11.4±22.8| 0.01±0.00| **43.6±27.6**  |\n\n\nThe following table shows the p-values from Student’s t-test:\n\n| Task                        | col   | ind   | iar   | wei   | pcg   | aga   |\n|----------------------------|-------|-------|-------|-------|-------|-------|\n| Harvest (Geomean, α=1)     | 0.04  | 0.00  | 0.00  | 0.02  | 0.00  | 0.00  |\n| Harvest (Min, α=∞)         | 0.02  | 0.00  | 0.00  | 0.01  | 0.03  | 0.01  |\n| Cleanup (Geomean, α=1)     | 0.07  | 0.00  | 0.00  | 0.05  | 0.01  | 0.00  |\n| Cleanup (Min, α=∞)         | 0.05  | 0.02  | 0.02  | 0.38  | 0.09  | 0.02  |\n\n\nAs shown in both tables, FCGrad consistently outperforms the baselines in terms of α-fairness (α=1 and α=∞), with statistical evidence. The only exception is the \"wei\" baseline in Cleanup (Min performance), which can exhibit high variance as α increases in α-fairness. However, FCGrad (43.6) still achieves better performance than Wei (31.1).\n\n---\n\n## [1-3] Regarding \"Regarding the expected behavior of agents trained under different regimes\"\n\nWe appreciate the reviewer’s thoughtful questions regarding the robustness of FCGrad agents when interacting with agents trained using different regimes or exhibiting off-policy behaviors.\n\nWe would first like to clarify that the theoretical guarantees of FCGrad (e.g., monotonic improvement and fairness) assume that all agents adopt the FCGrad update rule. Nevertheless, even in mixed training regimes, our design encourages coordination rather than exploitation due to the structure of the collective reward.\n\nIn particular, since the collective objective is designed to yield higher returns than purely individual optimization (i.e., col > ind), agents trained with other regimes are naturally incentivized to engage in coordination. If an agent trained with a different policy attempts to exploit FCGrad agents by behaving selfishly, it will often find that its own individual reward remains limited, potentially perceiving this as an unfair outcome. Over time, it would thus be driven toward more cooperative behavior to access the benefits of the collective reward.\n\nConversely, if an FCGrad agent perceives that others are not cooperative or are unable to adapt, it will naturally shift toward optimizing the individual objective. This dynamic ensures that coordination is pursued when beneficial, but exploitation is not blindly tolerated. In this way, FCGrad agents exhibit a degree of robustness when interacting with off-policy behaviors by adapting their gradient direction based on observed reward feedback.\n\nWe believe this structure makes FCGrad well-suited for mixed-agent scenarios, and we thank the reviewer for raising this important point."}, {"review_id": "34pEbmX7GA", "reviewer": "Reviewer_mRtA", "summary": "The paper presents a new gradient-based algorithm for MARL in which the returns of each agent are shaped to consider not only individual objectives, but also social welfare and fairness. The new algorithm balances each agent's learning to explicitly account for the individual and shared objectives so they are solved to maximize fairness. The paper includes two theorems that together form a theoretical basis for the learned policies to be optimal. Empirical evaluation includes three social dilemma domains and a variety of agents that are self-interested. The results show that the new method, FCGrad, enables the agents to learn a balanced policy that maintains fairness using various metrics.", "questions": "1. Did you perform any significance tests for the results in Table 1?", "limitations": "It can be beneficial to have a brief discussion about the dual purpose of this research: can someone use the same paradigm as FCGrad to intentionally *decrease* fairness?", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n+ This paper is one of the most precise and clear papers I have read recently. Well done!\n+ The related work is comprehensive, follows a clear narrative, and provides context to understand the novelty of the new contribution.\n+ Empirical evaluation is strong, with environments especially modified with asymmetries to emphasize the role of fairness in these domains.\n+ The results support the overall superior performance of the new approach across several metrics.\n\nWeaknesses:\n- The main weakness is the theoretical analysis. Theorems are considered a part of the main paper, but the proofs are solely in the appendix, so they cannot be considered a fully sound contribution. Theorem 3.2 is a significant leap and requires additional preliminary explanations.\n- In line 168, \"FCGrad uses the weighted sum of two gradients: g = ...\" I assume g and g_FCGrad are supposed to be the same, because if they're not, I'm not sure where the latter is used.\n- In Table 1, results are bolded not just for the top-performing algorithm, but also for the second-best. This is confusing, especially when no significance is reported, so there's no way to verify these results are indeed \"close\".", "was_revised": true, "final_justification": "This is a good paper, and I am happy that the authors responded to the concerns of the reviewers to the extent that the paper has reached a unanimous positive rating.", "rebuttal": "We appreciate the reviewer’s valuable comments and questions.\n\nBelow, we provide our responses to the reviewer’s concerns and questions.\n\n## [2-1] Regarding \"Details of proof\"\n\nDue to space limitations, we include only the interpretations of the statements in the main paper. We will provide a proof sketch in the final version.\n\n---\n\n## [2-2] Regarding \"FCGrad uses the weighted sum of two gradients: g = ...\"\n\nThat is correct. We will revise g to g_fcgrad in line 168.\n\n---\n\n## [2-3] Regarding \"Table 1: bold and significant tests\"\n\nWe will include the standard deviation in the final paper. The following table shows the mean and standard deviation of Jain and Gini indices for each algorithm. (We have added two more seeds for Harvest and Cleanup.)\n\n| Task (Metric)     | col       | ind       | iar       | wei       | pcg       | aga       | fcgrad (ours) |\n|-------------------|-----------|-----------|-----------|-----------|-----------|-----------|----------------|\n| Coin (Gini)       | 0.47±0.00 | 0.51±0.08 | 0.12±0.02 | 0.05±0.00 | 0.04±0.01 | 0.29±0.02 | 0.01±0.01     |\n| Coin (Jain)       | 0.53±0.00 | 0.50±0.08 | 0.94±0.02 | 0.99±0.00 | 0.99±0.00 | 0.75±0.02 | 1.00±0.00   |\n| Cleanup (Gini)    | 0.50±0.12 | 0.49±0.15 | 0.44±0.21 | 0.23±0.06 | 0.36±0.21 | 0.32±0.06 | 0.21±0.11     |\n| Cleanup (Jain)    | 0.50±0.12 | 0.53±0.18 | 0.61±0.24 | 0.83±0.06 | 0.69±0.24 | 0.74±0.08 | 0.84±0.12     |\n| Harvest (Gini)    | 0.24±0.13 | 0.12±0.02 | 0.09±0.01 | 0.16±0.06 | 0.09±0.01 | 0.10±0.03 | 0.08±0.05     |\n| Harvest (Jain)    | 0.82±0.15 | 0.95±0.02 | 0.97±0.00 | 0.91±0.07 | 0.97±0.01 | 0.96±0.02 | 0.97±0.03     |\n\nThe following table shows the p-values from Student’s t-test comparing FCGrad to each baseline:\n\n| Task            | col   | ind   | iar   | wei   | pcg   | aga   |\n|-----------------|-------|-------|-------|-------|-------|-------|\n| Coin (Gini)     | 0.00  | 0.01  | 0.01  | 0.00  | 0.06  | 0.00  |\n| Coin (Jain)     | 0.00  | 0.01  | 0.04  | 0.01  | 0.11  | 0.00  |\n| Cleanup (Gini)  | 0.01  | 0.04  | 0.10  | 0.74  | 0.25  | 0.12  |\n| Cleanup (Jain)  | 0.00  | 0.05  | 0.12  | 0.90  | 0.29  | 0.15  |\n| Harvest (Gini)  | 0.04  | 0.13  | 0.55  | 0.03  | 0.46  | 0.42  |\n| Harvest (Jain)  | 0.07  | 0.22  | 0.99  | 0.11  | 0.91  | 0.68  |\n\nNote that these results reflect fairness only, without considering task performance. In the Unfair Coin Game, FCGrad achieves higher fairness metrics than all baselines, with strong statistical significance. For Cleanup, FCGrad outperforms all except wei, which achieves similar fairness. In Harvest, most algorithms except col show comparable fairness to FCGrad. Even though FCGrad does not always outperform others in raw fairness scores, it achieves statistically significant improvement in terms of α-fairness (please refer to our response to Reviewer ZK8G03 [1-2]).\n\n\n\n## [2-4] Regarding \"a brief discussion about the dual purpose of this research: can someone use the same paradigm as FCGrad to intentionally decrease fairness?\"\n\nThank you for the helpful and meaningful suggestion. We would like to clarify that FCGrad does not directly optimize for fairness as an explicit objective; rather, fairness emerges as a consequence of balancing individual and collective objectives. Therefore, we do not believe FCGrad can be trivially repurposed to intentionally decrease fairness.\nIn fact, an extreme case of not considering fairness at all would be optimizing only for the collective objective, i.e., the Col baseline, which may lead to highly imbalanced outcomes despite maximizing the overall return.\n\nThat said, as the reviewer suggests, there are certainly ways an agent could attempt to manipulate fairness, such as by maximizing its own reward while actively minimizing others’. Detecting and preventing such adversarial behaviors presents an important and underexplored direction for future work in fairness-aware multi-agent systems."}, {"review_id": "Se4246naK4", "reviewer": "Reviewer_V4P2", "summary": "The paper introduces a conflict-aware gradient adjustment-based multi-agent RL method to deal with both cooperation and fairness in mixed-motive games. The algorithm determines whether a conflict occurs through the inner product of the gradients of individual and collective objectives. If there is no conflict, FCGrad uses weighted sum gradient adjustment. If there is conflict, FCGrad places more weight on the individual (collective) gradient when the collective (individual) objective is greater, in order to ensure fairness. Experimental results show that the proposed algorithm outperforms the baselines in fairness in conventional mixed-motive game tasks.", "questions": "1. What are the specific differences between the algorithms FCGrad and PCGrad?\n\n2. Do the reward functions for both individual and collective objectives come from feedback from the environment?\n\n3. In addition to fairness metrics, how well does the algorithm perform on the task itself?\n\n4. How scalable is the algorithm to multi-agent systems?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths：\n1. The paper explicitly guarantees fairness through conflict-aware gradient adjustment. The theory is complete with synchronous and monotonic improvement guarantees for individual and collective objectives, and ultimate goal consistency.\n\n2. The computational cost is only the gradient projection operation, so the computational overhead is low.\n\n3. In typical mixed-motive games, the proposed algorithm outperforms the baselines in fairness metrics.\n\nWeaknesses：\n1. The paper only considers individual-collective conflicts, but potential goal conflicts between agents (individual-individual) are not involved.\n\n2. The experimental indicators mainly reflect fairness, but lack measurement of \"task completion\" itself.\n\n3. It has not been verified in large-scale complex heterogeneous multi-agent systems.\n\n4. The proposed algorithm is highly dependent on the periodic occurrence of gradient conflicts, but periodic occurrence may not be guaranteed to always hold true in real-world.\n\n5. The accuracy of gradient estimation for the two objectives may also be a potential problem.", "was_revised": true, "final_justification": "The authors addressed my concerns in rebuttal and added corresponding experiments and discussion, so I raised my score.", "rebuttal": "We appreciate the reviewer’s valuable comments and questions.\n\nBelow, we provide our responses to the reviewer’s concerns and questions.\n\n## [3-1] Regarding \"individual-collective conflicts\"\n\nWe believe that individual–collective conflict implicitly includes individual–individual conflicts in the form of \"one agent versus the others.\" For example, consider a three-agent case where the collective reward is defined as the average of individual rewards. If agent 1’s reward is lower than the average, i.e., (R1+R2+R3)/3 > R1, then it must be the case that the average of R2 and R3 is greater than R1, implying at least one agent is outperforming agent 1.\n\n---\n\n## [3-2] Regarding \"lack measurement of 'task completion' itself\"\n\nWe respectfully clarify that the main results in Section 4 are based on $\\alpha$-fairness, which simultaneously captures both \"task completion\" and \"fairness.\" As discussed in the paper:\n- $\\alpha=0$-fairness (equivalent to average return) captures only task completion;\n- $\\alpha=1$-fairness (geometric mean) and $\\alpha=\\infty$-fairness (minimum individual return) incorporate both task performance and fairness, with increasing emphasis on fairness.\n\n---\n\n## [3-3] Regarding \"large-scale complex heterogeneous multi-agent systems\"\n\nOur current experiments focus on fairness and cooperation in environments that exhibit asymmetries, which we consider a form of heterogeneity. In addition, Cleanup and Harvest have been considered as fairly large-scale and complex environments. Scaling FCGrad to extremely large or structurally heterogeneous agent populations is an important challenge, and we agree it would be valuable future work.\n\n---\n\n## [3-4] Regarding \"the periodic occurrence of gradient conflicts\"\n\nWe agree with the reviewer that the theoretical framework assumes the periodic occurrence of gradient conflicts. While this assumption may not always hold in real-world scenarios, our empirical results demonstrate that FCGrad performs well in the tested environments, where such conflicts naturally arise during learning. Investigating robustness under less frequent or non-periodic conflict patterns would be a promising direction for future research.\n\n---\n\n## [3-5] Regarding \"The accuracy of gradient estimation\"\n\nWe agree with the reviewer that the quality of gradient estimation can impact learning. However, similar to general stochastic gradient methods, we mitigate this issue by using a sufficiently large number of samples when estimating gradients.\n\n---\n\n## [3-6] Regarding \"the specific differences between FCGrad and PCGrad\"\n\nFirst, PCGrad was originally proposed for multi-objective learning, whereas FCGrad is designed for achieving fair cooperation in multi-agent reinforcement learning. Second, PCGrad does not adaptively consider which gradient direction should be prioritized. It resolves conflicts by projecting each conflicting gradient onto the normal plane of the other and summing them (see Section 2.3 of our paper).\n\nIn contrast, FCGrad adaptively selects which gradient to project based on the expected returns: the gradient associated with the lower expected return is projected onto the normal plane of the other. This mechanism promotes fairness by favoring the underperforming objective in conflict situations.\n\n---\n\n## [3-7] Regarding \"reward functions\"\n\nIn our setup, each agent has access to all individual rewards provided by the environment. The collective reward is computed as the average of these individual rewards. Thus, both individual and collective rewards originate from the environment, with the latter being a derived aggregate.\n\n---\n\n## [3-8] Regarding \"perform on the task itself\"\n\nAs described in our main results (Section 4), we use $\\alpha$-fairness as a unified metric that captures both task performance (explicit reward) and fairness. Specifically, $\\alpha = 0$ corresponds to average return (pure task performance), while $\\alpha = 1$ and $\\alpha = \\infty$ incorporate increasing emphasis on fairness. FCGrad achieves competitive performance under $\\alpha = 0$ and outperforms other baselines under $\\alpha = 1$ and $\\alpha = \\infty$, indicating it balances both objectives effectively.\n\n---\n\n## [3-9] Regarding \"How scalable is the algorithm to multi-agent systems\"\n\nSince FCGrad is a decentralized algorithm, it is easy to scale, as reviewer ZK8G mentioned as one of our strengths (“2. It is a decentralized training approach which gives it scalability.”) We believe scaling to very large populations is a potential future direction."}, {"review_id": "bj80cK8fXA", "reviewer": "Reviewer_ATyi", "summary": "This paper proposes a conflict-aware gradient adjustment method (FCGrad) for multi-agent reinforcement learning that promotes cooperation in mixed-motive games while ensuring fairness among agents.", "questions": "These two works seem to have somewhat similar insights. I’m curious if there is any connection between them and would like to hear the author’s thoughts.\n- Duque, Juan Agustin, et al. \"Advantage Alignment Algorithms.\" _arXiv preprint arXiv:2406.14662_ (2024).", "limitations": "Mentioned in weaknesses.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n- The research problem concerns mixed-motive tasks, which is an issue that the MARL community should pay attention to.\n- The method is straightforward and aligns well with intuition.\n- The choice of experimental environments is consistent with the MARL setting and supports the authors’ claims.\n\n\nWeaknesses\n\n- The research focuses on mixed-motive tasks but lacks discussion of solution concepts from game theory. What equilibrium is being targeted here? How does changing the gradient direction help find a better equilibrium?\n- Based on the authors’ narrative, it seems to be about the direction of motivation/preference, but the method modifies the gradient direction instead. Is there a gap between these two?\n- Fairness and resolving conflict seem to be two separate issues. If I understand correctly, fairness here refers to taking the mean direction in the conflict-free case in Figure 1. But wouldn’t this affect the optimality of the problem?\n    \n- The motivation might not be very clear. How is the trade-off between optimality and fairness handled? For example, in the authors’ Cleanup environment, the optimal solution is division of labor — one agent continuously cleans the pollution while the other continuously collects apples, which maximizes efficiency since they don’t switch roles. However, this is unfair. Some prior work adjusts the reward allocation to address this, but here the authors do not introduce any additional mechanism. Since fairness is emphasized, it should affect optimality, so how should the metric be defined in this case?\n\nOverall, I think these are key questions. If there is a reasonable response, I am open to raising the score.", "was_revised": true, "final_justification": "All issues are resolved.", "rebuttal": "We appreciate the reviewer’s valuable comments and questions.\n\nBelow, we provide our responses to the reviewer’s concerns and questions.\n\n## [4-1] Regarding \"discussion of solution concepts from game theory\"\n\nRegarding \"What equilibrium is being targeted here?\"\nWe clarify that our work does not aim to explicitly compute or characterize a game-theoretic equilibrium (e.g., Nash or correlated equilibrium). Instead, our focus is on learning Pareto-optimal joint policies in mixed-motive multi-agent environments.\n\nIn these settings, pure equilibrium solutions (such as Nash equilibrium) often result in inefficient or unfair outcomes. Rather than relying on equilibrium concepts, we directly optimize the joint training dynamics such that all agents’ individual value functions converge, while maximizing the performance. This guarantees that the learned outcome lies on the Pareto front.\n\n\nTo evaluate the quality of these outcomes, we adopt the α-fairness framework, which allows us to capture different trade-offs between efficiency and fairness:\n- $\\alpha=0$-fairness (equivalent to average return) captures only task completion;\n- $\\alpha=1$-fairness (geometric mean) and $\\alpha=\\infty$-fairness (minimum individual return) incorporate both task performance and fairness, with increasing emphasis on fairness.\n\nWe will clarify in the final version that our solution concept is Pareto optimality.\n\n\n---\n\n## [4-2] Regarding \"the direction of motivation\"\n\nWe represent the direction of motivation through gradient adjustment. In reinforcement learning, the policy is guided by the value function. FCGrad modifies this guidance by adjusting the gradient direction to promote fairness and cooperation.\n\n---\n\n## [4-3] Regarding \"Fairness and resolving conflict seem to be two separate issues\"\n\nFCGrad aims to achieve fairness through conflict-aware gradient adjustment. Fairness in our context refers to the actual returns received by each agent, not to taking the mean of gradient directions. FCGrad allows agents to prioritize the collective objective when they are in an unfair situation (i.e., receiving less return than others), and vice versa. This adaptive adjustment promotes fairness while still optimizing expected returns. As shown in our theoretical results, this procedure yields a monotonic non-decreasing improvement in the value functions.\n\nRegarding optimality, we emphasize that it depends on the chosen metric. This is precisely why we adopt the $\\alpha$-fairness framework, which jointly considers both task completion and fairness. For example, in terms of average return ($\\alpha=0$), which is widely used in the literature and ignores fairness, FCGrad performs comparably to the Col baseline (which maximizes collective return) in the Cleanup and Harvest environments, and even slightly worse in Unfair Coin. However, in terms of $\\alpha=1$ and $\\alpha=\\infty$, which incorporate fairness to increasing degrees, FCGrad consistently outperforms the baselines. This demonstrates that FCGrad achieves a favorable trade-off between fairness and performance.\n\nMore broadly, $\\alpha$-fairness defines a family of Pareto-optimal solutions with varying emphasis on fairness. By adjusting gradients to navigate toward different regions of the Pareto front, FCGrad enables a flexible trade-off between fairness and efficiency, rather than sacrificing one for the other.\n\n\n---\n\n## [4-4] Regarding \"motivation\"\n\nThe example provided by the reviewer—division of labor that maximizes average return but is unfair—is precisely the kind of solution FCGrad is designed to address (as also mentioned in the Introduction). As shown in Figure 4(a), the Col baseline, which optimizes collective return, performs poorly in terms of Min ($\\alpha = \\infty$), indicating that some agents receive disproportionately low rewards under \"efficient but unfair\" policies. In contrast, FCGrad achieves a minimum return close to 40, suggesting that even the least-rewarded agent is actively collecting apples.\n\nWhile some prior works attempt to address fairness through reward shaping (as discussed in Section 2.4), these typically rely on gifting mechanisms—where agents transfer rewards to others. Fairness is then measured over the sum of task rewards and gifts. However, this can be problematic: for example, one agent may clean pollution alone while receiving only \"intrinsic\" rewards, which does not reflect actual participation in the task. In contrast, we define fairness solely based on task rewards to better capture each agent’s true contribution.\n\nFCGrad addresses this imbalance through gradient adjustment. Specifically, when an agent’s individual return is lower than the collective return, FCGrad updates its policy in the direction of its individual value function (e.g., toward collecting apples). Conversely, agents whose individual return exceeds the collective return shift their gradient toward the collective objective (e.g., cleaning pollution). This dynamic leads to more balanced task participation and promotes fairness while maintaining high overall performance.\n\nTo capture the trade-off between fairness and optimality, we adopt $\\alpha$-fairness as a unified metric. For $\\alpha > 0$, it balances both aspects, with higher $\\alpha$ placing more emphasis on fairness. As our results show, FCGrad outperforms baselines in terms of $\\alpha = 1$ and $\\alpha = \\infty$, achieving strong fairness while maintaining high task performance. To the best of our knowledge, we are the first to report $\\alpha$-fairness as an evaluation metric in mixed-motive multi-agent reinforcement learning.\n\n---\n\n## 4-5) Regarding \"connection with [Duque et al.]\"\n\nWe appreciate the reviewer pointing out the work by Duque et al., which indeed shares a high-level motivation with ours—namely, addressing the misalignment between individual and collective objectives, particularly in social dilemmas where selfish behavior leads to suboptimal outcomes.\n\nWhile both works aim to promote cooperation, their approaches differ fundamentally. Duque et al. attempt to resolve this misalignment by aligning agents' advantages, shaping the behavior of other agents to promote robust cooperation. Their method seeks to mitigate the conflict, rather than leveraging it.\n\nIn contrast, our method explicitly detects and addresses such conflicts during training via conflict-aware gradient adjustments. Moreover, fairness is a core objective in our framework. Unlike Duque et al., which does not incorporate fairness metrics, FCGrad is designed to promote both cooperation and fairness by dynamically adjusting the learning direction based on disparities in agent returns.\n\nIn summary, while the high-level motivation is similar, our objectives and mechanisms differ: Duque et al. focus on advantage alignment to foster cooperation, whereas we explicitly exploit gradient-level conflicts to achieve fair and cooperative outcomes."}]

Author rebuttal:

Meta-review: The paper is technically solid, makes an original contribution to fairness-aware MARL, and generated thoughtful discussion that improved its clarity and positioning. The final version should incorporate the promised clarifications regarding theoretical proofs, fairness definitions, connections to redistribution methods, and the scope of applicability.
Decision: Accept (spotlight)