Title: Fair Cooperation in Mixed-Motive Games via Conflict-Aware Gradient Adjustment
Venue: NeurIPS 2025
Accepted: yes
Keywords: Fairness

Cooperation

Mixed-Motive Games

Multi-agent Reinforcement Learning

Gradient adjustment
Abstract: Multi-agent reinforcement learning in mixed-motive settings presents a fundamental challenge: agents must balance individual interests with collective goals, which are neither fully aligned nor strictly opposed. To address this, reward restructuring methods such as gifting and intrinsic motivation have been proposed. However, these approaches primarily focus on promoting cooperation by managing the trade-off between individual and collective returns, without explicitly addressing fairness with respect to agents’ task-specific rewards. In this paper, we propose an adaptive conflict-aware gradient adjustment method that promotes cooperation while ensuring fairness in individual rewards. The proposed method dynamically balances policy gradients derived from individual and collective objectives in situations where the two objectives are in conflict. By explicitly resolving such conflicts, our method improves collective performance while preserving fairness across agents. We provide theoretical results that guarantee monotonic non-decreasing improvement in both the collective and individual objectives and ensure fairness. Empirical results in sequential social dilemma environments demonstrate that our approach outperforms baselines in terms of social welfare, while maintaining fairness.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a conflict-aware gradient adjustment method (FCGrad) for multi-agent reinforcement learning that promotes cooperation in mixed-motive games while ensuring fairness among agents.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n- The research problem concerns mixed-motive tasks, which is an issue that the MARL community should pay attention to.\n- The method is straightforward and aligns well with intuition.\n- The choice of experimental environments is consistent with the MARL setting and supports the authors’ claims.\n\n\nWeaknesses\n\n- The research focuses on mixed-motive tasks but lacks discussion of solution concepts from game theory. What equilibrium is being targeted here? How does changing the gradient direction help find a better equilibrium?\n- Based on the authors’ narrative, it seems to be about the direction of motivation/preference, but the method modifies the gradient direction instead. Is there a gap between these two?\n- Fairness and resolving conflict seem to be two separate issues. If I understand correctly, fairness here refers to taking the mean direction in the conflict-free case in Figure 1. But wouldn’t this affect the optimality of the problem?\n    \n- The motivation might not be very clear. How is the trade-off between optimality and fairness handled? For example, in the authors’ Cleanup environment, the optimal solution is division of labor — one agent continuously cleans the pollution while the other continuously collects apples, which maximizes efficiency since they don’t switch roles. However, this is unfair. Some prior work adjusts the reward allocation to address this, but here the authors do not introduce any additional mechanism. Since fairness is emphasized, it should affect optimality, so how should the metric be defined in this case?\n\nOverall, I think these are key questions. If there is a reasonable response, I am open to raising the score.\n\nQUESTIONS:\nThese two works seem to have somewhat similar insights. I’m curious if there is any connection between them and would like to hear the author’s thoughts.\n- Duque, Juan Agustin, et al. \"Advantage Alignment Algorithms.\" _arXiv preprint arXiv:2406.14662_ (2024).\n\nLIMITATIONS:\nMentioned in weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a conflict-aware gradient adjustment-based multi-agent RL method to deal with both cooperation and fairness in mixed-motive games. The algorithm determines whether a conflict occurs through the inner product of the gradients of individual and collective objectives. If there is no conflict, FCGrad uses weighted sum gradient adjustment. If there is conflict, FCGrad places more weight on the individual (collective) gradient when the collective (individual) objective is greater, in order to ensure fairness. Experimental results show that the proposed algorithm outperforms the baselines in fairness in conventional mixed-motive game tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths：\n1. The paper explicitly guarantees fairness through conflict-aware gradient adjustment. The theory is complete with synchronous and monotonic improvement guarantees for individual and collective objectives, and ultimate goal consistency.\n\n2. The computational cost is only the gradient projection operation, so the computational overhead is low.\n\n3. In typical mixed-motive games, the proposed algorithm outperforms the baselines in fairness metrics.\n\nWeaknesses：\n1. The paper only considers individual-collective conflicts, but potential goal conflicts between agents (individual-individual) are not involved.\n\n2. The experimental indicators mainly reflect fairness, but lack measurement of \"task completion\" itself.\n\n3. It has not been verified in large-scale complex heterogeneous multi-agent systems.\n\n4. The proposed algorithm is highly dependent on the periodic occurrence of gradient conflicts, but periodic occurrence may not be guaranteed to always hold true in real-world.\n\n5. The accuracy of gradient estimation for the two objectives may also be a potential problem.\n\nQUESTIONS:\n1. What are the specific differences between the algorithms FCGrad and PCGrad?\n\n2. Do the reward functions for both individual and collective objectives come from feedback from the environment?\n\n3. In addition to fairness metrics, how well does the algorithm perform on the task itself?\n\n4. How scalable is the algorithm to multi-agent systems?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a new gradient-based algorithm for MARL in which the returns of each agent are shaped to consider not only individual objectives, but also social welfare and fairness. The new algorithm balances each agent's learning to explicitly account for the individual and shared objectives so they are solved to maximize fairness. The paper includes two theorems that together form a theoretical basis for the learned policies to be optimal. Empirical evaluation includes three social dilemma domains and a variety of agents that are self-interested. The results show that the new method, FCGrad, enables the agents to learn a balanced policy that maintains fairness using various metrics.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n+ This paper is one of the most precise and clear papers I have read recently. Well done!\n+ The related work is comprehensive, follows a clear narrative, and provides context to understand the novelty of the new contribution.\n+ Empirical evaluation is strong, with environments especially modified with asymmetries to emphasize the role of fairness in these domains.\n+ The results support the overall superior performance of the new approach across several metrics.\n\nWeaknesses:\n- The main weakness is the theoretical analysis. Theorems are considered a part of the main paper, but the proofs are solely in the appendix, so they cannot be considered a fully sound contribution. Theorem 3.2 is a significant leap and requires additional preliminary explanations.\n- In line 168, \"FCGrad uses the weighted sum of two gradients: g = ...\" I assume g and g_FCGrad are supposed to be the same, because if they're not, I'm not sure where the latter is used.\n- In Table 1, results are bolded not just for the top-performing algorithm, but also for the second-best. This is confusing, especially when no significance is reported, so there's no way to verify these results are indeed \"close\".\n\nQUESTIONS:\n1. Did you perform any significance tests for the results in Table 1?\n\nLIMITATIONS:\nIt can be beneficial to have a brief discussion about the dual purpose of this research: can someone use the same paradigm as FCGrad to intentionally *decrease* fairness?", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn summary, this paper introduces the idea of fair and conflict-aware gradient adjustments for the updates of policies which can achieve cooperation and fairness. It outperforms current baselines in terms of fairness in the social welfare in mixed-motive multi-agent sequential social dilemmas.\n\nThey propose a fair and conflict-aware gradient adjustment method (FCGrad) which guarantees a monotonic non-decrease of both individual and collective objectives, while preserving fairness across each individuals objective.\nFCGrad, which works in a decentralized model, uses two value functions, for the individual and for the collective. This imposes the assumption that there is a shared global reward. They then find the gradients of the individual and the collective objectives respectively. They then check the sign inner product of these gradients. If positive, they use a weighted sum of the two gradients, and if it is negative, there is a conflict. It will then place more weight on the gradient that is lower to ensure fairness. They do this by projecting the individual or collective gradients onto the normal plane of another gradient vector, the exact form is described in equation (2).\n\nThis is shown both theoretically and empirically.\n\nThe empirical results are shown in the unfair coins, cleanup and harvest environments, which are slightly tweaked versions of previously used versions of these environments.\nThey compare with 6 baselines: Collective reward optimization (Col), Individual reward optimization (Ind), inequity aversion reward restructuring (IA), FCGrad without conflict handling, PCGrad, and Altruistic Gradient Adjustment. They also include a hyperparameter for the weighting of FCGrad, they show the choice of this value is significant to the method working, and is environment dependent.\n\nOverall they show that FCGrad (Their method) results in being in the top two fairest outcomes across the methods over multiple fairness metrics, Gini and Jain. (Except Jain in Harvest, but it still close to the top of the performance)\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1.\tThis is an interesting take on how to achieve fairness in these mixed-motive environments.\n\n2.\tIt is a decentralized training approach which gives it scalability.\n\n3.\tIt compares to a number of known algorithms for fairness over multiple well known environments.\n\nWeaknesses:\n\n1.\tIt does describe how the model is set up, however, I believe a diagram would help a lot, even it were in the appendix.\n\n2.\tThe reported results only being over 4 random seeds makes it seem like its still possible that these are specific ones which it works particularly well for. Increasing the number of runs could help.\n\nQUESTIONS:\n1.\tWhile it is not the focus of this study. What do you think the outcome of agents trained with this type of reward being mixed into an environment with other training regimes? \n\n2.\tWould they be exploited?\n\n3.\tWould they be able to coordinate with different behaviours that are off policy from their own?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}