Title: Learning to Balance Altruism and Self-interest Based on Empathy in Mixed-Motive Games
Venue: NeurIPS 2024
Accepted: yes
Keywords: mixed-motive games

multi-agent reinforcement learning

cooperation

gifting

empathy
Abstract: Real-world multi-agent scenarios often involve mixed motives, demanding altruistic agents capable of self-protection against potential exploitation. However, existing approaches often struggle to achieve both objectives. In this paper, based on that empathic responses are modulated by learned social relationships between agents, we propose LASE (**L**earning to balance **A**ltruism and **S**elf-interest based on **E**mpathy), a distributed multi-agent reinforcement learning algorithm that fosters altruistic cooperation through gifting while avoiding exploitation by other agents in mixed-motive games. LASE allocates a portion of its rewards to co-players as gifts, with this allocation adapting dynamically based on the social relationship --- a metric evaluating the friendliness of co-players estimated by counterfactual reasoning. In particular, social relationship measures each co-player by comparing the estimated $Q$-function of current joint action to a counterfactual baseline which marginalizes the co-player's action, with its action distribution inferred by a perspective-taking module. Comprehensive experiments are performed in spatially and temporally extended mixed-motive games, demonstrating LASE's ability to promote group collaboration without compromising fairness and its capacity to adapt policies to various types of interactive co-players.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces LASE, a novel distributed multi-agent reinforcement learning algorithm. LASE aims to foster altruistic cooperation through a gifting mechanism while avoiding exploitation in mixed-motive games. The algorithm dynamically adjusts the allocation of rewards based on social relationships inferred using counterfactual reasoning. The paper reports comprehensive experiments in various mixed-motive games, demonstrating that LASE effectively promotes group collaboration and adapts policies to different co-player types without compromising fairness.\n\nSTRENGTHS:\nThe paper presents an innovative combination of gifting mechanisms and counterfactual reasoning within a multi-agent reinforcement learning framework. This approach to dynamically adjust reward allocation based on inferred social relationships is novel and well-grounded in developmental psychology theories of empathy.\n\nThe paper is clearly written and well-organized. The methodology and experimental setup are described in detail, making it relatively easy to follow and reproduce the results. Additionally, the authors conducted comprehensive experiments across various mixed-motive game scenarios, thoroughly demonstrating the effectiveness of LASE in promoting cooperation and fairness. The breadth and depth of these experiments add significant credibility to the proposed method.\n\nWEAKNESSES:\nThe proposed method, while interesting, could be seen as incremental since it primarily combines existing techniques (gifting and counterfactual reasoning). The novelty could be better highlighted by contrasting more explicitly with prior works. \n\nThe more important weakness is that the paper lacks a detailed analysis of the individual contributions of the gifting mechanism and counterfactual reasoning. For example, there is no direct comparison with the original gifting method or an ablation study isolating the impact of the counterfactual reasoning module. This makes it difficult to discern the specific roles and contributions of these components to the overall performance. \n\nThus, I think the paper is below the acceptance bar for NeurIPS.\n\nQUESTIONS:\n1.How does LASE compare to the original gifting methods? Including a direct comparison or an ablation study focusing on the gifting mechanism could provide deeper insights into its contribution.\n2.Can you provide an ablation study that isolates the impact of the counterfactual reasoning module? This would help in understanding its specific role and effectiveness.\n3. Why not compare the Equality with other baseline methods? I am curious about the equality performance of baseline methods? from the report of their paper, I guess some of them will have not bad performance.\n\nLIMITATIONS:\nThe paper seems lack of discussion about limitations and broader societal impacts.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes LASE, a multi-agent reinforcement learning framework that aims to improve co-operation between agents in mixed-motive games using transfer of rewards between agents in a zero-sum manner. Specifically, each agent uses counterfactual reasoning to compute a social relationship metric that computes the effect of their actions on the Q-values of other agents. Notably, LASE uses fully decentralized training, in contrast to many related works in the area. Finally, LASE outperforms existing baselines in a variety of popular mixed-motive environments such as Cleanup.\n\nSTRENGTHS:\n1.\tThe paper is well-organized, clearly written and technically sound. The general flow of the paper is smooth and proposed methods are explained reasonably well. The paper has an appropriate number of citations and properly details existing work in the related work section. \n2.\tThe presented framework is fully decentralized, which gives it an advantage to most works in the area that use centralized training decentralized execution (CTDE).\n3.\tThe results are generally promising with LASE showing significant gains in most environments tested. The analysis of co-operation strategies learned by different methods is interesting. In particular, I liked the GO vs LASE BG ablation study in Section 6.3.\n\nWEAKNESSES:\n1) The framework is relatively complex, as it requires learning two additional networks (perspective-taking and Q networks) for each agent.\n2) Given the dependence of the method on joint action, I have doubts about the scalability of the method. As more agents are added into the system, credit assignment would become more difficult. \n3) Some experimental details need to be clarified further.\n\nQUESTIONS:\n1.\tIn general, given the complexity of the framework and the dependence on the joint action, I am sceptical about the scalability of the method. It would be interesting to see more results that test the scalability of the method. Convincing results with 8 agents for Cleanup and 1 other environment would significantly increase the strength of the paper. \n2.\tThe observation conversion network is not trained using the ground truth observations of agent j, but instead using the ground truth observations of agent i (this being key to the fully decentralized claim of the work). This makes me wonder why such a network is needed at all? What happens if a network that directly maps from agent i’s observation to agent j’s policy is learned. It would be interesting to see this in an ablation study. In other words, why is the \"perspective taking\" module not a single network?\n3.\tHow is the SR policy network trained? I am confused why it is trained using RL and not simply using supervised learning using the actual observed actions of agent j.\n4.\tI am not convinced by the argument that removing the cleaning beam in Cleanup makes the environment harder. Removing the cleaning beam also reduces the dimensionality of the action space. What was the reason behind removing it?\n5.\tFigure 1 is slightly misleading as it shows separate parameters for the observation conversion network and SR policy network. However, equation 4 shows them having the same parameters. \n6.\tI am unsure why the policy has a manual epsilon greedy added to it. Actor critic methods are on-policy, the epsilon greedy changes that and I am not sure this manual tuning for the policy is standard. Why was this required?\n\nLIMITATIONS:\nSufficient details provided.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a novel algorithm, LASE, that employs a gifting mechanism in order to steer agents toward equilibria of high social welfare in mixed motive games. A novelty in LASE is that it estimates the \"social relationship\" between a player and its co-players through a counterfactual Q-value baseline. The authors show that LASE performs favourably across a number of temporally-extended social dilemmas.\n\nSTRENGTHS:\nThe paper is well-written and well-presented.\n\nThe authors address a gap in the literature which does so far not attempt to estimate the influence of co-player policies on the joint Q-value function. In general, estimating counterfactuals involving Q-values has in the past been found to suffer from high variance issues (such as in [1]). Hence, I find it positively surprising that the authors' method performs decently across several different environments.\n\n[1] Counterfactual Multi-Agent Policy Gradients, Foerster et al., AAAI 2018\n\nWEAKNESSES:\nI believe the main weakness of the authors' approach is that, under partial observability, a player cannot generally see all parts of its co-players observations, hence making it impossible to fully reconstruct their policy inputs (hence the restriction to common knowledge fields of view in [2]). I believe the author's algorithm should crucially estimate the uncertainty of their social relationship estimates such as to avoid misunderstandings that could erode trust in real-world situations.  \n\nAdditionally, I am unsure about the authors' use of the term \"empathy\" - \"theory of mind\" would certainly work here, but \"empathy\" seems like an inherently emotional concept.\n\n[2] Multi-Agent Common Knowledge Reinforcement Learning, Schroeder de Witt et al., NeurIPS 2019\n\nQUESTIONS:\nWhat do you think could be a real-world application of this line of work?\n\nLIMITATIONS:\nI believe the authors are addressing limitations adequately.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces LASE (Learning to balance Altruism and Self-interest based on Empathy), a multi-agent reinforcement learning algorithm designed for mixed-motive games. LASE uses a gifting mechanism where agents share a portion of their rewards with others based on inferred social relationships. Counterfactual reasoning determines these relationships by comparing the actual joint action's value to a baseline that averages over the other agents' actions. The authors use a perspective-taking module to predict other agents' policies. Experimental results across various sequential social dilemmas show LASE's ability to promote cooperation while maintaining individual interests. The authors claim the following contributions: (1) a computational model of empathy that modulates responses based on inferred social relationships; (2) a decentralized MARL algorithm that balances altruism and self-interest in mixed-motive games, and (3) theoretical analysis of decision dynamics in iterated matrix games and experimental verification of LASE's performance in sequential social dilemmas.\n\nSTRENGTHS:\n* Novel approach: The introduction of LASE as a mechanism to balance altruism and self-interest in multi-agent settings is innovative and addresses an important challenge in mixed-motive games.\n\n* Theoretical contribution: I think the analysis of the algorithm's behavior in iterated matrix games does a good job of grounding the empirical results in game theory.\n\n* The overall evaluation was sufficient to demonstrate the performance of the algorithm in comparison to well-chosen baselines\n\nWEAKNESSES:\nI think the paper is quite strong overall. However, several places can still be improved to increase the potential impact of the work.\n\n* Clarity issues: The paper could benefit from improved clarity in several areas. For example, a bit more clarity on how gifts are determined would be helpful. Consider adding pseudocode for key components like the Social Relationships Inference module or the gifting mechanism and how the modules integrate. \n\n* Insight into counterfactual baseline design choice: The paper's choice of using the average behavior as the counterfactual baseline for determining social relationships warrants further investigation. The authors could consider conducting experiments that vary this baseline, comparing alternatives such as fixed neutral baselines, learned cooperative behavior baselines, or worst-case action baselines. This would provide insights into the robustness of the approach and potentially identify improvements to the algorithm.\n\n* Limited domain coverage: While the paper evaluates LASE across several sequential social dilemmas, which is adequate, the approach could be further strengthened by expanding the range of domains tested. For example, the authors could consider alternatives from MeltingPot domains (https://github.com/google-deepmind/meltingpot).\n\nQUESTIONS:\nDo you have any proposed changes to the paper in response to my critiques?\n\nLIMITATIONS:\nI think there could be more discussion of this, but the current evaluation is sufficient.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-23"}