Learning to Balance Altruism and Self-interest Based on Empathy in Mixed-Motive Games
Authors: Fanqi Kong
Yizhe Huang
Song-Chun Zhu
Siyuan Qi
Xue Feng
Venue: NeurIPS 2024 poster (2024)
Keywords: mixed-motive games
multi-agent reinforcement learning
cooperation
gifting
empathy
Primary area: reinforcement_learning

Abstract: Real-world multi-agent scenarios often involve mixed motives, demanding altruistic agents capable of self-protection against potential exploitation. However, existing approaches often struggle to achieve both objectives. In this paper, based on that empathic responses are modulated by learned social relationships between agents, we propose LASE (**L**earning to balance **A**ltruism and **S**elf-interest based on **E**mpathy), a distributed multi-agent reinforcement learning algorithm that fosters altruistic cooperation through gifting while avoiding exploitation by other agents in mixed-motive games. LASE allocates a portion of its rewards to co-players as gifts, with this allocation adapting dynamically based on the social relationship --- a metric evaluating the friendliness of co-players estimated by counterfactual reasoning. In particular, social relationship measures each co-player by comparing the estimated $Q$-function of current joint action to a counterfactual baseline which marginalizes the co-player's action, with its action distribution inferred by a perspective-taking module. Comprehensive experiments are performed in spatially and temporally extended mixed-motive games, demonstrating LASE's ability to promote group collaboration without compromising fairness and its capacity to adapt policies to various types of interactive co-players.

Learning to Balance Altruism and Self-interest Based on Empathy in Mixed-Motive Games
Fanqi Kong2.1
Yizhe Huang 2.1
Song-Chun Zhu
Siyuan Qi!
Xue Feng es
State Key Laboratory of General Artificial Intelligence, BIGAI
Institute for Artificial Intelligence, Peking University
Department of Automation, Tsinghua University kfq20@stu. pku. edu. en, fengxue@bigai.
Abstract
Real-world multi-agent scenarios often involve mixed motives, demanding altru- istic agents capable of self- protection against potential exploitation. However, existing approaches often struggle to achieve both objectives. In this paper, based on that empathic responses are modulated by inferred social relationships between agents, we propose LASE (Learning to balance Altruism and Self-interest based on Empathy), distributed multi-agent reinforcement learning algorithm that fosters altruistic cooperation through gifting while avoiding exploitation by other agents in mixed-motive games. LASE allocates portion of its rewards to co-players gifts, with this allocation adapting dynamically based on the social relationship metric evaluating the friendliness of co-players estimated by counterfactual reasoning. In particular, social relationship measures each co-player by compar- ing the estimated Q- Q-function of current joint action to counterfactual baseline which marginalizes the co-player's 's action, with its action distribution inferred by a perspective-taking module. Comprehensive experiments and temporally extended mixed-motive games, demonstrating LASE's ability to promote group collaboration without compromising fairness and its capacity adapt policies to various types of interactive co-players.
performed in spatially
Introduction
Multi-agent reinforcement learning (MARL) has exhibited impressive performance in numerous collaborative tasks and zero-sum games such as MPE, StarCraft, and Google Research Football These environments involve a predefined competitive or cooperative relationship between agents. Besides, mixed-motive games prevalent, which the relationships between agents are non-deterministic and dynamic. That is, agents could cooperate with some co-players and simultancously compete with someone else. Furthermore, along with interactions, friends may turn into foes, and vice versa. In such games, to maximize self-interest, agents need to cooperate altruistically in some relationships while keep self-interested to avoid being exploited in some others. Consequently, in mixed-motive environments, the ability to balance altruism and self-interest according to social relationships is crucial for agent performance.
19
25 38
The commonly used CTDE (Centralized Training and Decentralized Execution) methods [31 29
in MARL focus on global optimization goals and necessitate individual information sharing with centralized controllers, which is impractical for self-interest agents in mixed-motive games. On the other hand, simply training self-interest agents in decentralized way may converge to local
*Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

optima, failing to maximize individual interests. For example, in Iterated Prisoner's Dilemma (IPD). decentralized A2C agents converge to defection, getting the minimal reward (see details Fig. 4).
Within the framework of decentralized learning, the gifting mechanism has been used to address the decision-making problems in mixed-motive games by enabling agents transfer a portion of their rewards to others [20] 4]. Agents independently select gift recipients and determine gift amount, which complies with the decentralized requirement. Gifting can potentially shape co-players' policies and even incentivize them to behave more altruistically by influencing co-players' reward
structure. Previous work has studied handcrafted gifting scheme [20 34 and has learned end-to-end neural networks to determine the reward transfer scheme [36
37 However, a consideration of the correlation between social relationships and response strategies is absent, which is crucial for eerision-maaing in mixed-motive games.
To balance the dilemma between altruism and self-interest caused by the non-deterministic and dy- namic relationship between agents, it is a feasible way that adaptively modulating the gift amount to others according to the social relationships. This process is called (cognitive) empathy developmen- tal psychology 288). What's ' more, previous studies and human behavioral experiments have shown that empathy can promote the emergence of altruism among self-interested individuals I 2 30]. To the best of our knowledge, there has been a lack of computational models of empathy and the study of empathy- -based drcision-maainng In this work, we propose computational model of empathy, in which social relationship is measured by continuous variable, capturing the influence of co- co-players' behavior on the focal agent's 's reward and guiding gift scheme. On that basis, we provide distributed MARI algorithm LASE (Learning to balance Altruism and Self-interest based on Empathy) to address the dilemma between altruism and self-interest in mixed- motivee games.
LASE uses counterfactual reasoning to infer the different social relationships with different co- players separately by comparing the estimated Q-value for the joint action a counterfactual baseline established for each other agent. The counterfactual baseline marginalizes a single agent's is action while keeping the other agents' actions fixed. This computational approach to social relationships enables LASE to explicitly decompose the value contributions of other agents to it, thereby providing clearer guidance for gift allocation. That is, gift more to the co-players who contribute more. Additionally, to deal with the challenge of inferring co-players in partially observable and decentralized environments, LASE is equipped with a perspectivee taking module to predict others' policies by converting LASE's local observation to simulated observation of others.
To verify the effectiveness of LASE, we theoretically analyze its dynamics of decision-mate making in iterated mixed-motive games and conduct comprehensive experiments in spatially and temporally extended mixed-motive games. The results demonstrate LASE's ability to promote group cooperation without compromising fairness and its capability of self-protectonn against potential exploitation.
This paper makes three main contributions.
the first to com- putationally model empathy which modulates the response based on inferred social relationships. (2) We present LASE, a decentralized MARL algorithm that balances altruism and self-interest in mixed-motive games. It flexibly adapts strategy to promote cooperation while mitigating exploitation by others. (3) We provide theoretical analysis of decision dynamics in iterated matrix games and experimentally verify that LASE outperforms baselines in a variety of sequential social dilemmas.
(1)
To best knowledge.
2 Related work
In multi-agent learning, the game-theoretic notion of social dilemmas has been generalized from the classic two-player matrix-form games Tab. to sequential social dilemmas, a spatial-temporally extended complex behavior learning setting [15] 4l. Various approaches have been proposed to foster cooperative behavior among agents to advance societal welfare. One approach incorporates the rewards of others as intrinsic rewards into one's own optimization objectives, with the ratio of intrinsic to extrinsic rewards determined by different methods, such as pre-defined social value orientations like altruistic or prosocial [21 23). introducing the concept of inequity aversion [I]] or learning in a model-free way (33] However, this approach depends on direct access to others' reward functions, which may not be feasible in realistic mixed-motive games. Another line of work dispenses with this assumption, allowing agents to model others and influence them through their
actions 66 17 13 8 [0). Here, employ more direct form of opponent shaping named gifting.
2

---

As peer rewarding mechanism that allows agents to reward other agents as part of their action space (20), gifting can be viewed as a process of redistributing rewards among agents [2] the key difference is that in our work, gifting does not require a powerful centralized controller to decide on the allocation. Instead, individuals make their own decisions about gifting, which is more in line with the setting of decentralized training. [<4] [35] study the theoretical basis of how gifting promotes cooperative behavior in simple social dilemmas, while LIO [36] independently learns an incentive function gift others. However, the rewards used for gifting in LIO are determined by the incentive function rather than split from its own reward. LIO doesn't It adhere to zero-sum conditions, which to some extent alters the original game- theorete nature. LToS [37] models the optimization problem of gifting (sharing) weights with the zero-sum setting as a bi-level problem and uses an end-to-end approach to train weights and policies jointly. MOCA [3] introduces contracts to restrict gifting recipients to the agents that fulfill certain behavioral patterns.
but
Our perspective taking module simulates others' behavior by adopting their perspective, technique akin Self-Other Modeling (SOM) 24 The difference is that instead of inferring the agent's goal which may not necessarily be well defined in some environments, we directly imagine their observations, and decouple the agent's 's network for inferring others' actions from its own real policy network utilized for execution. In MARL, some prior studies have employed counterfactual reasoning to deduce the impact of individual actions on others [[3] or the whole team [5]. In contrast, our focus lies assessing others' influence the focal agent.
3. Preliminaries
3.1 Partially observable Markov games
We consider an N --player partially observable Markov game (POMG) 27 18
= (N,S,(O)) (0').{A') P, ( where N represents the number of agents, S represents the state of the environment. In the partially observable setting, agent only obtains the local observation independent policy 7 (a"lo") to select
M
of E O' based the current state s. Each agent i learns actions and form joint action a = (a'...,,'' a²) E AA x ..... X AA, resulting in the state change from s' according to the transition function P 8 A .... A - ASS), where ((8) represents a probability distribution over the set S. Agent i receives an individual extrinsic reward r' R' (s, at,." ... V) and tries maximize a long-term return:
Vi" (5o) M ((t.,as))
Σn >'R'
(s an)
(1)
where the variables in bold represent the joint information of all agents, and y is the discount factor.
3.2 Policy Gradient Learning
In decentralized MARL. each self-interest agent learns an independent policy n' parameterized by 0'. The optimization objective is to maximize the expected return in [g. 1 We use the policy- based Actor-Critic method as the learning algorithm for our agents. The gradient for actor is
(oJ(0) = Exg Et= v/2 a log Ne (uelo,)), where vi represents the critic's evaluation of the actor. The most popular form of the is TD-error th r. + >V* (st41) V (sc).
4 Methodology
To balance altruism and self-interest in mixed-motive games, we propose a distributed MARL algo- rithm LASE which empathically shares rewards with co-players based on inferred social relationships. The architecture of LASE is illustrated in Fig. composed of two main modules: Social Relationship Inference (SRI) and Gifting. SRI conducts counterfactual reasoning to get the social relationships with co-players, which reflects the impact of co-players' actions LASE's return. SRI compares the Q-value (estimated by the SR value network) of the current joint action to a counterfactual baseline which marginalizes the co-player's action, with its action distribution inferred by perspective-taking (PT) module. PT is provided to address the challenge of predicting co-players' policies in partially observable and decentralized environments. In particular, PT consists of an observation conversion network, simulating the co- 1o-player j's observation O² from the local observation o', and an SR policy network, learning function from o' to the inferred j's policy.
The Gifting module, according to the inferred social relationships, determines the amount of reward sharing with others. Meanwhile, agents receive others' gifts and get the final reward yo.tot which
3

---

a'aar'
Env
using the TD-error introduced in Social Relationships Inference
Gifting
n'(o'lo))
Clv (a".(""
Observation
Counterfartual
cunversion network
Persportive Taking
Ressoning
atatel (ef)
Eq.
SR policy network
Figure 1: Architecture of LASE, It consists of Social Relationships Inference (SRI) and Gifting. SRI conducts counterfactual reasoning to get the social relationships with co-players. The social relationship is measured by comparing the Q-value (estimated by the SR value network) of the current joint action counterfactual baseline which marginalizes the co-player's action, with its action distribution inferred by perspective-taking (PT) module. PT is provided to address the challenge of predicting co- co-players' policies in partially observable and decentralized environments. The Gifting module, according to the inferred social relationships, determines the amount of reward to share.
serves optimization target guide the training of policy n'. Section 4.1 and Section 4.2 will introduce our algorithm in detail. LASE's pseudocode is given as Algorithm
4.1 Zero-Sum Gifting
Here, we use the zero-sum gifting mechanism which adheres to the principle of "what give is what I lose" 20), indicating that the overall rewards for the group remain constant. Notably, gifting is
autonomous decision, wherein any agent holds gifting weight vector at time step w [w [ 1==1+
where w [0, 1] and Σ}= w.y = 1. w is the fraction of agent i's reward to gift j. is exactly the social relationship computed as Eq. 3 For an NV-layer group using the zero-sum gifting mechanism,
T' denotes the extrinsic rewards vector obtained through interactions with the environment at timestep t. Agent i's total reward is computed by
Ft (ut r't) = Σ j=1 wi
The policy 7 (aflof) is trained to maxmize E,, (Et 2412:009 2,000 Section 3.2 by replacing Tt with riot
4.2 Social Relationships Inference
The social relationship w modeled as a continuous variable, measures i's inference of j's friendli- ness to him. Based on counterfactual reasoning, w is inferred as:
Qis az) - En!' TडR (af (0)) (a, "a,')))
w
(3)
M
where Q' at), estimated by the SR value network, is i's Q-value of its local observation and the joint action. Eq. 3 compares O (ot at) with counterfactual baseline, which is the weighted sum of Q-values, with j taking all possible actions of while the other agents' actions a fixed. The weight मे (a{ Jā!) j's policy inferred by The denominator is for normalization. M (N - 1)(xx " Qis( (of, (q,3, of min of O'R (oi, (q.², ))), where (N 11) ensures w < N-T. After gifting, LASE keeps the remaining reward itself, wh = 1111 Ej wtd
Due to partial observability and decentralized learning, unable to accurately obtain j's policy mi and its observation of. So utilize PT module to estimate j's policy, denoted as πडд (aş (o/). गड is the SR policy network parameterized by μ which predicts j's actions conditioned on the simulated obervation of learned by the observation conversion network. The observation conversion network, parameterized by 17", enables LASE to adopt the perspective of others and generate a simulated observation of them. At timestep t, LASE processes its own observation of along with another agent

---

j's ID (represented as one-hot vector). yielding the output of which is of the same size as of. To update of and get more accurate 0%. the loss function is
بببد (1 (Gn') = E, E,
-1- S)CE(((())) Si (af of)) + 5110g o/
(4)
The first term aims ensure that the predicted policy TडR (a: (o)) aligns with co-player j's actual actions. The second term aims to minimize the deviation of the simulated observation of from i's true observation oi, that some common features in the environment can be reconstructed. The hyperparameter E [0, 1] balances the two goals.
Since SR policy network predicts actions from an agent's 's peffpeersee and SR value network estimates the agent's 's own returns, we integrated the training processes of the two networks within an actor-critic framework. In training, "ŠR takes one agent's observation as input and outputs probability distribution over his action space. QśR computes the Q-value of the joint action under the current observation. Both networks are updated based on the individual extrinsic rewards obtained in the environment, and the TD-rrrorr defined as 88 = +SsR (ois(oi+ ax+1)- -ssR (oi. az). Specifically, the SR policy network and SR value network, parameterized by p and ø', are updated by
n² n² + Oxe Σ, V n logTss (a)lo)), Ø² === oi + ax E, & O'R (oi, ax).
(5)
We can further intuitively comprehend Eq. when w attains the maximum value of 1/(N - 1). has taken the best action that maximized QuR and predicts with a probability of 1 that will select the action that minimizes QAR- At this point, the value j brings to i far exceeds i's psychological expectation, which corresponds to the real-world scenario where people feel particularly happy when they helped by someone they might have thought was unkind to them. So the amount of i's gifting to j reaches the maximum. It is worth noting that Eq. may yield a negative value, indicating that the agent is required to acquire rewards from other agents. Dealing with this scenario becomes intricate, particularly when the other agent is uncooperative. As this type of competition is not the primary focus of our research, we assign w when w 0.
4.3 Analysis in Iterated Matrix Game
We use iterated matrix games to theoretically analyze LASE's learn- ing process. The iterative matrix game is to play multiple rounds of a single game with the payoff matrix shown in Tab. where both players get payoff of R by mutual cooperation (C) and P by mutual defection (D). If one player defects and the other coop- erates, the defector receives a reward of T, while the cooperator receives a reward of S. We normalize R to 1 and S to O, and let 0 < T < 2, -1 S
Table 1: Matrix-form game
P1/P2
C
D
C
(R, R)
(S,7)
D
(T,S)
(P,P)
which is shown to be sufficient to characterize the three typical kinds of dilemmas in
Tab. 3 26)
We carry out closed-form gradient descent analysis on LASE in the two-player iterated matrix games and derive the policy update rule Eq. 15 and Eq. 16 where each agent i optimizes the reward after gifting pl The detailed deduction is provided in Apppeddix A Then we simulate the policy update iteratively with random initial value and plot LASE's cooperation probability after convergence under various game parameters as illustrated in Fig.
Harmany
10
SHI
The results demonstrate that LASE converges to pure cooperation in Harmony and SH. In the more intense games, SG and PD, LASE sta- bilizes at cooperating with probability greater than 0.5, successfully escaping from non-efficient Nash equilibria.
05
8.
The cooperation probability of LASE agents after convergence under dif- ferent matrix-game parame- ters. The X-axis and Y-axis represent two parameters T and S respectively, where T E [0 0.02 : 2] and S - 1).
Figure 2:
Experimental setup
5.1 Environments
Iterated Prisoner's Dilemma (IPD). Here, we use iterated prisoner's dilemma (IPD) as an illustration to validate the theoretical analysis of LASE conducted in (Section 4.3 and Appendix A The specific game parameters are set [R, S.
1. 0.2, 1.2, 0]. represented
5

---

as the red dot in Fig. We employ the memory-1 I IPD introduced in (6], with the state [CC, CC, DD, sol comprising the joint action in the previous round and the initial state sp- The action space consists of two discrete actions cooperation (C) and defection (D). Each episode lasts for 100 timesteps.
To evaluate the ability of LASE to address more complex environments, we study its performance in partially observable SSDs. SSDs extend matrix-form games in terms of space, time, and number of agents. Here, we study four specific SSDs: Coingame, Cleanup, Sequential Stag-Hunt (SSH), and Sequential Snowdrift Game (SSG) Fig. 3). Schelling diagrams (see 10p of the four environments validate that they are appropriate extensions of representative game paradigms (a detailed analysis is given Appendix B). Below is a detailed description of the four environments.
agent
coin
waste
apple
hare
stag
snowdrift
X
9K
SC
(a) Coingame
(b) Cleanup
(c) SSH
(d) SSG
Figure 3: Graphic representations of four SSDs: (a) Coingame (5x5 map), (b) Cleanup (8x8 map),
(c) SSH (8x8 map). (d) SSG (8x8 map).
Coingame. The Coingame was originally introduced in [16 higher dimensional alternative to the IPD with multi-step actions. In this environment, two agents, 'red' and 'blue", collect coins in a 5 x 5 map. A coin is either red or blue and there is only one coin on the map at any timestep. Agents will get reward of 1 by picking up coin of any color. However, when an agent collects a coin of a different color, the other agent will lose 2 points. After pickup, new coin with random color and random location appears immediately. Therefore, if each agent greedily picks up all the coins, the sum of their expected scores will be
Cleanup. In Cleanup, the goal of the agent is to gather as many apples as possible, with each apple carrying reward of +1. However, the accumulation of waste in the river steadily approaches depletion threshold, causing a linear decline in the apple growth rate to 0. At the beginning of each episode, the waste level exceeds the threshold and there in social dilemma: while individually focusing on collecting apples under the map leads to higher rewards, if all agents opt to refuse waste cleanup, no rewards are obtained. To maintain consistency with other environments, we partially modify the setting of cleanup from [. We eliminate the actions of firing beams (cleaning and zapping) and require the agent to move to the waste's position to clean it. This does not alter the nature of the dilemma but makes it more challenging because the cleaning beam could have helped the agent clean from a distance.
apples in the map. This places the agent
Sequential Stag-Hunt (SSH). This environment is inspired by Markov Stag Hunt in 23 Each agent get a reward of +1 by hunting a hare while hunting stags is more challenging and requires two or more agents. Each stag can bring a reward of +10, which is divided equally among the agents that jointly hunted it. The agent immediately removed from the environment after successfully hunting once. Therefore, if an agent chooses to hunt stags, it must contend with the risk of no one cooperating with it. In contrast, hunting rabbits is safer choice.
Sequential Snowdrift Game (SSG). In SSG, there are 6 piles of snowdrifts which can be removed by the agent. A pile of removed snowdrifts brings +6 reward for each agent, but the remover incurs a cost of 4. So the agent waiting for others to remove the snowdrift (free-rider) can obtain a higher return. However, if no one chooses remove it, no rewards be obtained by anyone.
5.2 Implementations
We employ fully decentralized training and execution for the agents, where all network parameters are independent. The policy structure of the agent comprises convolutional layers for encoding observations, an LSTM layer to capture temporal information, and several fully connected layers activated by ReLUU. The input is a multi-channel binary tensor, with the specific number of channels determined by the characteristics of different environments. For example, in Cleanup, 7 channels are

---

incorporated, wherein the first four channels denote the positions of the four agents, the fifth and sixth channels signify waste and apples, and the last channel distinguishes between the inside and outside of the map through masking. To ensure adequate ex exploration, we let ((aoo) (1 - ()((o))))))))))t)))))))) + A/ with decaying linearly from Estart to fend over <div episodes.
The SR policy network and SR value network of the SRI module share two convolutional layers, followed by their own linear layers activated by ReLU function. The difference is that the in- puts of the linear layers in SR value network include an additional concatenated one-hot vec-
1.50
Episodi
Col Celledive
030
UDLA
tor of the joint action. The observation conver- sion network uses a single convolutional layer to encode observation, which is then concate- nated with one-hot vector representing the of other agents. This concatenated input is then processed through two linear layers, with the resulting output normalized using the sigmoid function.
Agent
(w) Learning puth of LASE
(b) Collective reward
Figure 4: Results in IPD. (a) The learning path of two LASE agents. They start from the lower left and converge the upper right of the phase diagram, where both agents cooperate with prob- ability around 0.93. (b) The collective reward of LASE and baselines. Five seeds are randomly se- lected for the experiment. The solid line represents the mean performance, while the shaded area indi- cates the standard deviation.
In IPD, we modify the implementation by re- moving convolutional layers, reducing the pa- rameters of FC layers, and appropriately increas- ing the learning rate adapt the algorithm to this environment. See AAppendxx C. for more details about the implementations of LASE.
5.3 Baselines
Independent advantage actor-critic labeled A2C [22] is a classical gradient- based RL algorithm. LIO (66] learns an incentive function through the learning updates of reward recipients. LOLA 6
considers the learning process of other agents when updating its own policy parameters. IA
[1 32
modifies the individual reward function by introducing inequity aversion. SI 13 32] achieves coordination by rewarding agents for having causal influence over other agents' actions. We also show the approximate upper bound on performance by training the group optimal (GO) agents to
maximize the collective reward. And we conduct ablation experiments LASE w/o, by removing the observation conversion network and replacing Tड (af" (oğ) Eq. with 1/|,A%).
Results
6.1
LASE promotes cooperation in social dilemmas
LASE successfully escapes the dilemma of non-efficient Nash Equilibrium (D, D). Both LASE agents converge to cooperate with a high probability, around 0.93 (see Fig. 4p) accompanied by high collective reward. This is consistent with the theoretical results shown in Fig. 2 validating the effectiveness of LASE in dealing with social dilemmas. LASE is better at convergence speed and stability than LOLA, which also achieves a high collective reward. Unsurprisingly. GO,
In IPD, as shown in Fig. 4
aiming to maximize group reward, reaches the upper bound. On the other hand, A2C, optimizing for one's own return, easily falls into the Nash equilibrium, where everyone defects and the group reward reduces to the minimum.
In SSH and SSG, LASE nearly reaches the upper bound of the total rewards as shown in Fig. 5a and Fig. 5b) a total reward of 20 for successfully hunting two stags in SSH and a total reward of 120 in SSG for removing all the six snowdrifts. In Coingame (Fig. 3c) and Cleanup (Fig. 5d) LASE exhibits commendable performance by effectively avoiding the sub-optimal equilibriu, where everyone defects. GO outperforms LASE in SSG, Coingame, and Cleanup, because its optimization objective of maximizing the collective reward of all the agents and a strong assumption of the accessibility of everyone's reward function can help it avoid the dilemma.
On the other hand, it's worth emphasizing that the architecture of GO also gets it into the lazy problem [1], shown as its underperformance in SSH Fig. Sa In SSH, carly hunting leads to moving
7

---

Colective reward
500
Calecca rewerd
Colleciive 16
Collectve
Looa
420
(a) SSH
(b) SSG
(c) Coingame
(d) Cleanup
Figure 5: Learning curves in four SSDs. Shown is the collective reward. All the curves are plotted using 5 training runs with different random seeds, where the solid line is the mean and the shadowed area indicates the standard deviation.
out of the environment and failing to obtain the group rewards of others' later hunting. Thus, GO may not hunt until the last few steps, and likely misses the the opportunity of cooperating to hunt stags.
A2C and SI hardly improve the collective return in Coingame and Cleanup, and fail to outperform LASE in SSH and SSG, where the dilemma is less intense. Although IA can directly access the rewards of others and adjust its own intrinsic rewards accordingly, it is still unable to get higher collective return than LASE. Also environments. Meanwhile, the need to manually specify the hyperparameters used to scale the amount of gifting also creates challenges for LIO to apply different environments. In contrast, LASE breaks the limitations and achieves significantly better performance than LIO.
gifting algorithm, LIO struggles to perform well in the four
LASE w/o's convergence speed and performance are affected to a certain extent but not significantly. This is because LASE w/o differs from LASE by replacing the policy predictions of others with uni- form policy compute counterfactual baselines. It means that the gifting will continue as long as the first term of the numerator is substantial in [:q. 3. On the other hand, LASE builds a counterfactual baseline dynamically through perspective taking, and only gifts when the co-player's 's real action is superior the baseline. Tab. shows the mean of the gifting weights other agents over the last le4 episodes of training, showing that LASE's gifting weights are lower than those of LASE w/o. Regarding gifting as form of communication, LASE is valuable in reducing communication costs. Furthermore, to test LASE's scalability, we implement LASE in the extended Cleanup and SSG with agents and a larger map. Results show that LASE is able to deal with more complex environments (see details in Section D.I
Table 2: Mean of the gifting weights for LASE and LASE w/o during the last 10000 episodes of self-play training.
Coingame
Cleanup
SSG
SSH
LASE LASE w/o
0.222
0.346
0.234
0.072
0.343
0.457
0.354
0.081
6.2 LASE promotes fairness
Amount of mexte (bemed
22
A
D55
Extrdi reward
Aguit
Apri1
a15
ata
Apert3
Aprl4
ams
Steps
1st
Steps
Steps
Stegs
(a) Extrinsic rewards
(b) Waste cleaned
(c) Gifting weights
(d) Total rewards
Figure 6: Learning curves of each agent in Cleanup. Since the division of labor for different random
seeds is not the same, only the results under one seed are shown to distinguish individual performance.
Considering the fact that the division of labor in Cleanup is more pronounced than in other environ- ments and that cooperation Cleanup is purely altruistic, making the dilemma more challenging, we take Cleanup as example to show LASE's ability to promote fairness. Fig. 6a aaland] Fig. 6b show the extrinsic rewards of each agent and the amount of waste cleaned by each in Cleanup. We find Agent 4 is the only one to clean and does not receive any extrinsic reward.
8

---

Fig. 6c illustrates the gifting weights each agent received from the other three agents. Agent 4 gets the most gifts, indicating that its cleaning contribution to the team is recognized and rewarded by the other three agents. Fig. 6d shows reward curves after gifting, where the reward gap be- tween Agent 4 and the other three agents shrinks, implying fairness within the group is improved. Agents 1-3 always collect apples without cleaning, but this is not free- rider behavior. Both cleaning and collecting are indispensable to get rewards in Cleanup. Although Agent 4 does not obtain any reward directly from the environment, the acquired reward is redistributed through gifting, narrowing the gap of reward between the cleaner and collectors. In contrast, although GO obtains higher collective returns, it sacrifices the
Agents %
Agesz Agene
Aggent
individual interests of Agent 1 and Agent 2, while only Agent 3 and Agent 4 can get rewards Fig. 7
M R.-R, 12 to quantify
Figure 7: Four GOs' re- wards in Cleanup.
We use Equality (E) given by E= = the fairness, where the second term is the Gini inequality index. The greater the value of E, the fairer it is. We can get E(LASH) as 0.802,
2N 2N R.
F((GO) 0.496, showing that LASE achieves a higher level of faimess.
6.3 LASE distinguishes co-players and responds adaptively
To evaluate LASE's adaptation ability to interact with various types of agents, we conduct an experiment in which focal LASE agent interacts with three rule-based agents: cooperator (always clean up waste), defector (always try to collect apples), and random agent. The gifting weights of LASE to the other agents are shown in Fig. 8) LASE can explicitly distinguish between different types of co- co-players. Moreover, it responds in a manner that aligns with human values: preferring to share rewards with cooperators rather than defectors.
0.80
0.25
0.20
COMERIM sunan
camtnn
0.10
Ste
To study how LASE responds dynamically and how it affects collective behavior, we conduct an experiment where one focal LASE agent interacts with three A2C agents (Background agents, Bgs). A GO agent is trained in the same way for comparison with LASE. Fig. 9p displays each agent's rewards after training for 30k episodes, whereas the LASE group shows the reward after gifting.
Figure 8: LASE's gift- ing weights to three rule- based co-players.
LASE and GO improve the group reward to similar level, while four A2C agents will converge to the equilibrium of defection and gain almost no reward (see Sdp. With gift- ing, LASE incentives 3g3 to clean as shown in Fig. Dp. Thus, LASE and the other two Bgs can get rewards by gathering apples. But for GO, the focal agent sacrifices itself to undertake all the cleaning tasks and cannot get any reward.
Amount of wosse clammdd
LASE GO
36
-3
LAGZ
20
2.3 Steps
2.0
Group
Focal
Ug!
1922
Ugg
(a)
(b)
GO and LASE represent two different methods to foster cooperation. GO sacrifices its own interest to promote collective reward. LASE attempts to alleviate social dilemmas by incen- tivizing others to cooperate. When such an in- centive mechanism fails, LASE will no longer gift those agents who constantly exploit it, like the defector in Fig. 8 Overall, we believe that LASE a more efficient and secure policy for SSDs, it can promote cooperation well as avoid potential exploitation by others.
Figure 9: An LASE (or GO) agent interacts with three A2C agents in Cleanup. (a) The average re- ward for LASE and GO groups after training 30k episodes. The LASE group shows the reward af- ter gifting. The first two bars show whole groups' rewards. The remaining bars show the average re- ward for each agent. (b) The amount of the waste cleaned by each agent in the LASE group.
7 Conclusion
We introduce LASE, a decentralized MARL algorithm that fosters cooperation through gifting while safeguarding individual interests in mixed-motive games. LASE uses counterfactual reasoning to

---

infer the social relationships with others which captures the influences of others' actions on LASE and modulates the gifting strategy empathetically. In particular, empower LASE with the ability infer others' policies in partially observable and decentralized environments, we establish a perspective taking module for LASE. Both theoretical analyses in matrix-form games and experimental results across diverse SSDs show that LASE can effectively promote cooperative behavior while ensuring relative fairness within the group. Furthermore, LASE also able to recognize various types of co- players and adjust its gifting strategy adaptively avoid being exploited, enabling broad applicability in complex real-world multi-agent interactions, such as automated negotiations in E-commerce and decision-make making in autonomous driving. Whilst LASE exhibits superior abilities, there some limitations of our method. What would be the consequence of giving agents the ability to refuse gifts? How to extend the algorithm LASE to continuous action space? These problems will illuminate our future work.
Acknowledgments and Disclosure of Funding
This work is supported by the National Science and Technology Major Project (No. 0022ZD0114904 This work is also supported by the project "Wuhan East Lake High-Tech Development Zone, National Comprchensive Experimental Base for Governance of Intelligent Society".
References
[1] C. D. Batson, B. D. Duncan, P. Ackerman, T. Buckley, and K. Birch. Is empathic emotion source of altruistic motivation? Journal of personality and Social Psychology, 40(2):290, 1981.
[2] C. D. Batson, N. Ahmad, D. A. Lishner, Tsang, C. Snyder, and S. Lopez. Empathy and altruism. The Oxford handbook of hypo-egoic phenomena, pages 161-174, 2002.
[3] Christoffersen, A. A. Haupt, and D. Hadfield-Menell. Get it in writing: Formal contracts mitigate social dilemmas in multi-agent rl. arXiv preprini arXiv:2208. 10469, 2022.
[4] Y. Du, J. Z. Leibo, U. Islam, R. Willis, and P. Sunehag. A review of cooperation in multi-agent learning. arXiv preprint arXiv:2312.051622 2023.
[5] Foerster, G. Farquhar, Afouras, N. Nardelli, and S. Whiteson. Counterfactual multi-agent policy gradients. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
[6] J. N. Foerster, R. Y. Chen, M. Al-Shedivat, S. Whiteson, Abbeel, and Mordatch. Learning with opponent-learning awareness. arXiv preprint
2017.
[7] I. Gemp, K. R. McKee, R. Everett, E. A. Duéñez--uzzmán, Y. Bachrach, D. Balduzzi, and A. Tacchetti. D3c: Reducing the price of anarchy in multi-agent learning. arXiv preprint arXiv:2010.00575. 2020.
[8] H. Heemskerk. Social curiosity in deep multi-agent reinforcement learning. Master's thesis, 2020.
[9] Y. Hua, S. Gao, W. Li, B. Jin, X. Wang, and H. Zha. Learning optimal" pigovian tax" in sequential social dilemmas. arXiv preprint arXiv:2305.06222 2023.
[10] Y. Huang, A. Liu, Kong, Y. Yang, S.-C. Zhu, and X. Feng. Efficient adaptation in mixed-motive environments via hierarchical opponent modeling and planning. arXiv preprint arXiv:2406.08002, 2024.
[11] E. Hughes. J. Z. Leibo, M. Phillips. K. Tuyls, E. Dueñez-Guzman A. García Castañeda, I. Dunning, T. Zhu, K. McKee, R. Koster, al. Inequity aversion improves cooperation in intertemporal social dilemmas. Advances in neural information processing systems, 31, 2018.
[12] A. Ibrahim, A. Jitani, D. Piracha, and D. Precup. Reward redistribution mechanisms in multi- agent reinforcement learning. In Adaptive Learning Agents Workshop the International Conference Autonomous Agents and Multiagent Systems, 2020.
10

---

[13] N. Jaques, A. Lazaridou, E. Hughes, C. Gulcehre, P. Ortega, D. Strouse, J. Z. Leibo, and N. De Freitas. Social influence as intrinsic motivation for multi-agent deep reinforcement learning. In International conference machine learning, pages 3040-3049. PMLR, 2019.
[14] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6988, 2014.
[15] J. Z. Leibo, Zambaldi, M. Lanctot, J. Marecki, and T. Graepel. Multi-agent reinforcement learning in sequential social dilemmas. arXiv preprint
2017.
[16] A. Lerer and A. Peysakhovich. Maintaining cooperation in complex social dilemmas using deep reinforcement learning. arXiv preprint arXiv: 2017.
[17] A. Letcher, J. Foerster, D. Balduzzi, T. Rocktäschel, and S. Whiteson. Stable opponent shaping differentiable games. arXiv preprint arXiv: 18 1,08469, 2018.
[18] M. L. Littman. Markov games as framework for multi-agent reinforcement learning. In Machine learning proceedings 1994, pages 157-163. Elsevier, 1994.
[19] R. Lowe, I. Wu, A. Tamar, Harb, O. Pieter Abbeel, and Mordatch. Multi-agent actor-critic for mixed cooperative-compettive environments. Advances in neural information processing systems, 30, 2017.
[20] A. Lupu and D. Precup. Gifting in multi-agent reinforcement learning. In Proceedings of the 19th International Conference on autonomous agents and multiagent systems, pages 789-797, 2020.
[21] K. R. McKee, Gemp, B. McWilliams, E. A. Duéñezz Guefiez-Guzmán, E. Hughes, and J. Z. Leibo. Social diversity and social preferences in mixed-motive reinforcement learning, arXiv preprint arXiv:2002.02232 02.002....222..... 2020.
[22] V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference machine learning, pages 1928-1937. PMLR, 2016.
[23] A. Peysakhovich and A. Lerer. Prosocial learning agents solve generalized stag hunts better than selfish ones. arXiv preprint arXiv: /ttv::/09 709.022655 2017.
[24] R. Raileanu, E. Denton, A. Szlam, and R. Fergus. Modeling others using oneself in multi-agent reinforcement learning. In International conference on machine learning, pages 4257-4266. PMLR, 2018
[25] T. Rashid, M. Samvelyan, C. S. De Witt, G. Farquhar, J. Foerster, and S. Whiteson, Monotonic value function factorisation for deep multi-agent reinforcement learning. The Journal of Machine 2020.
Learning Research, 21(1):7234-728
[26] F. C. Santos, J. M. Pacheco, and T. Lenaerts. Evolutionary dynamics of social dilemmas in structured heterogeneous populations. Proceedings of the National Academy of Sciences, 103 (9):3490-344) 3494, 2006.
[27] L. S. Shapley. Stochastic games. Proceedings of the national academy of sciences, 39(10): 1095-1100, 1953.
[28] T. Singer, B. Seymour, J. P. O'Doherty, K. F. Stephan, R. J. Dolan, and C. D. Frith. Empathic neural responses are modulated by the perceived fairness of others. Nature, 439(7075):4664699 2006.
[29] K. Son, D. Kim, W. J. Kang, D. E. Hostallero, and Y. Yi. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In International conference on machine learning. pages 5887-5896. PMLR, 2019.
[30] E. L. Stocks, D. A. Lishner, and S. K. Decker. Altruism or psychological escape: Why does empathy promote prosocial behavior? European journal of social psychology, 39(5):649-665 2009.

---

[31] P. Sunchag, G. Lever, A. Gruslys, W. M. Czarnecki, V. Zambaldi, M. Jaderberg, M. Lanctot, N. Sonnerat, Z. Leibo, K. Tuyls, et al. Value-decomposition networks for cooperative multi-agent learning. arXiv preprint arXiv: 1706.05296 2017.
[32] E. Vinitsky, N. Jaques, J. Leibo, A. Castenada, and E. Hughes. An open source imple- mentation of sequential social dilemma games. https: Bectps://githar com.ttubboom/mmmm...oooommmmmoot com wtt..//oo//eeeeiittt.t.... eugenevinitssky sequential_soca laiaaalldlll dilemma, garaaaaaaaaaaaeaaeeeee iesiissssssssesssee 182 2019. GitHub repository.
[33] X. Wang, E. Hughes, C. Fernando, W. M. Czarnecki, E. A. Duéitez-Guzmán, and J. Leibo. Evolving intrinsic motivations for altruistic behavior. arXiv preprint arXiv: 1..0931, 2018.
[34] W. Wang, M. Beliaev, E. Bıyık, A. Lazar, Pedarsani, and D. Sadigh. Emergent prosociality in multi-agent games through gifting, arXiv preprint (exXi::200S006330 2021.
[35] R. Willis and M. Luck. Resolving social dilemmas through reward transfer commitments. In Proc. Adaptive and Learning Agents Workshop (ALA2023) 29-30 May 2023, London, UK, 2023.
[36] J. Yang, A. Li, M. Farajtabar, P. Sunehag, E. Hughes, and H. Zha. Learning to incentivize other learning agents. Advances in Neural Information Processing Systems, 33:15208-15219, 2020.
[37] Y. Yi, G. Li, Y. Wang, and Z. Lu. Learning to share multi-agent reinforcement learning. arXiv preprint arXiv: 2021.
[38] C. Yu, A. Velu, E. Vinitsky, J. Gao, Y. Wang, A. Bayen, and Y. Wu. The surprising effectiveness of ppo in cooperative multi-agent games, Advances in Neural Information Processing Systems, 15:24611-24665 2022.
NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We clearly set out the contribution of our work in the abstract and introduction, and support our claims with both theory and experiments.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss the limitations of our work and future directions for improvement at the conclusion section of the paper.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
12

---

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specing well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: We provide a complete proof of the theoretical results in Section 4.3 in
Appendix A
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide the model structure and parameters required for the experiment in detail in Section 5.2
5.2 and Appendix C. and describe the environment in which the experiment is run in detail in (Section 5. and SSection B.2
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
13

---

If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution a new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification:
We are still sorting out the code for future open source.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https //nips.cc cc/ http://lllgggg..eee/eeeeeeuueeee http:///otttuubmsssooooooooo.ooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
14

---

Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: We provide the model structure and parameters required for the experiment in detail in [Section 5.2 5.2 and Appendix C| and describe the environment in which the experiment is run in detail in SSection 5.1 and Section B.2
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We run five random seeds for the experiments. In Fig. g. the solid line represents the mean performance, while the shaded area indicates the range between the mean minus the standard deviation and the mean plus the standard deviation. In some figures like Fig. 6, we only show the results under one seed for clarity due to the different division of agents under different seeds.
Fig. and Fig. 8
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
As introduced in Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
15

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
We have reviewed the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
As shown in Appendix E
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks
16

---

Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
The open-source codes used in the paper are cited. The license is MIT license.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paaerssithcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
17

---

Answer:
[NA]
The paper does not involve crowdsourcing nor research with human subjects Guidelines:
Justification:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
18

---

A Analysis in Iterated Matrix Games
For general iterated matrix game whose payoff matrix is computed according Tab. 1 at each step, we can let 0* for i E (1,2} denote each agent's probability of taking the cooperative action, and let @ for j E {2, } denote each agent's 's prediction of the other's ' policy. To avoid the difficulties caused by the coupled update reinforcement learning for the theoretical analysis, we make the simplifying assumption that the three networks in SRI module have been fully trained. We further assume that this two-player game fully observable, and we can make the following approximation to.Fq. 3
Q'(oi at) mad (0)) (a5",a{) a.a')) ))
5 W
1
of
N max ('(o,, (ai at) min (' (a, af.))
(6)
of r' d)) - C) (1- Pr ())
af."
N-1
(aa, C) (a',
Here, r'aa, a represents player i's reward determined by the payoff matrix which only relies on the two players' actions without state or observation. The update rule Eq. states that it is feasible to replace Q' by r'. Then we can calculate the gifting weights under the four combinations of actions: CC, CD, DC, and DD. We take w2 for example:
r (C, (C,r' + (1 0 (C, P(C,C)-
R - (02. RRR (1 0 (2)))
whe
= 11 02
R-S
(7)
weD = (FC.D) (C, S (0 RR+ + (1 -02. .S)
0²
(CC,))-r((CD)
R S
(8)
- (1+(00))(+))+-110000))0)000 D)) T (02 . (1
0 P))
0²
DC r'(,CC)r'(D,D)
T
(9)
who r (D, - (D,r (1 0 (D, (D,
P -0 T + (1
0 P)) 0²
r (D
T-P
(10)
Similarly
Section 4 set w less than 0 to 0. So agent I's reward distribution scheme is:
rec = w = (1- (,, rc = --rcc = 0² R.
12
DC w wbc -1- 0T, rrrT 11
DC T DDC @TT,
12
(11)
TCD rDD 0, r,rbb P
12
Agent 2's computation is symmetric. The total reward received by each agent is
1. 10 - (1- 0-) S 0' 2 = 10' R - (1- 01, 02)
(12)
(13)
The value function for each agent is defined by
Vi 00) = Σ
r'p Fit0
(14)
0=
where p= 00 00 000(1'11111 (1-0)),(1-))))) (1-
)1 (1) {]]]
Agent 2 updates its policy by
8²_ = 8² + a a V 02)
= 8² + 1 1-y
a V 0000 0 R
( (1-
(15)
+0' (1 - 0T)''T + 0 01) [s 62]]
+ (1 - (1-0') -p))p
= 0 + a { 01 R+ (1 82) - 0'7 01 + 5 S + (1 (1- (1-01-1))) 02y p| (1 (1- 01} (1-01)
19

---

By symmetry, agent I updates its policy by
= 00 + a {{ (1-0
0² D'yr p| (1 (1-0% [
(16)
We assume that LASE'S prediction the others is accurate when simulating in SSection 4.3) i.e
01 0,, 02 02. And we let 10-3 7 = 0.99.
B Environments
B.1 Validating the environments
In this section, we will show that our four environments are all sequential social dilemmus defined in Mi: An N-player sequential social dilemma is tuple (M, II II. lIa) of a Markov game and two disjoint sets of policies, said to implement cooperation and defection respectively, satisfying the following properties. Consider the strategy profile (o)..... TT T .d..... xT"") E II x HIT with I + m = N. We shall denote the average payoff for the cooperating policies by R. and for the defecting policies by (t).). (M, IT) is sequential social dilemma iff the following hold:
1. Mutual cooperation is preferred over mutual defection: Re(N) > Ra(0).
2. Mutual cooperation is preferred to being exploited by defectors: Re(N) > Ro(0).
3. Either the fear property, the greed property, both:
Fear: mutual defection is preferred to being exploited. Ra(i) > Re(i) for sufficiently small i.
Greed: exploiting a cooperator is preferred to mutual cooperation. Ra(t) > (i( for sufficiently large i.
A Schelling diagram is a game representation that highlights interdependencies between agents, showing how the choices of others shape one's own incentives. It plots the curves Ra + 1) and Ra(() as shown in Fig. 10 All the environments satisfy the first two properties of sequential social dilemmas: (N) > (O) and (NV (0).). In SSH, fear promotes defection: Ra(0) > (0). In Cleanup and SSG, the problem is greed: (3) (3).. Coingame suffers from both temptations defect. This indicates that our experimental enviromments include all three different types of sequential social dilemmas corresponding to Tab.
15.
15
12
dulector
rumber
number
(a) Cleanup
(b) Coingame
(c) SSH
(d) SSG
Figure 10: The Schelling diagram of Cleanup, Coingame, Sequential Stag-Hum (SSH) and Sequential Snowdrift Game (SSG). The dotted line shows the overall average return where the individual chooses defection.
Table 3: Classification of social dilemmas. In all dilemmas, mutual cooperation yields higher payoff's than mutual defection, yet each dilemma provides incentive for defection. In the Snowdrift game, one player can gain higher payoff by defecting when the other cooperates, while in the Stag Hunt game, a player can achieve higher payoffs by defecting when the other defects. The Prisoner's Dilemma encompasses both types of incentives.
Social dilemmas
Abbreviation
Parameters
Snowdrift
SG
T>1>8>0
Stag Hunt
SH
1>T>>>>
Prisoner's Dilemma
PD
T>1>0>822
8,2
20

---

B.2
Environment details
IPD. Each agent makes decisions based on the actions of the two players in the previous step, so this is a fully observable environment unlike other environments. We trained for 10k episodes, each with 100 steps.
Coingame. Map size is 5 x 5. Agent's action space is A = {up, down, left, right). The state is represented as a x 5 x 5 binary tensor. The five channels are (blue agent, red agent. blue coin, red coin, mask), where the first two channels encode the location of each agent, the last channel distinguishes between parts within and beyond the boundary, and the other two channels encode the location of the coin if any exist. If two agents walk the coin at the same time, one of them is randomly selected to successfully pick it up. Cooperative agents will only try to collect coins with their own color, while self-interested agents tend to greetingly collect all coins. Each episode lasts for 100 steps.
Cleanup. Map size is 8 x 8. A {up, down, left, right, stay, clean, pick], where the last two actions require the agent to be in the same position as the waste or the apple. The seven channels are (agent 1, .... agent 4, waste, apple, m The parameters with the same meaning as the open-source implementation about Cleanup [32] are shown in Tab. 4 To achieve cooperation, agents need to take the initiative to undertake part of the cleaning task to help improve the group's 's revenue. A defecting agent will just keep waiting for the apple to grow and gather it. Each episode lasts for 100 steps.
Table Environment parameters in Cleanup
Parameter
Value
map_size size
8 8
appleeespawPProbabity
0.4
wasteSpawnProbzbblityy
0.5
thresholdDepletion
0.5
thresholdRestoratioon
0.0
view_size
5
max_steps
100
SSH. Map size is A = {up, down, left, right, stay, hunt hare, hunt stag). Agents must be in the same position as the prey to hunt and the prey doesn't 't respawn after being hunted. The seven channels are fagent 1, ... agent 4, hare, stag, mask). Cooperative agents are happy to hunt deer with others, while defectors only hunt rabbits to avoid the risk of not getting a payoff. Each episode lasts for 30 steps.
SSG. Map size is 8 x 8. A = (up, down, left, right, stay, remove snowdrift). Agents must be in the same position as the snowdrift to remove it and the removed snowdrift doesn't 't respawn. The six channels are (agent ..., agent 4, snowdrift, mask). Cooperators will proactively remove snowdrifts bring high rewards to the team, while defectors will just wait for others remove them. Each episode lasts for 50 steps.
C Implementation
The pscudocode for the LASE algorithm is shown in Algorithm
SSDs. The actor-critic model interacting with environments utilizes two cascaded CNNs to process input data, with a kernel size of 3, stride of size and 16 / 32 output channels. This is connected to one fully connected layer of size 128 activated by ReLU, and an LSTM with 128 cells. The actor head and critic head are two separate fully-connected layers that output the softmax normalized action policy and a scalar value respectively. The implementation of the intrinsic policy network and value network OM basically the same, but due to the need to further judge the actions of others, joint action space is added to the value network input dimension. In the observation transformation network, the input data passed through CNN with kernel of size 3, stride of size and 16 output channels, and is concatenated with one-hot vector representing the agent index to input in two FC
21

---

Algorithm Learning to balance Altruism and Self-interest based on Empathy (LASE)
Initialize action policy x', SR policy network ग(RR- SR value newtork QŠR and observation conver- sion network parameterized by 0² μ² o², '7' and trajectory buffer B², for each agent for eps = to maz_epiin munzeepsodes do
All agents interact with the environment for many steps, each agent gets trajectory x² (ot, de, -rt ((411) and stores it in B for each agent i do
for each other agent do
get j's simulated observation O by i's observation conversion network estimate j's policy TŚR (a² (6²) with SR policy network
for j's all possible actions a² do compute i's Q-value QšR (of, (ar aa')), )). with j taking all possible actions of while the other agents' actions fixed end for compute the counterfactual baseline Σ Egg" " (o) (a of.)) compute gifting weights w by Eq. 3 end for
get w by : end for get pltt <- and update policy x² to maximize the accumulated pox if eps mod mondatefrrqueee for each agent i do
Σ w
then
sample a minibatch from B, update n², of ny by Eq. end for end if end for
Eq.
layers of size 128. The output is reshaped to be the same size as the input observations. We use Adam optimizer [14] for all modules' training.
IPD. All the CNNs IPD are removed. The size of FC layer and the cell number in LSTM are scaled down to 32.
Table 5: Hyperparameters
(a) Hyperparameters SSDs
(b) Hyperparameters in IPD
Parameter
Value
Parameter
Value
Parameter
Value
Parameter
Value
Extart
Ogg
le-4
Estart
0.5
Okp
5e-3
<div
2e3
O'g
3e-5
édiv
1e3
0'p
le-3
fend
0.05
Otg
3e-5
fend
0.01
Org
le-3
Ysec
0.98
On
5e-5
Yscc
0.98
Oiy
le-3
?
0.98
update _edate_freq
20
?
0.95
update freq
20
0.1
batch_
1000
5
0.1
batch_size
64
Experiments Compute Resources
CPU: 128 Intel(R) Xeon(R) Platinum 8369B CPU @ 2.90GHz; Total memory: 263729336 kB GPU: NVIDIA GeForce RT' 3090; Memory per GPU: 24576 MiB The main experiments are as shown in Fig. 5 An experiment takes about 2000 MiB of the GPU and takes about 1 1.5 days to run. About experiments can be run simultaneously on the machine. Due to the need to debug and adjust parameters, approximately double the amount of computation is required.
22

---

D
Additional Results
D.1 Scalability of LASE
To test LASE's scalability, we have extended Cleanup and Snowdrift as Tab.
Table 6: Environmental parameters of extended Cleanup and SSG
Map Size
Player Num
Ohs Size
Init Waste/ Snowdrift num
Episode len
Cleanup.Extn
12
4
5
8 16
100 - 150
Extn
5
+ 12
50 - 70
Tab. shows the self-play results of LASE and baselines. LASE outperforms baselines in the two more complex environments.
Table Self-play results (total reward) in extended Cleanup and SSG
Total reward
LASE
IA
LIO
A2C
Cleanup.Extn
56.513
20.798
1.294
3.548 207.461
0.135
SSG.Extn
232.564
227.762
20.317
134.964
D.2 Estimate the uncertainty of their social relationship
We select the w data from the last 10 timesteps of training to calculate their mean value and standard deviation, which estimates the uncertainty of social relationships. The calculation method is as follows:
Led (w J
Tam
1= Tum
10
20
a
Σ
1 Lzjjzi
leb-1
106
n n (n 1n
J.j
We conduct comparative experiment to replace the input of SR policy network of with j's real observation o² Here is the results:
The results show that the mean value of LASE's inferred social relationships closely match actual observations when available, although partial observability significantly increases uncertainty. Con- sidering that the social relationships between people in real life tend to be relatively stable and do not change drastically, we think that possible solution handle the uncertainty of social relationships is to introduce some smoothing techniques for w to reduce the variance of social relationships over time. This approach will be explored in our future work.
Meanwhile, is important to note that Figure 8 and corresponding analysis show that LASE is able to correctly infer the relationships with different co-players and respond properly. Specifically, in the experiments conducted in Section 6.3, one LASE interacts with three rule-based co-players: cooperator, defector and random. The results show the wit given cooperative co-player is the
Table 8: The uncertainty of inferred social relationships
W, 8
SSH
SSH
Coingame
Cleanup
LASE w/o o'
0.07184 + 0.01921
0.02341 + 0.00939
0.22243 + 0.1 0.18594
0.34572 + 0.01509
LASE w/ of
0.06278 0.01136
0.03157 0.00527
0.19317 0.05493
0.29465 0.00889
23

---

largest, significantly higher than that given to the other two co-players. The w given to random co- player comes second, and the smallest given defector. These results demonstrate the consistency between LASE's estimates of the relations and the ground truth.
D.3 Compare the Equality with other baselines
As evaluation metric, fairness should be evaluated alongside reward measure algorithm perfor- mance effectively. Some algorithms may fail to address decision maasing issues in mixed-motive games where each agent receives a small reward, but the reward disparity between agents minimal, resulting in high fairness. Clearly, these methods are not effective. An effective method should both maximize group reward and ensure intra-group equity. Here, we include the fairness results of other baselines:
Table 9; All Equality results
Fairness
SSH
SSG
Coingame
Cleanup
LASE
0.994
0.951
0.835
0.802
LASE w/o
0.986
0.862
0.848
0.685
GO
0.968
0,856
0.785
0,496
IA
0.984
0.877
0.898
0.708
LIO
0.931
0.985
0.745
0.545
SI
0.995
0,937
0.750
0.892
A2C
0.997
0.854
0.831
0.824
E
Broader Impact
The rapid advancement of AI technology has brought about an explosion in the number of agents, making it unfeasible to rely on centralized controller to achieve coordinated collective behavior. The question how to design independent agents that excel in specific tasks and can demonstrate adequate social behavior when interacting with humans or other agents remains an open challenge. take a tentative step towards this problem by introducing the mechanism of gifting and the theory of empathy in human society. We believe work will have positive impact on the interaction of multi-agent systems, the alignment of AI with human values, and the development of AI safety.
24


Reviews:
[{"review_id": "YEJsN2kmNr", "reviewer": "Reviewer_vbJc", "summary": "This paper introduces LASE (Learning to balance Altruism and Self-interest based on Empathy), a multi-agent reinforcement learning algorithm designed for mixed-motive games. LASE uses a gifting mechanism where agents share a portion of their rewards with others based on inferred social relationships. Counterfactual reasoning determines these relationships by comparing the actual joint action's value to a baseline that averages over the other agents' actions. The authors use a perspective-taking module to predict other agents' policies. Experimental results across various sequential social dilemmas show LASE's ability to promote cooperation while maintaining individual interests. The authors claim the following contributions: (1) a computational model of empathy that modulates responses based on inferred social relationships; (2) a decentralized MARL algorithm that balances altruism and self-interest in mixed-motive games, and (3) theoretical analysis of decision dynamics in iterated matrix games and experimental verification of LASE's performance in sequential social dilemmas.", "questions": "Do you have any proposed changes to the paper in response to my critiques?", "limitations": "I think there could be more discussion of this, but the current evaluation is sufficient.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "* Novel approach: The introduction of LASE as a mechanism to balance altruism and self-interest in multi-agent settings is innovative and addresses an important challenge in mixed-motive games.\n\n* Theoretical contribution: I think the analysis of the algorithm's behavior in iterated matrix games does a good job of grounding the empirical results in game theory.\n\n* The overall evaluation was sufficient to demonstrate the performance of the algorithm in comparison to well-chosen baselines", "weaknesses": "I think the paper is quite strong overall. However, several places can still be improved to increase the potential impact of the work.\n\n* Clarity issues: The paper could benefit from improved clarity in several areas. For example, a bit more clarity on how gifts are determined would be helpful. Consider adding pseudocode for key components like the Social Relationships Inference module or the gifting mechanism and how the modules integrate. \n\n* Insight into counterfactual baseline design choice: The paper's choice of using the average behavior as the counterfactual baseline for determining social relationships warrants further investigation. The authors could consider conducting experiments that vary this baseline, comparing alternatives such as fixed neutral baselines, learned cooperative behavior baselines, or worst-case action baselines. This would provide insights into the robustness of the approach and potentially identify improvements to the algorithm.\n\n* Limited domain coverage: While the paper evaluates LASE across several sequential social dilemmas, which is adequate, the approach could be further strengthened by expanding the range of domains tested. For example, the authors could consider alternatives from MeltingPot domains (https://github.com/google-deepmind/meltingpot).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Clarify issues\n\nThanks for your valuable suggestions! The overall framework and flow of LASE as well as the relationship between each module can be seen in Figure 1 in the paper and Algorithm 1 in the appendix. For clarity, here we present the pseudocode for both Social Relationship Inference (SRI) and Gifting modules in detail:\n\n```python\n# Trajectories collection\n... ...\n\n# SRI\nfor i in Agents:\n\tfor j in Agents_without_i:\n\t\tpredicted_obs = obs_conversion(i_obs)\n\t\tpredicted_action_prob = SR_policy(predicted_obs)\n\t\tfor j_virtual_action in all_possible_actions:\n\t\t\tSR_q_values.append(SR_value(i_obs, j_virtual_action + others_real_actions))\n\t\tcounterfactual_baseline = predicted_action_prob @ SR_q_values\n\t\tw_ij = (SR_value(i_obs, real_joint_actions) - counterfactual_baseline) / M # Eq (3)\n\tw_ii = 1 - sum([w_ij for j in Agents_without_i])\n\n# Gifting\nfor i in Agents:\n\ti_weighted_reward = 0\n\tfor j in Agents:\n\t\ti_weighted_reward += w_ji * j_reward\n\n# Training with weighted rewards\n... ...\n```\n\nThese two modules are executed after all agents have completed an entire episode of interaction with the environment, adjusting the rewards to influence subsequent RL training. Specifically, in SRI, each agent $i$ predicts another agent $j$’s action probability using Perspective Taking and assesses the impact of all possible actions of $𝑗$ on its $Q$-value using the SR value network. Then $i$ infers the social relationship $w^{ij}$ with $j$ using Equation 3.\n\nAfter all agents have completed SRI, the Gifting module is responsible for distributing rewards. Specifically, the weighted reward for each agent $i$ is the sum of the gifted rewards $w^{ij}\\times r^j$ from other agents $j$, including $i$’s own reward left to itself $w^{ii}\\times r^i$.\n\n> Insight into counterfactual baseline design choice\n\nThank you very much for your suggestions on designing the counterfactual baseline! We apologize for the confusion we may have encountered while attempting to implement the three different baselines you mentioned. We are not entirely sure if we fully understood your points and would greatly appreciate the opportunity to discuss this matter in more detail to ensure we correctly implement your suggestions!\n\n- Fixed neutral baseline: If you are referring to using a fixed hyperparameter as a neutral baseline, we believe that this approach may not be appropriate due to the continuously evolving $Q$-values throughout the RL learning process. If you are referring to the neutral value of $i$'s $Q$-values conditioned on the various possible actions of $j$, we think that our ablation study **LASE w/o**, where $\\sum_{a_t^{j'}}\\frac{1}{|\\mathcal{A}^j|}Q^i(o^i_t,(a_t^{-j},a_t^{j'}))$ is used as a counterfactual baseline, has already addressed this.\n\n- Learned cooperative behavior baselines: Are you referring to \"cooperative behavior\" as $\\text{max}_{a^{j'}_t} Q^i(o^i_t, (a_t^{-j},a_t^{j'}))$? If so, based on Equation 3, this would lead to $w^{ij} \\leq 0$, resulting in no gifting occurring. This would cause the algorithm to fail and revert to the A2C baseline.\n- Worst-case action baselines: We interpret this baseline as $\\text{min}_{a^{j'}_t}Q^i(o^i_t, (a_t^{-j},a_t^{j'}))$. We replace LASE's counterfactual baseline with the Worst-case Action Baseline (WAB), and the self-play results are as follows:\n\n    |*Self-play*|SSH|SSG|Coingame|Cleanup|\n    |---|---|---|---|---|\n    |WAB|17.624|118.731|0.009|43.741|\n    |LASE|18.948|117.784|38.736|33.467|\n    \n  WAB outperforms LASE on SSG and Cleanup, which is because using $\\text{min}_{a^{j'}_t}Q^i(o^i_t, (a_t^{-j},a_t^{j'}))$ as the baseline results in $w^{ij}\\geq 0$. This overly optimistic estimation of social relationships promotes group cooperation in self-play settings, but it also raises the risk of exploitation when interacting with unknown agents. To demonstrate this, an adaptive experiment on WAB is conducted in the same manner as described in Section 6.3 of the paper. A WAB is trained with three rule-based agents and three A2C agents, respectively. We record the mean gifting weights assigned by the WAB to other agents and the rewards of the WAB after gifting:\n    \n    |*Gifting weight*|Cooperator|Random|Defector|A2C_1|A2C_2|A2C_3|\n    |---|---|---|---|---|---|---|\n    |WAB|0.321|0.254|0.184|0.266|0.221|0.222|\n    |LASE|0.268|0.091|0.027|0.0391|0.1339|0.2194|\n    \n    |*Reward after gifting*|Rule based agents|A2C agents|\n    |---|---|---|\n    |WAB|5.778|4.467|\n    |LASE|**15.129**|**8.752**|\n    \n    Due to WAB giving excessive gifts to other agents and not receiving reciprocation as it would in self-play, the actual reward obtained by WAB is significantly lower than that of LASE. This demonstrates the weakness of WAB being easily exploited.\n\n    Additionally, we observe that in Coingame, the social relationship converged to $w^{12}\\approx w^{21}\\approx 1$, indicating that the agents' optimization objective shifted towards maximizing the opponent's rewards. In Coingame, only collecting coins of the opponent's color impacts its reward. Finally, the two agents converged to a behavior of not collecting any coins, demonstrating that excessive gifting is detrimental to learning in this environment.\n    \n> Limited domain coverage\n\nThank you very much for your suggestions! As a standardized testing platform that integrates various environments including cooperation, competition, and mixed motives, MeltingPot can further assist in testing the scalability and robustness of algorithms. We will consider selecting some of these environments as experimental settings in our future work to enhance the persuasiveness of our paper.\n\nThank you again for your insightful comments! We will make corresponding changes in the revised version of the paper based on your and other reviewers’ suggestions, including but not limited to improving and clarifying the description of the algorithm, adding discussions and experiments on the counterfactual baseline, and expanding the experimental environments.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CLIyksqQjz", "reviewer": "Reviewer_7rD5", "summary": "The authors propose a novel algorithm, LASE, that employs a gifting mechanism in order to steer agents toward equilibria of high social welfare in mixed motive games. A novelty in LASE is that it estimates the \"social relationship\" between a player and its co-players through a counterfactual Q-value baseline. The authors show that LASE performs favourably across a number of temporally-extended social dilemmas.", "questions": "What do you think could be a real-world application of this line of work?", "limitations": "I believe the authors are addressing limitations adequately.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-written and well-presented.\n\nThe authors address a gap in the literature which does so far not attempt to estimate the influence of co-player policies on the joint Q-value function. In general, estimating counterfactuals involving Q-values has in the past been found to suffer from high variance issues (such as in [1]). Hence, I find it positively surprising that the authors' method performs decently across several different environments.\n\n[1] Counterfactual Multi-Agent Policy Gradients, Foerster et al., AAAI 2018", "weaknesses": "I believe the main weakness of the authors' approach is that, under partial observability, a player cannot generally see all parts of its co-players observations, hence making it impossible to fully reconstruct their policy inputs (hence the restriction to common knowledge fields of view in [2]). I believe the author's algorithm should crucially estimate the uncertainty of their social relationship estimates such as to avoid misunderstandings that could erode trust in real-world situations.  \n\nAdditionally, I am unsure about the authors' use of the term \"empathy\" - \"theory of mind\" would certainly work here, but \"empathy\" seems like an inherently emotional concept.\n\n[2] Multi-Agent Common Knowledge Reinforcement Learning, Schroeder de Witt et al., NeurIPS 2019", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Estimate the uncertainty of their social relationship.\n> \n\nWe select the $w^{ij}$ data from the last $10^6$ timesteps of training to calculate their mean value $\\overline{w}$ and standard deviation $s$, which estimates the uncertainty of social relationships. The calculation method is as follows:\n\n$$\n\\overline{w}^{ij}=\\frac{\\sum_{t=T_{max}-10^6}^{T_{max}}w_t^{ij}}{10^6}, \\overline{w}=\\sum_{i=1}^n\\sum_{j=1, j\\neq i}^n \\overline{w}^{ij}\n$$\n\n$$\ns=\\frac{\\sum_{i=1}^n\\sum_{j=1, j\\neq i}^n\\sqrt{\\frac{\\sum_{t=T_{max}-10^6}^{T_{max}}(w_t^{ij}-\\overline{w}^{ij})^2}{10^6-1}}}{n\\times (n-1)}\n$$\n\nWe conduct a comparative experiment to replace the input of SR policy network $\\hat{o}^j$ with $j$’s real observation $o^j$ . Here is the results:\n\n| $\\overline{w}$ | SSH | SSG | Coingame | Cleanup |\n| --- | --- | --- | --- | --- |\n| LASE w/o $o^j$ | $0.07184\\pm{0.01921}$ | $0.02341\\pm{0.00939}$ | $0.22243\\pm{0.18594}$ | $0.34572\\pm{0.01509}$ |\n| LASE w/ $o^j$ | $0.06278\\pm{0.01136}$ | $0.03157\\pm{0.00527}$ | $0.19317\\pm{0.05493}$ | $0.29465\\pm{0.00889}$ |\n\nThe results show that the mean value of LASE's inferred social relationships closely matches that of scenarios with actual observations, although partial observability significantly increases uncertainty. Considering that the social relationships between people in real life tend to be relatively stable and do not change drastically, we think that a possible solution to handle the uncertainty of social relationships is to introduce some smoothing techniques for $w^{ij}$ to reduce the variance of social relationships over time. This approach will be explored in our future work.\n\nMeanwhile, It is important to note that Figure 8 and the corresponding analysis show that LASE is able to correctly infer the relationships with different co-players and respond properly. Specifically, in the experiments conducted in Section 6.3, one LASE interacts with three rule-based co-players: cooperator, defector and random. The results show the $w^{ij}$ given to the cooperative co-player is the largest, significantly higher than that given to the other two co-players. The $w^{ij}$ given to the random co-player comes second, and the smallest is given to the defector. These results demonstrate the consistency between LASE’s estimates of the relations and the ground truth.\n\nThanks again for your valuable insights! We strongly agree that this issue has an important impact on trust in real-world scenarios, and we will add relevant results and analysis in a revised version of the paper. At the same time, we would also like to thank you for providing the literature about partial observability, which is very helpful to our follow-up work!\n\n> The usage of the term “empathy”\n> \n\nEmpathy includes both emotional empathy and cognitive empathy [1]. The former refers to the ability to feel and share others’ emotions. The latter involves imagining and understanding others’ thoughts, feelings and perspectives. In particular, [2] shows that human response is empathically modulated by the learned relations with others. Although there are similarities between Theory of Mind (ToM) and empathy, ToM mainly focuses on attributing mental states to others [1]. Based on [2], we design LASE, incentivizing the cooperative behavior of others through gifts, which is closely related to the emergence of cooperation in real world [3].\n\n[1] De Waal, Frans BM, and Stephanie D. Preston. \"Mammalian empathy: behavioral manifestations and neural basis.\" *Nature Reviews Neuroscience* 18.8 (2017): 498-509.\n\n[2] Singer, Tania, et al. \"Empathic neural responses are modulated by the perceived fairness of others.\" *Nature* 439.7075 (2006): 466-469.\n\n[3] Yalcin, Ӧzge Nilay, and Steve DiPaola. \"A computational model of empathy for interactive agents.\" *Biologically inspired cognitive architectures* 26 (2018): 20-25.\n\n> The real-world application\n> \n\nOur work employs decentralized learning to infer the influence of other agents on oneself in mixed-motive games, enabling broad applicability in complex real-world multi-agent interactions. A potential application scenario is multi-agent automated negotiation. With the rapid advancement and increasing application of machine learning and LLMs, we believe that in the future, agents may assist or even replace humans in various fields such as E-commerce for automated negotiations and autonomous driving. \n\nFor example, bargaining in e-commerce is a classic mixed-motive game scenario. In bargaining, both the buyer and seller share the common goal of reaching an agreement. However, each party also aims to maximize their own profit, leading to competition over the transaction price. Consequently, during this process, each side must infer the other's willingness to cooperate and adjust their negotiation strategy accordingly. If the other party shows a high willingness to cooperate, such as when a seller notices that the buyer is very eager to get the good, the seller can propose a higher price. Conversely, if the other side displays a very unfriendly attitude, the agent should consider making some concessions.\n\nAnother example is autonomous driving. When two vehicles traveling towards each other meet on a narrow road, they infer the other's manner based on its behavior. If the other one shows courtesy, the focal vehicle proceeds first; if the other one is brash or in a hurry, the focal agent yields to avoid conflict.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eidmJm3BMs", "reviewer": "Reviewer_AnKj", "summary": "This work proposes LASE, a multi-agent reinforcement learning framework that aims to improve co-operation between agents in mixed-motive games using transfer of rewards between agents in a zero-sum manner. Specifically, each agent uses counterfactual reasoning to compute a social relationship metric that computes the effect of their actions on the Q-values of other agents. Notably, LASE uses fully decentralized training, in contrast to many related works in the area. Finally, LASE outperforms existing baselines in a variety of popular mixed-motive environments such as Cleanup.", "questions": "1.\tIn general, given the complexity of the framework and the dependence on the joint action, I am sceptical about the scalability of the method. It would be interesting to see more results that test the scalability of the method. Convincing results with 8 agents for Cleanup and 1 other environment would significantly increase the strength of the paper. \n2.\tThe observation conversion network is not trained using the ground truth observations of agent j, but instead using the ground truth observations of agent i (this being key to the fully decentralized claim of the work). This makes me wonder why such a network is needed at all? What happens if a network that directly maps from agent i’s observation to agent j’s policy is learned. It would be interesting to see this in an ablation study. In other words, why is the \"perspective taking\" module not a single network?\n3.\tHow is the SR policy network trained? I am confused why it is trained using RL and not simply using supervised learning using the actual observed actions of agent j.\n4.\tI am not convinced by the argument that removing the cleaning beam in Cleanup makes the environment harder. Removing the cleaning beam also reduces the dimensionality of the action space. What was the reason behind removing it?\n5.\tFigure 1 is slightly misleading as it shows separate parameters for the observation conversion network and SR policy network. However, equation 4 shows them having the same parameters. \n6.\tI am unsure why the policy has a manual epsilon greedy added to it. Actor critic methods are on-policy, the epsilon greedy changes that and I am not sure this manual tuning for the policy is standard. Why was this required?", "limitations": "Sufficient details provided.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1.\tThe paper is well-organized, clearly written and technically sound. The general flow of the paper is smooth and proposed methods are explained reasonably well. The paper has an appropriate number of citations and properly details existing work in the related work section. \n2.\tThe presented framework is fully decentralized, which gives it an advantage to most works in the area that use centralized training decentralized execution (CTDE).\n3.\tThe results are generally promising with LASE showing significant gains in most environments tested. The analysis of co-operation strategies learned by different methods is interesting. In particular, I liked the GO vs LASE BG ablation study in Section 6.3.", "weaknesses": "1) The framework is relatively complex, as it requires learning two additional networks (perspective-taking and Q networks) for each agent.\n2) Given the dependence of the method on joint action, I have doubts about the scalability of the method. As more agents are added into the system, credit assignment would become more difficult. \n3) Some experimental details need to be clarified further.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> The complexity of the framework and the scalability of the method.\n\nThank you very much for your suggestions on LASE’s scalability! To test LASE’s scalability, we have extended Cleanup and Snowdrift as follows:\n\n|  | Map size | Player num | Obs size | Init Waste/ Snowdrift num | Episode length |\n| --- | --- | --- | --- | --- | --- |\n| Cleanup.Extn | 8→12 | 4→8 | 5→7 | 8→16 | 100→150 |\n| Snowdrift.Extn | 8→12 | 4→8 | 5→7 | 6→12 | 50→70 |\n\nHere are the experimental results:\n\n|  | LASE | IA | LIO | SI | A2C |\n| --- | --- | --- | --- | --- | --- |\n| Cleanup.Extn | **56.513** | 20.798 | 1.294 | 3.548 | 0.135 |\n| Snowdrift.Extn | **232.564** | 227.762 | 20.317 | 207.461 | 134.964 |\n\nAs shown above, LASE still outperforms the baselines in extended environments. Thus, we say that LASE has a certain degree of scalability. We believe that learning two additional networks for opponent modeling and social relationship inference in a decentralized manner is not overly complicated, ensuring the method's scalability. However, it is undeniable that the current version of LASE requires inferring the social relationships of each agent individually, leading to increased computational complexity as the number of agents increases greatly. The scalability of LASE will be a main focus of our future study. A possible approach is updating the relationship between agents less frequently.\n\n> The explanation of the observational conversion network and the SR policy network. An ablation study about mapping $i$’s observation to $j$’s policy directly.\n\nThank you very much for your suggestion to replace the PT module with a single network for the ablation study! We trained the network $p(\\mathbf{\\hat{a_t}} | o^i_t, \\mathbf{a_{t-1}})$ with supervised learning by minimizing the MSE loss of the predicted joint action $\\mathbf{\\hat{a_t}}$ and the real action $\\mathbf{a_t}$, and get the following results:\n\n|  | SSH | SSG | Cleanup | Coingame |\n| --- | --- | --- | --- | --- |\n| LASE w/o PT | 18.442 | 118.616 | 37.174 | 29.541 |\n| LASE | 18.948 | 117.784 | 38.736 | 33.467 |\n\nThe results show that the two methods perform comparably in SSH, SSG, and Cleanup, while LASE performs better in Coingame.  However, since both approaches predict others' actions based on the same local observations, they essentially serve as opponent modeling tools and thus exhibit similar performance.\n\nWe use the PT module in LASE for two main reasons: \n\n**First**, since the SR value network $\\phi$ is required to calculate the egocentric $Q$-value when carrying out counterfactual reasoning, and the SR policy network $\\mu$ predicts others' actions with the egocentric policy model, both networks share the CNN and some FC layers to extract the observation features.  And they are optimized with the same reward signal under the actor-critic framework (Eq. 5). This sharing is common in actor-critic framework and helps improve training efficiency. Therefore, although the PT module in Figure 1 consists of two networks, the additional parameters and computational overhead mostly stem from the observation conversion network, and it introduces essentially the same amount of parameters as using a single perspective taking network.\n\n**Second**, based on the psychological theory that perspective-taking is a crucial component of cognitive empathy [1], we employ the PT module instead of an end-to-end trained neural network to predict opponent actions, which helps computationally and comprehensively model empathy, the essential mechanism in human society. Since this approach does not result in significant performance loss or increased memory and computational demands, we believe our work contributes new insights into opponent modeling and encourages the community to incorporate human cognitive processes into AI agent design.\n\n[1] Davis, Mark H. \"Measuring individual differences in empathy: Evidence for a multidimensional approach.\" *Journal of personality and social psychology* 44.1 (1983): 113.\n\n> Removing the cleaning beam in Cleanup.\n\nIn our implementation, the cleaning beam is replaced by a cleaning action which takes effect only at the waste location, thereby maintaining the dimensionality of the action space. Since the river where the waste accumulates and the apple orchard are located on opposite sides of the map, removing the cleaning beam prevents the agent from directly cleaning waste near the apple orchard. Meanwhile, the time cost of traveling between the river and the apple orchard requires agents to collectively learn a more explicit division of labor strategy in the dilemma. This is why we claim that removal will make the environment harder.\n\n> The parameters of the observation conversion network and SR policy network.\n\nSorry for the confusion! The observation conversion network $\\eta$ and the SR policy network $\\mu$ do employ separate sets of parameters. In Equation 4, even though the loss is computed forward through $\\mu$, is not used to update $\\mu$, but only to update $\\eta$. The update method of $\\mu$ is in Eq.5.\n\n> Epsilon greedy\n\nWe use epsilon greedy mainly to enhance exploration and avoid falling into local optima, which is more likely to occur in mixed-motive games. The introduction of epsilon greedy allows for a controlled level of exploration. The utilization of epsilon greedy is not an anomaly. The classic on-policy algorithm SARSA [2] uses epsilon greedy. A low epsilon value does not cause drastic policy changes, so on-policy methods remain effective. This strategic balance between exploitation and exploration has proven to be empirically sound, fostering a robust approach to algorithmic performance. A similar example is PPO, which uses trajectories sampled from a slightly offset policy for update, but it is still classified as an on-policy algorithm, and it has demonstrated strong empirical performance.\n\n[2] Sutton, Richard S. \"Generalization in reinforcement learning: Successful examples using sparse coarse coding.\" NeurIPS 1995", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oXN7BjGayr", "reviewer": "Reviewer_vMEk", "summary": "The paper introduces LASE, a novel distributed multi-agent reinforcement learning algorithm. LASE aims to foster altruistic cooperation through a gifting mechanism while avoiding exploitation in mixed-motive games. The algorithm dynamically adjusts the allocation of rewards based on social relationships inferred using counterfactual reasoning. The paper reports comprehensive experiments in various mixed-motive games, demonstrating that LASE effectively promotes group collaboration and adapts policies to different co-player types without compromising fairness.", "questions": "1.How does LASE compare to the original gifting methods? Including a direct comparison or an ablation study focusing on the gifting mechanism could provide deeper insights into its contribution.\n2.Can you provide an ablation study that isolates the impact of the counterfactual reasoning module? This would help in understanding its specific role and effectiveness.\n3. Why not compare the Equality with other baseline methods? I am curious about the equality performance of baseline methods? from the report of their paper, I guess some of them will have not bad performance.", "limitations": "The paper seems lack of discussion about limitations and broader societal impacts.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper presents an innovative combination of gifting mechanisms and counterfactual reasoning within a multi-agent reinforcement learning framework. This approach to dynamically adjust reward allocation based on inferred social relationships is novel and well-grounded in developmental psychology theories of empathy.\n\nThe paper is clearly written and well-organized. The methodology and experimental setup are described in detail, making it relatively easy to follow and reproduce the results. Additionally, the authors conducted comprehensive experiments across various mixed-motive game scenarios, thoroughly demonstrating the effectiveness of LASE in promoting cooperation and fairness. The breadth and depth of these experiments add significant credibility to the proposed method.", "weaknesses": "The proposed method, while interesting, could be seen as incremental since it primarily combines existing techniques (gifting and counterfactual reasoning). The novelty could be better highlighted by contrasting more explicitly with prior works. \n\nThe more important weakness is that the paper lacks a detailed analysis of the individual contributions of the gifting mechanism and counterfactual reasoning. For example, there is no direct comparison with the original gifting method or an ablation study isolating the impact of the counterfactual reasoning module. This makes it difficult to discern the specific roles and contributions of these components to the overall performance. \n\nThus, I think the paper is below the acceptance bar for NeurIPS.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments on our work!\n\n> The comparison to the original gifting methods and the individual contributions of the gifting mechanism.\n\nOur primary contribution lies in designing an algorithm that can dynamically adjust the gifting strategy to different co-players, performing exceptionally well in multi-agent decision-making and adaptation. In contrast, most of the previous works predefine fixed gifting amounts, lacking adaptation ability. LIO and LToS [4] have learned gifting strategies, but underperform in mixed-motive scenarios.\n\nAs far as we know, [1] first used the gifting mechanism to promote cooperation in sequential social dilemmas. The implementation of the gifting in [1] is to equip the agent with the extra action of a \"gifting beam\", which gives other agents in the beam a reward of $g$, and incurs an immediate penalty of $−g$ to itself. We try to approximatively implement this method by adding an additional gifting action to the A2C agents. However, since this action incurs an immediate $−g$ penalty, even a small $g$ will quickly reduce the probability of choosing this action, and the result is almost the same as that of A2C without gifting in our baselines which underperforms LASE.\n\nThere are also some works that focus on how social dilemmas can be resolved by pre-defining the proportion of rewards to be distributed (gifted) [2] [3]. At one extreme, equal distribution makes everyone's goal the group's total benefit, the same as our GO baseline. For a clearer comparison, we conduct an additional ablation experiment using the converged gifting weight (as shown in Table 2) as a fixed gifting weight for each agent.\n\n||SSH|SSG|Cleanup|Coingame|\n|---|---|---|---|---|\n|Original gifting|12.937|80.347|0.172|0.209|\n|A2C|12.476|81.285|0.274|0.036|\n|Fixed weight|14.626|55.117|35.704|3.435|\n|LASE|18.948|117.784|38.736|33.467|\n\nLASE outperforms because it can encourage cooperative behavior by rewarding some specific agents. In addition, LASE can dynamically adjust the proportion of rewards shared with different agents, which helps avoid being exploited by others.\n\nTo sum up, our contribution to the gifting mechanism is mainly reflected in the fact that LASE can adjust the gifting strategy dynamically by inferring the social relationships with others, and can effectively promote group cooperation and avoid exploitation under various sequential social dilemmas.\n\n[1] Lupu et al., Gifting in multi-agent reinforcement learning, AAMAS 2020\n\n[2] Wang, et al., Emergent prosociality in multi-agent games through gifting, IJCAI 2021\n\n[3] Willis et al., Resolving social dilemmas through reward transfer commitments, ALA 2023\n\n> Ablation study, contribution and effectiveness about the counterfactual reasoning module.\n\nThe social relationships inferred through counterfactual reasoning guide our gifting strategy and form the core of our algorithm, making ablation experiments on this module challenging. A feasible method is to use an end-to-end trained neural network to determine gifting weights, as done by LToS [4]. So we add LToS as one baseline:\n\n||SSH|SSG|Cleanup|Coingame|\n|---|---|---|---|---|\n|LToS|12.476|77.386|1.912|-0.078|\n|LASE|18.948|117.784|38.736|33.467|\n\nIt shows that the counterfactual reasoning module helps LASE outperform LToS. Meanwhile, LIO as a method that does not use counterfactual reasoning but directly uses neural networks to output gifted rewards has also been compared as a baseline in the paper.\n\nCounterfactual reasoning is an effective idea in multi-agent learning, however, most previous research has focused on cooperative tasks using the CTDE framework [5] or competitive tasks using counterfactual regret minimization (CFR) [6]. In contrast, LASE employs counterfactual reasoning to infer social relationships in a decentralized manner, achieving excellent performance in mixed-motive games. We believe this innovation has the potential to significantly enhance community development.\n\n[4] Yi, et al., Learning to share in multi-agent reinforcement learning, NeurIPS 2022\n\n[5] Foerster et al., Counterfactual Multi-Agent Policy Gradients, AAAI 2018\n\n[6] Zinkevich, et al., Regret minimization in games with incomplete information, NeurIPS 2007\n\n> Compare the Equality with other baseline methods.\n\nAs an evaluation metric, fairness should be evaluated alongside reward to measure algorithm performance effectively. Some algorithms may fail to address decision-making issues in mixed-motive games where each agent receives a small reward, but the reward disparity between agents is minimal, resulting in high fairness. Clearly, these methods are not effective. An effective method should both maximize group reward and ensure intra-group equity. Thus, we compared only LASE and GO, which achieve the highest rewards. We have now included the fairness results of other baselines:\n\n|Fairness|SSH|SSG|Coingame|Cleanup|\n|---|---|---|---|---|\n|LASE|0.994|0.951|0.835|0.802|\n|LASE w/o|0.986|0.862|0.848|0.685|\n|GO|0.968|0.856|0.785|0.496|\n|IA|0.984|0.877|0.898|0.708|\n|LIO|0.931|0.985|0.745|0.545|\n|SI|0.995|0.937|0.750|0.892|\n|A2C|0.997|0.854|0.831|0.824|\n\nAs you guess, LASE does not always outperform the baseline on fairness metrics, such as SI and A2C in SSH, LIO in SSG, IA in Coingame, and SI in Cleanup. However, LASE can significantly enhance group benefits while maintaining high fairness, which other baselines can’t.\n\n> Limitations and broader societal impacts.\n\nWe have mentioned the limitations of our work in Section 7, Conclusion, including the assumption that each agent's reward function will be modified by other agents' gifts, whereas in reality, people may refuse gifts. Additionally, LASE currently focuses on finite discrete action spaces, and extending it to continuous action spaces is our next goal.\n\nWe discussed the broader impact of our work in Appendix D. If you have any further questions, please feel free to ask! We are happy to discuss this with you!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: All reviewers feel the paper makes a sound and useful contribution.  The authors effectively addressed the reviewers' concerns during the discussion phase.  I encourage the authors to implement what they mentioned in the discussion to improve the paper.
Decision: Accept (poster)