Learning to Balance Altruism and Self-interest Based on Empathy in Mixed-Motive Games

Authors: fanqi  kong
yizhe  huang
song-chun  zhu
siyuan  qi
xue  feng
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 11:37:49

Abstract: Real-world multi-agent scenarios often involve mixed motives, demanding altruistic agents capable of self-protection against potential exploitation. However, existing approaches often struggle to achieve both objectives. In this paper, based on that empathic responses are modulated by learned social relationships between agents, we propose LASE (**L**earning to balance **A**ltruism and **S**elf-interest based on **E**mpathy), a distributed multi-agent reinforcement learning algorithm that fosters altruistic cooperation through gifting while avoiding exploitation by other agents in mixed-motive games. LASE allocates a portion of its rewards to co-players as gifts, with this allocation adapting dynamically based on the social relationship --- a metric evaluating the friendliness of co-players estimated by counterfactual reasoning. In particular, social relationship measures each co-player by comparing the estimated $Q$-function of current joint action to a counterfactual baseline which marginalizes the co-player's action, with its action distribution inferred by a perspective-taking module. Comprehensive experiments are performed in spatially and temporally extended mixed-motive games, demonstrating LASE's ability to promote group collaboration without compromising fairness and its capacity to adapt policies to various types of interactive co-players.

Main paper: 
 introduction: 
 Multi-agent reinforcement learning (MARL) has exhibited impressive performance in numerous collaborative tasks and zero-sum games such as MPE, StarCraft, and Google Research Football [19, 25, 38] . These environments involve a predefined competitive or cooperative relationship between agents. Besides, mixed-motive games are prevalent, in which the relationships between agents are non-deterministic and dynamic. That is, agents could cooperate with some co-players and simultaneously compete with someone else. Furthermore, along with interactions, friends may turn into foes, and vice versa. In such games, to maximize self-interest, agents need to cooperate altruistically in some relationships while keep self-interested to avoid being exploited in some others. Consequently, in mixed-motive environments, the ability to balance altruism and self-interest according to social relationships is crucial for agent performance.The commonly used CTDE (Centralized Training and Decentralized Execution) methods [31, 29] in MARL focus on global optimization goals and necessitate individual information sharing with centralized controllers, which is impractical for self-interest agents in mixed-motive games. On the other hand, simply training self-interest agents in a decentralized way may converge to local Corresponding author. optima, failing to maximize individual interests. For example, in Iterated Prisoner's Dilemma (IPD), decentralized A2C agents converge to defection, getting the minimal reward 0 (see details in Fig. 4 ).Within the framework of decentralized learning, the gifting mechanism has been used to address the decision-making problems in mixed-motive games by enabling agents to transfer a portion of their rewards to others [20, 4] . Agents independently select gift recipients and determine gift amount, which complies with the decentralized requirement. Gifting can potentially shape co-players' policies and even incentivize them to behave more altruistically by influencing co-players' reward structure. Previous work has studied handcrafted gifting scheme [20, 34] and has learned end-to-end neural networks to determine the reward transfer scheme [36, 37] . However, a consideration of the correlation between social relationships and response strategies is absent, which is crucial for decision-making in mixed-motive games.To balance the dilemma between altruism and self-interest caused by the non-deterministic and dynamic relationship between agents, it is a feasible way that adaptively modulating the gift amount to others according to the social relationships. This process is called (cognitive) empathy in developmental psychology [28] . What's more, previous studies and human behavioral experiments have shown that empathy can promote the emergence of altruism among self-interested individuals [1, 2, 30] . To the best of our knowledge, there has been a lack of computational models of empathy and the study of empathy-based decision-making. In this work, we propose a computational model of empathy, in which social relationship is measured by a continuous variable, capturing the influence of co-players' behavior on the focal agent's reward and guiding gift scheme. On that basis, we provide a distributed MARL algorithm LASE (Learning to balance Altruism and Self-interest based on Empathy) to address the dilemma between altruism and self-interest in mixed-motive games.LASE uses counterfactual reasoning to infer the different social relationships with different coplayers separately by comparing the estimated Q-value for the joint action to a counterfactual baseline established for each other agent. The counterfactual baseline marginalizes a single agent's action while keeping the other agents' actions fixed. This computational approach to social relationships enables LASE to explicitly decompose the value contributions of other agents to it, thereby providing clearer guidance for gift allocation. That is, gift more to the co-players who contribute more. Additionally, to deal with the challenge of inferring co-players in partially observable and decentralized environments, LASE is equipped with a perspectivec taking module to predict others' policies by converting LASE's local observation to a simulated observation of others.To verify the effectiveness of LASE, we theoretically analyze its dynamics of decision-making in iterated mixed-motive games and conduct comprehensive experiments in spatially and temporally extended mixed-motive games. The results demonstrate LASE's ability to promote group cooperation without compromising fairness and its capability of self-protection against potential exploitation. This paper makes three main contributions. (1) To our best knowledge, we are the first to computationally model empathy which modulates the response based on inferred social relationships.(2) We present LASE, a decentralized MARL algorithm that balances altruism and self-interest in mixed-motive games. It flexibly adapts strategy to promote cooperation while mitigating exploitation by others. (3) We provide a theoretical analysis of decision dynamics in iterated matrix games and experimentally verify that LASE outperforms baselines in a variety of sequential social dilemmas. 
related work: 
 In multi-agent learning, the game-theoretic notion of social dilemmas has been generalized from the classic two-player matrix-form games Tab. 1 to sequential social dilemmas, a spatial-temporallyextended complex behavior learning setting [15, 4] . Various approaches have been proposed to foster cooperative behavior among agents to advance societal welfare. One approach incorporates the rewards of others as intrinsic rewards into one's own optimization objectives, with the ratio of intrinsic to extrinsic rewards determined by different methods, such as pre-defined social value orientations like altruistic or prosocial [21, 23] , introducing the concept of inequity aversion [11] , or learning in a model-free way [33] . However, this approach depends on direct access to others' reward functions, which may not be feasible in realistic mixed-motive games. Another line of work dispenses with this assumption, allowing agents to model others and influence them through their actions [6, 17, 13, 8, 10] . Here, we employ a more direct form of opponent shaping named gifting.As a peer rewarding mechanism that allows agents to reward other agents as a part of their action space [20] , gifting can be viewed as a process of redistributing rewards among agents [12, 9, 7] , but the key difference is that in our work, gifting does not require a powerful centralized controller to decide on the allocation. Instead, individuals make their own decisions about gifting, which is more in line with the setting of decentralized training. [34, 35] study the theoretical basis of how gifting promotes cooperative behavior in simple social dilemmas, while LIO [36] independently learns an incentive function to gift others. However, the rewards used for gifting in LIO are determined by the incentive function rather than split from its own reward. LIO doesn't adhere to zero-sum conditions, which to some extent alters the original game-theoretic nature. LToS [37] models the optimization problem of gifting (sharing) weights with the zero-sum setting as a bi-level problem and uses an end-to-end approach to train weights and policies jointly. MOCA [3] introduces contracts to restrict gifting recipients to the agents that fulfill certain behavioral patterns.Our perspective taking module simulates others' behavior by adopting their perspective, a technique akin to Self-Other Modeling (SOM) [24] . The difference is that instead of inferring the agent's goal which may not necessarily be well defined in some environments, we directly imagine their observations, and decouple the agent's network for inferring others' actions from its own real policy network utilized for execution. In MARL, some prior studies have employed counterfactual reasoning to deduce the impact of individual actions on others [13] or the whole team [5] . In contrast, our focus lies on assessing others' influence on the focal agent. 
partially observable markov games: 
 We consider an N -player partially observable Markov game (POMG) [27, 18] , M = N, S, O i , A i , P, R i , where N represents the number of agents, s ∈ S represents the state of the environment. In the partially observable setting, agent i only obtains the local observation o i ∈ O i based on the current state s. Each agent i learns an independent policy π i (a i |o i ) to select actions and form a joint action a = (a 1 , ..., a N ) ∈ A 1 × • • • × A N , resulting in the state change from s to s ′ according to the transition functionP : S × A 1 × • • • × A N → ∆(S), where ∆(S) represents a probability distribution over the set S. Agent i receives an individual extrinsic reward r i = R i (s, a 1 , • • • , a N ) and tries to maximize a long-term return:EQUATIONwhere the variables in bold represent the joint information of all agents, and γ is the discount factor.In decentralized MARL, each self-interest agent i learns an independent policy π i parameterized by θ i . The optimization objective is to maximize the expected return in Eq. 1. We use the policybased Actor-Critic method as the learning algorithm for our agents. The gradient for actor is∇ θ J(θ) = E π θ [ T t=0 ψ t ∇ θ log π θ (a t |o t )], where ψ t represents the critic's evaluation of the actor. The most popular form of ψ t is TD-error:ψ t = r t + γV π θ (s t+1 ) -V π θ (s t ). 
methodology: 
 To balance altruism and self-interest in mixed-motive games, we propose a distributed MARL algorithm LASE which empathically shares rewards with co-players based on inferred social relationships. The architecture of LASE is illustrated in Fig. 1 , composed of two main modules: Social Relationship Inference (SRI) and Gifting. SRI conducts counterfactual reasoning to get the social relationships with co-players, which reflects the impact of co-players' actions on LASE's return. SRI compares the Q-value (estimated by the SR value network) of the current joint action to a counterfactual baseline which marginalizes the co-player's action, with its action distribution inferred by a perspective-taking (PT) module. PT is provided to address the challenge of predicting co-players' policies in partially observable and decentralized environments. In particular, PT consists of an observation conversion network, simulating the co-player j's observation ôj from the local observation o i , and an SR policy network, learning a function from ôj to the inferred j's policy.The Gifting module, according to the inferred social relationships, determines the amount of reward sharing with others. Meanwhile, agents receive others' gifts and get the final reward r i,tot , which SRI conducts counterfactual reasoning to get the social relationships with co-players. The social relationship is measured by comparing the Q-value (estimated by the SR value network) of the current joint action to a counterfactual baseline which marginalizes the co-player's action, with its action distribution inferred by a perspective-taking (PT) module. PT is provided to address the challenge of predicting co-players' policies in partially observable and decentralized environments. The Gifting module, according to the inferred social relationships, determines the amount of reward to share.serves as an optimization target to guide the training of policy π i . Section 4.1 and Section 4.2 will introduce our algorithm in detail. LASE's pseudocode is given as Algorithm 1.Here, we use the zero-sum gifting mechanism which adheres to the principle of "what I give is what I lose" [20] , indicating that the overall rewards for the group remain constant. Notably, gifting is an autonomous decision, wherein any agent i holds a gifting weight vector at time step t:w i t = [w ij t ] N j=1 , where w ij t ∈ [0, 1] and N j=1 w ij t = 1.w ij t is the fraction of agent i's reward to gift j. It is exactly the social relationship computed as Eq. 3. For an N -player group using the zero-sum gifting mechanism, r t denotes the extrinsic rewards vector obtained through interactions with the environment at timestep t. Agent i's total reward is computed byEQUATIONThe policy π i (a i t |o i t ) is trained to maxmize E π i [ T t=0 γ t r i,tot t ] using the TD-error introduced in Section 3.2 by replacing r t with r tot t .The social relationship w ij , modeled as a continuous variable, measures i's inference of j's friendliness to him. Based on counterfactual reasoning, w ij is inferred as:EQUATIONwhere Q i SR (o i t ,M = (N -1)(max a j ′ t Q i SR (o i t , (a -j t a -j t a -j t , a j ′ t ) -min a j ′ t Q i SR (o i t , (a -j t a -j t a -j t , a j ′ t ))), where (N -1) ensures w ij ≤ 1 N -1 . After gifting, LASE keeps the remaining reward for itself, w ii = 1 -N j=1,j̸ =i w ij . Due to partial observability and decentralized learning, i is unable to accurately obtain j's policy π j and its observation o j t . So we utilize PT module to estimate j's policy, denoted as π i SR (a j ′ t |ô j t ). π i SR is the SR policy network parameterized by µ i which predicts j's actions conditioned on the simulated obervation ôj t learned by the observation conversion network. The observation conversion network, parameterized by η i , enables LASE to adopt the perspective of others and generate a simulated observation of them. At timestep t, LASE processes its own observation o i t along with another agent j's ID (represented as a one-hot vector), yielding the output ôj t which is of the same size as o i t . To update η i and get a more accurate ôj t , the loss function isEQUATIONThe first term aims to ensure that the predicted policy π i SR (a j t |ô j t ) aligns with co-player j's actual actions. The second term aims to minimize the deviation of the simulated observation ôj t from i's true observation o i t , so that some common features in the environment can be reconstructed. The hyperparameter δ ∈ [0, 1] balances the two goals.Since SR policy network predicts actions from an agent's self-perspective and SR value network estimates the agent's own returns, we integrated the training processes of the two networks within an actor-critic framework. In training, π i SR takes one agent's observation as input and outputs a probability distribution over his action space. Q i SR computes the Q-value of the joint action under the current observation. Both networks are updated based on the individual extrinsic rewards obtained in the environment, and the TD-error defined asδ i t = r i t +γQ i SR (o i t+1 , a t+1 )-Q i SR (o i t , a t ).Specifically, the SR policy network and SR value network, parameterized by µ i and ϕ i , are updated byEQUATIONWe can further intuitively comprehend Eq. 3: when w ij t attains the maximum value of 1/(N -1), j has taken the best action that maximized Q i SR and i predicts with a probability of 1 that j will select the action that minimizes Q i SR . At this point, the value j brings to i far exceeds i's psychological expectation, which corresponds to the real-world scenario where people feel particularly happy when they are helped by someone they might have thought was unkind to them. So the amount of i's gifting to j reaches the maximum. It is worth noting that Eq. 3 may yield a negative value, indicating that the agent is required to acquire rewards from other agents. Dealing with this scenario becomes intricate, particularly when the other agent is uncooperative. As this type of competition is not the primary focus of our research, we assign w ij = 0 when w ij ≤ 0. We carry out a closed-form gradient descent analysis on LASE in the two-player iterated matrix games and derive the policy update rule Eq. 15 and Eq. 16, where each agent i optimizes the reward after gifting r i,tot . The detailed deduction is provided in Appendix A. Then we simulate the policy update iteratively with random initial value and plot LASE's cooperation probability after convergence under various game parameters as illustrated in Fig. 2 .The results demonstrate that LASE converges to pure cooperation in Harmony and SH. In the more intense games, SG and PD, LASE stabilizes at cooperating with a probability greater than 0.5, successfully escaping from non-efficient Nash equilibria. To evaluate the ability of LASE to address more complex environments, we study its performance in partially observable SSDs. SSDs extend matrix-form games in terms of space, time, and number of agents. Here, we study four specific SSDs: Coingame, Cleanup, Sequential Stag-Hunt (SSH), and Sequential Snowdrift Game (SSG) (Fig. 3 ). Schelling diagrams (see Fig. 10 ) of the four environments validate that they are appropriate extensions of representative game paradigms (a detailed analysis is given in Appendix B). Below is a detailed description of the four environments. Coingame. The Coingame was originally introduced in [16] as a higher dimensional alternative to the IPD with multi-step actions. In this environment, two agents, 'red' and 'blue', collect coins in a 5 × 5 map. A coin is either red or blue and there is only one coin on the map at any timestep. Agents will get a reward of 1 by picking up a coin of any color. However, when an agent collects a coin of a different color, the other agent will lose 2 points. After a pickup, a new coin with a random color and random location appears immediately. Therefore, if each agent greedily picks up all the coins, the sum of their expected scores will be 0.Cleanup. In Cleanup, the goal of the agent is to gather as many apples as possible, with each apple carrying a reward of +1. However, the accumulation of waste in the river steadily approaches a depletion threshold, causing a linear decline in the apple growth rate to 0. At the beginning of each episode, the waste level exceeds the threshold and there are no apples in the map. This places the agent in a social dilemma: while individually focusing on collecting apples under the map leads to higher rewards, if all agents opt to refuse waste cleanup, no rewards are obtained. To maintain consistency with other environments, we partially modify the setting of cleanup from [11] . We eliminate the actions of firing beams (cleaning and zapping) and require the agent to move to the waste's position to clean it. This does not alter the nature of the dilemma but makes it more challenging because the cleaning beam could have helped the agent clean from a distance.Sequential Stag-Hunt (SSH). This environment is inspired by Markov Stag Hunt in [23] . Each agent can get a reward of +1 by hunting a hare while hunting stags is more challenging and requires two or more agents. Each stag can bring a reward of +10, which is divided equally among the agents that jointly hunted it. The agent is immediately removed from the environment after successfully hunting once. Therefore, if an agent chooses to hunt stags, it must contend with the risk of no one cooperating with it. In contrast, hunting rabbits is a safer choice.Sequential Snowdrift Game (SSG). In SSG, there are 6 piles of snowdrifts which can be removed by the agent. A pile of removed snowdrifts brings a +6 reward for each agent, but the remover incurs a cost of 4. So the agent waiting for others to remove the snowdrift (free-rider) can obtain a higher return. However, if no one chooses to remove it, no rewards can be obtained by anyone. 
implementations: 
 We employ fully decentralized training and execution for the agents, where all network parameters are independent. The policy structure of the agent comprises 2 convolutional layers for encoding observations, an LSTM layer to capture temporal information, and several fully connected layers activated by ReLU. The input is a multi-channel binary tensor, with the specific number of channels determined by the characteristics of different environments. For example, in Cleanup, 7 channels are incorporated, wherein the first four channels denote the positions of the four agents, the fifth and sixth channels signify waste and apples, and the last channel distinguishes between the inside and outside of the map through masking. To ensure adequate exploration, we let π(a|o) = (1 -ϵ)π(a|o) + ϵ/|A|, with ϵ decaying linearly from ϵ start to ϵ end over ϵ div episodes. The SR policy network and SR value network of the SRI module share two convolutional layers, followed by their own linear layers activated by ReLU function. The difference is that the inputs of the linear layers in SR value network include an additional concatenated one-hot vector of the joint action. The observation conversion network uses a single convolutional layer to encode observation, which is then concatenated with a one-hot vector representing the ID of other agents. This concatenated input is then processed through two linear layers, with the resulting output normalized using the sigmoid function.In IPD, we modify the implementation by removing convolutional layers, reducing the parameters of FC layers, and appropriately increasing the learning rate to adapt the algorithm to this environment. See Appendix C for more details about the implementations of LASE.Independent advantage actor-critic labeled A2C [22] is a classical gradient-based RL algorithm. LIO [36] learns an incentive function through the learning updates of reward recipients. LOLA [6] considers the learning process of other agents when updating its own policy parameters. IA [11, 32] modifies the individual reward function by introducing inequity aversion. SI [13, 32] achieves coordination by rewarding agents for having causal influence over other agents' actions. We also show the approximate upper bound on performance by training the group optimal (GO) agents to maximize the collective reward. And we conduct ablation experiments LASE w/o, by removing the observation conversion network and replacing π i SR (a j ′ t |ô j t ) in Eq. 3 with 1/|A j |.6 Results 
lase promotes cooperation in social dilemmas: 
 In IPD, as shown in Fig. 4 , LASE successfully escapes the dilemma of non-efficient Nash Equilibrium (D, D). Both LASE agents converge to cooperate with a high probability, around 0.93 (see Fig. 4a ), accompanied by a high collective reward. This is consistent with the theoretical results shown in Fig. 2 , validating the effectiveness of LASE in dealing with social dilemmas. LASE is better at convergence speed and stability than LOLA, which also achieves a high collective reward. Unsurprisingly, GO, aiming to maximize group reward, reaches the upper bound. On the other hand, A2C, optimizing for one's own return, easily falls into the Nash equilibrium, where everyone defects and the group reward reduces to the minimum.In SSH and SSG, LASE nearly reaches the upper bound of the total rewards as shown in Fig. 5a and Fig. 5b : a total reward of 20 for successfully hunting two stags in SSH and a total reward of 120 in SSG for removing all the six snowdrifts. In Coingame (Fig. 5c ) and Cleanup (Fig. 5d ), LASE exhibits commendable performance by effectively avoiding the sub-optimal equilibrium, where everyone defects. GO outperforms LASE in SSG, Coingame, and Cleanup, because its optimization objective of maximizing the collective reward of all the agents and a strong assumption of the accessibility of everyone's reward function can help it avoid the dilemma.On the other hand, it's worth emphasizing that the architecture of GO also gets it into the lazy problem [31] , shown as its underperformance in SSH (Fig. 5a ). In SSH, early hunting leads to moving out of the environment and failing to obtain the group rewards of others' later hunting. Thus, GO may not hunt until the last few steps, and likely misses the the opportunity of cooperating to hunt stags.A2C and SI hardly improve the collective return in Coingame and Cleanup, and fail to outperform LASE in SSH and SSG, where the dilemma is less intense. Although IA can directly access the rewards of others and adjust its own intrinsic rewards accordingly, it is still unable to get a higher collective return than LASE. Also as a gifting algorithm, LIO struggles to perform well in the four environments. Meanwhile, the need to manually specify the hyperparameters used to scale the amount of gifting also creates challenges for LIO to apply to different environments. In contrast, LASE breaks the limitations and achieves significantly better performance than LIO. Considering the fact that the division of labor in Cleanup is more pronounced than in other environments and that cooperation in Cleanup is purely altruistic, making the dilemma more challenging, we take Cleanup as an example to show LASE's ability to promote fairness. Fig. 6a and Fig. 6b show the extrinsic rewards of each agent and the amount of waste cleaned by each in Cleanup. We find Agent 4 is the only one to clean and does not receive any extrinsic reward.Fig. 6c illustrates the gifting weights each agent received from the other three agents. Agent 4 gets the most gifts, indicating that its cleaning contribution to the team is recognized and rewarded by the other three agents. Fig. 6d shows reward curves after gifting, where the reward gap between Agent 4 and the other three agents shrinks, implying fairness within the group is improved.Figure 7 : Four GOs' rewards in Cleanup.Agents 1-3 always collect apples without cleaning, but this is not a freerider behavior. Both cleaning and collecting are indispensable to get rewards in Cleanup. Although Agent 4 does not obtain any reward directly from the environment, the acquired reward is redistributed through gifting, narrowing the gap of reward between the cleaner and collectors. In contrast, although GO obtains higher collective returns, it sacrifices the individual interests of Agent 1 and Agent 2, while only Agent 3 and Agent 4 can get rewards Fig. 7 .We use Equality (E) given by E = 1-N i=1 N j=1 |Ri-Rj | 2N N i=1Ri [20] to quantify the fairness, where the second term is the Gini inequality index. The greater the value of E, the fairer it is. We can get E(LASE) ≈ 0.802, E(GO) ≈ 0.496, showing that LASE achieves a higher level of fairness. To evaluate LASE's adaptation ability to interact with various types of agents, we conduct an experiment in which a focal LASE agent interacts with three rule-based agents: cooperator (always clean up waste), defector (always try to collect apples), and a random agent. The gifting weights of LASE to the other agents are shown in Fig. 8 . LASE can explicitly distinguish between different types of co-players. Moreover, it responds in a manner that aligns with human values: preferring to share rewards with cooperators rather than defectors.To study how LASE responds dynamically and how it affects collective behavior, we conduct an experiment where one focal LASE agent interacts with three A2C agents (Background agents, Bgs). A GO agent is trained in the same way for comparison with LASE. Fig. 9a displays each agent's rewards after training for 30k episodes, whereas the LASE group shows the reward after gifting. LASE and GO improve the group reward to a similar level, while four A2C agents will converge to the equilibrium of defection and gain almost no reward (see Fig. 5d ). With gifting, LASE incentives Bg3 to clean as shown in Fig. 9b . Thus, LASE and the other two Bgs can get rewards by gathering apples. But for GO, the focal agent sacrifices itself to undertake all the cleaning tasks and cannot get any reward.GO and LASE represent two different methods to foster cooperation. GO sacrifices its own interest to promote collective reward. LASE attempts to alleviate social dilemmas by incentivizing others to cooperate. When such an incentive mechanism fails, LASE will no longer gift those agents who constantly exploit it, like the defector in Fig. 8 . Overall, we believe that LASE is a more efficient and secure policy for SSDs, as it can promote cooperation as well as avoid potential exploitation by others. 
conclusion: 
 We introduce LASE, a decentralized MARL algorithm that fosters cooperation through gifting while safeguarding individual interests in mixed-motive games. LASE uses counterfactual reasoning to infer the social relationships with others which captures the influences of others' actions on LASE and modulates the gifting strategy empathetically. In particular, to empower LASE with the ability to infer others' policies in partially observable and decentralized environments, we establish a perspective taking module for LASE. Both theoretical analyses in matrix-form games and experimental results across diverse SSDs show that LASE can effectively promote cooperative behavior while ensuring relative fairness within the group. Furthermore, LASE is also able to recognize various types of coplayers and adjust its gifting strategy adaptively to avoid being exploited, enabling broad applicability in complex real-world multi-agent interactions, such as automated negotiations in E-commerce and decision-making in autonomous driving. Whilst LASE exhibits superior abilities, there are some limitations of our method. What would be the consequence of giving agents the ability to refuse gifts? How to extend the algorithm LASE to continuous action space? These problems will illuminate our future work. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: The paper poses no such risks Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: The open-source codes used in the paper are cited. The license is MIT license. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA]Justification: The paper does not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: For a general iterated matrix game whose payoff matrix is computed according to Tab. 1 at each step, we can let θ i for i ∈ {1, 2} denote each agent's probability of taking the cooperative action, and let θj for j ∈ {2, 1} denote each agent's prediction of the other's policy. To avoid the difficulties caused by the coupled update of reinforcement learning for the theoretical analysis, we make the simplifying assumption that the three networks in SRI module have been fully trained. We further assume that this two-player game is fully observable, and we can make the following approximation to Eq. 3:EQUATIONHere, r i (a i , a j ) represents player i's reward determined by the payoff matrix which only relies on the two players' actions without state or observation. The update rule Eq. 5 states that it is feasible to replace Q i by r i . Then we can calculate the gifting weights under the four combinations of actions: CC, CD, DC, and DD. We take w 12 for example:EQUATIONEQUATIONEQUATIONEQUATION) Similarly as Section 4, we set w 12 less than 0 to 0. So agent 1's reward distribution scheme is:EQUATIONAgent 2's computation is symmetric. The total reward received by each agent isEQUATIONEQUATIONThe value function for each agent is defined byEQUATION)Agent 2 updates its policy byEQUATIONBy symmetry, agent 1 updates its policy byEQUATIONWe assume that LASE's prediction to the others is accurate when simulating in Section 4.3, i.e θ1 = θ 1 , θ2 = θ 2 . And we let α = 10 -3 , γ = 0.99.In this section, we will show that our four environments are all sequential social dilemmas defined in [11] : Table 3 : Classification of social dilemmas. In all dilemmas, mutual cooperation yields higher payoffs than mutual defection, yet each dilemma provides an incentive for defection. In the Snowdrift game, one player can gain a higher payoff by defecting when the other cooperates, while in the Stag Hunt game, a player can achieve higher payoffs by defecting when the other defects. The Prisoner's Dilemma encompasses both types of incentives.An N -player sequential social dilemma is a tuple (M, Π = Π c ⊔ Π d )Snowdrift SG T > 1 > S > 0 Stag Hunt SH 1 > T > 0 > S Prisoner's Dilemma PD T > 1 > 0 > S, 2 > T + S B.2 Environment details IPD.Each agent makes decisions based on the actions of the two players in the previous step, so this is a fully observable environment unlike other environments. We trained for 10k episodes, each with 100 steps.Coingame. Map size is 5 × 5. Agent's action space is A = {up, down, left, right}. The state is represented as a 5 × 5 × 5 binary tensor. The five channels are {blue agent, red agent, blue coin, red coin, mask}, where the first two channels encode the location of each agent, the last channel distinguishes between parts within and beyond the boundary, and the other two channels encode the location of the coin if any exist. If two agents walk on the coin at the same time, one of them is randomly selected to successfully pick it up. Cooperative agents will only try to collect coins with their own color, while self-interested agents tend to greetingly collect all coins. Each episode lasts for 100 steps.Cleanup. Map size is 8 × 8. A = {up, down, left, right, stay, clean, pick}, where the last two actions require the agent to be in the same position as the waste or the apple. The seven channels are {agent 1, ..., agent 4, waste, apple, mask}, The parameters with the same meaning as the open-source implementation about Cleanup [32] are shown in Tab. 4. To achieve cooperation, agents need to take the initiative to undertake part of the cleaning task to help improve the group's revenue. A defecting agent will just keep waiting for the apple to grow and gather it. Each episode lasts for 100 steps. SSH. Map size is 8 × 8. A = {up, down, left, right, stay, hunt hare, hunt stag}. Agents must be in the same position as the prey to hunt and the prey doesn't respawn after being hunted. The seven channels are {agent 1, ..., agent 4, hare, stag, mask}. Cooperative agents are happy to hunt deer with others, while defectors only hunt rabbits to avoid the risk of not getting a payoff. Each episode lasts for 30 steps.SSG. Map size is 8 × 8. A = {up, down, left, right, stay, remove snowdrift}. Agents must be in the same position as the snowdrift to remove it and the removed snowdrift doesn't respawn. The six channels are {agent 1, ..., agent 4, snowdrift, mask}. Cooperators will proactively remove snowdrifts to bring high rewards to the team, while defectors will just wait for others to remove them. Each episode lasts for 50 steps.The pseudocode for the LASE algorithm is shown in Algorithm 1.The actor-critic model interacting with environments utilizes two cascaded CNNs to process input data, with a kernel size of 3, stride of size 1 and 16 / 32 output channels. This is connected to one fully connected layer of size 128 activated by ReLU, and an LSTM with 128 cells. The actor head and critic head are two separate fully-connected layers that output the softmax normalized action policy and a scalar value respectively. The implementation of the intrinsic policy network and value network in OM is basically the same, but due to the need to further judge the actions of others, joint action space is added to the value network input dimension. In the observation transformation network, the input data is passed through a CNN with a kernel of size 3, stride of size 1 and 16 output channels, and is concatenated with a one-hot vector representing the agent index to input in two FC Algorithm 1 Learning to balance Altruism and Self-interest based on Empathy (LASE)Initialize action policy π i , SR policy network π i SR , SR value newtork Q i SR and observation conversion network parameterized by θ i , µ i , ϕ i , η i and trajectory buffer B i , for each agent i for eps = 1 to max_episodes do All agents interact with the environment for many steps, each agent i gets a trajectory τ i = (o i t , a t a t a t , r i,env t , o i t+1 ) and stores it in B i for each agent i do for each other agent j do get j's simulated observation ôj by i's observation conversion network estimate j's policy π i SR (a j ′ |ô j ) with SR policy network for j's all possible actionsa j ′ do compute i's Q-value Q i SR (o i t , (a -j t a -j t a -j t , a j ′ t )), with j taking all possible actions a j ′ t while the other agents' actions a a a -j fixed end for compute the counterfactual baselinea j ′ t π i SR (a j ′ t |ô j t )Q i SR (o i t , (a -j t a -j t a -j t , a j ′ t)) compute gifting weights w ij by Eq. 3 end for get w ii by w ii = 1 -N j=1,j̸ =i w ij end for get r tot ← r and update policy π i to maximize the accumulated r tot if eps mod update_f requency = 0 then for each agent i do sample a minibatch from B, update µ i , ϕ i , η i by Eq. 4, Eq. 5 end for end if end for layers of size 128. The output is reshaped to be the same size as the input observations. We use Adam optimizer [14] for all modules' training.IPD. All the CNNs in IPD are removed. The size of FC layer and the cell number in LSTM are scaled down to 32. To test LASE's scalability, we have extended Cleanup and Snowdrift as Tab. 6. We select the w ij data from the last 10 6 timesteps of training to calculate their mean value and standard deviation, which estimates the uncertainty of social relationships. The calculation method is as follows:w ij = Tmax-10 6 t=Tmax w ij t 10 6 , w = n i n j,j̸ =i w ij , s = n i n j,j̸ =i Tmax -1e6 t=Tmax (w ij t -w ij ) 2 1e6-1 n × (n -1)We conduct a comparative experiment to replace the input of SR policy network ôj with j's real observation o j . Here is the results:The results show that the mean value of LASE's inferred social relationships closely match actual observations when available, although partial observability significantly increases uncertainty. Considering that the social relationships between people in real life tend to be relatively stable and do not change drastically, we think that a possible solution to handle the uncertainty of social relationships is to introduce some smoothing techniques for w ij to reduce the variance of social relationships over time. This approach will be explored in our future work.Meanwhile, It is important to note that Figure 8 and corresponding analysis show that LASE is able to correctly infer the relationships with different co-players and respond properly. Specifically, in the experiments conducted in Section 6.3, one LASE interacts with three rule-based co-players: cooperator, defector and random. The results show the w ij given to cooperative co-player is the largest, significantly higher than that given to the other two co-players. The w ij given to random coplayer comes second, and the smallest is given to defector. These results demonstrate the consistency between LASE's estimates of the relations and the ground truth.As an evaluation metric, fairness should be evaluated alongside reward to measure algorithm performance effectively. Some algorithms may fail to address decision-making issues in mixed-motive games where each agent receives a small reward, but the reward disparity between agents is minimal, resulting in high fairness. Clearly, these methods are not effective. An effective method should both maximize group reward and ensure intra-group equity. Here, we include the fairness results of other baselines: The rapid advancement of AI technology has brought about an explosion in the number of agents, making it unfeasible to rely on a centralized controller to achieve coordinated collective behavior. The question of how to design independent agents that excel in specific tasks and can demonstrate adequate social behavior when interacting with humans or other agents remains an open challenge. We take a tentative step towards this problem by introducing the mechanism of gifting and the theory of empathy in human society. We believe our work will have a positive impact on the interaction of multi-agent systems, the alignment of AI with human values, and the development of AI safety.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 


Decision: All reviewers feel the paper makes a sound and useful contribution.  The authors effectively addressed the reviewers' concerns during the discussion phase.  I encourage the authors to implement what they mentioned in the discussion to improve the paper.

Reviews: review: paper_summary: paper_summary: This paper introduces LASE (Learning to balance Altruism and Self-interest based on Empathy), a multi-agent reinforcement learning algorithm designed for mixed-motive games. LASE uses a gifting mechanism where agents share a portion of their rewards with others based on inferred social relationships. Counterfactual reasoning determines these relationships by comparing the actual joint action's value to a baseline that averages over the other agents' actions. The authors use a perspective-taking module to predict other agents' policies. Experimental results across various sequential social dilemmas show LASE's ability to promote cooperation while maintaining individual interests. The authors claim the following contributions: (1) a computational model of empathy that modulates responses based on inferred social relationships; (2) a decentralized MARL algorithm that balances altruism and self-interest in mixed-motive games, and (3) theoretical analysis of decision dynamics in iterated matrix games and experimental verification of LASE's performance in sequential social dilemmas.
strength_weakness: strength_weakness: I think the paper is quite strong overall. However, several places can still be improved to increase the potential impact of the work.

* Clarity issues: The paper could benefit from improved clarity in several areas. For example, a bit more clarity on how gifts are determined would be helpful. Consider adding pseudocode for key components like the Social Relationships Inference module or the gifting mechanism and how the modules integrate. 

* Insight into counterfactual baseline design choice: The paper's choice of using the average behavior as the counterfactual baseline for determining social relationships warrants further investigation. The authors could consider conducting experiments that vary this baseline, comparing alternatives such as fixed neutral baselines, learned cooperative behavior baselines, or worst-case action baselines. This would provide insights into the robustness of the approach and potentially identify improvements to the algorithm.

* Limited domain coverage: While the paper evaluates LASE across several sequential social dilemmas, which is adequate, the approach could be further strengthened by expanding the range of domains tested. For example, the authors could consider alternatives from MeltingPot domains (https://github.com/google-deepmind/meltingpot).
questions: questions: Do you have any proposed changes to the paper in response to my critiques?
limitations: limitations: I think there could be more discussion of this, but the current evaluation is sufficient.
paper_summary: paper_summary: The authors propose a novel algorithm, LASE, that employs a gifting mechanism in order to steer agents toward equilibria of high social welfare in mixed motive games. A novelty in LASE is that it estimates the "social relationship" between a player and its co-players through a counterfactual Q-value baseline. The authors show that LASE performs favourably across a number of temporally-extended social dilemmas.
strength_weakness: strength_weakness: I believe the main weakness of the authors' approach is that, under partial observability, a player cannot generally see all parts of its co-players observations, hence making it impossible to fully reconstruct their policy inputs (hence the restriction to common knowledge fields of view in [2]). I believe the author's algorithm should crucially estimate the uncertainty of their social relationship estimates such as to avoid misunderstandings that could erode trust in real-world situations.  

Additionally, I am unsure about the authors' use of the term "empathy" - "theory of mind" would certainly work here, but "empathy" seems like an inherently emotional concept.

[2] Multi-Agent Common Knowledge Reinforcement Learning, Schroeder de Witt et al., NeurIPS 2019
questions: questions: What do you think could be a real-world application of this line of work?
limitations: limitations: I believe the authors are addressing limitations adequately.
paper_summary: paper_summary: This work proposes LASE, a multi-agent reinforcement learning framework that aims to improve co-operation between agents in mixed-motive games using transfer of rewards between agents in a zero-sum manner. Specifically, each agent uses counterfactual reasoning to compute a social relationship metric that computes the effect of their actions on the Q-values of other agents. Notably, LASE uses fully decentralized training, in contrast to many related works in the area. Finally, LASE outperforms existing baselines in a variety of popular mixed-motive environments such as Cleanup.
strength_weakness: strength_weakness: 1) The framework is relatively complex, as it requires learning two additional networks (perspective-taking and Q networks) for each agent.
2) Given the dependence of the method on joint action, I have doubts about the scalability of the method. As more agents are added into the system, credit assignment would become more difficult. 
3) Some experimental details need to be clarified further.
questions: questions: 1.	In general, given the complexity of the framework and the dependence on the joint action, I am sceptical about the scalability of the method. It would be interesting to see more results that test the scalability of the method. Convincing results with 8 agents for Cleanup and 1 other environment would significantly increase the strength of the paper. 
2.	The observation conversion network is not trained using the ground truth observations of agent j, but instead using the ground truth observations of agent i (this being key to the fully decentralized claim of the work). This makes me wonder why such a network is needed at all? What happens if a network that directly maps from agent i’s observation to agent j’s policy is learned. It would be interesting to see this in an ablation study. In other words, why is the "perspective taking" module not a single network?
3.	How is the SR policy network trained? I am confused why it is trained using RL and not simply using supervised learning using the actual observed actions of agent j.
4.	I am not convinced by the argument that removing the cleaning beam in Cleanup makes the environment harder. Removing the cleaning beam also reduces the dimensionality of the action space. What was the reason behind removing it?
5.	Figure 1 is slightly misleading as it shows separate parameters for the observation conversion network and SR policy network. However, equation 4 shows them having the same parameters. 
6.	I am unsure why the policy has a manual epsilon greedy added to it. Actor critic methods are on-policy, the epsilon greedy changes that and I am not sure this manual tuning for the policy is standard. Why was this required?
limitations: limitations: Sufficient details provided.
paper_summary: paper_summary: The paper introduces LASE, a novel distributed multi-agent reinforcement learning algorithm. LASE aims to foster altruistic cooperation through a gifting mechanism while avoiding exploitation in mixed-motive games. The algorithm dynamically adjusts the allocation of rewards based on social relationships inferred using counterfactual reasoning. The paper reports comprehensive experiments in various mixed-motive games, demonstrating that LASE effectively promotes group collaboration and adapts policies to different co-player types without compromising fairness.
strength_weakness: strength_weakness: The proposed method, while interesting, could be seen as incremental since it primarily combines existing techniques (gifting and counterfactual reasoning). The novelty could be better highlighted by contrasting more explicitly with prior works. 

The more important weakness is that the paper lacks a detailed analysis of the individual contributions of the gifting mechanism and counterfactual reasoning. For example, there is no direct comparison with the original gifting method or an ablation study isolating the impact of the counterfactual reasoning module. This makes it difficult to discern the specific roles and contributions of these components to the overall performance. 

Thus, I think the paper is below the acceptance bar for NeurIPS.
questions: questions: 1.How does LASE compare to the original gifting methods? Including a direct comparison or an ablation study focusing on the gifting mechanism could provide deeper insights into its contribution.
2.Can you provide an ablation study that isolates the impact of the counterfactual reasoning module? This would help in understanding its specific role and effectiveness.
3. Why not compare the Equality with other baseline methods? I am curious about the equality performance of baseline methods? from the report of their paper, I guess some of them will have not bad performance.
limitations: limitations: The paper seems lack of discussion about limitations and broader societal impacts.

Comments: title: Thanks for your feedback.
Thanks for your Response.
Two possible approaches to integrating LLM and LASE in negotiation tasks.
Thanks for Your Response, and One More Question.
Thanks for the reply
Thanks for the response
Main concerns are addressed.
Reply to rebuttal
comment: Thank you for your continued engagement and valuable feedback on our work! We are grateful for your time and effort in reviewing our paper, and we will carefully consider your questions and comments in the revised version and our future work.
I thank the authors for their response. I appreciate the authors' ideas concerning LLM agents, which I find interesting. I will leave my score unchanged for now.
Given that effective communication and complex strategies in automated negotiation agents often require fluent natural language, it is intuitive to leverage LLMs in this real-world application. Previous work has already formulated the negotiation task and developed datasets that support reinforcement learning, supervised learning, and other methods. [1] Building on this foundation, we propose integrating LLMs into our approach in two ways:

First, inspired by Cicero [2], which uses RL to train an intent model and generates messages based on that intent using a pre-trained language model, we think that we can prompt an LLM based on the predicted cooperation willingness of the opponent derived from LASE’s SRI module. Considering that our SRI module outputs a scalar value and that LLMs can often misinterpret the magnitude and meaning of numerical values, it may be necessary to establish a mapping system that correlates different cooperation willingness scores with corresponding natural language expressions. For example, if the SRI module predicts that the opponent has a high willingness to cooperate, the LLM, once prompted with this information, might negotiate more assertively, potentially increasing its bargaining position and maximizing profit.

Second, we can consider transferring the framework of our algorithm to the design of an LLM-based agent, aiming for superior performance in this task. For instance, we could replace the PT module with the LLM's world knowledge by prompting the LLM to infer the opponent's next action based on the current dialogue history. When the LLM observes the opponent's actual action, it could be prompted to engage in counterfactual reasoning, similar to the ReAct [3] framework, updating its belief about the opponent's cooperation willingness. An example template of such a prompt might be: "You initially believed the buyer would offer {}, but the actual offer was {}. Considering your previous belief about the buyer's cooperation willingness was {}, do you think you need to make adjustments? If so, how should it be made?" We hope this approach will help address the issue of LLMs lacking a broader understanding of the overall dialogue progression [4], thereby improving performance in negotiation tasks.

[1] Post, Thierry, et al. "Deal or no deal? decision making under risk in a large-payoff game show." American Economic Review 98.1 (2008): 38-71.

[2] Meta Fundamental AI Research Diplomacy Team (FAIR)†, et al. "Human-level play in the game of Diplomacy by combining language models with strategic reasoning." Science 378.6624 (2022): 1067-1074.

[3] Yao, Shunyu, et al. "React: Synergizing reasoning and acting in language models." arXiv preprint arXiv:2210.03629 (2022).

[4] Cheng, Yi, et al. "Cooper: Coordinating specialized agents towards a complex dialogue goal." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 16. 2024.
I thank the authors for their reply. I am satisfied with the authors' response to my concerns about relationship uncertainty and the usage of the term "empathy".

Concerning the real-world applicability, I, however, I would like to ask whether and how the authors believe that their work is relevant to LLM agents given the difficulty of performing RL directly with these.
Thank you for your reply! We greatly appreciate the time and effort you put into evaluating our work. Your feedback has provided us with meaningful insights, and we will make sure to address your comments and incorporate the necessary changes in our future revisions.
Thank you very much for your valuable feedback and for taking the time to review our submission! We truly appreciate your insights and suggestions, which have helped us identify areas for improvement. We will carefully consider your comments and add the experiments in our revised version!
Dear Authors,

Thanks for the additional experiments provided. The most of my concerns are addressed. And I suggest the authors to add the new experiments in the future version.  And I will increase my score.
I thank the authors for the rebuttal. My doubts have been clarified and I have raised my score to reflect the same.