Reciprocal Reward Influence Encourages Cooperation From Self-Interested Agents

Authors: john l zhou
weizhe  hong
jonathan c kao
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 02:08:50

Abstract: Cooperation between self-interested individuals is a widespread phenomenon in the natural world, but remains elusive in interactions between artificially intelligent agents. Instead, naïve reinforcement learning algorithms typically converge to Pareto-dominated outcomes in even the simplest of social dilemmas. An emerging literature on opponent shaping has demonstrated the ability to reach prosocial outcomes by influencing the learning of other agents. However, such methods differentiate through the learning step of other agents or optimize for meta-game dynamics, which rely on privileged access to opponents' learning algorithms or exponential sample complexity, respectively. To provide a learning rule-agnostic and sample-efficient alternative, we introduce Reciprocators, reinforcement learning agents which are intrinsically motivated to reciprocate the influence of opponents' actions on their returns. This approach seeks to modify other agents' $Q$-values by increasing their return following beneficial actions (with respect to the Reciprocator) and decreasing it after detrimental actions, guiding them towards mutually beneficial actions without directly differentiating through a model of their policy. We show that Reciprocators can be used to promote cooperation in temporally extended social dilemmas during simultaneous learning. Our code is available at https://github.com/johnlyzhou/reciprocator/.

Main paper: 
 introduction: 
 Many species exhibit cooperative behaviors of remarkable variety and complexity. Even among prosocial animals, however, humans are especially notable for their ability to cooperate with unrelated individuals and maintain that cooperation even in highly adversarial environments (Melis & Semmann, 2010) . These qualities are often credited with the development of human technology, culture, and advanced cognition (Burkart et al., 2014) . As artificially intelligent (AI) agents become more commonplace in human society, it is increasingly important to ensure that they share similar prosocial qualities so that interactions between AI agents, as well as between AI agents and humans, may converge to mutually beneficial outcomes.However, state-of-the-art reinforcement learning (RL) methods are typically designed for the single agent setting and are ill-suited for the nonstationarities introduced by multiple agents learning simultaneously. Treating other agents as fixed elements of the environment, referred to as the "naïve" learning (NL) approach, can destabilize training and produce collectively suboptimal outcomes. This is particularly evident in a class of games known as sequential social dilemmas (SSDs), which contain tradeoffs between collective and individual return (Leibo et al., 2017) . SSDs present a particularly challenging problem for multi-agent RL (MARL) because of this mixed motivational structure, which precludes the use of common centralized training algorithms designed for cooperative settings (Kraemer & Banerjee, 2016; Sunehag et al., 2017; Gupta et al., 2017; Rashid et al., 2018) .Previous work has shown that NL agents optimizing only for their individual returns converge to non-cooperative, Pareto-dominated outcomes in even the simplest of SSDs (Foerster et al., 2018) .In this work, we propose an intrinsic reward that encourages agents called Reciprocators to learn a tit-for-tat-like strategy which reciprocates the influence that others exert on their returns. We first define value influence, a notion of influence that quantifies the effect that one agent's action has on another's expected return. Given a pair of agents, a Reciprocator rc and another learning agent i, we track the cumulative influence of agent i's sequence of actions on rc's expected return, which we refer to as the influence balance "owed" to rc by agent i. At each time step, the Reciprocator receives an additional intrinsic reciprocal reward proportional to the product of its current influence balance with agent i and the value influence of its action on agent i. This encourages the Reciprocator to take actions whose influence matches the sign of the influence balance: for example, if agent i has produced a net positive influence on rc, then rc will be rewarded for actions that have a reciprocally positive influence on agent i's expected return. Mechanistically, our method seeks to manipulate the opponent's Q-values by altering their expected return in particular directions depending on their actions. A reward-maximizing agent i should then be incentivized to take mutually beneficial actions and avoid harmful externalities.The contributions of this work are as follows: (1) We formulate a novel intrinsic reward that encourages an agent to incentivize cooperation from other, simultaneously learning agents without modifying the structure of the environment, computing higher-order derivatives, or learning metagame dynamics. (2) Agents trained with this reward achieve state-of-the-art cooperative outcomes in sequential social dilemmas and are able to shape purely self-interested naïve learners into mutually beneficial behavior. (3) Reciprocators demonstrate resistance to exploitation by higher-order baselines, despite using only first-order reinforcement learning algorithms.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: The main claims made in the abstract and introduction do accurately reflect the paper's contributions and scope. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 In order to improve convergence towards Pareto-optimal solutions among independently learning agents, previous work has modified agents' reward structures to explicitly consider either group or percapita return in order to promote cooperative behavior, e.g., via inequity aversion (Hughes et al., 2018) , "empathic" harm reduction (Bussmann et al., 2019) , or "altruistic" gradient adjustments (Li et al., 2024) . While agents that abide by such restrictions may seek to reduce their harmful externalities, they have no way of regulating the behavior of other, less magnanimous agents, including purely self-interested "exploiters" in the worst case (Agapiou et al., 2023) . Their efficacy in SSDs, as well as in most other types of multi-agent systems, is therefore limited in the absence of strong guarantees over the other agents' altruistic tendencies.A more robust form of cooperation can be achieved by actively exerting influence over other agents in the environment rather than unilaterally adopting a prosocial policy, especially in SSDs where extrinsic rewards incentivize defection. Jaques et al. (2019) provides an intrinsic reward for "social influence" on other agents, which is computed as the Kullback-Leibler (KL) divergence between an opponent's policy distribution conditioned on an influence agent's action a i and a counterfactual distribution which marginalizes out that influence agent. However, as a metric-agnostic quantity, KL divergence is unable to capture key details of how the action distribution changes (Park et al., 2024) . In particular, this definition of influence cannot selectively modify opponents' behavior with respect to extrinsic task returns or state transition dynamics, and therefore has limited utility as a mechanism to incentivize cooperation.Other works seek to directly influence opponent returns via the social mechanisms of reward and punishment, which are believed to assist in stabilizing cooperative relationships by controlling freeriding, cheating, and other antisocial behaviors (Melis & Semmann, 2010) . Incorporating these mechanisms into groups of agents, whether by providing rewards to incentivize good behavior (Yang et al., 2020) , or doling out punishments to discourage bad behavior (Schmid et al., 2021; Yaman et al., 2023) , has been shown to encourage cooperation among independent agents in a variety of SSDs. However, these methods require extensions of the original action space that allow agents to directly modify other agents' rewards. On the other hand, we instead propose to quantify the influence of each naturally available action a ∈ A in a given state s on another agent's expected return R, and use this influence as the medium for reciprocation. As we later show, selectively rewarding or punishing an opponent's actions can be seen as a form of opponent shaping which seeks to manipulate the Q-value of given state-action pair in order to influence the likelihood of that action in future policies.Considering future policies points to a key issue with the canonical reinforcement learning (RL) framework, in that it optimizes only for the expected return within a single episode of the environment. However, taking actions that seek to optimize the long-term behavior of the other agents in the environment will not receive any immediate feedback within an episode, and must wait for one, or possibly many, learning steps. Opponent-shaping methods of this kind address this issue by differentiating across pairs of episodes (Yang et al., 2020) , through opponents' gradient updates (Foerster et al., 2018; Zhao et al., 2022; Willi et al., 2022) , or repeated sequences of learning steps organized into "meta-episodes" (Lu et al., 2022) . While these methods have demonstrated convergence to cooperative behavior in simple SSDs, they are impractical or intractable in realistic multi-agent scenarios, requiring additional independent action channels for providing incentives, white-box access to the learning rules and gradients of other agents, or exponential sample complexity (Fung et al., 2023) to learn the dynamics of meta-games, respectively.In particular, the Learning with Opponent-Learning Awareness (Foerster et al., 2018, LOLA) class of approaches differentiate through the opponent's learning update but have only been demonstrated to work with full access to either the opponent's policy gradients and Hessians or their policy parameters. When using gradient approximations and modeling the opponent's policy, LOLA with opponent modeling (LOLA-OM) showed significantly worse results, even against vanilla policy gradients. Modeling efforts become even more implausible when faced with modern RL techniques, which employ adaptive optimizers that set different per-weight learning rates (Kingma & Ba, 2017, Adam) , randomized experience replay (Schaul et al., 2016) , and various auxiliary terms such as policy divergence penalties (Schulman et al., 2017) and entropy exploration bonuses (Williams, 1992; Mnih et al., 2016) to improve learning. Model-Free Opponent Shaping (Lu et al., 2022, MFOS) was introduced as a more general meta-learning approach that requires neither privileged access to nor inconsistent assumptions on opponent learning rules. However, MFOS's meta-policies are trained across multiple training runs repeated in sequence, each of which are treated as a single "meta-episode," in order to learn how to exploit opponent learning dynamics. This requires the ability to freely perform rollouts in the environment against opponents whose policies can be repeatedly reset to initialization, an unrealistic assumption in environments with partially adversarial motivations.Our work is conceptually most similar to Learning with Opponent Q-Learning Awareness (Aghajohari et al., 2024, LOQA) , which also seeks to shape opponent policies by influencing the Q-values for different actions, under the assumption that opponents are Q-learners. However, LOQA still differentiates through a model of the opponent's policy and optimizes according to the joint advantage function A(s t , a 1 , a 2 ) computed with respect to the state-value function V (s t ). On the other hand, we use a counterfactual baseline that marginalizes out only the opponent's action to perform agentspecific credit assignment and use an intrinsic reward instead of gradients over opponent policies to encourage agents to influence the opponent's Q-values in the correct direction. Most importantly, LOQA focuses on the problem of learning a general end-policy that performs well against a variety of other agents at evaluation and has therefore only demonstrated the ability to shape other LOQA opponents in a controlled self-play scenario, while our method focuses on shaping the policies of diverse agents over multiple episodes of simultaneous learning.In the context of these higher-order opponent-shaping approaches, we position our intrinsic reciprocal reward as a form of immediate, within-episode feedback that encourages otherwise naïve learners to implicitly consider the long-term, cross-episode effects of their actions on the policy changes of other agents, demonstrating many of the properties of higher-order opponent-shaping methods while using only first-order reinforcement learning algorithms and standard rollout-based training procedures.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We include a Limitations section in the main text. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
preliminaries: 
 We construct a series of sequential social dilemmas which can each be described as a stochastic game G, defined by a tuple G = ⟨S, A, P, r, n, γ⟩. In G, n agents, indexed by i ∈ {1, . . . , n}, observe the state of the environment s ∈ S and simultaneously choose actions a i ∈ A to form a joint action a ∈ A ≡ A n . The environment then undergoes a change according to the state transition function P (s ′ |s, a) : S × A × S → [0, 1]. Each agent receives an individual reward r i = r i (s, a), and future rewards are discounted at each time step by a discount factor γ ∈ [0, 1]. We consider the fully observable setting where agents have access to the full state of the environment s ∈ S at every time step, joint action a ∈ A, and rewards received by each agent. Each agent i conditions a stochastic recurrent policy π i (a i |τ i ) on its action-observation history, which is denoted asτ i ∈ T ≡ (S × A) * . 4 Reciprocal Reward Influence 4.1 1-Step Value InfluenceIn order to compute a general measure of value-based influence, we modify an intrinsic exploration reward proposed by Wang et al. (2020) called Value of Interaction (VoI). VoI measures how much agent i's actions affect agent j's expected return by computing the difference between the Q-value conditioned on the joint state and action Q π j (s, a) and a counterfactual baseline Q π j|i s, a -i which marginalizes out the state and action of the influencing agent i to compute a "default" expected return [Equation 1]. Here, the bolded π superscript indicates the dependence of these Q-functions on the parameters of all agents' policies. The VoI can be decomposed into an immediate influence term r(s, a) -r(s, a -i ) and a discounted future influence term computed over changes in state transition probabilities [Equation 2]. Although the original VoI was formulated as an expectation over trajectories τ and used as a regularizer, we modify it to compute the one-step Value Influence (V I) where V oI = E τ [V I], allowing us to quantify the influence of individual actionsEQUATIONwhere r(s, a -i ) = E a i ∼π i r(s, (a -i , a i )) and p(s ′ |s, a -i ) = E a i ∼π i p π i (s ′ |s, (a -i , a i )) . This definition of influence relies on the notion of a counterfactual baseline to assign credit to a particular agent's action while holding all other agents' actions constant. This counterfactual baseline is so named because it estimates the counterfactual expected return if the agent's true action is replaced with a "default" action, which is computed by marginalizing out the agent's action to get the expected on-policy return. This baseline return is given byEQUATIONIn practice, we approximate this by regressing towards the Q-value while masking out a i from the joint state-action input. Marginalizing out an agent's action to quantify influence is a common paradigm in MARL, having also been used to define 1-step adversarial power (Li & Dennis, 2023) , which estimates the maximum reduction in agent j's expected reward that can be achieved by agent i as the minimum V I π i|j (s, a) over all a i ∈ A. Similarly, Counterfactual Multi-Agent policy gradients (Foerster et al., 2017, COMA) marginalizes out a single agent's action to assess the advantage (i.e., influence) of that agent's selected action relative to the counterfactual on-policy return ceteris paribus. 4] can be expressed a special case of V I π i|j (s, a) where i = j.EQUATIONQuestion: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
keeping score with influence balances: 
 The amount of influence that a Reciprocator is able to exert in a single time step is heavily environmentdependent, so that it may not always be possible to immediately reciprocate previous influences. To encourage reciprocation over extended timescales, we continuously accumulate a measure of net influence over sequences of actions. We draw inspiration from the "debit" tally used by approximate Markov tit-for-tat (Lerer & Peysakhovich, 2018, amTFT) , which tracks the cumulative advantage gained by an opponent compared to a fully cooperative baseline policy known a priori.Extending this idea to our framework, we sum agent i's influence V I π i|rc on the Reciprocator's expected return at each timestep rather than its own. However, using influence in only one direction as motivation for reciprocation can lead to continuous punishment or rewarding without a way to settle the score. To mitigate this, we also accumulate the net influence V I π rc|i in the opposite direction and subtract it from the influence balance at every timestep as a way to "pay off" the balance, limiting the degree to which reciprocation is rewarded. Formally, we define the influence balance B rc|i (t) maintained by a Reciprocator rc with another agent i at time t asEQUATIONThe influence balance can be thought of as a score of net influence over time between agents, and can be used to motivate reciprocation in the correct direction, i.e., either positive reinforcement of net positive influence or positive punishment of net negative influence.If agent i takes a series of actions that improves the expected return of the Reciprocator over a baseline estimate, i.e., produces a net positive influence balance, then the Reciprocator should be motivated to reinforce this behavior by exerting a reciprocal positive influence on agent i's expected return R i , in order to encourage a higher likelihood of that behavior during policy updates. Similar logic applies for detrimental deviations, negative influence balance, and subsequent reciprocal punishment. We then define the intrinsic reciprocal reward r R rc|i (t) received by rc asEQUATIONTaking the product of existing influence balance and current action's VI encourages the Reciprocator to take actions that reinforce agent i's behavior in the correct direction by matching signs, and scales the reward by the magnitude of the outstanding influence balance and the reinforcing influence. The intrinsic reward is then added to the agent's extrinsic reward to form the total reward used in training.We use experience replay to periodically update target networks and iteratively update our counterfactual baseline estimates towards these target values (Mnih et al., 2015) . This provides two key benefits: first, periodically updating these policy-dependent functions stabilizes training and allows us to approximately ignore their gradients, and therefore the gradient of the intrinsic reciprocal reward, with respect to the agents' policy parameters (Wang et al., 2020) . With this assumption, we are able to use standard policy gradient methods to train our agents to jointly optimize the combined extrinsic and intrinsic rewards.Second, drawing samples from multiple previous episodes to the train the counterfactual baseline target functions makes the Reciprocator less susceptible to exploitation. If the counterfactual estimators were updated concurrently with the policy using only the most recent on-policy data, then the Reciprocator's baseline would be immediately adjusted to its opponent's new policy after each episode. Because assessment of influence hinges on these counterfactual baselines, updating them too frequently would allow adversaries to easily manipulate these estimates of on-policy returns. 
experiments: 
 We conduct experiments using two commonly used SSDs of varied complexity to demonstrate the shaping abilities of Reciprocators against other types of learning agents. For IPD, we consider memory-1 iterated games as in Foerster et al. (2018) and Lu et al. (2022) , following the proof from Press & Dyson (2012) that longer-memory strategies provide no advantage over strategies conditioned on shorter memories. We use two methods to evaluate head-to-head performance in IPD: allowing agents to directly differentiate through the analytic, closed-form solution of the game as originally derived in Foerster et al. (2018) , and more standard batched policy rollouts for a fixed episode length. We append "analytic" and "rollout" to the game names to denote the evaluation method used. For Coins, we augment the observation given to the critics and value influence estimators with the time remaining in the episode to prevent state aliasing and stabilize learning from experience replay (Pardo et al., 2022) , but do not provide them as input to the policy networks.Iterated Prisoners' Dilemma (IPD): The iterated prisoners' dilemma (IPD) is a temporally extended version of the classical thought experiment, in which two prisoners are given a choice to either stay silent/cooperate (C) or confess/defect (D) with rewards given in Table 1a . The Pareto-optimal strategy is for both agents to cooperate by maintaining their silence, but the only Nash equilibrium (in the non-iterated, single-shot case) is mutual defection.Coins: Coins is a temporally extended variant of the IPD introduced by Lerer & Peysakhovich (2018) . In this game, two coins spawn randomly in a fixed-size grid, with each coin corresponding to one of the two players (designated by color matching). Each player moves around the grid and receives a reward of 1 for collecting any coin, and a punishment of -1 if the other agent collects their coin. If agents collect coins indiscriminately where P (collect own coin) ≈ 1/n, the net expected reward is 0. We show example rewards for various scenarios in Figure 1b . We select the following baselines because they focus specifically on the problem of shaping other agents' policies during simultaneous learning and without modifications to the environment. In Coins, we exclude Meta-MAPG and LOLA-DICE as baselines following work by Yu et al. (2022) showing that neither method is able to achieve significant results, even with a simplified shared reward.Naïve Learner (NL): As previously defined, NLs optimize their expected return with respect only to their own policy parameters θ. In this work, we implement NLs using policy gradient methods (Sutton et al., 1999) , which perform updates of the form θ t+1 = θ t + α∇ θ J(π θ )| θt , where α is the learning rate and ∇ θ J(π θ )| θt is the gradient of the objective function with respect to the policy parameters θ t at step t.Learning with Opponent-Learning Awareness (LOLA): LOLA uses either whitebox access to an opponent's gradients and Hessians or an explicit model of their policy parameters, assuming they are NLs, and differentiates through their learning step using the update ruleθ i t+1 = θ i t + α i ∇ θ i t J i θ i t , θ -i t + ∆θ -i t ∆θ -i t = α -i ∇ θ i t J -i θ i t , θ -i t .Multiagent Model-Agnostic Meta-Learning (M-MAML): M-MAML (Lu et al., 2022) learns initial parameters and then meta-learns over both its own and its opponent's policy updates, conceptually similar to Meta-Multiagent Policy Gradient (Kim et al., 2021, Meta-MAPG) but modified to differentiate directly through the analytic form of the return in matrix games. Model-Free Opponent Shaping (MFOS): As briefly discussed in the introduction, MFOS metalearns over multiple episodes of policy updates in order to accomplish long-horizon opponent shaping. In Lu et al. (2022) , MFOS is implemented with inner and outer policies, where the outer policy either directly outputs an inner policy to play in each episode or a conditioning vector which is element-wise multiplied with an inner policy vector, as done in IPD-Analytic and Coins, respectively.In IPD-Analytic, we differentiate directly through the analytic solution to the matrix game. For rolloutbased experiments, we implement all policy gradient-based agents using actor-critic architectures trained with proximal policy optimization using a clipped surrogate objective (Schulman et al., 2017, PPO-Clip) . Results including naïve learners trained with this algorithm are denoted by NL-PPO. Target networks to estimate the Q-values in Equation 1 are updated every k episodes using uniformly sampled experience from a replay buffer (Lin, 1992) . Additional hyperparameter values and network architecture details can be found in Appendix A. 
results: 
 We evaluate the Reciprocators' performance against a variety of baselines in a round-robin tournament for IPD-Analytic. In Coins, we assess opponent-shaping performance against NLs and against another agent of the same type, which we refer to as symmetric Coins. Shaded regions indicate standard error of the mean (SE) over eight random seeds. We do not perform full round-robin experiments in non-analytic environments for the following reasons: the Reciprocators' complex and stochastic learning rule renders LOLA's assumptions invalid and the extended time to convergence due to reciprocal behavior makes collecting meta-training episodes for MFOS computationally prohibitive.IPD-Analytic: In the analytic form of IPD, we show that Reciprocators are able to reach cooperative equilibria with all other baselines, resisting exploitation by higher-order methods such as LOLA and MFOS despite being only a first-order method using vanilla gradient descent [Table 1 ]. Although the Reciprocator is extorted by MFOS to small extent (-1.06 vs. -0.98, respectively), we emphasize that MFOS relies on extensive liberties such as the ability to observe thousands of parallel training runs against their opponents and roll out pre-trained meta-policies against newly initialized agents.IPD-Rollout: Stochastic policy rollouts allow the Reciprocator to influence opponent returns differently for different action sequences and provide a stronger learning signal, motivating additional experiments using sampled rollouts. In this setting, we show that Reciprocator is able to consistently shape an NL over the course of a single learning trajectory with limited rollout samples and observe interesting oscillations in the Reciprocator's rate of cooperation. Figure 2 suggests that these oscillations are driven by the opposing intrinsic and extrinsic rewards, where the derivative of the rate of cooperation corresponds to the reciprocal reward value.We provide the following explanation for this phenomenon: although the Reciprocator initially learns a cooperative tit-for-tat strategy guided by a strong reciprocal reward, as the NL's policy becomes deterministic and a i becomes predictable, the V I π i|rc component of the influence balance decreases to 0. As the extrinsic reward begins to dominate, the Reciprocator essentially reverts into an NL, leading to exploitative defection. However, exploitation of the NL agent i causes the V I π rc|i term, and subsequently the reciprocal reward, to become negative. Combined with an increase in the NL's frequency of defection, which leads to an increase in V I π i|rc , the intrinsic reward produces a reversal back to cooperative behavior that is reinforced by positive reciprocal rewards, and the cycle repeats.In this temporally extended social dilemma, we see that the Reciprocator is able to shape NL-PPO into picking up more of its own coins at rates significantly higher than MFOS. This allows both agents to achieve a positive reward, while needing only a fraction of the samples to converge to mutually beneficial behavior (each meta-episode consists of 16 sequential episodes) [Figure 3 ]. We show that this change is driven by changes in coin preference rather than in total collection, with both agents collecting at near-optimal pace. This is in contrast to MFOS, which does not shape the NL towards cooperation, but rather uses its pretraining advantage to suppress opponent learning altogether (Khan et al., 2023) When two Reciprocators are pitted against each other, we see that they quickly learn a cooperative strategy of collecting their own coins [Figure 4 ], resulting in an average reward of ∼ 8 per 32 steps without needing self-play. This significantly outperforms MFOS and the reported performance of LOLA with opponent modeling (LOLA-OM), which achieves an average reward of only ∼ 2 per 32 steps according to Foerster et al. (2018) . One concern is that the intrinsic reciprocal reward dominates the extrinsic reward, effectively changing the motivational structure of these SSDs into a cooperative game. To address this, we recorded the cumulative reciprocal reward received per episode [Appendix B.1] to show that it quickly decreases to a negligible fraction of the total reward, similar to the scale of the reciprocal reward in IPD in Figure 2 .Together, these results demonstrate that Reciprocators are able to robustly shape the behavior of other agents towards prosocial equilibria during simultaneous learning, achieving state-of-the-art results with fewer assumptions and limitations than existing methods. Apart from opponent-shaping properties, we also show that the intrinsic reciprocal reward discourages Reciprocators from exploiting others, showing promise for the development of a more cooperative multi-agent learning framework. 
limitations and future directions: 
 Our implementation of Reciprocators using naïve RL algorithms remains limited in that it seeks to maximize its compound return within single episodes rather than across multiple episodes. Using handpicked weights to balance intrinsic and extrinsic rewards opens the door for a suboptimal tradeoff between long-term opponent shaping and short-term return maximization. This is partially mitigated by the counterfactual target baseline updates, which allow the Reciprocator to lower the magnitude of the reciprocal reward in response to opponent policies that remain stationary over multiple episodes [Figure 2 ]. Future work will focus on methods to evaluate reciprocation efficacy across multiple episodes and dynamically tune the balance between reciprocal and extrinsic rewards.In terms of evaluation, Khan et al. (2023) found that Coins does not require history to enable opponent shaping, since the current state is often indicative of past actions. Due to computational limitations, we were unable to assess our method's performance on the suite of Spatio-Temporal Representations of Matrix Games (STORM) designed to test shaping over longer time horizons. We do note that Reciprocators capture both types of memory necessary to achieve shaping as identified by Khan et al. (2023) : the counterfactual baselines trained on replay buffers serve as a way to capture inter-episode context, and the recurrent policies and influence balance capture intra-episode history. Therefore, our method can in theory generalize to STORM environments, although we leave this to future work. 
conclusion and broader impacts: 
 Emerging interest in cooperative AI has been led by approaches that endow agents with higherorder shaping capabilities. Although these agents exhibit cooperative behaviors when pitted against equivalent opponents, they readily manipulate and exploit agents with simpler learning rules or lower computational capabilities. This is a fundamentally undesirable outcome in partially adversarial interactions between learning agents, with the potential to exacerbate existing computational resource gaps beyond out-of-the-box pretrained performance. We presented Reciprocators, agents which seek to influence the behavior of other, simultaneously learning agents towards mutually beneficial outcomes. We showed that Reciprocators can both learn prosocial behaviors and induce them from other agents in a variety of sequential social dilemmas, while remaining resistant to exploitation by higher-order agents. To the best of our knowledge, Reciprocators represent the first class of reinforcement learning algorithms to achieve cooperation during simultaneous learning between two independent agents without needing meta-learning methods, knowledge of other agents' learning algorithms, or pretraining routines such as self-play or tracing procedures to control opponent selection. We believe that these results show a promising avenue forward for inducing cooperative outcomes from a diverse array of learning opponents.Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: Compute resources are described in the Appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] Justification: The research conducted in the paper conforms with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: Broader impacts of this work are discussed in the Conclusion and Broader Impacts section.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: These models are evaluated on simple game-theoretic environments and do not pose a high risk for misuse.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We use previously released code from the MFOS paper and an adapted opensource implementation of PPO, both of which are credited in the relevant code files and in the Appendix.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: New assets, i.e., code, is well-documented via comments and docstrings and is linked in the abstract. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: All experiments were run on Nvidia 3070 GPUs with 8 GB of VRAM. Results were averaged over eight random seeds, with a batch size of 8192 for IPD-Analytic and 2048 for IPD-Rollout and Coins. For all experiments involving MFOS agents, we adapted the original code from Lu et al. (2022) , leaving all architectural choices and hyperparameters the same.The architecture for both the actor and the critic consists of a state encoder (two convolutional layers followed by a linear layer, with ReLU activations between layers), followed by hidden linear layers as detailed in the table below and a final output linear layer. The actor has a softmax activation to output a policy over the action space, whereas the critic simply outputs a scalar value estimate. With the exception of a linear layer size of 32 (instead of 16) for Coins, the network architecture and parameters to estimate the various target functions to compute the V I are identical to those described for the PPO components in Table 2 . For IPD-Analytic, target estimates for opponent policies in IPD were computed as the frequencies of observed choices for each of the five possible states (start, CC, CD, DC, DD).For Coins, we implement the decomposition of V I given in Equation 2, separately predicting the immediate reward and the transition probabilities for each possible next state. We collect one batch of episodes at the beginning of each experiment to initialize the influence estimators. 38th Conference on Neural Information Processing Systems (NeurIPS 2024). 


Decision: This paper proposes a new co--player shaping method that is meant to be better for cooperatively resolving sequential social dilemmas. Most reviewers were happy with the experiments and baselines, as well as the connection to broader themes in the literature concerned with these questions.

Reviewer SrMN disagreed with the others and argued that the baselines were not appropriate. But the other reviewers disagreed with them. I looked through the paper myself also and I concur with the other reviewers. I do think MFOS is an appropriate baseline here, and I notice that the results achieved for it here look very closely comparable to those reported in the MFOS paper itself. So this looks fine to me.

I note also that the authors promised to include the discussion which emerged in the conversation around the reviews here of the relationship between this work and Jaques et al. (2019) "Social Influence...". Like the reviewer who asked about it, I was also wondering about the relationship between the two papers, and I thought your explanation was interesting. It's a very good idea to include it in the related work for the camera ready version.

Reviews: review: paper_summary: paper_summary: The authors introduce Reciprocators, RL agents that are intrinsically motivated to reciprocate the influence of other player's actions on the agent's returns. They show that this promotes cooperation in social dilemmas.
strength_weakness: strength_weakness: This paper claims that it does not modify the reward structure of the environment. However, adding an intrinsic reward to cooperate seems like it very much does modify the reward of the agents. It's not particularly interesting that, when given reward for promoting cooperation, agents perform behaviors that promote cooperation. What makes prior work on opponent shaping interesting is that when given a *purely self-interested objective* (ie anticipating the opponent's learning updates), cooperation emerges naturally.

In particular, the highlighting of results in Table 1 is misleading: The goal of prior works in opponent shaping is not to simply promote cooperation, but to promote *rational* cooperation (ie without being exploitable). For example, in Table 1, if I were self-interested and were told to pick an algorithm, I should always pick MFOS since it dominates all other choices. I do believe that this difference in intention between this work and the prior literature / baselines should be highlighted, as direct comparisons do not seem appropriate.

Misc: The paper references the exponential sample complexity of M-FOS a few times. It might be good to reference the relevant paper [1] discussing that. 

Fung, K., Zhang, Q., Lu, C., Wan, J., Willi, T., & Foerster, J. Analysing the Sample Complexity of Opponent Shaping.
questions: questions: 1. Have you considered trying your method on other SSD's beyond Coin Game? [1] discusses some of the downsides of using the coin game and introduces some new environments for doing so, available in [2] and [3]. I do not expect experiments to complete in time for the rebuttal, but it could be interesting for a camera-ready copy or future work.

2. For Line 212: doesn't augmenting coin game with the time remaining de-construct the social dilemma? At the very last time step, one should always defect. By induction, this then applies all the way back to the first time step. State aliasing prevents this from happening.

[1] Khan, Akbir, et al. "Scaling Opponent Shaping to High Dimensional Games."

[2] https://github.com/ucl-dark/pax

[3] Rutherford, Alexander, et al. "JaxMARL: Multi-Agent RL Environments and Algorithms in JAX." Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems. 2024.
limitations: limitations: The authors have addressed the limitations of their work.
paper_summary: paper_summary: The paper discusses an analysis of the effects of a combination of methods (1-step influence, debt mechanism and intrinsic reciprocal reward) on the emergence of cooperation in a society of artificial agents. The authors evaluate their approach for the case of n=2 using an IPD and the Coin game.
strength_weakness: strength_weakness: - The contribution of each mechanism (1-step influence, debt mechanism, and intrinsic reciprocal reward) is not discussed and evaluated in the paper. In fact, the author essentially combines them without clearly showing evidence that all three are actually needed. There is also a key question about the actual improvement that is possible to obtain by introducing them (and with which parameters). The mechanisms themselves are not new. Value influence comes from (Wang et al., 2020) and the idea of debt comes from (Lerer & Peysakhovich, 2018). That is perfectly fine, but I think it is important to understand if and how they are needed for observing the emergence of cooperation.

- In general, the reviewer would have welcome some discussions (in terms of intuitions) about the design choices that are at the basis of this paper. In fact, it is not completely clear *why* the proposed mechanisms should work (and also *how* in a sense).

- The mechanism presented by the authors appears to be somewhat related to the idea of social learning presented in [1]. However, this work is not discussed by the authors, even though it seems closely related. I believe a discussion about the relationship between that class of approaches and the one presented in this paper would be beneficial.

- It would be interesting to see a comparison with other mechanisms that foster the emergence of cooperation, particularly in the evaluation of the proposed method. Instead, the authors present comparisons with algorithms (like LOLA), which seem unrelated to the problem at hand. In fact, we know beforehand that these mechanisms do not lead to the emergence of cooperation.

- The theory presented in the paper essentially refer to a multi-agent scenario. However, the authors present their results only for the case of $n=2$. This case is not really informative (also comparing with existing works in the literature cited by the authors). The emergence of cooperation with $n=2$ is a phenomenon that has characteristics quite different compared to the case of $n>2$ in my opinion.
	

References	

[1] Jaques N, Lazaridou A, Hughes E, Gulcehre C, Ortega P, Strouse DJ, Leibo JZ, De Freitas N. Social influence as intrinsic motivation for multi-agent deep reinforcement learning. In International Conference on Machine Learning 2019 May 24 (pp. 3040-3049). PMLR.
questions: questions: - Do you have any results in terms of “ablation” considering the presence or not of the various mechanisms presented in the paper?

- How does your work relate to the idea of social learning and the use of sort of social influence as intrinsic reward?

- How does your work compare with Hughes et al. 2017? In fact it seems to me that the underlying mechanism might be similar to inequality aversion in a sense.

- In Equation (1), the reviewer is not sure if there should be a sum over $a_i$: that appears to be an error. Could you please clarify the mathematical formulation?

- Also in Equation (2), the sum over $a_i$ does not appear to be correct. Could you please discuss this formula as well?

- In Section 4.4. you listed two “assumptions”. The reviewer wonders if it would have been preferable to validate them. Do you have results supporting these assumptions?

- Why did you select LOLA, M-MAML, etc. as baselines? They do not appear to be relevant in a study about the emergence of cooperation. In fact, it seems to me that we know a priori that they do not lead to cooperation. It would have been more helpful to show comparison with other papers (like those that are cited in the related work) that actually lead to the emergence of cooperation.

- Do you have results for the case of n>2? In fact, the theory is presented for n>2, but the evaluation is carried out for the case of n=2, which is very limiting and not sufficiently informative in my opinion.
limitations: limitations: There is not a real discussion of the limitations of the work in terms of generalization to the case of the proposed approach to the case of $n>2$ (in terms of evaluation).
paper_summary: paper_summary: The authors develop a new intrinsic reward that tracks the balance of influence compared to a counterfactual baseline and provides positive/negative rewards to incentivize naive learners to cooperate. They test this algorithm in an iterated prisoner dilemma and a simplified coin game, showing positive results compared to some strong baselines on opponent shaping.
strength_weakness: strength_weakness: - There are some missing implementation details (number of seeds, averaging methods). 
- I’d have liked to see more extensive evaluation on the grid environments where the RL results are more important. 
- The evaluation could be more robust and compare in more sophisticated games. Even the coin game is highly simplified compared to the original versions. 
- While the algorithm is novel in the RL literature this is highly reminiscent of the following work on commutative recipcority which develops analytical results for a similar approach. Ideally the methods here could be compared to the algorithm and analyses proposed in this work: Li, J., Zhao, X., Li, B., Rossetti, C. S., Hilbe, C., & Xia, H. (2022). Evolution of cooperation through cumulative reciprocity. Nature Computational Science, 2(10), 677-686.
questions: questions: - While MarkovTFT is discussed — how is this approach significantly different? Given that the algorithm learns a TFT like algorithm — when would this approach be preferred and how does it advance the literature? The authors should also discuss a more probabilistic approach to TFT published in this work: Kleiman-Weiner, M., Ho, M. K., Austerweil, J. L., Littman, M. L., & Tenenbaum, J. B. (2016). Coordinate to cooperate or compete: abstract goals and joint intentions in social interaction. In CogSci.
- The opponent shaping work has the potential to shape towards a wide variety of goals and aims — this work seems more focused on just cooperation. Is that right? If so — this should be discussed in more detail as a limitation. 
- Unless the algorithm correctly models what it could be doing counterfactually — it won’t know how to accurately update the balance of influence. Will this always be learned rapidly? Could an adversary bias its learning away from forming an accurate model?
limitations: limitations: Yes
paper_summary: paper_summary: The paper proposes a method for including the influence of other agent’s into an agent’s own reward function, which is shown to promote cooperation in SSDs like IPD and the temporal version Coins. It also highlights that the method only requires first order RL algorithms, does not require access to privileged information, and does not require meta-learning.
strength_weakness: strength_weakness: - Not all results are shown clearly for IPD-Rollout or Coins. I suggest a result matrix similar to IPD-Analytics or a clear explanation as to why it is not included, even if only in the Appendix. 
- Nit: Add titles to Section 5 Figure 2 and 3.
- Nit: Section 5.2 Figure 2 description: percentage -> probability.
questions: questions: - LOLA vs. MFOS in Section 5.2 Table 1 does not match the original MFOS paper results. In MFOS Section 6.1, Table 4, LOLA vs. MFOS resulting in -2.09 for LOLA, while your section claims -1.02. Your result implies that LOLA has learned to cooperate with MFOS, when the MFOS paper shows that MFOS should learn a dominating strategy against LOLA. Where is the discrepancy coming from here? 
- Section 6 Figure 4 — Does each agent in this symmetric head-to-head receive the same reward and have the same probability of own coins? 
- Is there an experiment for Reciprocator vs. MFOS in Coin Game? 
- L 469: It’s unclear why the mean intrinsic reward begins as a negative number and converges to 0. I would have expected the intrinsic reward to decrease from a positive amount down to 0.
- L 293: Is it possible that implementation details created a difference? 
- Will the codebase be made available upon acceptance? Concrete examples for your paper’s method will help future researchers implement baselines in accordance with your paper.
limitations: limitations: Yes.

Comments: title: References
comment: We thank the reviewer for their consideration of our work and will be sure to include the clarifications in our future manuscript!
We appreciate the reviewer's engagement in this process and address both our choice of baselines and our choice of verbiage for "emergent cooperation."

Regarding our choice of baselines:

- Because we position our work as an opponent shaping method with specific advantages over existing work (namely, improved sample efficiency, reliance only on first-order derivatives, and resistance to exploitation), we maintain that the baselines used in this paper such as LOLA and MFOS are appropriate. We are not aware of other shaping/cooperation-related baselines in the literature that are equally appropriate, keeping in mind that our method demonstrates cooperation in a setting of **simultaneous** learning in an **unmodified** environment. 

  -   Agents with modified prosocial objectives such as Empathic DQN (Bussmann et al., 2019), inequity aversion (Hughes et al., 2018), Altruistic Gradient Adjustment (Li et al., 2024) cannot induce cooperative behavior from self-interested opponents in SSDs and would therefore be trivial baselines. 

  -   Other opponent shaping methods such as LIO (Yang et al., 2020) and Learning to Penalize other Learning Agents (Schmid et al., 2021) explicitly modify the environment by expanding the action space to allow agents to directly influence the rewards of other agents, making them inapplicable to the original unmodified games.

  -  The most similar works to ours are amTFT (Lerer & Peysakovich) and the hierarchical model of Kleiman-Weiner (2016) require a priori knowledge of cooperative and defective strategies and therefore cannot be extended to the simultaneous learning setting.

- Finally, we note that shaping methods, unlike prosocial methods (as enumerated above), are typically evaluated head-to-head against a variety of other agent types. Our agent's resistance to exploitation and strong performance against higher-order shapers is a key result of this paper that would only be possible to demonstrate in these head-on comparisons against strong modern opponent-shaping baselines - the success of our method against prosocial agents would be trivial and of little interest to the community, in our opinion.

Regarding our usage of the term "emergent cooperation":

-   We appreciate the reviewer's perspective and concede that "emergent cooperation" may be an overloaded term in the context of opponent shaping, since the mutual cooperation is induced by the reward structure of SSDs **combined** with opponent shaping abilities rather than the latter alone. Therefore, we have toned down the language regarding this claim in our current draft.
Many thanks for the further clarifications.

Just a quick note: by doing opponent shaping, you do not have emergence of cooperation, but cooperation by design in my opinion. Again, LOLA and MFOS are generic frameworks in themselves, not designed for the problem of emergence of cooperation in my opinion.
The authors have clarified my questions and I would like to maintain my score.
W1/2:

It would be good to see the clarifications mentioned in the updated manuscript. That being said, the authors' response has largely addressed my concerns, so I will raise my score.

W3: 

There was a typo on my end, where I put [1] I was referring to this paper:

Fung, K., Zhang, Q., Lu, C., Wan, J., Willi, T., & Foerster, J. Analysing the Sample Complexity of Opponent Shaping.
Thank you for the detailed response. I will discuss these responses with the other reviewers. I have no further questions at this time.
> However, the reviewer still has major concerns about this paper, especially with respect to the contribution of each specific mechanism and their inter-play. Essentially, it is unclear if all these mechanisms (and inherent complexity) are necessary. 

-   Once again, we apologize for our lack of clarity and will use two motivating examples from the literature to further establish the basis of not just one specific mechanism, but all of them combined together to produce a single TFT-like strategy. 

-   We reference amTFT, which seeks to achieve cooperation in SSDs by combining two pretrained strategies: one cooperative and one defective. amTFT monitors the partner's gains (debit) from deviating from a cooperative baseline and switches to defection for a fixed amount of time $k$ as punishment once the debit crosses a threshold, s.t. any gains from defection are fully neutralized. Our work takes a similar TFT-like approach, but tackles two key problems: 1) it is rarely possible to classify actions as purely cooperative or defective in temporally-extended SSDs such as Coins. 2) switching to defection for a fixed number of steps to neutralize a fixed debit threshold requires several manually selected parameters and does not generalize to complex environments. For the first problem, we use VI to assess the influence of one agent's action on another agent's expected return as a general measure of cooperativity. For the second, we develop the notion of an influence balance that is analogous to debit, but can be dynamically "paid off" via reciprocal influence at each timestep instead of an abrupt switch from cooperation to defection.  

-   Our work can also be seen as a "cumulative reciprocator" (CURE; Li et al., 2022), which has been shown to improve the stability of cooperation compared to memory-1 TFT strategies, especially in the presence of errors (e.g., a stochastic policy). CURE depends on countable instances of cooperation and defection, which are used to compute the defection difference statistic $d(k)$ - maintaining a history of cooperation can improve stability by allowing agents to tolerate "accidental" defection from mostly cooperative opponents. While this works in simple matrix games such as IPD, it is more difficult to directly classify individual actions as cooperative or defective in temporally-extended games such as Coins, inspiring our use of the modified VI metric to measure continuous changes to expected return as a quantification of cooperation/defection. Our influence balance is then a continuous analogue to $d(k)$.

-   We hope that this context makes clear to the reviewer the contributions of each part **as generalized components within a single TFT-like strategy, rather than disparate mechanisms**. We would be happy to revise our manuscript to make this clearer.

> With respect to the baselines, it seems to me that the type of "cooperation" in LOLA/MFOS is different from that defined by the authors. The authors of LOLA use "cooperation" as a term, but it is a different phenomenon from the emergence of cooperation of interest for the authors. 

-   We would like to ask the reviewer for further clarification on specific reasons why it seems to them that these two types of cooperation are different. However, we provide our case here as to why we believe they are very similar. As stated in the paper and in our responses to other reviews, our method performs a form of opponent shaping (as do MFOS and LOLA) by influencing opponent returns following particular actions via reciprocation, with the goal of shaping policy updates towards actions that are favorable w.r.t. the Reciprocator's own expected return.

-   In social dilemmas, the environment's reward structure induces mutually defective behavior from naive learners, in which they myopically optimize for their own individual returns to converge to a Pareto-dominated outcome. Opponent shaping work such as LOLA, MFOS, and our work seek to take actions which guide opponents towards cooperative solutions, for the reason that **cooperative behavior from the opponent leads to increased return for the shaping agent** in SSDs. We distinguish this class of cooperation from a large body of work in MARL in which agent objectives are explicitly modified to include "altruistic" goals - which we once again stress fails to achieve cooperation and is exploited when faced with self-interested opponents.

> With respect to the notation, I see what the authors mean, but I would suggest trying to improve/correct the notation in terms since those terms are not essentially defined in the formula, in my opinion. 

-   We appreciate the reviewer's attention to detail - as stated in our initial rebuttal, we have updated our draft to define the unclear terms (the counterfactual reward function) and fixed the summation term. Additionally, we will clarify that $\mathbf{a}^{-i}$ denotes the vector of joint actions excluding that of agent $i$.
As stated in the clarifying email sent by the conference PCs, “**Comments to paper and reviews will be fine. Comments can be seen in time. Please set the readers correctly when you post them. Reviewers are not required to take comments into consideration**.” However, we note that our follow-up comment was simply a list of references to the main rebuttal, making this a relatively minor point.
It seems that the authors also used two boxes for the answer - not sure if it is over the limit, I cannot check from the interface (for a general matter of fairness).
I would like to thank the authors for their rebuttal. However, the reviewer still has major concerns about this paper, especially with respect to the contribution of each specific mechanism and their inter-play. Essentially, it is unclear if all these mechanisms (and inherent complexity) are necessary.

With respect to the baselines, it seems to me that the type of “cooperation” in LOLA/MFOS is different from that defined by the authors. The authors of LOLA use “cooperation” as a term, but it is a different phenomenon from the emergence of cooperation of interest for the authors.

With respect to the notation, I see what the authors mean, but I would suggest trying to improve/correct the notation in terms since those terms are not essentially defined in the formula, in my opinion.

For these reasons, I will maintain my assessment of this work.
Thank you for your detailed response to my concerns. I have read your rebuttal and will provide further comments soon.
[1] Foerster, J. N. et al. Learning with Opponent-Learning Awareness. in Proceedings of the 17th International Conference on Autonomous Agents and Multiagent Systems (2018). doi:10.48550/arXiv.1709.04326.

[2] Wang, T., Wang, J., Wu, Y. & Zhang, C. Influence-Based Multi-Agent Exploration. in Eighth International Conference on Learning Representations (2020).

[3] Lerer, A. & Peysakhovich, A. Maintaining cooperation in complex social dilemmas using deep reinforcement learning. Preprint at https://doi.org/10.48550/arXiv.1707.01068 (2018).

[4] Lu, C., Willi, T., Witt, C. A. S. D. & Foerster, J. Model-Free Opponent Shaping. in Proceedings of the 39th International Conference on Machine Learning 14398–14411 (PMLR, 2022).

[5] Yang, J. et al. Learning to Incentivize Other Learning Agents. in Advances in Neural Information Processing Systems vol. 33 15208–15219 (Curran Associates, Inc., 2020).