Reciprocal Reward Influence Encourages Cooperation From Self-Interested Agents
Authors: John Luoyu Zhou
Weizhe Hong
Jonathan Kao
Venue: NeurIPS 2024 poster (2024)
Keywords: cooperation
reinforcement learning
opponent shaping
multi-agent reinforcement learning
Primary area: reinforcement_learning
TL;DR: We introduce an intrinsic reciprocal reward that encourages an agent to reciprocate another's influence on its own expected return, and show that resulting policies promote cooperative outcomes in sequential social dilemmas.

Abstract: Cooperation between self-interested individuals is a widespread phenomenon in the natural world, but remains elusive in interactions between artificially intelligent agents. Instead, naïve reinforcement learning algorithms typically converge to Pareto-dominated outcomes in even the simplest of social dilemmas. An emerging literature on opponent shaping has demonstrated the ability to reach prosocial outcomes by influencing the learning of other agents. However, such methods differentiate through the learning step of other agents or optimize for meta-game dynamics, which rely on privileged access to opponents' learning algorithms or exponential sample complexity, respectively. To provide a learning rule-agnostic and sample-efficient alternative, we introduce Reciprocators, reinforcement learning agents which are intrinsically motivated to reciprocate the influence of opponents' actions on their returns. This approach seeks to modify other agents' $Q$-values by increasing their return following beneficial actions (with respect to the Reciprocator) and decreasing it after detrimental actions, guiding them towards mutually beneficial actions without directly differentiating through a model of their policy. We show that Reciprocators can be used to promote cooperation in temporally extended social dilemmas during simultaneous learning. Our code is available at https://github.com/johnlyzhou/reciprocator/.

Reciprocal Reward Influence Encourages Cooperation From Self-Interested Agents
John L. Zhou
Weizhe Hong Jonathan Kao
University of California, Los Angeles
john. 1y..hoo@gmaii... com
Abstract
Cooperation between self- eeff-interested individuals is widespread phenomenon in the natural world, but remains elusive in interactions between artificially intelligent agents. Instead, naïve reinforcement learning algorithms typically converge Pareto-dominated outcomes even the simplest of social dilemmas. An emerging literature on opponent shaping has demonstrated the ability to reach prosocial outcomes by influencing the learning of other agents. However, such methods differentiate through the learning step of other agents or optimize for meta-game dynamics, which rely on privileged access to opponents' learning algorithms or ex- ponential sample complexity, respectively. To provide a learning rule-agnoue aggnosici and comple-efficient alternative, we introduce Reciprocators, reinforcement learning agents which are intrinsically motivated to reciprocate the influence of opponents' actions on their returns. This approach seeks to modify other agents' Q-values by increasing their return following beneficial actions (with respect to the Recipro- cator) and decreasing it after detrimental actions, guiding them towards mutually beneficial actions without directly differentiating through a model of their policy. We show that Reciprocators can be used to promote cooperation in temporally extended social dilemmas during simultancous learning. Our code is available httpp:///iit //github//t comm://oullyyhhooorrerpprraororr
Introduction
Many species exhibit cooperative behaviors of remarkable variety and complexity. Even among prosocial animals, however, humans are especially notable for their ability to cooperate with unrelated individuals and maintain that cooperation even in highly adversarial environments (Melis & Semmann 2010). These qualities are often credited with the development of human technology, culture, and advanced cognition (Burkart et al.) 2014). As artificially intelligent (AI) agents become more commonplace in human society, it is increasingly important to ensure that they share similar prosocial qualities so that interactions between AI agents, well as between AI agents and humans, may converge to mutually beneficial outcomes.
However, state-of-the-as reinforcement learning (RL) methods typically designed for the single agent setting and are ill-suited for the nonstationarities introduced by multiple agents learning simultaneously. Treating other agents fixed elements of the environment, referred to the "naïve" learning (NL) approach, can destabilize training and produce collectively suboptimal outcomes. This is particularly evident in class of games known sequential social dilemmas (SSDs), which contain tradeoff's between collective and individual return (Leibo et al.) 2017). SSDs present a particularly challenging problem for multi-agent RL (MARL) because of this mixed motivational structure, which precludes the use of common centralized training algorithms designed for cooperative settings (Kraemer & Banerjee 2016 Sunehag al. 2017 Gupta al. 2017 Rashid et al. 2018
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Previous work has shown that NL agents optimizing only for their individual returns converge to non-cooperative, Pareto-dominated outcomes in even the simplest of SSDs (Foerster et al. 2018).
In this work, we propose an intrinsic reward that encourages agents called Reciprocators to learn a tit-for-tat-like strategy which reciprocates the influence that others exert their returns. We first define value influence, a notion of influence that quantifies the effect that one agent's 's action has on another's expected return. Given a pair of agents, a Reciprocator re and another learning agent i, we track the cumulative influence of agent i's sequence of actions on re's expected return, which we refer as the influence balance "owed" to re by agent i. At each time step, the Reciprocator receives an additional intrinsic reciprocal reward proportional to the product of its current influence balance with agent and the value influence of its action on agent This encourages the Reciprocator to take actions whose influence matches the sign of the influence balance: for example, if agent i has produced a net positive influence on rc, then re will be rewarded for actions that have a reciprocally positive influence on agent i's expected return. Mechanistically, our method seeks to manipulate the opponent's Q-values by altering their expected return in particular directions depending on their actions. A aaward-maximizing. agent i should then be incentivized to take mutually beneficial actions and avoid harmful externalities.
The contributions of this work are as follows: (1) We formulate a novel intrinsic reward that encourages an agent to incentivize cooperation from other, simultaneously learning agents without modifying the structure of the environment, computing higher-order derivatives, or learning meta- game dynamics. (2) Agents trained with this reward achieve state-of-the-an cooperative outcomes in sequential social dilemmas and are able to shape purely self-interested naïve learners into mutually beneficial behavior. (3) Reciprocators demonstrate resistance to exploitation by higher-order baselines, despite using only first-order reinforcement learning algorithms.
2 Related Work
In order to improve convergence towards Pareto-optimal solutions among independently learning agents, previous work has modified agents' reward structures to explicitly consider either group or per- capita return in order to promote cooperative behavior, e.g., via inequity aversion (Hughes et al.) 2018)) "empathic" harm reduction (Bussmann et al.] 2019), or "altruistic" gradient adjustments (Li et al.. 2024). While agents that abide by such restrictions may seek to reduce their harmful externalities, they have no way of regulating the behavior of other, less magnanimous agents, including purely self-interested "exploiters" in the worst case (Agapiou al.) 2023). Their efficacy in SSDs, well as in most other types of multi-agent systems, is therefore limited in the absence of strong guarantees over the other agents' altruistic tendencies.
2.1 Cooperation through Influence
A more robust form of cooperation can be achieved by actively exerting influence over other agents in the environment rather than unilaterally adopting prosocial policy, especially in SSDs where extrinsic rewards incentivize defection. Crrrcction.auuuss et al. (2019) provides an intrinsic reward for "social influence" on other agents, which is computed as the Bullback-Leible (KL) divergence between an opponent's 's policy distribution conditioned on an influence agent's action a' and counterfactual distribution which marginalizes out that influence agent. However, as a metric-agnostic quantity. KL divergence is unable to capture key details of how the action distribution changes (Park et al. 2024). In particular, this definition of influence cannot selectively modify opponents' behavior with respect to extrinsic task returns or state transition dynamics, and therefore has limited utility as mechanism to incentivize cooperation.
Other works seek to directly influence opponent returns via the social mechanisms of reward and punishment, which are believed assist stabilizing cooperative relationships by controlling free- riding, cheating, and other antisocial behaviors (Melis & Semmann, 2010). Incorporating these mechanisms into groups of agents, whether by providing rewards to incentivize good behavior (Yang et al. 2020). or doling out punishments to discourage bad behavior (Schmid et al. [021] Yaman et al. 2023), has been shown to encourage cooperation among independent agents in a variety of SSDs. However, these methods require extensions of the original action space that allow agents to directly modify other agents' rewards. On the other hand, we instead propose to quantify the influence of
each naturally available action E A in given state on another agent's expected return R, and use

---

this influence as the medium for reciprocation. As we later show, selectively rewarding or punishing an opponent's 's actions can be seen as a form of opponent shaping which seeks manipulate the Q-value of given state-action pair in order to influence the likelihood of that action in future policies.
2.2 Opponent Shaping
Considering future policies points to a key issue with the canonical reinforcement learning (RL) framework, in that it optimizes only for the expected return within single episode of the environment. However, taking actions that seek to optimize the long-term behavior of the other agents in the environment will not receive any immediate feedback within an episode, and must wait for one, or possibly many, learning steps. Opponent-shaping methods of this kind address this issue by differentiating across pairs of episodes (Yang et al.] 2020). through opponents' gradient updates (Foerster et al. 2018 Zhao et al.] 20222] Willi et al. 2022). or repeated sequences of learning steps organized into "meta-episodes" (Lu al. 2022). While these methods have demonstrated convergence to cooperative behavior in simple SSDs, they are impractical or intractable in realistic multi-agent scenarios, requiring additional independent action channels for providing incentives, white-box access to the learning rules and gradients of other agents, or exponential sample complexity (Fung et al., 2023) to learn the dynamics of meta- meta-games. respectively.
In particular, the Learning with Opponent-Leerrinn approaches differentiate through the opponent's 's learning update but have only been demonstrated to work with full access to either the opponent's policy gradients and Hessians or their policy parameters. When using gradient approximations and modeling the opponent's 's policy, LOLA with opponent modeling (LOLA-OM) showed significantly worse results, even against vanilla policy gradients. Modeling efforts become even more implausible when faced with modern RL techniques, which employ adaptive optimizers that set different per-weight learning rates (Kingma & Ba 2017 Adam), randomized experience replay (Schaul et al., 2016), and various auxiliary terms such as policy divergence penalties (Schulman et al. 2017) and entropy exploration bonuses (Williams 1992
Awareness (Foerster et al. 2018 LOLA) class of
Mnih et al., 2016) to improve learning. Model-Free Opponent Shaping (Lu et al. [0022] MFOS) was introduced as a more general meta-learning approach that requires neither privileged access to nor inconsistent assumptions on opponent learning rules. However, MFOS's meta-policies are trained across multiple training runs repeated in sequence, each of which are treated as a single "meta-episode," in order to learn how to exploit opponent learning dynamics. This requires the ability to freely perform rollouts in the environment against opponents whose policies can be repeatedly reset to initialization, an unrealistic assumption in environments with partially adversarial motivations.
Our work is conceptually most similar to Learning with Opponent Q-Learning Awareness (Aghajohari 2024, LOQA), which also seeks to shape opponent policies by influencing the Q-values for different actions, under the assumption that opponents are Q-learners. However, LOQA still differentiates through a model of the opponent's 's policy and optimizes according to the joint advantage function A(s,, a1, 42) computed with respect to the state-value function Vstt). On the other hand, we use a counterfactual baseline that marginalizes out only the opponent's ' action to perform agent- specific credit assign assignment and use intrinsic reward instead of gradients over opponent policies to encourage agents to influence the opponent's 's Q-values the correct direction. Most importantly, LOQA focuses on the problem of learning general end-policy that performs well against a variety of other agents at evaluation and has therefore only demonstrated the ability to shape other LOQA opponents in a controlled self-play scenario, while our method focuses on shaping the policies of diverse agents over multiple episodes of simultaneous learning.
In the context of these higher-order opponent-shaping approaches, we position our intrinsic reciprocal reward as a form of immediate, within-episode feedback that encourages otherwise naive learners to implicitly consider the long-term, cross-episode effects of their actions on the policy changes of other agents, demonstrating many of the properties of higher-order opponent-shaping methods while using only first-order reinforcement learning algorithms and standard rollout-based training procedures.
3 Preliminaries
We construct series of sequential social dilemmas which can each be described as a stochastic game G, defined by tuple G (S, A, r, P,r,n, 7). In G, n agents, indexed by E (I...., n), observe the state of the environment s e S and simultancously choose actions a" e A to form joint action
3

---

E LEA=A A". The environment then undergoes a change according to the state transition function P(s'|s,a) A S - [0. 1]. Each agent receives an individual reward r' (s, a), and future rewards are discounted at each time step by discount factor y E [0. 1]. We consider the fully observable setting where agents have access to the full state of the environment & E S at every time step. joint action a E A. and rewards received by each agent. Each agent conditions a stochastic recurrent policy x <'aa'r"" on its action-observation history, which is denoted as T² E T = (S x A)"
4 Reciprocal Reward Influence
4.1 1-Step Value Influence
In order to compute general measure of value-based influence, we modify an intrinsic exploration reward proposed by Waan et al. (2020) called Value of Interaction (Vol). Vol measures how much agent i's actions affect agent j expected return by computing the difference between the Q-value conditioned on the joint state and action Q7 (s, a) and counterfactual baseline Qii (s, a²)) which marginalizes the state and action of the influencing agent compute a "default" expected return [Equation]] T]. Here, the bolded π superseript indicates the dependence of these Q-functions on the parameters of all agents' policies. The Vol can be decomposed into immediate influence term r(s, a) rss, a ") and discounted future influence term computed over changes in state transition probabilities [Equation 2]. Although the original Vol was formulated as an expectation over trajectories T and used as regularizer, we modify it to compute the one-step Value Influence (VI) where Vol E, [V]] 1]. allowing us quantify the influence of individual actions
VT (8, = Q7 (s, a) -0 (s,a
p" (5' 8, s,a
= = a) - r(s, a " + Σ p(s' 8, s)
V(5').
(2)
where = [r(s, (a
af))]]
and p(s'|s,a"*) )) Eginzi p (s'|s, (a aa)) This
definition of influence relies on the notion of a counterfactual baseline to assign credit to a particular agent's 's action while holding all other agents' actions constant. This counterfactual baseline is so named because it estimates the counterfactual expected return if the agent's 's true action replaced with "default" action, which is computed by marginalizing out the agent's 's action to get the expected on-policy return. This baseline return given by
O3e (s,a Boraxi 197 (s. (a (a - =E((())9 (a=t,a))
In practice, approximate this by regressing towards the Q-value while masking out o; from the joint state-action input. Marginalizing out an agent's ' action to quantify influence is a common paradigm in MARL, having also been used to define 1- 1-step adversarial power (Li & Dennis 2023), which estimates the maximum reduction in agent j's expected reward that can be achieved by agent i the minimum V VT, (s, a.) over all a' A. Similarly, Counterfactual Multi-Agent policy gradients (Foerster et al. 2017 COMA) marginalizes out single agent's 's action assess the advantage (i.e.. influence) of that agent's 's selected action relative to the counterfactual on-policy return ceteris paribus. Note that the COMA advantage function [Equation 4 can be expressed a special case of V VI (s, a)))) where == j.
Asss,a) (s)ss [ (a,(aa) a')))
(4)
4.2 Keeping Score with Influence Balances
The amount of influence that Reciprocator is able to exert in a single time step is heavily environment- dependent, so that may not always possible immediately reciprocate previous influences. To encourage reciprocation over extended timescales, we continuously accumulate mcasure of net

---

Algorithm Training with Reciprocal Reward Influence vs, Agent
Initialize agent re's policy parameters Oz, VI target function parameters é, influence balance vector Brdj O, policy memory M 0, and influence memory H. = for each episode do Observe initial state so
for t = to T do
Choose action at (s:)
Observe Te, St+1
Store transition tuple (st, at it, St+1) 1 in M and joint transition tuple (st, ae rt. St+1) in H
end for
Update influence target function parameters & with H every episodes
R Compute reciprocal rewards rees (1), ...,, rejs (T) w.r.t. agent i and sum with rewards in M
Compute advantage estimates AI...., Ar for K epochs do
Optimize the surrogate PPO-clip objective w.r.t. Oz end for Reset .M 0 end for
influence over sequences of actions. We draw inspiration from the "debit" tally used by approximate Markov tit-for-tat (Lerer & Peysakhovich 2018 am amFFT), which tracks the cumulative advantage gained by an opponent compared to a fully cooperative baseline policy known priori.
Extending this idea to our framework, we sum agent i's influence V Vdipee on the Reciprocator's expected return at each timestep rather than its own. Using influence in only one direction as motivation for reciprocation can lead continuous punishment or rewarding without way settle the score. To mitigate this, we also accumulate the net influence Pels in the opposite direction and subtract it from the influence balance at every timestep way to "pay off" the balance, limiting the degree to which reciprocation rewarded. Formally, we define the influence balance (fc maintained by Reciprocator re with another agent at time i
(t) = Bro - [V (st. a²)
reje (st, az)].
(5)
The influence balance can thought of as score of net influence over time between agents, and can be used to motivate reciprocation in the correct direction, i.e., either positive reinforcement of net positive influence or positive punishment net negative influence.
4.3 Intrinsic Reciprocal Reward
agent takes series of actions that improves the expected return of the Reciprocator over a baseline estimate, i.e., produces a positive influence balance, then the Reciprocator should be motivated to reinforce this behavior by exerting a reciprocal positive influence on agent v's expected return R2 in order encourage higher likelihood of that behavior during policy updates. Similar logic applies for detrimental deviations, negative influence balance. and subsequent reciprocal punishment. We then define the intrinsic reciprocal reward IR (t)((t) received by re as
R rro (t) = Broji(t- - 1). VT, (st, at).
6
Taking the product existing influence balance and current action's VI encourages the Reciprocator take actions that reinforce agent i's behavior in the correct direction by matching signs, and scales the reward by the magnitude of the outstanding influence balance and the reinforcing influence. The intrinsic reward is then added to the agent's extrinsic reward form the total reward used in training.
4.4 Policy Optimization with Reciprocal Rewards
We use experience replay to periodically update target networks and iteratively update our counter- factual baseline estimates towards these target values (Mnih et al., 2015). This provides two key
5

---

Acquire own coin
Acquire coin
Acquire same coin
C
D
C
-1,-1
-3,0 0
D
0,-3
-2,-2
11/0
+1-22
(a) Rewards for IPD.
(b) Rewards for Coins.
Figure 1: (a) The first number in each cell denotes the reward received by the agent taking the row action, and the second the reward received by the agent taking the column action, where C: cooperate (stay silent) and defect (confess). (b) Two agents (red and blue) are tasked with collecting randomly spawning coins. If an agent collects its own coin, it receives reward of +1 (left). If an agent collects another's coin, then receives a reward of +1 but the other agent receives punishment of -2.
benefits: first, periodically updating these policy-dependee functions stabilizes training and allows us approximately ignore their gradients, and therefore the gradient the intrinsic reciprocal reward, with respect to the agents' policy parameters (Wang et al., (020). With this assumption, we are able use standard policy gradient methods to train our agents to jointly optimize the combined extrinsic and intrinsic rewards.
Second, drawing samples from multiple previous episodes the train the counterfactual baseline target functions makes the Reciprocator less susceptible to exploitation. If the counterfactual esti- mators were updated concurrently with the policy using only the most recent on-policy data, then the Reciprocator's baseline would be immediately adjusted to its opponent's 's new policy after each episode. Because assessment of influence hinges on these counterfactual baselines, updating them too frequently would allow adversaries to easily manipulate these estimates of on-policy returns.
5 Experiments
We conduct experiments using two commonly used SSDs of varied complexity to demonstrate the shaping abilities of Reciprocators against other types of learning agents. For IPD, consider memory-1 iterated games as in Foerster et al. (2018) and et al. (2022), following the proof from Press & Dyson 20T2) that longer-memory strategies provide advantage strategies conditioned on shorter memories. We use two methods to evaluate head-to-head performance in IPD: allowing agents to directly differentiate through the analytic, closed-form solution of the game originally derived in Foerster et al. (2018), and more standard batched policy rollouts for a fixed episode length. We append "analytic" and "rollout" the game names denote the evaluation method used. For Coins, we augment the observation given to the critics and value influence estimators with the time remaining in the episode to prevent state aliasing and stabilize learning from experience replay (Pardo et al. 2022), but do not provide them as input to the policy networks.
5.1 Sequential Social Dilemmas
Iterated Prisoners' Dilemma (IPD): The iterated prisoners' dilemma (IPD) is temporally extended version of the classical thought experiment, in which two prisoners are given a choice to either stay silent/cooperate (C) confess/defect (D) with rewards given in Table Ta The Pareto-optimal strategy for both agents to cooperate by maintaining their silence, but the only Nash equilibrium (in the non-iterated, single-shot case) is mutual defection.
Coins: Coins temporally extended variant of the IPD introduced by Lerer & Peysakhovichi (2018). In this game, two coins spawn randomly in a fixed-size grid, with each coin corresponding to one of the two players (designated by color matching). Each player moves around the grid and receives a reward of for collecting any coin, and a punishment of -1 if the other agent collects their coin. If agents collect coins indiscriminately where P(collect own coin) Rs 1/n, the net expected reward is We show example rewards for various scenarios in Figure lb

---

Table 1: IPD-Analytic round robin results
RC
NL
LOLA
M-MAML
MFOS
RC
-1.06
-.03
-1.05
-1.05
-1.6
NL
-1.06
-1.98
-1.52
-1.28
-1.88
LOLA
-1.08
-1.30
-1.09
-1.04
-1.02
M-MAML
-1.13
-1.25
-1.15
-1.17
-1.56
MFOS
-0.98
-0.65
-1.02
-0.81
-1.01
Each entry is the average reward per episode achieved by the row agent against the column agent.
Standard error of the mean (SE) is less than 0.01 for all experiments and M-MAML results are averaged across the 10 initial policies provided by Lu et al. (2022).
0.150
1.0
0.125
0.100
Reciprocal Rewerd
0.050
0.6
PiCooparate)
0.025
0.025
Reciprocal Reward
-02
000550
Reciprocaton: PICooperate)
AL-PPO. P(Cooperate)
200
300
400
500
Episode
Figure 2: Representative run of Reciprocator NL in IPD-Rollout. Average reciprocal reward per step (left axis) and probability of cooperation (right axis) over the course of an episode.
5.2 Baselines
We select the following baselines because they focus specifically on the problem shaping other agents' policies during simultaneous learning and without modifications to the environment. In Coins, we exclude Meta-MAPG and LOLA-DICE as baselines following work by Yu al. (2022) showing that neither method is able to achieve significant results, even with simplified shared reward.
Naïve Learner (NL): As previously defined, NL. optimize their expected return with respect only to their own policy parameters In this work, we implement NI.s using policy gradient methods (Sutton al. 1999), which perform updates of the form 0.+1 0, V (me) e, where is the learning rate and g. g p. is the gradient of the objective function with respect to the policy parameters O, at step t.
Learning with Opponent-Learreeeeeeeeee opponent's 's gradients and Hessians explicit model of their policy parameters, assuming they NLs, and differentiates through their learning step using the update rule
Awareness (LOLA): LOLA uses either whitebox access to an
0 = 0: + a 0j J (0 + 20T")
A0 a oV J (0,0,))
Multiagent Model-Agnostic Meta-Learning (M-MAML): M-MAML (Lu et al., 2022) learns initial parameters and then meta-learns over both its own and its opponent's policy updates, concep- tually similar to Meta-Multiagent Policy Gradient (Kim al. 2021| Meta-MAPG) but modified to differentiate directly through the analytic form of the return in matrix games.

---

Meaa Epicod
0.625
Raciarocater
MFCS
0.600
NL-PPO
NOWn
0.500
15-
0300
10-
0.455
Reciorectior
105
200
100
400
560
Eskood
Figure 3: Shaping an NL in Coins. Proportion of own coins collected by NL during training when facing each opponent (left) and coin counts by type for Reciprocator vs. NL (right). Reciprocator and NL-PPO results plotted scale of single episodes (bottom axis) whereas MFOS results are plotted on a scale of meta-episodes, where one meta-episode contains episodes (top axis).
Model-Free Opponent Shaping (MFOS); As briefly discussed in the introduction, MFOS meta- learns multiple episodes of policy updates in order to accomplish long-horizon opponent shaping. In Lu et al. (2022). MFOS implemented with inner and outer policies, where the outer policy either directly outputs inner policy play in each episode conditioning vector which is element multiplied with an inner policy vector, as done in IPD-Analytic and Coins, respectively.
5.3 Implementation Details
In IPD-Analytic, we differentiate directly through the analytic solution to the matrix game. For rollout- based experiments, implement all policy gradient- based agents using actor-eritic architectures trained with proximal policy optimization using a clipped surrogate objective (Schulman et al.) 2017] PPO-Clip). Results including naïve learners trained with this algorithm are denoted by NL-PPO. Target networks to estimate the Q-values in Equation Tare updated every & episodes using uniformly sampled experience from a replay buffer (Lin 1992). Additional hyperparameter values and network architecture details can be found in Appendix
6 Results
We evaluate the Reciprocators' performance against variety of baselines in round-robin tournament for Analyiay In Coins, we assess apponent-shaping performance against NLs and against another agent of the eиror of the mean (SE) over eight random seeds. We do not perform full round-robin experiments in non-analytic environments for the following reasons: the Reciprocators' complex and stochastic learning rule renders LOLA's assumptions invalid and the extended time to convergence due to reciprocal behavior makes collecting meta-training episodes for MFOS computationally prohibitive.
type, which we refer to symmetric Coins. Shaded regions indicate standard
6.1 IPD
IPD-Analytic: In the analytic form of IPD, we show that Reciprocators able to reach cooperative cquilibria with all other baselines, resisting exploitation by higher-order methods such as LOLA and MFOS despite being only first-order method using vanilla gradient descent [Table D. Although the Reciprocator is extorted by MFOS to small extent (-1.06 vs. -0.98, respectively), we emphasize that MFOS relies extensive liberties such as the ability observe thousands of parallel training runs against their opponents and roll out pre- pre-trained meta-policies against newly initialized agents.
Stochastic policy rollouts allow the Reciprocator to influence opponent returns dif- ferently for different action sequences and provide a stronger learning signal, motivating additional experiments using sampled rollouts. In this setting, show that Reciprocator is able to consistently
IPD-Rollout:

---

Mets Eslsode
Moroa Eatodd
Ruciarocator
Recipro.anoo
krros NUPPO
NTCS NL-PPO
Pown C4r)
060
0.50
-2
Figure 4: Head-to-heat results in symmetric Coins (two agents of the same kind). Total (extrinsic) reward per episode (left), proportion of own coins collected (right), Again, Reciprocator and NL-PPO results are plotted on scale of episodes and MFOS results are plotted on scale of meta-episodes.
shape an NL over the course of a single learning trajectory with limited rollout samples and observe interesting oscillations the Reciprocator's rate cooperation. Figure SegureeSugeen suggests that these oscilla- tions are driven by the opposing intrinsic and extrinsic rewards, where the derivative of the rate of cooperation corresponds to the reciprocal reward value.
We provide the following explanation for this phenomenon: although the Reciprocator initially learns cooperative tit-for-tat strategy guided by strong reciprocal reward, as the NL's policy becomes deterministic and a² becomes predictable, the VTiee component of the influence balance decreases to O. As the extrinsic reward begins to dominate, the Reciprocator essentially reverts into an NI, leading to exploitative defection. However, exploitation of the NI. agent i causes the VT roli term, and subsequently the reciprocal reward, to become negative. Combined with an increase the NL's frequency of defection, which leads to increase in VTiree the intrinsic reward produces reversal back cooperative behavior that is reinforced by positive reciprocal rewards, and the cycle repeats.
6.2 Coins
In this temporally extended social dilemma, we see that the Reciprocator able to shape NL-PPO into picking up more of its own coins at rates significantly higher than MFOS. This allows both agents to achieve a positive reward, while needing only a fraction of the samples to converge to mutually beneficial behavior (cach meta-episode consists of 16 sequential episodes) [Figure 3 We show that this change is driven by changes in coin preference rather than in total collection, with both agents collecting at near-optimal pace. This is in contrast to MFOS, which does not shape the NL towards cooperation, but rather uses its pretraining advantage to suppress opponent learning altogether (Khan al. 2023 by collecting coins faster than the NL can reach them [Appendix B.1
When two Reciprocators are pitted against each other, we see that they quickly learn a cooperative strategy of collecting their own coins (Figure[] resulting in average reward of ~ 8 per 32 steps without needing self-play. This significantly outperforms MFOS and the reported performance of LOLA with opponent modeling (1.OLA-OM), which achieves an average reward of only ~ 2 per 32 steps according to Foerster et al. (2018). While can be argued that having intrinsic rewards for both opponents in the symmetric setting effectively alters the reward structure of the social dilemma into cooperative game, we emphasize that the reciprocal reward encourages apponent-shaping behavior that can take the form of cooperative strategy, rather than explicitly incentivizing cooperation for cooperation's 's sake.
Together, these results demonstrate that Reciprocators are able to robustly shape the behavior of other agents towards prosocial equilibria during simultaneous learning, achieving state-of-the-an results with fewer assumptions and limitations than existing methods. Apart from opponent-shaping properties, we also show that the intrinsic reciprocal reward discourages Reciprocators from exploiting others, showing promise for the development of more cooperative multi-agent learning framework.

---

7 Limitations and Future Directions
Our implementation of Reciprocators using naive RL algorithms remains limited in that it seeks to maximize its compound return within single episodes rather than across multiple episodes. Using handpicked weights balance intrinsic and extrinsic rewards opens the door for suboptimal tradeoff between long-term teg-ttermm opponent shaping and short-term return maximization. This is partially mitigated by the counterfactual target baseline updates, which allow the Reciprocator to lower the magnitude of the reciprocal reward in response to opponent policies that remain stationary over multiple episodes [Figure 2]. Future work will focus on methods to evaluate reciprocation efficacy across multiple episodes and dynamically tune the balance between reciprocal and extrinsic rewards.
In terms of evaluation, Khan et al. (2023) found that Coins does not require history to enable opponent shaping, since the current state is often indicative of past actions. Due to computational limitations, we were unable assess our method's performance on the suite of of Matrix Games (STORM) designed to test shaping over longer time horizons. We do note that Reciprocators capture both types of memory necessary to achieve shaping as identified by Khan et al.) (2023): the counterfactual baselines trained on replay buffers serve as way capture inter-episode context, and the recurrent policies and influence balance capture intra-episode history. Therefore, our method can in theory generalize to STORM environments, although we leave this to future work.
Representations
8 Conclusion and Broader Impacts
Emerging interest in cooperative AI has been led by approaches that endow agents with higher- order shaping capabilities. Although these agents exhibit cooperative behaviors when pitted against equivalent opponents, they readily manipulate and exploit agents with simpler learning rules or lower computational capabilities. This is fundamentally undesirable outcome in partially adversarial interactions between learning agents, with the potential to exacerbate existing computational resource gaps beyond out-of-the-box pretrained performance.
We presented Reciprocators, agents which seek to influence the behavior of other, simultaneously learning agents towards mutually beneficial outcomes. We showed that Reciprocators prosocial behaviors and induce them from other agents in variety of sequential social dilemmas, while remaining resistant to exploitation by higher-order agents. To the best of our knowledge, Recip- rocators represent the first class of reinforcement learning algorithms to achieve cooperation during simultaneous learning between two independent agents without needing meta-leaming methods, knowledge of other agents' learning algorithms, or pretraining routines such as self-play or tracing procedures to control opponent selection. We believe that these results show promising avenue forward for inducing cooperative outcomes from a diverse array of learning opponents.
both learn
10

---

Acknowledgements
This work was supported by the following awards to JCK: National Institutes Health DP2NS and NSF CAREER 1943467.
References
Agapiou, J. P., Vezhnevets, A. S., Duéñiez-Gzzmän E. A., Matyas, J., Mao, Y., Sunchag. P., Köster, R., Madhushani, U., Kopparapu, K., Comanescu, R., Strouse, D. J., Johanson, M. B., Singh, S.. Haas, J., Mordatch, I., Mobbs, D., and Leibo, J. Z. Melting Pot 2.0, October 2023. URL http://arxiy: wtgg/bb///a/ 13746 arXiv:2211.1374 [es].
Aghajohari, M., Duque, A., Cooijmans, T., and Courville, A. LOQA: LEARNING WITH OPPONENT Q-LEARNING AWARENESS. 2024.
Burkart, J. M., Allon, O., Amici, F., Fichtel, C., Finkenwirth, C., Heschl, A., Huber, J., Isler, K., Kosonen, K., Martins, E., Meulman, E. J., Richiger, R., Rueth, K., Spillmann, B., Wiesen- danger, S., and van Schaik, C. P. The evolutionary origin of human hyper-cooperratinn Nature Communications, 5(1): August 2014. ISSN 2041-1723. doi: 10. 00.1038//commm57477 URL. https://www. http://wwwwauu..eeeee//. nature..aeee..........a http:///e.eeeettttaae/aooo////// Group.
Number: Publisher: Nature Publishing
Bussmann, B., Heinerman, J., and Lehman, J. Towards Empathic Deep Q-Learning, June 2019. URL
http://xxxxvv.rr//////// /90///996..11 10918 arXiv: 1906. 10918 [cs].
Foerster, J., Farquhar, G., Afouras, T., Nardelli, N., and Whiteson, S. Counterfactual Multi-Agent Pol- icy Gradients, December 2017. URL http::/arxiv, org. attp///s.////..../.. 08926 arXiv:1705.08926 [es].
Foerster, J. N., Chen, R. Y., Al-Shedivat, M., Whiteson, S., Abbeel, P., and Mordatch, I. Learning with Opponent-Learning Awareness. Proceedings of the 17th International Conference on Autonomous Agents and Multiagent Systems, September 2018. DXXiv:1709.0433 [cs].
abs/
1709.04326
Fung, K., Zhang, Q., Lu, C., Willi, T., and Foerster, N. Analyzing the Sample Complexity of Model- Free Opponent Shaping. July 2023. URL /uttps://openreviees wtt.://www.....eeoo.oommmemmm/aa
Gupta, J. K., Egorov, M., and Kochenderfer, M. Cooperative Multi-agent Control Using Deep Reinforcement Learning. In Sukthankar, G. and Rodriguez-Aguila;; J. A. (eds.), Autonomous Agents and Multiagent Systems, volume 10642, pp. 66-83. Springer International Publishing, doi: 10. 1-000/997777--319-1666---- 4_5
Cham, 2017. ISBN 998 URL http::/linkk http/liinkspiinge..... 10w0///.00000/000000001 Notes in Computer Science.
Series Title: Lecture
Hughes, E., Leibo, J. Z., Phillips, M., Tuyls, K., Dueiiez-Guzman, E., Gareía Castañeda, A., Dunning, I., Zhu, T., McKee, K., Koster, R., Roff, H., and Graepel, T. Inequity aversion improves coopera- tion in intertemporal social dilemmas. In Advances in Neural Information Processing Systems, volume 31, Curran Associates, Inc.. 2018. URL URL neurips...ppp..... oct.:eeddeaaaanooirpssscc//ppppp html
Jaques, N., Lazaridou, A., Hughes, E., Gulcehre, C., Ortega, P., Strouse, D., Leibo, J. Z., and Freitas, N. D. Social Influence as Intrinsic Motivation for Multi-Agent Deep Reinforcement Learning. In Proceedings of the 36th International Conference on Machine Learning, pp. 3040- 3049. PMLR, May 2019. URL https::/pproceednngs. mlr wwwppseeevvviee monsss/v7/jaage http//////aaaaa.aaa.a html ISSN: 2640-3498.
Khan, A., Willi, T., Kwan, N., Tacchetti, A., Lu, C., Grefenstette, E.,, Rocktäschel, T., and Foerster, J. Scaling Opponent Shaping to High Dimensional Games, December 2023. URL, http::/ tattt::/arxivy /arxiv org/abs 2212. 12568, arXiv:2312. 12568 [cs].

---

Kim, D. K., Liu, M., Riemer, M. D., Sun, C., Abdulhai, M., Habibi, G., Lopez-Cot, S., Tesauro, Policy Gradient Algorithm for Learning to Learn in Multiagent Reinforce-
G., and How, ment Learning. In Proceedings of the 38th International Conference on Machine Learning, pp. 5541-5550. PMIR, July 2021, htt.htttps:/ppocceeeiii httpppoooeeiigg.......eeeeee/eaa www.pepessssvvvee /1//s////////// wttpp///////////m//g//g/ kim21g....................mm..1 html ISSN: 2640-3498.
Kingma, D. and Ba, J. Adam: Method for Stochastic Optimization, January 2017. URL
httt:://axxii: org/ /a/b//////// 111. 6980 arXiv:1412.6980 [es].
Kraemer, L. and Banerjee, B. Multi-agent reinforcement learning as a rehearsal for decentral-
ized planning. Neurocomputing.
J.neucom.201..01.00 URL 190:82-94, May 2016. ISSN 0925-2312. doi: 10.1016/ 50925281216000783
Leibo, J. Z., Zambaldi, V., Lanctot, M., Marecki, J., and Graepel. T. Multi-agent Reinforcement Learning in Sequential Social Dilemmas, February 2017. URL http://arxiv.orrg/bbbb. [es].
03037.
Lerer, A. and Peysakhovich, A. Maintaining cooperation in complex social dilemmas us- ing deep reinforcement learning, March 2018. URL htttp///arxiv.orrr/aaa//777// arXiv: 1ee11177001600 [cs].
Li, M. and Dennis, M. The Benefits of Power Regularization in Cooperative Reinforcement Learning. In Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems, AAMAS '23, pp. 457-465, Richland, SC, May 2023. International Foundation for Autonomous Agents and Multiagent Systems. ISBN 978- 978-1-450339942.1
Li, Y., Zhang, W. Wang, J., Zhang, S., Du, Y., Wen, Y., and Pan, W. Aligning Individual and Collective Objectives in Multi-Agent Cooperation, February 2024. URL http://////axii org/ 12416, arXiv:2402. 12416 [cs].
Lin, L.-J. Self-improving reactive agents based on reinforcement learning, planning and teaching. Machine Learning, 83))2293-3 May 1992. ISSN 1573-0565. doi: 10. 10.1007/BF0999299, URL https://doi http:///1110000//0000900000
Lu, C., Willi, T., Witt, C. S. the 39th International Conference on Machine Learning, pp. 1339-11411, PMLR, June 2022. URL
S. D., and Foerster, J. Model-Free Opponent Shaping. In Proceedings of
mml. www.:emww................eemeere prress/v622/uu221 html ISSN: 2640-3498.
Melis, A. P. and Semmann, D. How is human cooperation different? Philosophical Transactions of the Royal Society B: Biological Sciences, 365(1553) September 2010. ISSN 0962- 8436. doi: 10. 1t.1009/stb.22200.1555 URI.htts::/ httppp:://wwww. http://wwwwccbbbcc nlm. nih, http://c/aatres//////p PMC2936178/
Mnih, V., Kavukcuoglu, K., Silver. D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wierstra, D., Legg. S., and Hassabis, D. Human- deep reinforcement learning. Nature, 518(7540):529 533, February 2015. ISSN 1476-4687. doi: 10. 1a.1000/nture 14236. URL hotohhttps///// //ww://wwww///// www.ra......... com cwwc http//rtiiclss.ee/ mttpuruuuuuree666 /ata 7540 Publisher: Nature Publishing Group.
control through
Number:
Mnih, V. Badia, A. P., Mirza, M., Graves, A., Lillierap, T., Harley, T., Silver, D., and Kavukcuoglu, K. Asynchronous Methods for Deep Reinforcement Learning. In Proceedings of The 33rd International Conference Machine Learning, pp. 1928-1937, PMLR, June 2016. URL https //proceedings.. mlr, wwwps/eesee.epppp http///8mmmmnina www.////..iaa......mmmmmmm.m..1. html
ISSN: 1938-7228.
Pardo, F., Tavakoli, A., Levdik, V., and Kormushev, P. Time Limits in Reinforcement Learning, January 2022. http: http://aavvvvggrr..///// hts///bb/1112 00378 arXiv:1712.00378 [cs].
Park, S., Rybkin, O., and Levine, S. METRA: Scalable Unsupervised RI. with Metric-Aware Abstraction, March 2024. URL http::///a/aii org/ abs/2310 org/abs/2333
08887 arXiv:2310.08883 [cs].
12

---

Press, W. H. and Dyson, F. Iterated Prisoner's Dilemma contains strategies that dominate any evolutionary opponent. Proceedings of the National Academy of Sciences, 109926;10449-1041
June 2012. doi: 10. 10.10///ppss. 1206569109 URL wttph:tt//////..... /ww:://wwwww.o ppas. . org wrg/doi//ie doi aas///bs//// 10. 1073/ Publisher: Proceedings of the National Academy of Sciences.
pnas. 1206569109
Rashid, T., Samvelyan, M., de Witt, C. S., Farquhar, G., Foerster, J., and Whiteson, S. QMX:
Monotonic Value Function Factorisation for Deep Multi--gent Reinforcement Learning, June 2018.
wttp://..rrggb////////aaae/ 1803. 11485 arXiv:1803.1 [cs, stat].
Schaul, T., Quan, J. Antonoglou, I,, and Silver, D. Prioritized Experience Replay, February 2016. arXiv:1511.0523 [es].
URL http://aaa.....rr//////55111111
Schmid, K., Belzner, L., and Linnhoff-Popien, C. Learning to Penalize Other Learning Agents. MIT
Press, July 2021. doi: 10.1 10.1/al/ 00369. URL httph:tts///// //tx. h/tp:://xxxddoodooooooo www.oo....... 11t///0011111.a11....1
00369
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal Policy Optimization Algorithms, August 2017. URL http://arxiv org/ abs/ 1707. 06347 arXiv:1707.065: 066477 [cs].
Sunchag, P., Lever, G., Gruslys, A.. Czaneecii, W. M.. Zambaldi, V., Jaderberg, M.. Lanctot, M., Sonnerat, N., Leibo, J. Z., Tuyls. K., and Graepel, T. Value- Seluu--Dcoomositionn Networks For Cooperative Multi- Agent Learning, June 2017. URL htttp://axii.... wwtp://wwww.aa....oo..oo 05296 arXiv:1706.05296 [cs].
Sutton, R. S., McAllester, D., Singh, S., and Mansour, Policy Gradient Methods for Reinforcement Learning with Function Approximation. In Advances in Neural Information Processing Systems. volume 12. MIT Press, 1999. URL https: p/roceedi/ns/ceeess//e/o
Wang, T., Wang, J., Wu, Y., and Zhang, Influence-Based Multi-Agent Exploration. In Fighth International Conference on Learning Representations, April 2020. URL htttt:tt////aatt org/ abs abs/1910. 05512.
Willi, T., Letcher, A. H., Treutlein, J., and Foerster, COLA: Consistent Learning with Opponent- Learning Awareness. In Proceedings of the 39th International Conference on Machine Learning, pp. PMLR, June 2022. URL html ISSN: 2640-3498.
mlr wpwps//vve/wiillll////
Williams, R. J. Simple statistical gralient-foloowing algorithms for connectionist reinforcement leamn ing. Machine Learning. 8(3):229-255 May 1992. ISSN 1573-0565. doi: 10.1007/BBF00992666
URL https://doi org 10. 1007 1 BBF0099266
Yaman, A., Leibo, Z., Iacca, G., and Wan Lee, S. The emergence of division of labour through
decentralized social sanctioning. Proceedingy of the Royul Society B: Biologicul Sciences, 290 (2009):00331716, October 2023. ISSN 0962-8452, 1451-2954, doi: 10. 10.100//ppb.222.. 2023. URI. [tt.:tttee///gggllooeeeppaaaaaap org ott.://////////////////////////0 1098/ rsp.. 2023. 1716
Yang. J., Li, A., Farajtabar, M., Sunehag, P., Hughes, E., and Zha, H. Learning Incentivize
Other Learning Agents. In Advances in Neural Information Processing Systems, volume 33, Curran Associates, Inc., 2020. URL httptppee///rrooeinngg... wttp:/cceeiinnn..eriiss... cc/ paper/2020/hasile ad? htt.a//aaaaaaaaaaa.o/////////2e0
pp. hatp:///.......aaccceeeaaeeeaaea
html
Yu, X., Jiang, J., Zhang, W., Jiang, H., and Lu, Z. Model-Based Opponent Modeling, May 2022.
URL https:// http://wwww...e..oooo/oo////////
Zhao, S., Lu, C., Grosse, R. B., and Foerster, N. Proximal Learning With Opponent-Learning
Awareness, October 2022. URL
wwwp://..rg/a/b//////aaa/ 10125 AXiv:2210.10012 [es].
13

---

A Experimental Details
All experiments were Nvidia 3070 GPUs with 8 GB of VRAM. Results averaged over cight random seeds, with a batch size of 8192 for IPD-Analytic and 2048 for IPD-Rollout and Coins. For all experiments involving MFOS agents, adapted the original code from Lu et al. (002)) leaving all architectural choices and hyperparameters the same.
The architecture for both the actor and the critic consists of a state encoder (two convolutional layers followed by linear layer, with ReLU activations between layers), followed by hidden linear layers as detailed in the table below and a final output linear layer. The actor has softmax activation to output a policy over the action space, whereas the critic simply outputs a scalar value estimate.
Table 2: General PPO parameters.
Hyperparameter
IPD-Rollout Coins
Number of Convolutional Layers
2
Convolutional Kernel Size
3
Number of Linear Layers
2
1
Size of Linear Layers
2
16
Number of GRUs
1
Size of GRUs
16
Episode Length
32
32
Adam Learning Rate
0.005
0.005
PPO Epochs Per Episode
40
PPO-Clip
0.1
0.15
Discount Factor
0.96
0.96
Entropy Coefficient
0.02
0.01
With the exception of a linear layer size of 32 (instead of 16) for Coins, the network architecture and parameters estimate the various target functions to compute the V are identical to those described for the PPO components in Table For IPD-Analytic, target estimates for opponent policies in IPD were computed as the frequencies of observed choices for each of the five possible states (start, CC, CD, DC, DD).
For Coins, implement the decomposition of VI given in Equation 2 separately predicting the immediate reward and the transition probabilities for each possible next state. We collect one batch of episodes at the beginning of each experiment to initialize the influence estimators.
Table 3: Reciprocator-specffc parameters.
Hyperparameter
IPD- IPD-Analytic
IPl-Rollout
Coins
Replay Buffer Size (in episodes)
5
4
Target Training Batch Size
4096
Batches per Epoch
64
Target Function Update Period
10
Target Function Epochs Per Episode
20
Adam Learning Rate
0.01
Reciprocal Reward Weight
5.0
5.0
1.0
Replay buffer sizes are in units of episodes, where each episode consists of batch of 32 steps. A replay buffer of size 4 for Coins corresponds to 32 steps x episodes 2048 batch size 262,144 steps of experience.
B Additional Results
B.1 Coin Counts vs. NL-PPO
We display the total coin counts of an NI when faced against each other type of agent in order to show that changes in P(Own Coins) when faced by Reciprocator driven by changes in coin
14

---

color preference rather than changes in the total number of coins collected. In contrast, we see that the NL collects far fewer coins when faced with an MFOS agent. providing support for Khan et al.) (2023)'s i claim that MFOS suppresses rather than shapes learning. Note that this figure corresponds the same experimental data as Figure 3
Meta Episode
200
400
500
25
Reciprocator
MFOS
32
20
NL-PPO
15
NL PLP
100
200
300
400
500
Episode
Figure 5: Total number of coins per 32 steps collected by NL-PPO (right) vs. each baseline in Coins.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction do accurately reflect
the paper's 's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We include Limitations section in the main text.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
16

---

Justification:
This paper focuses on empirical evaluation and does not include theoretical
results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
The paper includes all network parameters, algorithmic details, and hyperpa- rameters necessary to reproduce the main experimental results. A zip file containing the code is also provided in the supplementary material.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
17

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: Code is linked in the abstract.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
priips.cc/publicc/uuieeee CoodeSumeeieeeeee ssi cessonPolll mollicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Training details are described in the Implementation Details (section 5.3) and hyperparameters are provided in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Number of replicates are provided in the Appendix and error bars (depicting standard error of the mean) are defined in Implementation Details.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
18

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Compute resources are described in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htttp:://euurpp::: http:///waaarr..ccpppaaaacccaaaa
Answer: [Yes]
Justification:
The research conducted in the paper conforms with the NeurIPS Code of
Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Broader impacts of this work are discussed in the Conclusion and Broader Impacts section.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
19

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e-g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: These models are evaluated on simple game- theee-theoretic environments and do not pose high risk for misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We use previously released code from the MFOS paper and an adapted open- source implementation of PPO, both of which are credited in the relevant code files and in the Appendix.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
20

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
batasets.papersstt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: New assets, i.e., code, is well-documented via comments and docstrings and is linked in the abstract.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
21

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
22


Reviews:
[{"review_id": "NhZFCkDtIa", "reviewer": "Reviewer_wp16", "summary": "The authors introduce Reciprocators, RL agents that are intrinsically motivated to reciprocate the influence of other player's actions on the agent's returns. They show that this promotes cooperation in social dilemmas.", "questions": "1. Have you considered trying your method on other SSD's beyond Coin Game? [1] discusses some of the downsides of using the coin game and introduces some new environments for doing so, available in [2] and [3]. I do not expect experiments to complete in time for the rebuttal, but it could be interesting for a camera-ready copy or future work.\n\n2. For Line 212: doesn't augmenting coin game with the time remaining de-construct the social dilemma? At the very last time step, one should always defect. By induction, this then applies all the way back to the first time step. State aliasing prevents this from happening.\n\n[1] Khan, Akbir, et al. \"Scaling Opponent Shaping to High Dimensional Games.\"\n\n[2] https://github.com/ucl-dark/pax\n\n[3] Rutherford, Alexander, et al. \"JaxMARL: Multi-Agent RL Environments and Algorithms in JAX.\" Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems. 2024.", "limitations": "The authors have addressed the limitations of their work.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Originality:\n\n- I do not believe this method has been proposed before\n\nQuality:\n\n- The results shown are strong. The fact that reciprocators are more resistant to extortion from MFOS is a previously unseen result.\n\n- The authors get strong results on the coin game, a challenging sequential social dilemma.\n\nClarity:\n\n- The paper is clearly written.\n\nSignificance:\n\n- As learning models become more prevalent, algorithms that learn to cooperate and are aware of each other's learning becomes increasingly important and significant. This paper proposes one such algorithm for this setting.", "weaknesses": "This paper claims that it does not modify the reward structure of the environment. However, adding an intrinsic reward to cooperate seems like it very much does modify the reward of the agents. It's not particularly interesting that, when given reward for promoting cooperation, agents perform behaviors that promote cooperation. What makes prior work on opponent shaping interesting is that when given a *purely self-interested objective* (ie anticipating the opponent's learning updates), cooperation emerges naturally.\n\nIn particular, the highlighting of results in Table 1 is misleading: The goal of prior works in opponent shaping is not to simply promote cooperation, but to promote *rational* cooperation (ie without being exploitable). For example, in Table 1, if I were self-interested and were told to pick an algorithm, I should always pick MFOS since it dominates all other choices. I do believe that this difference in intention between this work and the prior literature / baselines should be highlighted, as direct comparisons do not seem appropriate.\n\nMisc: The paper references the exponential sample complexity of M-FOS a few times. It might be good to reference the relevant paper [1] discussing that. \n\nFung, K., Zhang, Q., Lu, C., Wan, J., Willi, T., & Foerster, J. Analysing the Sample Complexity of Opponent Shaping.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Weaknesses\n\nW1\n\n-   Although the central finding of this paper is that we are able to produce cooperative behavior through our intrinsic reward, we emphasize that the underlying mechanism of the reciprocal reward is one of opponent shaping rather than a prosocial inclination to cooperate. The intrinsic reward's purpose is to alter the opponent's expected returns following various actions, in order to shape their policy updates towards a desired behavior. For example, suppose that the opponent takes an action which marginally benefits itself but significantly harms the return of the Reciprocator. Then, the Reciprocator is motivated to punish this behavior by reducing the opponent's return following that action and therefore negating the action's advantage. Future learning updates should then lower the probability of this action under the opponent's policy.\n\n-   Concretely, this differentiates it from other methods which add an intrinsic reward for the sole purpose of cooperation such as the inequity aversion of Hughes et al., which are able to produce cooperative behaviors ONLY if the agent population is entirely composed of other agents with the modified cooperative reward. On the other hand, we show that our agent is able to induce cooperative behavior from purely self-interested agents, leading to higher returns in the long run.\n\n-   However, we acknowledge that this is a real concern in the symmetric game when Reciprocators go head-to-head. To address this, we recorded the cumulative reciprocal reward received in each episode in Appendix Figure B1 (incorrectly linked as A5 in the text) and provided an explanation in L 295 to emphasize that the intrinsic reward is small relative to the extrinsic reward and does not transform the overall reward structure of the game into a purely cooperative one.\n\n-   We will update our manuscript to clarify these points.\n\nW2\n\n-   Our original motivation for including this figure came from the key concerns laid out in both the MFOS and LOLA papers: specifically, the idea of an agent-order \"arms race,\" in which agents of successively higher orders (e.g., meta-MFOS or nth-order LOLA) can readily exploit lower-order versions of themselves.\n\n-   We therefore positioned the reciprocal reward as an addition to basic first-order RL agents that could resist exploitation by higher-order agents without requiring meta-learning or higher-order derivatives, while NOT exploiting other naive learners itself, a phenomenon which we sought to demonstrate in the IPD-Rollout experiments as explained in section 6.1, L269.\n\n-   However, we agree that this is a very fair point - since MFOS and LOLA are specifically designed as opponent shaping algorithms rather than cooperation-inducing algorithms, they should not be evaluated in terms of their ability to achieve cooperation with different opponents. We have removed the bolding in the table and instead modified the caption and text to explain these advantages of the Reciprocator in our draft.\n\nW3\n\n-   An oversight on our part, thank you for pointing this out! We have added the relevant citations.\n\nQuestions\n\nQ1\n\n-   To the best of our knowledge, previous works in the opponent shaping literature have conducted evaluations only on simple iterated matrix games and Coins, and so we focused on these settings. However, we note that Reciprocators capture both types of memory necessary to achieve shaping as identified in [1]: the counterfactual baselines trained on replay buffers serve as a way to capture inter-episode context, and the recurrent policies and influence balance capture intra-episode history. Therefore, we expect our method to handle the augmented SSDs introduced in [1]. We thank the reviewer for bringing this to our attention and will conduct experiments on \"CoinGame in the matrix\" for a camera-ready copy. \n\nQ2\n\n-   This is an interesting insight! We apologize for the lack of clarity here - we augment the observations input to the joint and counterfactual Q-function estimators as a way to stabilize the estimates used to compute the value influence (VI), but do not provide them as part of the input to the policy networks. We have updated the text of the paper to clarify this distinction.\n\n-   We note that recurrent policies are mandatory in order to reciprocate/shape the opponent as detailed in [1] and used (in the form of GRUs) in other shaping methods such as MFOS, POLA, LOQA, etc., and RNNs are by nature capable of tracking elapsed time during an episode regardless of a time-augmented observation. However, we see empirically that the presence of recurrence in NLs or Reciprocators does not prevent the induction of cooperation in ours or previous works.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CrijSwJQEj", "reviewer": "Reviewer_SrMN", "summary": "The paper discusses an analysis of the effects of a combination of methods (1-step influence, debt mechanism and intrinsic reciprocal reward) on the emergence of cooperation in a society of artificial agents. The authors evaluate their approach for the case of n=2 using an IPD and the Coin game.", "questions": "- Do you have any results in terms of “ablation” considering the presence or not of the various mechanisms presented in the paper?\n\n- How does your work relate to the idea of social learning and the use of sort of social influence as intrinsic reward?\n\n- How does your work compare with Hughes et al. 2017? In fact it seems to me that the underlying mechanism might be similar to inequality aversion in a sense.\n\n- In Equation (1), the reviewer is not sure if there should be a sum over $a_i$: that appears to be an error. Could you please clarify the mathematical formulation?\n\n- Also in Equation (2), the sum over $a_i$ does not appear to be correct. Could you please discuss this formula as well?\n\n- In Section 4.4. you listed two “assumptions”. The reviewer wonders if it would have been preferable to validate them. Do you have results supporting these assumptions?\n\n- Why did you select LOLA, M-MAML, etc. as baselines? They do not appear to be relevant in a study about the emergence of cooperation. In fact, it seems to me that we know a priori that they do not lead to cooperation. It would have been more helpful to show comparison with other papers (like those that are cited in the related work) that actually lead to the emergence of cooperation.\n\n- Do you have results for the case of n>2? In fact, the theory is presented for n>2, but the evaluation is carried out for the case of n=2, which is very limiting and not sufficiently informative in my opinion.", "limitations": "There is not a real discussion of the limitations of the work in terms of generalization to the case of the proposed approach to the case of $n>2$ (in terms of evaluation).", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "- The reviewer personally believes that the study of cooperation in multi-agent systems is both interesting and timely. Despite the significant amount of work already conducted in this area, the reviewer welcomes further research and contributions", "weaknesses": "- The contribution of each mechanism (1-step influence, debt mechanism, and intrinsic reciprocal reward) is not discussed and evaluated in the paper. In fact, the author essentially combines them without clearly showing evidence that all three are actually needed. There is also a key question about the actual improvement that is possible to obtain by introducing them (and with which parameters). The mechanisms themselves are not new. Value influence comes from (Wang et al., 2020) and the idea of debt comes from (Lerer & Peysakhovich, 2018). That is perfectly fine, but I think it is important to understand if and how they are needed for observing the emergence of cooperation.\n\n- In general, the reviewer would have welcome some discussions (in terms of intuitions) about the design choices that are at the basis of this paper. In fact, it is not completely clear *why* the proposed mechanisms should work (and also *how* in a sense).\n\n- The mechanism presented by the authors appears to be somewhat related to the idea of social learning presented in [1]. However, this work is not discussed by the authors, even though it seems closely related. I believe a discussion about the relationship between that class of approaches and the one presented in this paper would be beneficial.\n\n- It would be interesting to see a comparison with other mechanisms that foster the emergence of cooperation, particularly in the evaluation of the proposed method. Instead, the authors present comparisons with algorithms (like LOLA), which seem unrelated to the problem at hand. In fact, we know beforehand that these mechanisms do not lead to the emergence of cooperation.\n\n- The theory presented in the paper essentially refer to a multi-agent scenario. However, the authors present their results only for the case of $n=2$. This case is not really informative (also comparing with existing works in the literature cited by the authors). The emergence of cooperation with $n=2$ is a phenomenon that has characteristics quite different compared to the case of $n>2$ in my opinion.\n\t\n\nReferences\t\n\n[1] Jaques N, Lazaridou A, Hughes E, Gulcehre C, Ortega P, Strouse DJ, Leibo JZ, De Freitas N. Social influence as intrinsic motivation for multi-agent deep reinforcement learning. In International Conference on Machine Learning 2019 May 24 (pp. 3040-3049). PMLR.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Weaknesses\n\nW1 - 2\n\n- We apologize for our lack of clarity and hope the following provides intuition: reciprocating strategies (such as TFT) are known to induce cooperation [1]. Reciprocation requires the ability to distinguish cooperation from defection, which is difficult to define *a priori* in SSDs. This inspired our adaptation of VoI [2], which allows us to assess the \"cooperativity\" of an action without predefined notions of cooperation/defection. \n\n- However, agents often have limited agency in SSDs, so they may not be able to immediately reciprocate. Keeping a running tally of influence over time motivates the agent to reciprocate when it has the opportunity to do so. To keep this tally, we constructed the influence balance inspired by debit [3].\n\n- The intrinsic reciprocal reward is a mechanism to implement a tit-for-tat (TFT)-style strategy using the aforementioned components, where the product of the influence balance and the VI of the current action is positive for reciprocating actions.\n\n- If this is a helpful clarification, we will be glad to revise our manuscript to include these intuitions.\n\nW3 - W5\n\n- Addressed in question responses.\n\nQuestions\n\nQ1\n\n- Because our three mechanisms are combined nonlinearly into a single reward term, it does not make sense to ablate individual parts. For example, if we ablate the influence balance, we cannot determine the correct direction of reciprocation. Additionally, since these components have not been used in the context of opponent shaping, there are no suitable \"replacements\" that can serve as an ablation baseline.\n\nQ2\n\n- While both approaches can be considered as forms of social influence, Jaques et al. seek to influence the actions of other agents via a mutual information objective, without considering how the behavior is modified. This undirected measure of social influence has unclear utility, and may even be meaningless in cases where different actions lead to equivalent rewards or transitions.\n\n- On the other hand, our intrinsic reward explicitly considers how opponent actions affect the Reciprocator's expected returns, and vice versa. It encourages the agent to influence the opponent's returns in a specific direction for opponent-shaping purposes.\n\n- We will include a discussion of these similarities and distinctions in our Related Works.\n\nQ3\n\n- Inequity aversion differs from our method in that it adds a prosocial objective by penalizing agents who under- or over-perform relative to others. Conversely, our reciprocal reward is not inherently prosocial, but seeks to encourage other agents to take actions which benefit the Reciprocator as an opponent-shaping mechanism.\n\n- A key result of this difference: Hughes et al. note that their \"guilty\" agents are easily exploited if the population is not composed **only** of guilty agents (see L60 - 65). However, the opponent-shaping properties of Reciprocators allow them to induce cooperative behavior from purely self-interested agents while conferring a robust resistance to exploitation against other shaping agents, e.g., MFOS.\n\nQ4\n\n- We clarify that Equation (1) is correct. Please note that we define the 2nd term, $Q(s, a^{-i})$, as an expectation/sum over $a^i$'s in Equation (3), which we believe addresses the confusion.\n\nQ5\n\n- Similar to the definition of the counterfactual Q-function in Equation (3), the counterfactual $r(s, a^{-i})$ should be an expectation over $a^i$ and the summation over states should also be over $a^i$. We have defined $r(s, a^{-i})$ and fixed the summation term in our revision. \n\nQ6\n\n- The first assumption is the same justification used for the original VoI from Equation 13 in [2], where the replay buffer and periodic update of the counterfactual target functions makes the gradients of these counterfactual estimators (and therefore the intrinsic reward) approximately constant w.r.t. to policy parameters. Because of its validation in previous work, we did not think it necessary to validate them here. We will update the manuscript to clarify this.\n\n- As for the second claim, we found that immediate updates using only the most recent on-policy experience from the last episode produced training instabilities resulting in lopsided exploitation - we will add results to the Appendix.\n\nQ7\n\n- We respectfully disagree with this point - one of the key results of LOLA is that they learn to cooperate: \"LOLA agents **learn to cooperate**, as agents pick up coins of their own colour with high probability while naive learners pick up coins indiscriminately\" [1]. Similarly, MFOS notes that it is \"the first learning algorithm **to achieve cooperation** in self-play without using higher-order derivatives or inconsistent models\" [4].\n\n- Because the goal of the reciprocal reward is to shape opponent behavior towards self-benefiting actions, we see our work as an opponent-shaping method similar to LOLA and MFOS which can lead to emergent mutual cooperation, and distinct from MARL algorithms which have the sole purpose of cooperation such as Hughes et al. (2018).\n\nQ8\n- While the $n > 2$ setting introduces increased complexity, cooperation in the $n = 2$ setting with a purely self-interested opponent in an *unmodified environment* remains nontrivial. Additionally, and to the best of our knowledge, there are no works in the literature that attempt to shape multiple opponents simultaneously. Therefore, we chose these settings for compatibility with modern opponent shaping baselines. However, we concur that these evaluations are simple compared to the broader RL literature and will demonstrate the merits of our approach on more complex games in future work.\n\n- We also reiterate that methods which have induced cooperation among $n > 2$ agents, such as the \"guilty\" agents of Hughes et al. (2018), require homogeneous prosocial populations and are easily exploited otherwise, whereas shaping methods such as [5] modify the environment to allow direct influence over others' rewards [5].", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SAY3lwHtIg", "reviewer": "Reviewer_PQwp", "summary": "The authors develop a new intrinsic reward that tracks the balance of influence compared to a counterfactual baseline and provides positive/negative rewards to incentivize naive learners to cooperate. They test this algorithm in an iterated prisoner dilemma and a simplified coin game, showing positive results compared to some strong baselines on opponent shaping.", "questions": "- While MarkovTFT is discussed — how is this approach significantly different? Given that the algorithm learns a TFT like algorithm — when would this approach be preferred and how does it advance the literature? The authors should also discuss a more probabilistic approach to TFT published in this work: Kleiman-Weiner, M., Ho, M. K., Austerweil, J. L., Littman, M. L., & Tenenbaum, J. B. (2016). Coordinate to cooperate or compete: abstract goals and joint intentions in social interaction. In CogSci.\n- The opponent shaping work has the potential to shape towards a wide variety of goals and aims — this work seems more focused on just cooperation. Is that right? If so — this should be discussed in more detail as a limitation. \n- Unless the algorithm correctly models what it could be doing counterfactually — it won’t know how to accurately update the balance of influence. Will this always be learned rapidly? Could an adversary bias its learning away from forming an accurate model?", "limitations": "Yes", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "This is a paper addresses an important topic in multi-agent reinforcement learning that will be of active interest to the NeurIPS community. The manuscript is well written and the results are well situated within the current literature. The algorithm is benchmarked against strong modern baselines and the results are discussed appropriately and clearly.", "weaknesses": "- There are some missing implementation details (number of seeds, averaging methods). \n- I’d have liked to see more extensive evaluation on the grid environments where the RL results are more important. \n- The evaluation could be more robust and compare in more sophisticated games. Even the coin game is highly simplified compared to the original versions. \n- While the algorithm is novel in the RL literature this is highly reminiscent of the following work on commutative recipcority which develops analytical results for a similar approach. Ideally the methods here could be compared to the algorithm and analyses proposed in this work: Li, J., Zhao, X., Li, B., Rossetti, C. S., Hilbe, C., & Xia, H. (2022). Evolution of cooperation through cumulative reciprocity. Nature Computational Science, 2(10), 677-686.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Weaknesses\n\nW1\n\n- We have added these details in the Appendix: each experiment was run with 8 random seeds, and results were averaged and plotted with SEM.\n\nW2-3\n\n- To the best of our knowledge, previous works in the opponent shaping literature have conducted evaluations only on simple iterated matrix games and this simplified version of Coins, so we focused on these settings in order to provide baseline comparisons (MFOS would be especially computationally prohibitive in more complex environments--see Lu et al. 2022). However, we concur that these evaluations are simple compared to the broader RL literature and plan to demonstrate the merits of our approach on more complex games in future work.\n\nW4\n\n- Our work can be considered a general implementation of a \"cumulative reciprocator\" (CURE). The CURE strategy depends on discrete, countable instances of cooperation and defection, which are used to compute the defection difference statistic d(k). While this works in simple matrix games such as IPD, it is more difficult to directly classify individual actions as cooperative or defective in temporally-extended games such as Coins, inspiring our use of the modified VI metric to measure continuous changes to expected return. Our influence balance is also similar to the defection difference statistic, although we subtract VI's at every timestep to directly track this statistic rather than maintaining running tallies.\n\n-   We find their analyses on the role of the tolerance level in increasing cooperation stability vs. noisy decisions particularly interesting as potential future directions for our work, e.g. adding an influence balance threshold to the reciprocal reward.\n\n-   We will include this discussion in our related works. As for the payoff analyses, we note that CURE is a fixed strategy that does not have to consider learning dynamics. However, our method seeks to influence the learning of other, simultaneously learning agents. Therefore, any solution must consider learning algorithms and associated parameters, making closed-form solutions far more difficult (if even possible) to derive.\n\nQuestions\n\nQ1\n\n- Our approach can be considered a more general version of amTFT, which seeks to achieve cooperation in SSDs by combining two pretrained strategies: one cooperative and one defective (note that defining such pure strategies is rarely possible in more complex social dilemmas). amTFT monitors the partner's gains from deviating from a cooperative baseline (debit) and switches to defection as punishment once the debit crosses a threshold. While our reward draws inspiration from amTFT, amTFT relies on pretrained policies and handpicked parameters (the defection duration and debit threshold) and focuses on performance against fixed opponents or as teachers for naive learners. It does not tackle the problem of emergent cooperation among simultaneously learning agents -- after all, it is not a learning agent itself but a composition of frozen components. On the other hand, our reciprocal reward encourages agents to learn when and how to cooperate or defect in an end-to-end policy. Notably this is learned in an environment with other, simultaneously learning agents, presenting a significantly more difficult, nonstationary learning problem.\n\n- Kleiman-Weiner et al. also learn a hierarchical policy consisting of cooperative and defective \"modes.\" A high-level planner infers the high-level intention (I) of an opponent and responds in kind (using a TFT strategy or learned high-level policy) by selecting between two sub-strategies. Notably, they use a low-level dynamics model to both plan sub-strategies and infer opponent intentions using value iteration, which requires modeling of both of the opponent's sub-policies and inherently assumes that they are following either a pure cooperative or defective strategy at any given time. On the other hand, our algorithm makes no such assumptions on the \"purity\" of strategies, the intent behind them, or their homogeneity throughout an episode, instead operating solely on the actual effects of the opponent's behavior on the Reciprocator's expected returns to dynamically shift between cooperative and defective behavior. \n\nQ2\n\n- Although the Reciprocator does focus on cooperation, it can be extended to other shaping goals by modifying the $VI_{i|rc}$ formulation to compute the influence of agent i's action on some other measure. The key mechanism is that the intrinsic reward motivates the agent to alter the returns of the other agent in a specified direction following an action - matching the direction of alteration to other criteria against which opponent actions can be evaluated would permit more general shaping goals.\n\n- We will expand on this limitation as well as the potential for alternative shaping goals in the Future Directions section.\n\nQ3\n\n- This is a great question! Another adversary with a meta-policy operating across episodes could launch an adversarial attack by manipulating the baseline into producing overly pessimistic estimates, e.g., by maintaining a highly defective policy for multiple timesteps to \"lower expectations\" of on-policy returns, s.t. even objectively harmful future actions may be viewed as having a positive influence and induce unwarranted reciprocation. This method of exploitatively lowering expectations is a common manipulation tactic in real-life, e.g., \"weaponized\" incompetence.\n\n- Performing updates of the counterfactual baseline too rapidly can make the Reciprocator more susceptible to such attacks. By periodically updating the baseline with experience across multiple episodes, we allow the Reciprocator to maintain an inter-episodic form of memory that prevents opponents from rapidly manipulating baseline expectations within single episodes. As a future direction, we propose to preferentially select episodes with higher return using prioritized replay in order to produce optimistic counterfactual baselines.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UdsnQrEOyQ", "reviewer": "Reviewer_os8m", "summary": "The paper proposes a method for including the influence of other agent’s into an agent’s own reward function, which is shown to promote cooperation in SSDs like IPD and the temporal version Coins. It also highlights that the method only requires first order RL algorithms, does not require access to privileged information, and does not require meta-learning.", "questions": "- LOLA vs. MFOS in Section 5.2 Table 1 does not match the original MFOS paper results. In MFOS Section 6.1, Table 4, LOLA vs. MFOS resulting in -2.09 for LOLA, while your section claims -1.02. Your result implies that LOLA has learned to cooperate with MFOS, when the MFOS paper shows that MFOS should learn a dominating strategy against LOLA. Where is the discrepancy coming from here? \n- Section 6 Figure 4 — Does each agent in this symmetric head-to-head receive the same reward and have the same probability of own coins? \n- Is there an experiment for Reciprocator vs. MFOS in Coin Game? \n- L 469: It’s unclear why the mean intrinsic reward begins as a negative number and converges to 0. I would have expected the intrinsic reward to decrease from a positive amount down to 0.\n- L 293: Is it possible that implementation details created a difference? \n- Will the codebase be made available upon acceptance? Concrete examples for your paper’s method will help future researchers implement baselines in accordance with your paper.", "limitations": "Yes.", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The paper is well written. \n- The papers has clear comparisons with state-of-the-art methods. Despite lower scores in IPD against MFOS, the authors are clear to state that improvements include not only higher scores, but also in a reduction of steps required to reach cooperation, reduced complexity in opponent shaping methods, and the ability to cooperate without access to privileged information. First order RL algorithms and standard roll out processes make reproducibility and adoption of the method easier than higher order methods requiring privileged information or meta-learning.", "weaknesses": "- Not all results are shown clearly for IPD-Rollout or Coins. I suggest a result matrix similar to IPD-Analytics or a clear explanation as to why it is not included, even if only in the Appendix. \n- Nit: Add titles to Section 5 Figure 2 and 3.\n- Nit: Section 5.2 Figure 2 description: percentage -> probability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Weaknesses\n\nW1\n\n-   One key issue with LOLA and IPD-Analytic is that the original LOLA implementation assumed that the opponent takes vanilla gradient steps directly through the closed-form analytic solution to the IPD return. LOLA then directly computes the gradient of the opponent's return w.r.t. the opponent's parameters to use for its update. This approach does not work in our case, where the Reciprocator's policy update takes into account the intrinsic reward (which is recurrently computed at each step), precluding a closed-form solution to IPD. Since the rollout environment introduces stochasticity rather than analytic solutions, we found that stochastic optimizers such as Adam provided much better results, at the expense of a fixed update rule. Because of this, we did not think it would be a fair comparison/equivalent evaluation to create a new,  \"hamstrung\" formulation of LOLA that is differentiating through an incorrect gradient update and present it as a baseline. We will make these choices clear in the Appendix.\n\n-   Unfortunately, running MFOS against the Reciprocator is computationally intractable in Coins (we elaborate on this in our response to Q3), and LOLA has failed to show significant results in Coins, even with a simplified shared reward [1]. We mention the latter in the beginning of Section 6, but will be sure to add in our rationale for only doing symmetric and not all head-to-head comparisons for Coins and provide more detailed statistics for the remaining experiments in the Appendix.\n\n-   We do want to emphasize that, while these factors reduced the scope of our evaluation, they also point to the clear limitations on existing opponent shaping methods and the relative generality of our simple intrinsic reward.\n\nW2\n\n-   We will add appropriate titles to these figures, thank you!\n\nW3\n\n-   We have made this fix, thank you!\n\nQuestions\n\nQ1\n\n-   We are not sure of the discrepancy between these results, as we reproduced the IPD-Analytic results by directly running their published code on Github with no changes, averaged over 8 runs. However, we will investigate further to see if there might be differences in averaging or random seed that may explain this large gap.\n\nQ2\n\n-   Agents in symmetric head-to-head games do receive similar rewards and cooperate with similar probabilities. We provide a representative run in Appendix Figure B.1 (incorrectly linked as A5 in the text) and attempt to communicate this in Figure 4 by providing the standard error of the mean computed over both agents in symmetric games. If the two agents achieved significantly inequitable outcomes, then there would be much higher variance in these figures, i.e., the shaded areas would be much wider. We have clarified this in the figure legend.\n\nQ3\n\n-   We did not provide a result for Reciprocator vs. MFOS since it is computationally infeasible to do so: as a meta-learning algorithm, MFOS treats full training runs to convergence as *single episodes*, requiring not only exponential sample complexity [2] but also an order of magnitude more gradient updates for each agent (num_meta_episodes x num_episodes x num_epochs). Reciprocators take significantly longer to converge than NLs due to their intrinsic opponent shaping objective and additional network updates. To provide a fair comparison, we would need to train at least 16 runs to convergence *sequentially*, which would require training times on the order of hundreds of hours for just 8 replications, assuming no hyperparameter tuning. We did perform one Reciprocator vs. MFOS experiment and found that MFOS failed to shape the Reciprocator or learn in any meaningful way, but due to the lack of replication power we did not include it in our paper - however, we would be happy to run these experiments for a camera-ready copy.\n\nQ4\n\n-   We attribute this to the conflicting motivations of the extrinsic and intrinsic reward. The extrinsic reward pushes the agent to naively collect all coins regardless of color, whereas the intrinsic reward encourages the agent to selectively collect coins depending on the influence balance: collecting the opponent's coin when punishment is desired, and avoiding it when it is not. In the case where the extrinsic reward initially dominates, the mean intrinsic reward should be negative since the the agent may be collecting coins at suboptimal times w.r.t. reciprocation. For example, if the influence balance is positive, the Reciprocator should avoid collecting the other agent's coins and therefore punishing it -- however, the extrinsic reward encourages it to take any coin as fast as possible, which would lead to a negative reward if the closest coin happens to be that of the opponent's.  \n\nQ5\n\n-   We constructed our agent based on the MFOS codebase and with hyperparameters as similar to the LOLA agent as possible, matching the reported architecture (including all layer sizes and activations), learning rate, discount factor, and environment implementation. We use a smaller batch size (2048 vs. 4000) due to memory constraints. They used a vanilla actor-critic method, which does not have the clipped gradient of PPO but is otherwise similar. We also point to subsequent work by [1], which found that LOLA-DiCE (LOLA with an improved differentiable Monte Carlo estimator) was unable to model opponents quickly enough and thus failed to achieve significant opponent shaping in Coins against a naive learner.\n\nQ6\n\n-   The codebase is already available as a zipped attachment in the supplementary materials, and will also be linked as a Github repository in the paper after the blinded review.  \n\nReferences\n\n[1] Yu, X., Jiang, J., Zhang, W., Jiang, H. & Lu, Z. Model-Based Opponent Modeling. (2022).\n\n[2] Fung, K., Zhang, Q., Lu, C., Willi, T. & Foerster, J. N. Analyzing the Sample Complexity of Model-Free Opponent Shaping. (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a new co--player shaping method that is meant to be better for cooperatively resolving sequential social dilemmas. Most reviewers were happy with the experiments and baselines, as well as the connection to broader themes in the literature concerned with these questions.

Reviewer SrMN disagreed with the others and argued that the baselines were not appropriate. But the other reviewers disagreed with them. I looked through the paper myself also and I concur with the other reviewers. I do think MFOS is an appropriate baseline here, and I notice that the results achieved for it here look very closely comparable to those reported in the MFOS paper itself. So this looks fine to me.

I note also that the authors promised to include the discussion which emerged in the conversation around the reviews here of the relationship between this work and Jaques et al. (2019) "Social Influence...". Like the reviewer who asked about it, I was also wondering about the relationship between the two papers, and I thought your explanation was interesting. It's a very good idea to include it in the related work for the camera ready version.
Decision: Accept (poster)