Title: Similarity-based cooperative equilibrium
Venue: NeurIPS 2023
Accepted: yes
Keywords: Program equilibrium

multi-agent learning

game theory

opponent shaping

superrationality

decision theory

cooperative AI

Newcomb's problem
Abstract: As machine learning agents act more autonomously in the world, they will increasingly interact with each other. Unfortunately, in many social dilemmas like the one-shot Prisoner’s Dilemma, standard game theory predicts that ML agents will fail to cooperate with each other. Prior work has shown that one way to enable cooperative outcomes in the one-shot Prisoner’s Dilemma is to make the agents mutually transparent to each other, i.e., to allow them to access one another’s source code (Rubinstein, 1998; Tennenholtz, 2004) – or weights in the case of ML agents. However, full transparency is often unrealistic, whereas partial transparency is commonplace. Moreover, it is challenging for agents to learn their way to cooperation in the full transparency setting. In this paper, we introduce a more realistic setting in which agents only observe a single number indicating how similar they are to each other. We prove that this allows for the same set of cooperative outcomes as the full transparency setting. We also demonstrate experimentally that cooperation can be learned using simple ML methods.

Similarity-based cooperative equilibrium

Abstract

As machine learning agents act more autonomously in the world, they will increasingly interact with each other. Unfortunately, in many social dilemmas like the one-shot Prisoner's Dilemma, standard game theory predicts that ML agents will fail to cooperate with each other. Prior work has shown that one way to enable cooperative outcomes in the one-shot Prisoner's Dilemma is to make the agents mutually transparent to each other, i.e., to allow them to access one another's source code (Rubinstein, 1998; Tennenholtz, 2004) - or weights in the case of ML agents. However, full transparency is often unrealistic, whereas partial transparency is commonplace. Moreover, it is challenging for agents to learn their way to cooperation in the full transparency setting. In this paper, we introduce a more realistic setting in which agents only observe a single number indicating how similar they are to each other. We prove that this allows for the same set of cooperative outcomes as the full transparency setting. We also demonstrate experimentally that cooperation can be learned using simple ML methods.

1 Introduction

As AI systems start to autonomously interact with the world, they will also increasingly interact with each other. We already see this in contexts such as trading agents (CFTC & SEC, 2010), but the number of domains where separate AI agents interact with each other in the world is sure to grow; for example, consider autonomous vehicles. In the language of game theory, AI systems will play general-sum games with each other. For example, autonomous vehicles may find themselves in Game-of-Chicken-like dynamics with each other (cf. Fox et al., 2018). In many of these interactions, cooperative or even peaceful outcomes are not a given. For example, standard game theory famously predicts and recommends defecting in the one-shot Prisoner's Dilemma. Even when cooperative equilibria exist, there are typically many equilibria, including uncooperative and asymmetric ones. For instance, in the infinitely repeated Prisoner's Dilemma, mutual cooperation is played in some equilibria, but so is mutual defection, and so is the strategy profile in which one player cooperates 70% of the time while the other cooperates 100% of the time. Moreover, the strategies from different equilibria typically do not cooperate with each other. A recent line of work at the intersection of AI/(multi-agent) ML and game theory aims to increase AI/ML systems' ability to cooperate with each other (Stastny et al., 2021; Dafoe et al., 2020; Conitzer & Oesterheld, 2023).

Prior work has proposed to make AI agents mutually transparent to allow for cooperation in equilibrium (McAfee 1984; Howard 1988; Rubinstein 1998, Section 10.4; Tennenholtz 2004; Barasz et al. 2014; Critch 2019; Oesterheld 2019b). Roughly, this literature considers for any given 2-player normal-form game Γ the following program meta game : Both players submit a computer program, e.g., some neural net, to choose actions in Γ on their behalf. The computer program then receives as input the computer program submitted by the other player. The aforecited works have shown that the program meta game has cooperative equilibria in the Prisoner's Dilemma.

Unfortunately, there are multiple obstacles to cooperation based on full mutual transparency. 1) Settings of full transparency are rare in the real world. 2) Games played with full transparency in 37th Conference on Neural Information Processing Systems (NeurIPS 2023).

Principal 1

Principal 2

Policy 1

π

1

:

R

→

∆(

A

1

)

Policy 2

π

2

:

R

→

∆(

A

2

)

diff : ∆(

A

1

)

R

×

∆(

A

2

)

R

→

R

2

(diff(

π

1

, π

2

))

1

(diff(

π

1

, π

2

))

2

Mixed strategy 1

σ

1

∈

∆(

A

1

)

Mixed strategy 2

σ

2

∈

∆(

A

2

)

general have many equilibria, including ones that are much worse for some or all players than the Nash equilibria of the underlying game (see the folk theorems given by Rubinstein 1998, Section 10.4, and Tennenholtz 2004). In particular, full mutual transparency can make the problem of equilibrium selection very difficult. 3) The full transparency setting poses challenges to modern ML methods. In particular, it requires at least one of the models to receive as input a model that has at least as many parameters as itself. Meanwhile, most modern successes of ML use models that are orders of magnitudes larger than the input. Consequently, we are not aware of successful projects on learning general-purpose models such as neural nets in the full transparency setting.

Contributions. In this paper we introduce a novel variant of program meta games called difference (diff) meta games that enables cooperation in equilibrium while also addressing obstacles 1-3. As in the program meta game, we imagine that two players each submit a program or policy to instruct an agent to play a given game, such as the Prisoner's Dilemma. The main idea is that before choosing an action, the agents receive credible information about how similar the two players' policies are to each w.r.t. how they make the present decision. In the real world, we might imagine that this information is provided by a mediator (cf. Monderer & Tennenholtz, 2009; Ivanov et al., 2023; Christoffersen et al., 2023) who wants to enable cooperation. We may also imagine that this signal is obtained more organically. For example, we might imagine that the agents can see that their policies were generated using the same code base. We formally introduce this setup in Section 3. Because it requires a much lower degree of mutual transparency, we find the diff meta game setup more realistic than the full mutual transparency setting. Thus, it addresses Obstacle 1 to cooperation based on full mutual transparency.

Diff meta games can still have cooperative equilibria when the underlying base game does not. Specifically, in Prisoner's Dilemma-like games, there are equilibria in which both players submit policies that cooperate with similar policies and thus with each other. We call this phenomenon similarity-based cooperation (SBC) . For example, consider the Prisoner's Dilemma as given in Table 1 for G = 3 . (We study such examples in more detail in Section 3.) Imagine that the players can only submit threshold policies that are parameterized only by a single real-valued threshold θ i and cooperate if and only if the perceived difference to the opponent is at most θ i . As a measure of difference, the policies observe diff( θ 1 , θ 2 ) = | θ 1 -θ 2 | + Z , where Z is sampled independently for each player according to the uniform distribution over [0 , 1] . For instance, if Player 1 submits a threshold of 1 / 2 and Player 2 submits a threshold of 3 / 4 , then the perceived difference is 1 / 4 + Z . Hence, Player 1 cooperates with probability P ( 1 / 4 + Z ≤ 1 / 2 ) = 1 / 4 and Player 2 cooperates with probability P ( 1 / 4 + Z ≤ 3 / 4 ) = 1 / 2 . It turns out that ( θ 1 = 1 , θ 2 = 1) , which leads to mutual cooperation with probability 1 , is a Nash equilibrium of the meta game. Intuitively, the only way for either player to defect more is to lower their threshold. But then | θ 1 -θ 2 | will increase, which will cause the opponent to defect more (at a rate of 1 / 2 ). This outweighs the benefit of defecting more oneself.

In Section 4, we prove a folk theorem for diff meta games. Roughly speaking, this result shows that observing a diff value is sufficient for enabling all the cooperative outcomes that full mutual

transparency enables. Specifically, we show that for every individually rational strategy profile σ (i.e., every strategy profile that is better for each player than their minimax payoff), there is a function diff such that σ is played in an equilibrium of the resulting diff meta game.

Next, we address Obstacle 2 to full mutual transparency - the multiplicity of equilibria. First, note that any given measure of similarity will typically only enable a specific set of equilibria, much smaller than the set of individually rational strategy profiles. For instance, in the above example, all equilibria are symmetric. In general, one would hope that similarity-based cooperation will result in symmetric outcomes in symmetric games. After all, the new equilibria of the diff game are based on submitting similar policies and if two policies play different strategies against each other, they cannot be similar. In Section 5, we substantiate this intuition. Specifically, we prove, roughly speaking, that in symmetric, additively decomposable games, the Pareto-optimal equilibrium of the meta game is unique and gives both players the same utility, if the measure of difference between the agents satisfies a few intuitive requirements (Section 5). For example, in the Prisoner's Dilemma, the unique Pareto-optimal equilibrium of the meta game must be one in which both players cooperate with the same probability.

Finally we show that diff meta games address Obstacle 3: we demonstrate that in games with higherdimensional action spaces, we can find cooperative equilibria of diff meta games with ML methods. In Section 6.4, we show that, if we initialize the two policies randomly and then let each of them learn to be a best response to the other, they generally converge to the Defect-Defect equilibrium. This is expected based on results in similar contexts, such as in the Iterated Prisoner's Dilemma. However, in Section 6.1, we introduce a novel, general pretraining method that trains policies to cooperate against copies and defect (i.e., best respond) against randomly generated policies. Our experiments show that policies pretrained in this way find partially cooperative equilibria of the diff game when trained against each other via alternating best response training.

We discuss how the present paper relates to prior work in Section 7. We conclude in Section 8 with some ideas for further work.

2 Background

Elementary game theory definitions. We assume familiarity with game theory. For an introduction, see Osborne (2004). A (two-player, normal-form) game Γ = ( A 1 , A 2 , u ) consists of sets of actions or pure strategies A 1 and A 2 for the two players and a utility function u : A 1 × A 2 → R 2 . Table 1 gives the Prisoner's Dilemma as a classic example of a game. A mixed strategy for Player i is a distribution over A i . We denote the set of such distributions by ∆( A i ) . We can extend u to mixed strategies by taking expectations, i.e., u ( σ 1 , σ 2 ) := ∑ a 1 ∈ A 1 ,a 2 ∈ A 2 σ 1 ( a 1 ) σ 2 ( a 2 ) u ( a 1 , a 2 ) . For any player i , we use -i to denote the other player. We call σ i a best response to a strategy σ -i ∈ ∆( A -i ) , if supp( σ i ) ⊆ arg max a i ∈ A i u i ( a i , σ -i ) , where supp denotes the support. A strategy profile σ ∈ ∆( A 1 ) × ∆( A 2 ) is a vector of strategies, one for each player. We call a strategy profile ( σ 1 , σ 2 ) a (strict) Nash equilibrium if σ 1 is a (unique) best response to σ 2 and vice versa . As first noted by Nash (1950), each game has at least one Nash equilibrium. We say that a strategy profile σ is individually rational if each player's payoff is at least her minimax payoff, i.e., if u i ( σ ) ≥ min σ -i ∈ ∆( A -i ) max a i ∈ A i u i ( a i , σ -i ) for i = 1 , 2 . We say that σ is Pareto-optimal if there exists no σ ′ s.t. u i ( σ ′ ) ≥ u i ( σ ) for i = 1 , 2 and u i ( σ ′ ) > u i ( σ ) for at least one i .

Symmetric games and additively decomposable games. We say that a game is (player) symmetric if A 1 = A 2 and for all a 1 , a 2 for i = 1 , 2 , we have that u i ( a 1 , a 2 ) = u -i ( a 2 , a 1 ) . The Prisoner's Dilemma in Table 1 is symmetric. We say that a game additively decomposes into ( u i,j : A j → R ) i,j ∈{ 1 , 2 } if u i ( a 1 , a 2 ) = u i, 1 ( a 1 ) + u i, 2 ( a 2 ) for all i = { 1 , 2 } and all a 1 ∈ A 1 , a 2 ∈ A 2 . Intuitively, this means that each action a j of Player j generates some amount of utility u i,j ( a j ) for Player i independently of what Player -j plays. For example, the Prisoner's Dilemma in Table 1 is additively decomposable, where u i,i : Cooperate ↦→ 0 , Defect ↦→ 1 and u i, -i : Cooperate ↦→

G, Defect ↦→ 0 for i = 1 , 2 . Intuitively, Cooperate generates G for the opponent and 0 for oneself, while Defect generates 1 for oneself and 0 for the opponent.

Alternating best response learning. The orthodox approach to learning in games is to learn to best respond to the opponent, essentially ignoring that the opponent is also a learning agent. In this paper, we specifically consider alternating best response (ABR) learning. In ABR, the players take turns. In each turn, one of the two players updates the parameters θ i of her strategy to optimize u i ( θ i , θ -i ) , i.e., updates her model to be a best response to the opponent's current model (Brown cf. 1951; Zhang et al. 2022; Heinrich et al. 2023). Since learning an exact best response is generally intractable, we will specifically consider the use of gradient ascent in each turn to optimize u i ( θ i , θ -i ) over θ i . In continuous games if ABR with exact (locally) best response updates converges to ( θ 1 , θ 2 ) , then ( θ 1 , θ 2 ) is a (local) Nash equilibrium. Note, however, that ABR may fail to converge (e.g., in the face of Rock-Paper-Scissors dynamics). Moreover, if the best response updates of θ i are only approximated, ABR may converge to non-equilibria (Mazumdar et al., 2020, Proposition 6).

3 Diff Meta Games

We now formally introduce diff meta games, the novel setup we consider throughout this paper. Given some base game Γ , we consider a new meta game played by two players whom we will call principals . Each principal i submits a policy . The two players' policies each observe a real-valued measure of how similar they are to each other. Based on this, the policies then output a (potentially mixed) strategy for the base game. Finally, the utility is realized as per the base game. Below we define this new game formally. This model is illustrated in Figure 1.

Definition 1. Let Γ = ( A 1 , A 2 , u ) be a game. A (diff-based) policy for Player i for Γ is a function π : R → ∆( A i ) mapping the perceived real-valued difference between the diff-based policies to a mixed strategy of Γ . For i = 1 , 2 let A i ⊆ ∆( A i ) R be a set of difference-based policies for Player i . Then a policy difference (diff) function for ( A 1 , A 2 ) is a stochastic function diff : A 1 ×A 2 ⇝ R 2 . For any two policies π 1 , π 2 and difference function diff , we say that ( π 1 , π 2 ) plays the strategy profile σ ∈ ∆( A 1 ) × ∆( A 2 ) of Γ if σ i = E [ π i (diff i ( π 1 , π 2 ))] for i = 1 , 2 . For sets of policies A 1 , A 2 and difference function diff we then define the diff meta game (Γ , A 1 , A 2 , diff) to be the normal-form game ( A 1 , A 2 , V ) , where V ( π 1 , π 2 ) := E [ u (( π i (diff i ( π 1 , π 2 ))) i =1 , 2 )] for all π 1 ∈ A 1 , π 2 ∈ A 2 .

Note that Definition 1 does not put any restrictions on diff . For example, the above definition allows (diff( π i , π -i )) i to be a real number whose binary representation uniquely specifies π -i . This paper is dedicated to situations in which diff specifically represents some intuitive notion of how different the policies are, thus excluding such diff functions. Unfortunately, there are many different ways in which one could formalize this constraint, especially in asymmetric games. In Section 5 we will impose some restrictions along these lines, including symmetry. Our folk theorem (Theorem 3 in Section 4) will similarly impose constraints on diff to avoid diff functions like the above.

The rest of this section will study concrete examples of Definition 1. First, we define a particularly simple type of diff-based policy. Almost all of our theoretical analysis will be based on this class of policies.

Definition 2. Let θ ∈ R ∪ {-∞ , ∞} and σ ⩽ i , σ > i ∈ ∆( A i ) be strategies for Player i for i = 1 , 2 . Then we define ( σ ⩽ i , θ, σ > i ) to be the policy π s.t. π ( d ) = σ ⩽ i if d ≤ θ and π ( d ) = σ > i otherwise. We call policies of this form threshold policies. Let ¯ A i denote the set of such threshold policies.

Throughout the rest of this section, we analyze the Prisoner's Dilemma as a specific example. We limit attention to threshold agents of the form ( C, θ, D ) , i.e., policies that cooperate against similar opponents (diff below threshold θ ) and defect against dissimilar opponents. This is because such policies can be used to form cooperative equilibria, while policies that always cooperate (say, ( C, 1 , C ) ) or policies that are more cooperative against less similar opponent policies (e.g., ( D, 1 , C ) ) cannot be used to form cooperative equilibria in the PD with a natural diff function. Policies of the form ( C, θ, D ) are uniquely specified by a single real number θ . A natural measure of the similarity between two policies θ 1 , θ 2 is then the absolute difference | θ 1 -θ 2 | . We allow diff to be noisy, however. We summarize this in the following.

Example 1. Let Γ be the Prisoner's Dilemma as per Table 1. Then consider the (Γ , ˆ A 1 , ˆ A 2 , diff) meta game where ˆ A i = { ( C, θ i , D ) | θ i ∈ R } and diff i (( C, θ 1 , D ) , ( C, θ 2 , D ))) = | θ 1 -θ 2 | + Z i for i = 1 , 2 where Z i is some real-valued random variable.

The only open parameters of Example 1 are G (the parameter used in our definition of the Prisoner's Dilemma) and the noise distribution. Nevertheless, Example 1 is a rich setting that allows for nontrivial results. We leave a detailed analysis for Appendix B and only give two specific results about equilibria here.

In the first result, we imagine that the noise Z i is distributed uniformly between 0 and ϵ > 0 and that G is at least 2 . Then, roughly, there are two kinds of equilibria. First, there are equilibria in which both players always defect, because their threshold for cooperation is at most 0 (such that they defect with probability 1 even against exact copies). Second, and more interestingly, there are equilibria in which both players submit the same threshold strictly between 0 and ϵ . Note that this means that if both players submit a threshold of ϵ , they both cooperate with probability 1 .

Proposition 1. Consider Example 1 with Z i ∼ Uniform([0 , ϵ ]) i.i.d. for some ϵ > 0 and with G ≥ 2 . Then (( C, θ 1 , D ) , ( C, θ 2 , D )) is a Nash equilibrium if and only if θ 1 , θ 2 ≤ 0 or 0 < θ 1 = θ 2 ≤ ϵ . In case of the latter, the equilibrium is strict if G > 2 .

What happens if, instead of the uniform distribution, we let the Z i be, say, normally distributed? It turns out that for all unimodal distributions (which includes the normal distribution) and G = 2 , we get an especially simple result: in equilibrium, both players submit the same threshold and that threshold must be left of the mode.

Proposition 2. Consider Example 1 with G = 2 . Assume Z i is i.i.d. for i = 1 , 2 according some unimodal distribution with mode ν with positive measure on every interval. Then (( C, θ 1 , D ) , ( C, θ 2 , D )) is a Nash equilibrium if and only if θ 1 = θ 2 ≤ ν .

4 A folk theorem for diff meta games

What are the Nash equilibria of a diff meta game on Γ ? A first answer is that Nash equilibria of Γ carry over to the diff meta game regardless of what diff function is used (assuming that at least all constant policies are available); see Proposition 16 in Appendix C.1. Any other equilibria of the diff meta game hinge on the use of the right diff function. In fact, if diff is constant and thus uninformative, the Nash equilibria of the diff meta game are exactly the Nash equilibria of Γ ; see Proposition 17 in Appendix C.1. So the next question to ask is for what strategy profiles σ there exists some diff function s.t. σ is played in an equilibrium of the resulting diff meta game. The following result answers this question. In particular, a folk theorem similar to the folk theorems for infinitely repeated games (e.g., Osborne 2004, Ch. 15) and for program equilibrium (see Section 7).

Theorem 3 (folk theorem for diff meta games) . Let Γ be a game and σ be a strategy profile for Γ . Let A i ⊇ ¯ A i for i = 1 , 2 . Then the following two statements are equivalent:

There is a diff function such that there is a Nash equilibrium ( π 1 , π 2 ) of the diff meta game (Γ , diff , A 1 , A 2 ) s.t. ( π 1 , π 2 ) play σ .

The strategy profile σ is individually rational (i.e., better than everyone's minimax payoff).

The result continues to hold true if we restrict attention to deterministic diff functions with diff 1 = diff 2 and diff i ( π 1 , π 2 ) ∈ { 0 , 1 } for i = 1 , 2 .

We leave the full proof to Appendix C.2, but give a short sketch of the construction for 2 ⇒ 1 here. For any σ , we construct the desired equilibrium from policies π ∗ i = ( σ i , 1 / 2 , ˆ σ i ) for i = 1 , 2 , where ˆ σ i is Player i 's minimax strategy against Player -i . We then take any diff function s.t. diff( π ∗ i , π -i ) = (0 , 0) if π -i = π ∗ -i and diff( π ∗ i , π -i ) = (1 , 1) otherwise.

5 A uniqueness theorem

Theorem 3 allows for highly asymmetric similarity-based cooperation. For example, in the PD with, say, G = 2 , Theorem 3 shows that with the right diff function, the strategy profile ( C, 2 / 3 ∗ C + 1 / 3 ∗ D ) is played in an equilibrium of the diff meta game of the PD. This seems odd, as one would expect SBC to result in playing symmetric strategy profiles. Note that, for example, all equilibria of Propositions 1 and 2 are symmetric. In this section, we show that under some restrictions on diff and the base game Γ , we can recover the symmetry intuition. This is good because in symmetric games the symmetric outcomes are the fair and otherwise desirable ones (Harsanyi et al., 1988, Sect. 3.4) and because SBC thus avoids equilibrium selection problems of other forms of cooperation (including cooperation based on full mutual transparency and cooperation in the iterated Prisoner's Dilemma).

Wefirst need a few definitions of properties of diff . Let Γ be a symmetric game. We say that diff is minimized by copies if for all policies π, π ′ , all y and i = 1 , 2 , P (diff i ( π, π ′ ) <y ) ≤ P (diff i ( π, π ) <y ) . For example, the diff function in Example 1 is minimized by copies. The diff functions in the proof of Theorem 3 are not in general minimized by copies when the given base game is symmetric. For example, to achieve ( C, 2 / 3 ∗ C + 1 / 3 ∗ D ) in equilibrium, the proof of Theorem 3 (as sketched above) uses the policies π ∗ 1 = ( C, 1 / 2 , D ) and π ∗ 2 = ( 2 / 3 ∗ C + 1 / 3 ∗ D, 1 / 2 , D ) and a diff function with diff( π ∗ 1 , π ∗ 2 ) = (0 , 0) but diff( π ∗ 1 , π ∗ 1 ) = (1 , 1) . If the base game is symmetric, we call diff symmetric if for all π 1 , π 2 , diff( π 1 , π 2 ) is distributed the same as diff( π 2 , π 1 ) and (diff 1 ( π 1 , π 2 ) , diff 2 ( π 1 , π 2 )) is distributed the same as (diff 2 ( π 1 , π 2 ) , diff 1 ( π 1 , π 2 )) .

Finally, we need a more complicated but nonetheless intuitive property of diff functions. In this paper, we generally imagine that low values of diff are informative about the other player's policy. In contrast, we will her assume that high values of diff are uninformative. That is, for any σ i and π -i , we will assume that there is a policy π i that plays σ i against π -i and triggers the above-threshold policy of π -i with the highest-possible probability. Formally, let π -i = ( σ ⩽ -i , θ -i , σ > -i ) be any threshold policy. Let p be the supremum of numbers p ′ for which there is π i s.t. in ( π i , π -i ) , Player -i plays (1 -p ′ ) σ ⩽ -i + p ′ σ > -i . Let σ max π -i = (1 -p ) σ ⩽ -i + pσ > -i . Intuitively, σ max π -i is the strategy played by π -i against the most different opponent policies. For the examples of Section 3 we have p = 1 and thus simply σ max π -i = σ > -i . But if diff is bounded, then we might even have p = 0 or anything in between.

Definition 3. We call diff : ¯ A 1 × ¯ A 2 ⇝ R 2 high value uninformative if for each threshold policy π -i , σ i and ϵ > 0 there is a threshold policy π i such that in ( π i , π -i ) , a strategy profile within ϵ of ( σ i , σ max π -i ) is played.

We are now ready to state a uniqueness result for the Nash equilibria of diff meta games.

Theorem 4. Let Γ be a player-symmetric, additively decomposable game. Let diff be symmetric, high-value uninformative, and minimized by copies. Then if ( π 1 , π 2 ) is a Nash equilibrium that is not Pareto-dominated by another Nash equilibrium, we have that V 1 ( π 1 , π 2 ) = V 2 ( π 1 , π 2 ) . Hence, if there exists a Pareto-optimal Nash equilibrium, its payoffs are unique, Pareto-dominant among Nash equilibria and equal across the two players.

We prove Theorem 4 in Appendix D.3. Roughly, we prove that under the given assumptions, equilibrium policies are more beneficial to the opponent when observing a diff value below the threshold than if they observe a diff value above the threshold. Second, we show that if in a given strategy profile Principal i receives a lower utility than Principal -i , then Principal i can increase her utility by submitting a copy of Principal -i 's policy. Appendix D.1 shows why the assumptions (additive decomposability of the game and and high-value uninformativeness and symmetry of diff ) are necessary.

6 Machine learning for similarity-based cooperation in complex games

Our results so far demonstrate the theoretical viability of similarity-based cooperation, but leave open questions regarding its practicality. In complex environments, where cooperating and defecting are by themselves complex operations, can we find the cooperative equilibria for a given diff function with machine learning methods?

6.1 A novel pretraining method for similarity-based cooperation

We now describe Cooperate against Copies and Defect against Random (CCDR) , a simple ML method to find cooperative equilibria in complex games. To use this method, we consider neural net policies π θ parameterized by a real vector θ . First, for any given diff game, let V d : ( R m ) ( R n +1 ) × ( R m ) ( R n +1 ) → R 2 be the utility of a version of the game in which diff is non-noisy. CCDR trains a model π θ i to maximize V d ( π θ i , π θ i ) + V d ( π θ i , π θ ′ -i ) for randomly sampled θ ′ -i . That is, each player i pretrains their policy π θ i to do well in both of the following scenarios: principal -i copies principal i 's model; and principal -i generates a random model. The method is named for its intended effect in Prisoner's Dilemma-like games. Note, however, that it is well-defined in all symmetric games, not just Prisoner's Dilemma-like games.

CCDR pretraining is motivated by two considerations. First, in games like the Prisoner's Dilemma, there exist cooperative equilibria of policies that cooperate at a diff value of 0 and defect as the

f C

f D

f

i

0.2

0.4

0.6

0.8

1.0

0.2

0.4

0.6

0.8

1.0

f C

f D

f

i

0.2

0.4

0.6

0.8

1.0

0.2

0.4

0.6

0.8

1.0

perceived diff value increases. We give a toy model of this in Appendix E. CCDR puts in place the rudimentary structure of these equilibria. Note, however, that CCDR does not directly optimize for the model's ability to form an equilibrium. Second, CCDR can be thought of as a form of curriculum training. Before trying to play diff games against other (different but similar) learned agents, we might first train a policy to solve two (conceptually and technically) easier related problems.

6.2 A high-dimensional one-shot Prisoner's Dilemma

To study similarity-based cooperation in an ML context, we need a more complex version of the Prisoner's Dilemma. The complex Prisoner's Dilemma-like games studied in the multi-agent learning community generally offer other mechanisms that establish cooperative equilibria (e.g., playing a game repeatedly). For our experiments, however, we specifically need SBC to be the only mechanism to establish cooperation.

We therefore introduce a new game, the High-Dimensional (one-shot) Prisoner's Dilemma (HDPD). The goal is to give a variant of the one-shot Prisoner's Dilemma that is conceptually simple but introduces scalable complexity that makes finding, for example, exact best responses in the diff meta game intractable. In addition to G , the HDPD is parameterized by two functions f C , f D : R n → R m representing the two actions Cooperate and Defect, respectively, as well as a probability measure µ over R n . Each player's action is also a function f i : R n → R m . This is illustrated in Figure 2 for the case of n = 1 and m = 1 . For any pair of actions f 1 , f 2 , payoffs are then determined as follows. First, we sample some x according to µ from R n . Then to determine how much Player i cooperates, we consider the distance d ( f i ( x ) , f C ( x )) to determine, roughly speaking, how much Player 1 cooperates. The larger the distance the less cooperative is f i . In the case of n = m = 1 and µ uniform, the expected distance between f i ( x ) and f D ( x ) is simply the area between the curves of f i and f D , as visualized in Figure 2. We analogously determine how much the players defect. Formally, we define u i ( f 1 , f 2 ) = -E x ∼ µ [ d ( f i ( x ) , f D ( x )) + Gd ( f -i ( x ) , f C ( x ))] / E x ∼ µ [ d ( f C ( x ) , f D ( x ))] . Thus, the action f i = f D corresponds to defecting and the action f C corresponds to cooperating, e.g., u ( f C , f C ) = ( -1 , -1) and u ( f D , f D ) = ( -G, -G ) . The unique equilibrium of this game is ( f D , f D ) . In our experiments, we specifically used G = 5 .

We consider a diff meta game on the HDPD. Formally, a diff-based policy for the HDPD is a function R → ( R m ) ( R n ) . For notational convenience, we will instead write policies as functions R n +1 → R m . We then define our diff function by diff i ( π 1 , π 2 ) = E ( y, x ) ∼ ν [ d ( π 1 ( y, x ) , π 2 ( y, x ))] + Z i , where ν is some probability distribution over R n +1 and Z i is some real-valued noise.

6.3 Experiments

Experimental setup. We trained on the environment from Section 6.2. We selected a fixed set of hyperparameters based on prior exploratory experiments and the theoretical considerations in Appendix E. We then randomly initialized θ 1 and θ 2 , CCDR-pretrained them (independently), and then trained θ 1 and θ 2 against each other using ABR. We repeated the experiment with 28 random seeds. As control, we also ran the experiment without CCDR on 26 seeds. We also ran experiments with Learning with Opponent-Learning Awareness (LOLA) (Foerster et al., 2018), which we report in Appendix G.

Results. First, we observe that in the runs without CCDR pretraining, the players generally converge to mutual defection during alternating best response learning. In particular, in all 26 runs, at least one

(a)

(b)

5.0

1.4

Distance of ne(·,y) to fc + distance of ne(·,y) to fp

4.5

1.2

Distance of ne(.y) to fc,i.e.,Ex[d(fc(x)),ng(x,y)]

Distance of nte(·.y)to f

4.0

1.0

3.5

0.8

SS0

3.0

5

0.6

2.5

0.4

2.0

0.2

1.5

0.0

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0

200

400

600

800

1000

y

Timestep

player's utility was below -5 . Only two runs had a utility above -5 for one of the players ( -4 . 997 and -4 . 554 ). The average utility across the 26 runs and across the two players was -5 . 257 with a standard deviation of 0 . 1978 . Anecdotally, these results are robust - ABR without pretraining practically never finds cooperative equilibria in the HDPD.

Second, we observe that in all 28 runs, CCDR pretraining qualitatively yields the desired policy models, i.e., a policy that cooperates at low values of diff and gradually comes closer to defecting at high values of diff. Figure 3a shows a representative example.

Our main positive experimental result is that after CCDR pretraining, the models converged in alternating best response learning to a partially cooperative equilibrium in 26 out of 28 runs. Thus, the cooperative equilibria postulated in general by Theorem 3 and in simplified examples by Propositions 1 and 2 (as well as Proposition 25), do indeed exist and can be found with simple methods. The minimum utility of either player across the 26 successful runs was -4.854. The average utility across all runs and the two players was about -2.77 and thus a little closer to u ( f C , f C ) = -1 than to u ( f D , f D ) = -5 . The standard deviation was about 1.19. Figure 3b shows the losses (i.e., the negated utilities) across ABR learning. Generally, the policies also converge to receiving approximately the same utility (cf. Section 5). The average of the absolute differences in utility between the two players at the end of the 28 runs is about 0.04 with a standard deviation of 0.05. We see that in line with Theorem 4, we tend to learn egalitarian equilibria in this symmetric, additively decomposable setting. After alternating best response learning, the models generally have a similar structure as the model in Figure 3a, though often they cooperate only a little at low diff values. Based on prior exploratory experiments, CCDR's success is moderately robust.

6.4 Discussion

Without pretraining, ABR learning unsurprisingly converges to mutual defection. This is due to a bootstrapping problem. Submitting a policy of the form 'cooperate with similar policies, defect against different policies' is a unique best response against itself. If the opponent model π -i is not of this form, then any policy π i that defects, i.e., that satisfies π i (diff( π i , π -i )) = f D , is a best response. Because f C is complex, learning a model that cooperates at all is unlikely. (Even if f C was simple, the appropriate use of the perceived diff value would still be specific and thus unlikely to be found by chance.) Similar failures to find the more complicated cooperative equilibria by default have also been observed in the iterated PD (Sandholm & Crites 1996; Foerster et al. 2018; Letcher et al. 2019) and in the open-source PD (Hutter, 2020). Opponent shaping methods have been used successfully to learn to cooperate both in the iterated Prisoner's Dilemma (Foerster et al. 2018; Letcher et al. 2019) and the open-source Prisoner's Dilemma (Hutter, 2020). Our experiments in Appendix G show that LOLA can also learn SBC, but unfortunately not as robustly as CCDR pretraining.

CCDR pretraining reliably finds models that cooperate with each other and that continue to partially cooperate with each other throughout ABR training. This shows that when given some guidance, ABR can find SBC equilibria - SBC equilibria have at least some 'basin of attraction'. Our experiments therefore suggest that SBC is a promising means of establishing cooperation between ML agents.

That said, CCDR has many limitations that we hope can be addressed in future work. For one, in many games the best response against a randomly generated opponents does poorly against a rational opponent. Second, our experiments show that while the two policies almost fully cooperate after CCDR pretraining, they quickly partially unlearn to cooperate in the ABR phase. We would prefer a method that preserves closer to full cooperation throughout ABR-style training. Third, while CCDR seems to often work, it can certainly fail in games in which SBC is possible. Learning to distinguish randomly sampled opponent policies from copies will in many settings not prepare an agent to distinguish cooperative/SBC opponents from uncooperative but trained (not randomly sampled) opponents. Consequently, CCDR may sometimes result in insufficiently steep incentive curves, cooperating with too dissimilar opponents. We suspect that to make progress on the latter issues we need training procedures that more explicitly reason about incentives à la opponent shaping (cf. our experiments with LOLA Appendix G).

7 Related work

We here relate our project to the two most closely related lines of work. In Appendix H we discuss more distantly related lines of work.

Program equilibrium. We already discussed in Section 1 the literature on program meta games in which players submit computer programs as policies and the programs fully observe each other's code (McAfee 1984; Howard 1988; Rubinstein 1998, Section 10.4; Tennenholtz 2004). Interestingly, some constructions for equilibria in program meta games are similarity based. For example, the earliest cooperative program equilibrium for the Prisoner's Dilemma, described in all four of the above-cited papers, is the program 'Cooperate if the opponent's program is equal to this program; else Defect'. The program 'cooperate if my cooperation implies cooperation from the opponent' proposed by Critch et al. (2022) is also similarity-based. Other approaches to program equilibrium cannot be interpreted as similarity based, however (see, e.g., Barasz et al., 2014; Critch, 2019; Oesterheld, 2019b). To our knowledge, the only published work on ML in program equilibrium is due to Hutter (2020). It assumes the programs to have the structure proposed by Oesterheld (2019b) on simple normal-form games, thus leaving only a few parameters open. Similar to our experiments, Hutter shows that best response learning fails to converge to the cooperative equilibria. In Hutter's experiments, the opponent shaping methods LOLA (Foerster et al., 2018) and SOS (Letcher et al., 2019) converge to mutual cooperation.

Decision theory and Newcomb's problem. Brams (1975) and Lewis (1979) have pointed out that the Prisoner's Dilemma against a similar opponent closely resembles Newcomb's problem , a problem first introduced to the decision-theoretical literature by Nozick (1969). Most of the literature on Newcomb's problem is about the normative, philosophical question of whether one should cooperate or defect in a Prisoner's Dilemma against an exact copy. Our work is inspired by the idea that in some circumstances one should cooperate with similar opponents. However, this literature only informally discusses the question of whether to also cooperate with agents other than exact copies (Hofstadter e.g., 1983; Drescher 2006, Ch. 7; Ahmed 2014, Sect. 4.6.3). We address this question formally.

One idea behind the present project, as well as the program game literature, is to analyze a decision situation from the perspective of (actual or hypothetical) principals who design policies. The principals find themselves in an ordinary strategic situation. This is how our analysis avoids the philosophical issues arising in the agent's perspective. Similar changes in perspective have been discussed in the literature on Newcomb's problem (e.g., Gauthier 1989; Oesterheld & Conitzer 2022).

8 Conclusion and future work

We make a strong case for the promise of similarity-based cooperation as a means of improving outcomes from interactions between ML agents. At the same time, there are many avenues for future work. On the theoretical side, we would be especially interested in generalizations of Theorem 4, that is, theorems that tell us what outcomes we should expect in diff meta games. Is it true more generally that under reasonable assumptions about the diff function, we can expect SBC to result in fairly specific, symmetric, Pareto-optimal outcomes? We are also interested in further experimental investigations of SBC. We hope that future work can improve on our results in the HDPD in terms of robustness and degree of cooperation. Besides that, we think a natural next step is to study settings in which the agents observe their similarity to one another in a more realistic fashion. For example, we conjecture that SBC can occur when the agents can determine that their policies were generated by similar learning procedures.

Acknowledgments

We thank Stephen McAleer, Emery Cooper, Daniel Filan, John Mori, and our anonymous reviewers helpful discussions and comments. We thank Maxime Riché and the Center on Long-Term Risk for compute support. Caspar Oesterheld and Vincent Conitzer would like to thank the Cooperative AI Foundation, Polaris Ventures (formerly the Center for Emerging Risk Research) and Jaan Tallinn's donor-advised fund at Founders Pledge for financial support. Roger Grosse acknowledges financial support from Open Philanthropy. Caspar Oesterheld and Johannes Treutlein are grateful for support by FLI PhD Fellowships. Johannes Treutlein was additionally supported by an OpenPhil AI PhD Fellowship.

References

Melissa Acevedo and Joachim I Krueger. Evidential reasoning in the prisoner's dilemma. The American Journal of Psychology , 118(3):431-457, 2005.

Arif Ahmed. Evidence, Decision and Causality . Cambridge University Press, 2014.

Ozan Aksoy. Effects of heterogeneity and homophily on cooperation. Social Psychology Quarterly , 78(4):324-344, 2015.

Max Albert and Ronald Asher Heiner. An indirect-evolution approach to Newcomb's problem, 2001. URL https://www.econstor.eu/bitstream/10419/23110/1/2001-01_newc.pdf . CSLE Discussion Paper, No. 2001-01.

Robert Axelrod. The Evolution of Cooperation . Basic Books, 1984.

Mihaly Barasz, Paul Christiano, Benja Fallenstein, Marcello Herreshoff, Patrick LaVictoire, and Eliezer Yudkowsky. Robust cooperation in the prisoner's dilemma: Program equilibrium via provability logic, 1 2014. URL https://arxiv.org/abs/1401.5577 .

James Bell, Linda Linsefors, Caspar Oesterheld, and Joar Skalse. Reinforcement learning in Newcomblike environments. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems , volume 34, pp. 22146-22157. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper/2021/file/ b9ed18a301c9f3d183938c451fa183df-Paper.pdf .

Steven J. Brams. Newcomb's problem and prisoners' dilemma. The Journal of Conflict Resolution , 19(4):596-612, 12 1975.

Gordon W. Brown. Iterative solutions of games by fictitious play. In Tjalling C. Koopmans (ed.), Activity Analysis of Production and Allocation , chapter XXIV, pp. 371-376. John Wiley & Sons and Chapman & Hall, 1951. URL https://archive.org/details/in.ernet.dli.2015. 39951/ .

CFTC and SEC. Findings regarding the market events of may 6, 2010. report of the staffs of the cftc and sec to the joint advisory committee on emerging regulatory issues., 9 2010. URL https://www.sec.gov/files/marketevents-report.pdf .

Phillip J.K. Christoffersen, Andreas A. Haupt, and Dylan Hadfield-Menell. Get it in writing: Formal contracts mitigate social dilemmas in multi-agent rl. In Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems , AAMAS '23, pp. 448-456. International Foundation for Autonomous Agents and Multiagent Systems, Richland, SC, 2023. ISBN 9781450394321.

Vincent Conitzer and Caspar Oesterheld. Foundations of cooperative AI. Proceedings of the AAAI Conference on Artificial Intelligence , 37(13):15359-15367, Sep. 2023. doi: 10.1609/aaai.v37i13. 26791. URL https://ojs.aaai.org/index.php/AAAI/article/view/26791 .

Andrew Critch. A parametric, resource-bounded generalization of Löb's theorem, and a robust cooperation criterion for open-source game theory. Journal of Symbolic Logic , 84(4):1368-1381, 12 2019. doi: 10.1017/jsl.2017.42.

Andrew Critch, Michael Dennis, and Stuart Russell. Cooperative and uncooperative institution designs: Surprises and problems in open-source game theory, 2022. URL https://arxiv.org/ pdf/2208.07006.pdf .

Caterina Cruciani, Anna Moretti, and Paolo Pellizzari. Dynamic patterns in similarity-based cooperation: An agent-based investigation. Journal of Economic Interaction and Coordination , 12: 121-141, 2017.

Allan Dafoe, Edward Hughes, Yoram Bachrach, Tantum Collins, Kevin R. McKee, Joel Z. Leibo, Kate Larson, and Thore Graepel. Open problems in cooperative AI, 2020. URL https://arxiv. org/pdf/2012.08630.pdf .

Richard Dawkins. The Selfish Gene . Oxford University Press, 1976.

Gary L. Drescher. Good and Real. Demystifying Paradoxes from Physics to Ethics. The MIT Press, 2006.

Ilan Fischer. Friend or foe: subjective expected relative similarity as a determinant of cooperation. Journal of Experimental Psychology: General , 138(3):341, 2009.

Jakob Foerster, Richard Y Chen, Maruan Al-Shedivat, Shimon Whiteson, Pieter Abbeel, and Igor Mordatch. Learning with opponent-learning awareness. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems , pp. 122-130. International Foundation for Autonomous Agents and Multiagent Systems, 2018.

W. Fox, F. Camara, G. Markkula, R. A. Romano, R. Madigan, and N. Merat. When should the chicken cross the road?: Game theory for autonomous vehicle - human interactions. In Proceedings of the 4th International Conference on Vehicle Technology and Intelligent Transport Systems , volume 1, pp. 431-439. SciTePress, 2018. doi: 10.5220/0006765404310439.

Feng Fu, Martin A Nowak, Nicholas A Christakis, and James H Fowler. The evolution of homophily. Scientific reports , 2(1):1-6, 2012.

Andy Gardner and Stuart A. West. Greenbeards. Evolution: International Journal of Organic Evolution , 64(1):25-38, 2010.

David Gauthier. In the neighbourhood of the Newcomb-predictor (reflections on rationality). In Proceedings of the Aristotelian Society, New Series, 1988-1989 , volume 89, pp. 179-194. [Aristotelian Society, Wiley], 1989.

Jeffrey Goldberg, Lívia Markóczy, and G Lawrence Zahn. Symmetry and the illusion of control as bases for cooperative behavior. Rationality and Society , 17(2):243-270, 2005.

William D. Hamilton. The genetical evolution of social behaviour. Journal of theoretical biology , 7 (1):1-52, 1964.

John C Harsanyi, Reinhard Selten, et al. A general theory of equilibrium selection in games. MIT Press Books , 1, 1988.

Torsten Heinrich, Yoojin Jang, Luca Mungo, Marco Pangallo, Alex Scott, Bassel Tarbush, and Samuel Wiese. Best-response dynamics, playing sequences, and convergence to equilibrium in random games. International Journal of Game Theory , 52:703-735, 9 2023. doi: 10.1007/ s00182-023-00837-4.

Douglas Hofstadter. Dilemmas for superrational thinkers, leading up to a luring lottery. Scientific America , 248(6), 6 1983.

V. Howard. Cooperation in the prisoner's dilemma. Theory and Decision , 24:203-213, 5 1988. doi: 10.1007/BF00148954.

Adrian Hutter. Learning in two-player games between transparent opponents, 2020. URL https: //arxiv.org/abs/2012.02671 .

Dmitry Ivanov, Ilya Zisman, and Kirill Chernyshev. Mediated multi-agent reinforcement learning. In Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems , AAMAS'23, pp. 49-57. International Foundation for Autonomous Agents and Multiagent Systems, Richland, SC, 2023. ISBN 9781450394321.

Laurent Keller and Kenneth G Ross. Selfish genes: a green beard in the red fire ant. Nature , 394 (6693):573-575, 1998.

Joachim I Krueger and Melissa Acevedo. Social projection and the psychology of choice. The self in social perception , pp. 17-41, 2005.

Joachim I Krueger, Theresa E DiDonato, and David Freestone. Social projection can solve social dilemmas. Psychological Inquiry , 23(1):1-27, 2012.

Alistair Letcher, Jakob Foerster, David Balduzzi, Tim Rocktäschel, and Shimon Whiteson. Stable opponent shaping in differentiable games. In Proceedings of the 7th International Conference on Learning Representations (ICLR) . OpenReview.net, 2019.

David Lewis. Prisoners' dilemma is a Newcomb problem. Philosophy & Public Affairs , 8(3):235-240, 1979.

Johannes Martens. Hamilton meets causal decision theory, 1 2019. URL https://johannesmartensblog.files.wordpress.com/2019/01/ hamilton-meets-causal-decision-theory-1.pdf .

Dominik Mayer, Johannes Feldmaier, and Hao Shen. Reinforcement learning in conflicting environments for autonomous vehicles. In International Workshop on Robotics in the 21st century: Challenges and Promises , 2016. URL https://arxiv.org/abs/1610.07089 .

Eric Mazumdar, Lillian J. Ratliff, and S. Shankar Sastry. On gradient-based learning in continuous games. SIAM Journal on Mathematics of Data Science , 2(1):103-131, 2020.

Preston McAfee. Effective computability in economic decisions, 5 1984. URL https://www. mcafee.cc/Papers/PDF/EffectiveComputability.pdf .

Dov Monderer and Moshe Tennenholtz. Strong mediated equilibrium. Artificial Intelligence , 173(1): 180-195, 2009.

John F. Nash. Equilibrium points in n-person games. Proceedings of the National Academy of Sciences of the United States of America , 36(1):48-49, 1 1950.

Martin A Nowak and Robert M May. Evolutionary games and spatial chaos. Nature , 359(6398): 826-829, 1992.

Martin A Nowak and Karl Sigmund. Evolution of indirect reciprocity by image scoring. Nature , 393 (6685):573-577, 1998.

Robert Nozick. Newcomb's problem and two principles of choice. In Nicholas Rescher et al. (ed.), Essays in Honor of Carl G. Hempel , pp. 114-146. Springer, 1969. URL http://faculty.arts. ubc.ca/rjohns/nozick_newcomb.pdf .

Caspar Oesterheld. Approval-directed agency and the decision theory of Newcomb-like problems. Synthese , Feb 2019a. ISSN 1573-0964. doi: 10.1007/s11229-019-02148-2.

Caspar Oesterheld. Robust program equilibrium. Theory and Decision , 86(1):143-159, 2 2019b.

Caspar Oesterheld and Vincent Conitzer. Can de se choice be ex ante reasonable in games of imperfect recall? Working paper, 2022. URL https://www.andrew.cmu.edu/user/coesterh/ DeSeVsExAnte.pdf .

Caspar Oesterheld, Abram Demski, and Vincent Conitzer. A theory of bounded inductive rationality. In Rineke Verbrugge (ed.), Proceedings of the Nineteenth Conference on Theoretical Aspects of Rationality and Knowledge , volume 379 of Electronic Proceedings in Theoretical Computer Science , pp. 421-440. Open Publishing Association, 7 2023. doi: 10.4204/eptcs.379.33.

Martin J. Osborne. An Introduction to Game Theory . Oxford University Press, 2004.

David C. Queller, Eleonora Ponte, Salvatore Bozzaro, and Joan E. Strassmann. Single-gene greenbeard effects in the social amoeba dictyostelium discoideum. Science , 299(5603):105-106, 2003.

Rick L Riolo, Michael D Cohen, and Robert Axelrod. Evolution of cooperation without reciprocity. Nature , 414(6862):441-443, 2001.

Ariel Rubinstein. Modeling Bounded Rationality . Zeuthen Lecture Book Series. The MIT Press, 1998.

Tuomas W Sandholm and Robert H Crites. Multiagent reinforcement learning in the iterated prisoner's dilemma. Biosystems , 37(1-2):147-166, 1996.

Barry Sinervo, Alexis Chaine, Jean Clobert, Ryan Calsbeek, Lisa Hazard, Lesley Lancaster, Andrew G. McAdam, Suzanne Alonzo, Gwynne Corrigan, and Michael E. Hochberg. Self-recognition, color signals, and cycles of greenbeard mutualism and altruism. Proceedings of the National Academy of Sciences , 103(19):7372-7377, 2006.

Scott Smukalla, Marina Caldara, Nathalie Pochet, Anne Beauvais, Stephanie Guadagnini, Chen Yan, Marcelo D. Vinces, An Jansen, Marie Christine Prevost, Jean-Paul Latgé, Gerald R. Fink, Kevin R. Foster, and Kevin J. Verstrepen. Flo1 is a variable green beard gene that drives biofilm-like cooperation in budding yeast. Cell , 135(4):726-737, 2008.

Julian Stastny, Maxime Riché, Alexander Lyzhov, Johannes Treutlein, Allan Dafoe, and Jesse Clifton. Normative disagreement as a challenge for cooperative AI. In Cooperative AI workshop , 2021.

Moshe Tennenholtz. Program equilibrium. Games and Economic Behavior , 49(2):363-373, 11 2004.

Maggie E Toplak and Keith E Stanovich. The domain specificity and generality of disjunctive reasoning: Searching for a generalizable critical thinking skill. Journal of educational psychology , 94(1):197, 2002.

Arne Traulsen. Mechanisms for similarity based cooperation. The European Physical Journal B , 63:363-371, 2008. doi: 10.1140/epjb/e2008-00031-3. URL https://link.springer.com/ content/pdf/10.1140/epjb/e2008-00031-3.pdf .

Arne Traulsen and Jens Christian Claussen. Similarity-based cooperation and spatial segregation. Physical Review E , 70(4):046128, 2004.

Amos Tversky and Eldar Shafir. Choice under conflict: The dynamics of deferred decision. Psychological science , 3(6):358-361, 1992.

Jane X Wang, Edward Hughes, Chrisantha Fernando, Wojciech M Czarnecki, Edgar A DuéñezGuzmán, and Joel Z Leibo. Evolving intrinsic motivations for altruistic behavior. arXiv preprint arXiv:1811.05931 , 2018.

Timon Willi, Alistair Letcher, Johannes Treutlein, and Jakob Foerster. COLA: consistent learning with opponent-learning awareness. In International Conference on Machine Learning , pp. 23804-23831. PMLR, 2022.

Guodong Zhang, Yuanhao Wang, Laurent Lessard, and Roger B Grosse. Near-optimal local convergence of alternating gradient descent-ascent for minimax optimization. In Gustau Camps-Valls, Francisco J. R. Ruiz, and Isabel Valera (eds.), Proceedings of The 25th International Conference on Artificial Intelligence and Statistics , volume 151 of Proceedings of Machine Learning Research , pp. 7659-7679. PMLR, 28-30 Mar 2022. URL https://proceedings.mlr.press/v151/ zhang22e/zhang22e.pdf .

Štˇ epán Veselý. Psychology of decision making: Effect of learning, end-effect, cheap talk, and other variables influencing decision making in the prisoner's dilemma game, 4 2011. URL https://is.muni.cz/th/yz40u/diplomka.pdf . Diploma thesis.

1.0

0.8

0.6

0.4

0.2

00

0.2

0.4

0.4

0.2

0.0

0.2

0.4

0.6

0.8

1.0

A Preliminary game theory results

We say that σ very weakly Pareto-dominates ˆ σ if for all i , we have that u i ( σ ) ≥ u i (ˆ σ ) .

Proposition 5. Let Γ be a two-player additively decomposable normal-form game. Then Γ has a Nash equilibrium that very weakly Pareto-dominates all other Nash equilibria. If Γ is furthermore symmetric, then in the Pareto-dominant equilibrium, both players receive the same utility.

Proof. First note that for all σ -i , Player i 's best responses are given by

Now among this set of universal best responses, let let a ∗ i be one that maximizes u -i,i ( a i ) for i = 1 , 2 . Clearly a ∗ is a Nash equilibrium.

Now let σ be any Nash equilibrium. Note that for i = 1 , 2 the support of σ i must be in the above argmax. It follows that for i = 1 , 2 ,

B A detailed analysis of Example 1

Example 1 is already surprisingly rich. We here provide a detailed analysis.

Example 1. Let Γ be the Prisoner's Dilemma as per Table 1. Then consider the (Γ , ˆ A 1 , ˆ A 2 , diff) meta game where ˆ A i = { ( C, θ i , D ) | θ i ∈ R } and diff i (( C, θ 1 , D ) , ( C, θ 2 , D ))) = | θ 1 -θ 2 | + Z i for i = 1 , 2 where Z i is some real-valued random variable.

Figures 4, 5a and 5b illustrate Example 1. Specifically, Figure 4 considers the case without noise ( Z i = 0 ) and shows for each pair of thresholds θ 1 , θ 2 whether both agents cooperate (yellow), only one agent (the one with the lower threshold) cooperates (orange), or both defect (blue). Figure 5a and Figure 5b consider the case Z i ∼ Uniform([0 , 1]) , i.e., the case where noise is drawn uniformly from [0 , 1] . Figure 5a shows for each pair of thresholds the minimum probability of cooperation across the two players. For instance, if Player 1 submits 0 . 5 and Player 2 submits 1 , then Player 1 cooperates with probability 0 and and Player 2 cooperates with probability 1 / 2 , so the plot in Figure 5a is 0 (blue) at (0 . 5 , 1) (and symmetrically at (1 , 0 . 5) ). Figure 5b shows the action of the agent whose threshold is given by the x axis.

Because we here restrict attention to policies of type ( C, θ, D ) , policies are uniquely specified by a single real number θ . So we will denote them as such.

In addition to threshold policies that correspond to real numbers, we will here consider the agents -∞ by which we mean the agent that always defects, and the agent ∞ , by which we mean the agent that always cooperates.

(a)

(b)

1.5

1.5

1.0

6°0

1.0

0.9

0.8

0.8

0.7

0.7

0.6

0.6

0.5

0.5

0.5

0.5

0.4

0.4

0.3

0.3

0.2

0.2

0.0

0.1

0.0

0.1

0.5

0.5

0.5

0.0

0.5

1.0

1.5

0.5

00

0.5

1.0

1.5

One might suspect that if there is too much noise, there can be no cooperative equilibria. But it's easy to see that the setting of Example 1 is scale-invariant.

Proposition 6 (Scale invariance of noise) . Let (Γ , A 1 , A 2 , diff) be a diff -based meta game with utility V , where diff( θ 1 , θ 2 ) = | θ 1 -θ 2 | + Z i . Further, let (Γ , A 1 , A 2 , diff ′ ) be a diff -based meta game with utility V ′ , where diff ′ ( θ 1 , θ 2 ) = | θ 1 -θ 2 | + αZ i for some α > 0 . Then for all θ 1 , θ 2 , V ( θ 1 , θ 2 ) = V ′ ( αθ 1 , αθ 2 ) . It follows that for all θ 1 , θ 2 , ( θ 1 , θ 2 ) is a Nash equilibrium in the diff meta game if and only if ( αθ 1 , αθ 2 ) is a Nash equilibrium in the diff ′ meta game.

B.1 Best responses

In the regular Prisoner's Dilemma, defecting strictly dominates cooperating. Similarly, in the diff meta game of Example 1, always defecting strictly dominates always cooperating (without looking at the difference to the opponent).

Definition 4. Let ( A 1 , A 2 , u ) be a normal-form game. Let a 1 , a ′ 1 ∈ A 1 be strategies for Player 1. We say that a 1 very weakly dominates a ′ 1 if for all a 2 ∈ A 2 we have that u 1 ( a 1 , a 2 ) ≥ u 1 ( a 1 , a 2 ) . We further say that a 1 weakly dominates a ′ 1 if the inequality is strict for at least one a 2 and that a 1 strictly dominates a 1 if the inequality is strict for all a 2 .

Proposition 7. The threshold policy -∞ strictly dominates the threshold policy ∞ .

Intuitively, in our model there is never a reason submit a policy that defects when it can be sure that it faces an exact copy. If the noise is lower-bounded, this puts a lower bound on what kind of agent it makes sense to submit, as we now show.

Proposition 8. Let Z i ≥ θ i with certainty. Let θ ′ i < θ i . Then θ i very weakly dominates θ ′ i . If P ( Z = θ i ) > 0 , θ i weakly dominates θ ′ i .

The next result shows that if the player who submits the higher threshold decreases her threshold while still staying above the other player's threshold, she cooperates with the same probability. Conversely, one cannot (in the setting of Example 1) decrease one's probability of threshold by increasing one's threshold.

Lemma 9. Let θ i , θ ′ i , θ -i ∈ R with θ i ≥ θ ′ i ≥ θ -i . Then in ( θ i , θ -i ) , Player i cooperates with equal probability as in ( θ ′ i , θ -i ) .

Proof.

Theorem 10. Let θ i , θ -i ∈ R with θ i > θ -i . Then u i ( θ -i , θ -i ) ≥ u i ( θ i , θ -i ) . The inequality is strict if and only if P (2 θ -i -θ i ≤ Z -i ≤ θ -i ) > 0 .

Intuitively, there's never a reason to submit a higher threshold than the opponent.

Proof. With Lemma 9, we need only prove that by decreasing θ i to θ -i the probability that Player 2 cooperates (weakly) increases. This is easy to see, though for the strictness condition, we need the details:

Clearly, P ( Z -i ≤ θ -i ) ≥ P ( Z -i ≤ 2 θ -i -θ i ) . Moreover, the inequality is strict if and only if P (2 θ -i -θ i ≤ Z -i ≤ θ -i ) > 0 .

B.2 (Pure) Nash equilibria

We now give some results on the Nash equilibria of Example 1. We start with two simple results to warm up.

Proposition 11. For all distributions of the noise:

( -∞ , -∞ ) is a Nash equilibrium.

( ∞ , ∞ ) is not a Nash equilibrium.

Proposition 12. If there is no upper bound to noise, then there is no fully cooperative equilibrium.

Proof. If there is no upper bound to noise, then the only policy profile with universal cooperation is ( ∞ , ∞ ) . But by Proposition 11.2, this is not a Nash equilibrium.

Next we use our results on best responses to show that to form a Nash equilibrium it is never necessary for the two players to submit different thresholds.

Theorem 13. Let ( θ i , θ -i ) be a Nash equilibrium with θ i > θ -i . Then ( θ -i , θ -i ) is also a Nash equilibrium.

Proof. WLOG assume i = 1 for notational clarity. Assume for contradiction that ( θ 2 , θ 2 ) is not a Nash equilibrium. First, notice that by Theorem 10 and the assumption that θ 1 is a best response for Player 1 to θ 2 , it follows that for Player 1 θ 2 is a best response to θ 2 . So if ( θ 2 , θ 2 ) is not a Nash equilibrium, then it must be because for Player 2 θ 2 is not a best response to θ 2 as submitted by Player 1 . So there must be θ ′ 2 such that u 2 ( θ 2 , θ ′ 2 ) > u 2 ( θ 2 , θ 2 ) . By Theorem 10, θ ′ 2 < θ 2 .

We now show that we would then also have that u 2 ( θ 1 , θ ′ 2 ) > u 2 ( θ 1 , θ 2 ) in contradiction with the assumption that ( θ 1 , θ 2 ) is a Nash equilibrium. We do this via the following sequence of (in)equalities:

By Lemma 9, Player 1 cooperates with equal probability in ( θ 1 , θ ′ 2 ) and ( θ 2 , θ ′ 2 ) . It is easy to see that Player 2's probability of cooperating is weakly lower in ( θ 1 , θ ′ 2 ) . It follows that u 2 ( θ 1 , θ ′ 2 ) ≥ u 2 ( θ 2 , θ ′ 2 ) .

(2) (A) By Lemma 9, Player 1 cooperates with equal probability in ( θ 1 , θ 2 ) and ( θ 2 , θ 2 ) . (B) From A and the assumption that θ 1 ∈ BR 1 ( θ 2 ) it follows that Player 2 cooperates with equal probability in ( θ 1 , θ 2 ) and ( θ 2 , θ 2 ) . (Because if this were not the case, then θ 2 would be a strictly better response for Player 1 to θ 2 .) From A and B it follows that the distributions over actions are the same in ( θ 1 , θ 2 ) and ( θ 2 , θ 2 ) and thus that u 2 ( θ 2 , θ 2 ) = u 2 ( θ 1 , θ 2 ) as claimed.

We are now ready to show the first of our two results about the main text.

Proposition 1. Consider Example 1 with Z i ∼ Uniform([0 , ϵ ]) i.i.d. for some ϵ > 0 and with G ≥ 2 . Then (( C, θ 1 , D ) , ( C, θ 2 , D )) is a Nash equilibrium if and only if θ 1 , θ 2 ≤ 0 or 0 < θ 1 = θ 2 ≤ ϵ . In case of the latter, the equilibrium is strict if G > 2 .

Proof. " ⇐ ": First we show that the given strategy profiles really are equilibria.

1. θ 1 , θ 2 ≤ 0 is just the like the earlier ( -∞ , -∞ ) equilibrium. If one player plays θ -i ≤ 0 , then clearly the unique best response is to also always defect.

By Theorem 10 we only need to consider whether one of the players, WLOG Player 1, can increase her utility by decreasing their threshold. So for the following consider θ 1 < θ 2

This is equal to max(0 , (2 θ 1 -θ 2 ) /ϵ ) . Clearly, if Player 1 can profitably deviate to some θ 1 , then she can profitably deviate to some θ 1 s.t. (2 θ 1 -θ 2 ) /ϵ is nonnegative. After all, Player 1 wants to maximize Player 2's probability of cooperation.

Similarly,

Now θ 1 = θ 2 is a best response to θ 2 if and only if the rate at which P ( Pl. 1 C's | θ 1 , θ 2 ) decreases is at most G times as high as the rate at which P ( Pl. 2 C's | θ 1 , θ 2 ) decreases. Now the rates of change / derivatives are 2 /ϵ and 1 /ϵ . So this condition is satisfied (for our payoff matrix).

" ⇒ ": It is left to show that no other profile is a Nash equilibrium.

First, notice that for all θ -i > ϵ , the unique best response is θ i = ϵ , which minimizes the probability of i cooperating, while ensuring that Player -i cooperates with probability 1 . For this, use part 2 of " ⇐ ". From this it follows directly that there is no equilibrium in which both players play > ϵ . By the strictness part of ⇐ , all equilibria in which one player plays ≤ ϵ are as described in the result.

We now prove a lemma in preparation for proving our second result for the main text.

Lemma 14. Assume G = 2 and assume that the two players have the same noise distribution. Then ( θ, θ ) is a Nash equilibrium if and only if for all ∆ > 0 , P ( θ -2∆ < Z < θ -∆) ≤ P ( θ -∆ < Z < θ ) . It is a strict Nash equilibrium if all of these inequalities are strict.

Proof. By Theorem 10 we only need to consider deviations to a lower threshold. So consider WLOG the case where Player 1 deviates from θ to submit θ -∆ . First, we calculate the probabilities of cooperation under ( θ, θ ) and ( θ -∆ , θ ) :

Thus by Player 1 switching from θ to θ -∆ , Player 1 's probability of cooperating decreases by

Meanwhile, Player 2 's probability of cooperating decreases by

Thus, for this switch to not be profitable for player 1 , it needs to be the case that

or, equivalently, as claimed.

Proposition 2. Consider Example 1 with G = 2 . Assume Z i is i.i.d. for i = 1 , 2 according some unimodal distribution with mode ν with positive measure on every interval. Then (( C, θ 1 , D ) , ( C, θ 2 , D )) is a Nash equilibrium if and only if θ 1 = θ 2 ≤ ν .

Proof. By Theorem 10 and the assumption of positive measure on any interval, all Nash equilibria have the form θ 1 = θ 2 . The second part follows directly from Lemma 14 and the fact that the noise distribution is unimodal with mode ν .

B.3 A different type of noise

Intuitively, we might expect that more noise is an obstacle to similarity-based cooperation. The above results do not vindicate this intuition (see Proposition 6). We here give an alternative setup with a different kind of noise in which more noise is an obstacle to cooperation.

Example 2. Consider a variant of Example 1 where for i = 1 , 2 we have with probability p i that diff i (( C, θ 1 , D ) , ( C, θ 2 , D ))) = | θ 1 -θ 2 | + Z i with Z i ∼ Unif([0 , ϵ ]) for some ϵ > 0 ; and with the remaining probability diff i (( C, θ 1 , D ) , ( C, θ 2 , D ))) = 0 .

Note that for p 1 = p 2 = 1 the setting is exactly the setting of Proposition 1.

Intuitively, this models a scenario in which each player can try to manipulate the diff value to 0 and the manipulation succeeds with probability 1 -p i . (It is further implicitly assumed, that if manipulation fails, the other player never learns of the attempt to manipulate. Instead, the diff value is observed normally if manipulation fails. That way we can assume that each player always attempts to manipulate.)

We can generalize Proposition 1 to this new setting as follows:

Proposition 15. In Example 2, (( C, θ 1 , D ) , ( C, θ 2 , D )) is a Nash equilibrium if and only if

θ 1 , θ 2 ≤ 0 ; or

0 < θ 1 = θ 2 ≤ ϵ and Gp i ≥ 0 for i = 1 , 2 .

The proof works the same as the proof of Proposition 1.

C Proofs for Section 4

C.1 Nash equilibria of the base game as Nash equilibria of the meta game

We first note two simple results. The first is that every Nash equilibrium of the base game is also a Nash equilibrium of the diff meta game in which both players submit a policy that simply ignores the diff value.

Proposition 16. Let Γ be a game and σ be a Nash equilibrium of Γ . For i = 1 , 2 , let A i be any set of policies that contains the policy π i : d ↦→ σ i . Then ( π 1 , π 2 ) is a Nash equilibrium of the (Γ , diff , A 1 , A 2 ) meta game.

If the diff function is uninformative, then the Nash equilibria are in fact the only Nash equilibria of the diff meta game, as we now state.

Proposition 17. Let Γ be a game and (Γ , diff , A 1 , A 2 ) be a meta game on Γ where diff( · , · ) = y for some y 2 . Then ( π 1 , π 2 ) is a Nash equilibrium of the meta game if and only if ( π 1 ( y 1 ) , π 2 ( y 2 )) is a Nash equilibrium of Γ .

C.2 Proof of Theorem 3

Theorem 3 (folk theorem for diff meta games) . Let Γ be a game and σ be a strategy profile for Γ . Let A i ⊇ ¯ A i for i = 1 , 2 . Then the following two statements are equivalent:

There is a diff function such that there is a Nash equilibrium ( π 1 , π 2 ) of the diff meta game (Γ , diff , A 1 , A 2 ) s.t. ( π 1 , π 2 ) play σ .

The strategy profile σ is individually rational (i.e., better than everyone's minimax payoff).

The result continues to hold true if we restrict attention to deterministic diff functions with diff 1 = diff 2 and diff i ( π 1 , π 2 ) ∈ { 0 , 1 } for i = 1 , 2 .

Proof. '1 ⇒ 2': We show the contrapositive, i.e., that if σ is not individually rational, it is not implemented by any equilibrium of any diff game. Let ˆ σ i be the strategy of Player i that guarantees her her minimax utility. Then submitting (ˆ σ i , θ, ˆ σ i ) for any threshold θ guarantees minimax utility regardless of what diff function is used. Thus, anything that gives i less than threat point utility cannot be an equilibrium.

̸

'2 ⇒ 1': Let ˜ σ i be Player i 's minimax strategy against Player -i . Then consider the strategy profile ( π 1 = (˜ σ 1 , 1 / 2 , σ 1 ) , π 2 = (˜ σ 2 , 1 / 2 , σ 2 )) and any diff function s.t. diff( π 1 , π 2 ) = 0 and for i = 1 , 2 and π ′ -i = π -i , diff( π i , π ′ -i ) = 1 . Clearly, in ( π 1 , π 2 ) , the players play ( σ 1 , σ 2 ) . Finally, ( π 1 , π 2 ) is a Nash equilibrium of the resulting diff meta game, because if either player deviates they will receive their minimax utility, which is by assumption no larger than their utility in σ and thus in ( π 1 , π 2 ) .

D On the uniqueness theorem

D.1 Examples to show the need for the assumptions of Theorem 4

D.1.1 Why the diff function must be high-value uninformative in Theorem 4

We now give an example for why we need diff to be high-value uninformative, both for Lemma 22 and for our uniqueness theorem below.

Proposition (Example) 18. Consider the game of Table 2. Note that the game is symmetric and additively decomposable. Consider the diff function defined by diff(( σ ⩽ 1 , ∗ , σ > 1 ) i =1 , 2 ) = 1 if supp( σ ⩽ 1 ) ∪ supp( σ > 1 ) and supp( σ ⩽ 2 ) ∪ supp( σ > 2 ) are disjoint and diff(( σ ⩽ 1 , ∗ , σ > 1 ) i =1 , 2 ) = 0 otherwise. Then (( a 2 , 1 / 2 , a 1 ) , ( a 0 , ∗ , a 0 )) is an equilibrium of the diff meta game.

Intuitively, the policy ( a 2 , 1 / 2 , a 1 ) with the described diff function implements the following idea: 'I want to play a 1 (which is good for me and moderately bad for you). I don't want you to also play a 1 . If you are similar to me (which you are if you give weight to the same action I give weight to), I'll play a 2 , which is very bad for you.' Assuming that Player 1 submits such a policy, Player 2 optimizes her utility by always playing a 0 . Player 1 thus obtains her favorite outcome.

D.1.2 Why the game must be additively decomposable in Theorem 4

The following example shows why we need to restrict attention to additively decomposable games. Intuitively, the game is a Prisoner's Dilemma, except that if the players cooperate, they also play a Game of Chicken for an additional payoff. Then (with a natural diff function) similarity-based cooperation takes care of the cooperate versus defect part, but leaves open the Dare versus Swerve part. In particular, there are multiple Pareto-optimal equilibria.

Proposition (Example) 19. Let Γ be the game of Table 3. Define diff( π 1 , π 2 ) = 0 if a 0 / ∈ supp( π i (0)) for i = 1 , 2 and diff( π 1 , π 2 ) = 1 otherwise. Then for i = 1 , 2 ,

is a Pareto-optimal Nash equilibrium of the diff meta game on Γ .

Player 2

D.1.3 Why diff must be observer-symmetric for Theorem 4

We here give an example to show why the diff function in Theorem 4 needs to have (diff 1 ( π 1 , π 2 ) , diff 2 ( π 1 , π 2 )) and (diff 2 ( π 1 , π 2 ) , diff 1 ( π 1 , π 2 )) have the same distribution. One might call this observer symmetry.

Proposition (Example) 20. Let Γ be the Prisoner's Dilemma. Let diff 1 ( π 1 , π 2 ) = 0 if π 1 = π 2 and diff 1 ( π 1 , π 2 ) = 1 otherwise. Let diff 2 ( π 1 , π 2 ) be defined in the same way, except that if π 1 = π 2 , then there is still an ϵ probability of diff 2 ( π 1 , π 2 ) = 1 -ϵ . Note that this diff meta game satisfies the other conditions of Theorem 4, i.e., Γ is symmetric and additively decomposable, diff( π 1 , π 2 ) and diff( π 2 , π 1 ) are equally distributed for all π 1 , π 2 and diff is high-value-uninformative and minimized by copies. Then (( C, 1 / 2 , D ) , ( C, 1 / 2 , D )) has asymmetric payoffs but is a Pareto-optimal Nash equilibrium of the diff meta game on Γ .

In this example, the Pareto-optimal Nash equilibrium is still unique, but it is easy to come up with examples in which there are multiple Pareto-optimal Nash equilibria.

Note also that in this example the player who has less information about the other does better in the cooperative equilibria.

D.1.4 Why diff must be policy-symmetric for Theorem 4

Finally we give an example to show why the diff function in Theorem 4 needs to have diff( π 1 , π 2 ) and diff( π 2 , π 1 ) have the same distribution. One might call this policy symmetry.

Proposition (Example) 21. Let Γ be the game of Table 4. Let diff( π 1 , π 2 ) = (0 , 0) if ( π 1 , π 2 ) equals one of the following

and diff( π 1 , π 2 ) = (1 , 1) otherwise. Note that this diff meta game satisfies the other conditions of Theorem 4, i.e., Γ is symmetric and additively decomposable, diff 1 ( π 1 , π 2 ) = diff 2 ( π 1 , π 2 ) for all π 1 , π 2 and diff is high-value-uninformative and minimized by copies. However, the only Pareto-optimal (pure) Nash equilibrium of the diff meta game is (( C 2 , 1 / 2 , D ) , ( C 1 , 1 / 2 , D )) .

As in the previous example, the Pareto-optimal Nash equilibrium is still unique, but it is easy to come up with examples in which there are multiple Pareto-optimal Nash equilibria.

D.2 Results on the structure of equilibria

We have various intuitions about similarity-based cooperation. For example, we have the intuition that ( C, θ, D ) is a sensible policy but ( D,θ,C ) is not. In this section we prove results of this type

under appropriate assumptions. We find these results interesting independently, but we also need all results here to prove Theorem 4.

The following two lemmas capture the idea that under some assumptions it is rational to be more cooperative if the opponent is similar, i.e., if the observed difference is below the threshold.

Lemma 22. Let ( π 1 , π 2 ) be a Nash equilibrium of the meta game and diff be high-value uninformative. Then max σ i u i ( σ i , σ max π -i ) ≤ u i ( π 1 , π 2 ) for i = 1 , 2 .

Proof. Assume for contradiction that there is some strategy σ ′ i s.t. u i ( σ ′ i , σ max π -i ) > u i ( π 1 , π 2 ) . Because diff is high-value uninformative, there exists a policy π ′ i s.t. ( π ′ i , π -i ) resolves to a strategy profile arbitrarily close to σ ′ i , σ max π -i . Hence, u i ( π ′ i , π -i ) > u i ( π 1 , π 2 ) and so ( π 1 , π 2 ) cannot be a Nash equilibrium after all.

Lemma 23. Let Γ additively decompose into ( u i,j : A i → R ) i,j ∈{ 1 , 2 } and let diff be high-value uninformative. Let ( σ ⩽ i , θ i , σ > i ) i =1 , 2 be a Nash equilibrium. Then u i, -i ( σ > -i ) ≤ u i, -i ( σ ⩽ -i ) for i = 1 , 2 .

Proof. Follows directly from Lemma 22.

Lemma 24. Let Γ be symmetric and additively decomposable. Let diff be symmetric, high-value uninformative and minimized by copies. Let ( π i = ( σ ⩽ i , ∗ , σ > i )) i =1 , 2 be a Nash equilibrium of the diff meta game that induces strategies σ 1 , σ 2 . If σ i = σ > i for some i , then ( σ 1 , σ 2 ) is a Nash equilibrium of Γ .

Proof. With high-value uninformativeness, it follows immediately that σ -i is a best response to σ i . The case that σ -i = σ > -i is trivial, so we focus on the case where σ -i gives positive probability to σ ⩽ -i . It then follows that σ ⩽ -i is a best response to σ i . Because the game is additively decomposable, this means that σ ⩽ -i (independent of the opponent's strategy) maximizes -i 's utility (i.e., σ -i ∈ arg max σ ′ -i ∈ ∆( A -i ) u -i, -i ( σ ′ -i ) ). So in particular, σ -i is a best response to σ i .

It is left to show that σ i is a best response to σ -i . We will argue that if σ i is not optimizing Player i 's utility (i.e., if σ i / ∈ arg max σ ′ i ∈ ∆( A i ) u i,i ( σ ′ i ) ), then Player i could better-respond to π -i by also playing π -i instead of π i . Let σ ′ -i be the strategy induced for both players by ( π -i , π -i ) . Because diff is minimized by copies, σ ′ -i gives weakly more weight to σ ⩽ -i than σ -i . By Lemma 23, this means that u i, -i ( σ ′ -i ) ≥ u i, -i ( σ -i ) . Second, by the assumption that σ i doesn't optimize i 's utility but σ -i and σ ⩽ -i do, it follows that u i,i ( σ i ) < u i,i ( σ ′ -i ) .

Putting it all together we obtain that

as claimed.

D.3 Proof of Theorem 4

Theorem 4. Let Γ be a player-symmetric, additively decomposable game. Let diff be symmetric, high-value uninformative, and minimized by copies. Then if ( π 1 , π 2 ) is a Nash equilibrium that is not Pareto-dominated by another Nash equilibrium, we have that V 1 ( π 1 , π 2 ) = V 2 ( π 1 , π 2 ) . Hence, if there exists a Pareto-optimal Nash equilibrium, its payoffs are unique, Pareto-dominant among Nash equilibria and equal across the two players.

For the proof we define for additively decomposable games, u Σ ,j := u 1 ,j + u 2 ,j : A j → R . Intuitively, u Σ ,j denotes the utilitarian welfare generated by Player j 's actions. In symmetric games, u Σ , 1 = u Σ , 2

so that we can simply write u Σ . For example, in the Prisoner's Dilemma u Σ : Cooperate ↦→ G, Defect ↦→ 1 .

Proof. We will prove that if ( π i = ( σ ⩽ i , θ i , σ > i )) is a Pareto-optimal equilibrium of the meta game, then both players receive the same utility. The uniqueness of the Pareto-optimal equilibrium follows immediately.

We prove this in turn by contradiction. So assume that ( π 1 , π 2 ) is a Pareto-optimal equilibrium of the meta game and that the two players receive different utilities.

Assume WLOG that in ( π 1 , π 2 ) Player 1 receives higher utility. Let σ 1 , σ 2 be the strategies played in ( π 1 , π 2 ) . Then we distinguish two cases:

Player 1 'takes' more than Player 2, i.e.,

Player 1 does not take more but Player 2 'gives' more than Player 1, i.e.,

It is easy to see that one of these cases must obtain.

A) We in turn distinguish two cases:

A.1) First consider the case where and

We will show that in this case ( π 1 , π 2 ) cannot be a Nash equilibrium. Player 2 can better-respond by playing the policy ˜ π 1 that plays σ > 1 and maximizes (as per the high value uninformativeness condition) Player 1's probability of playing σ > 1 . This can be seen as follows:

A.2) Now consider the case where

We will show that in this case Player 2 can better respond by also playing π 1 instead of π 2 , such that ( π 1 , π 2 ) (again) cannot be a Nash equilibrium.

Let σ ′ 1 be the strategy played by both players in ( π 1 , π 1 ) . Note that because diff is minimized by copies, σ ′ 1 gives at least as much weight to σ ⩽ 1 as σ 1 .

We again distinguish two cases.

B.1) First consider the case where

We will show that in this case ( π 2 , π 2 ) is also a Nash equilibrium and that ( π 2 , π 2 ) Pareto-dominates ( π 1 , π 2 ) .

First, we show Pareto dominance. Let σ ′ 2 be the strategy played by both players in ( π 2 , π 2 ) . Because diff is minimized by copies, σ ′ 2 gives at least as much weight to σ ⩽ 2 as σ 2 . Then for Player 1 we have that

Player 2's utility is strictly higher in ( π 2 , π 2 ) , which we can see as follows:

It is left to show that ( π 2 , π 2 ) is a Nash equilibrium. By assumption, π 1 is a best response to π 2 . Line 7 therefore implies that π 2 is also a best response to π 2 . Because of symmetry, this is true for both players. We conclude that ( π 2 , π 2 ) is a Nash equilibrium.

B.2) Let

We now must make one more distinction. Consider first the case where σ 2 = σ > 2 . By Lemma 24, ( σ 1 , σ 2 ) must be a Nash equilibrium of Γ . The contradiction follows immediately from Proposition 5.

̸

Now consider the case where σ 2 = σ > 2 . With Ineq. 8 it follows that

We will show that ( π 1 , π 2 ) is not an equilibrium because Player 1 can better-respond by playing the policy ˜ π 1 that plays σ > 2 and maximizes as per the definition of high-value uninformativeness Player 2's probability of playing σ > 2 . Then

E Theoretical analysis beyond threshold policies

We here analyze a meta game in which players can not only submit threshold policies but continuous functions. The goal is to show an equilibrium based on linear functions, similar to the equilibria found by CCDR pretraining.

A policy now is a function π : R ≥ 0 → [0 , 1] , where π ( x ) denotes π 's probability of cooperation. For differences x 1 , x 2 , the payoff of Player i is given by

1.0

0.8

0.6

0.4

0.2

0.5

1.0

1.5

2.0

2.5

3.0

It is left to specify the difference function. Let ϵ > 0 and K > ϵ . Then define the function difference

Further, define the probabilistic difference mapping diff( π 1 , π 2 ) = d ( π 1 , π 2 )+ Z i , where Z i is drawn uniformly from [0 , ϵ ] . As the set of policies for each player consider the set of integrable functions.

Proposition 25. Let π ( x ) = max(1 -x/a, 0) for some a > ϵ . Then ( π, π ) is a Nash equilibrium if and only if 1 + aK/ϵ ≤ G .

Note that π decreases cooperation linearly in x down to 0 (which is hit at x = a ). This function is shown in Figure 6 for a = 1 . Note that our policies look roughly like this after Step 2.

Proof. Consider ( π, π ) and imagine that, WLOG, Player 1 moves away from π by ∆ , i.e., deviates to play some π ′ s.t. d ( π, π ′ ) = ∆ . It is easy to see that it is enough to consider small deviations. Specifically, we assume ∆ ≤ a -ϵ . First, if the difference between the policies increases by ∆ , what happens to Player 2's expected amount of cooperation? It is easy to see that this decreases by ∆ /a .

Next, we need to ask: by increasing the difference by ∆ , how much can Player 1 increase her probability of defection? We need to consider two effects. First, if the difference increases by ∆ , then automatically Player 1 defects more by the same effect as Player 2. So this gives Player 1 an extra ∆ /a probability of defection. Moreover, Player 1 can decrease the probability of cooperation on the relevant interval [∆ , ∆+ ϵ ] . This decreases the probability of cooperation by (at most) K ∆ /ϵ .

Taking stock, Player 1 can increase her probability of defecting by ∆ /a + K ∆ /ϵ at the cost of Player 2 increasing her probability of defecting by ∆ /a . This is good for Player 1 if and only if 1 + aK/ϵ > G .

F Details on our experiments

F.1 Software

We used pytorch for implementing CCDR and ABR and functorch for implementing LOLA. We used floats with double precision (by running torch.set_default_dtype(torch.float64) ), because preliminary experiments had shown numerical issues as ABR converged. We used Weights and Biases ( wandb.ai ) for tracking.

F.2 Game and meta game

We here give some details on the game and diff meta game we consider throughout our experiments.

Constructing f D , f C In our experiments f D , f C have input dimension 10 and output dimension 3 . (Thus, including one dimension for the similarity value, our policies have input dimension 11 .) First we generate s C,i and s D,i for i = 1 , 2 , 3 from { 0 , 1 } 10 uniformly at random. Then we define

We define f D analogously based on s D .

We chose this function because it is very simple to understand and implement and at the same time requires using larger nets. The only other approach we tried in preliminary experiments is to generate f C , f D by randomly generating neural nets. The problem is that large randomly generated fully connected neural nets are close to constant functions.

Constructing µ Recall that for any pair of actions f 1 , f 2 , the payoffs of the HDPD are given by u i ( f 1 , f 2 ) = -E x ∼ µ [ d ( f i ( x ) , f D ( x )) + Gd ( f i ( x ) , f C ( x ))] , where d is the Euclidean distance and µ is some measure of R 10 . Thus, to construct a specific instance of the HDPD we need to also construct µ . We do this by generating 50 vectors uniformly from [0 , 1] 10 and then taking the uniform distribution over these 50 vectors.

Constructing the noisy diff mapping Recall that diff i ( π 1 , π 2 ) = E ( y, x ) ∼ ν [ d ( π 1 ( y, x ) , π 2 ( y, x ))] + Z i , where ν is some probability distribution over R n +1 and Z i is some real-valued noise. We need to specify ν and the distribution Z i . For ν we first generate 50 reals from [0 , 0 . 1] uniformly at random as our test diffs. We increment each of these by a random draw from the underlying noise distribution, i.e., by a number drawn uniformly at random from [0 , 0 . 1] . We then define ν to be the uniform distribution over 50 values that result from pairing the support of µ with these 50 values.

For the noise we generate for each player 50 values uniformly from [0 , 0 . 1] and then use the uniform distribution over these 50 points.

Note that by using the uniform distribution over a finite support, we can compute expected utilities in the meta game exactly.

F.3 Neural net policies

Throughout our experiments, our policies π θ are represented by neural networks with three fully connected hidden layers of dimensions 100, 50 and 50 with biases and LeakyReLU activation functions. Thus, these networks overall have 11+100+50+50+3+11 · 100+100 · 50+50 · 50+50 · 3 = 8964 parameters.

F.4 Methods as used in our experiments

CCDR pretraining We here describe in more detail the CCDR pretraining step in our results. Recall that CCDR pretraining consists in maximizing V d ( π θ i , π θ i ) + V d ( π θ i , π ′ θ -i ) for randomly generated opponents π ′ θ -i . Call -V d ( π θ i , π θ i ) + V d ( π θ i , π ′ θ -i ) the CCDR loss.

In our experiments, we maximized this by running Adam for 100 steps. In each step, the CCDR loss is calculated by averaging over 100 randomly generated opponents. The learning rate is 0.02.

Alternating best response training We ran alternating best response training for T = 1000 turns. We need to specify how we updated θ i to maximize V ( θ i , θ -i ) (holding θ -i fixed) in each turn. For this we run gradient descent for T ′ = 1000 steps. However, we only take gradient steps that are successful, i.e., that in fact reduce loss. The learning rate γ ′ is sampled uniformly from [0 , γ = 0 . 00003] in each step. Note that by randomly sampling the learning rate, the algorithms avoids getting stuck when a gradient step is unsuccessful. We summarize this in Algorithm 1.

F.5 Convergence to spurious stable points?

In theory, alternating best response learning could converge to a 'very local Nash equilibrium', i.e., a pair of models that are best responses only within a very small neighborhood of these models. One might also worry about convergence to other stable points (Mazumdar et al., 2020).

However, as far as we can tell, the limit points of our learning procedure are not spurious. To confirm this, we performed the following test (implemented by the function best_response_test of our code). For a given pair of models, we pick either model and perturb each of its parameters a little. We then see whether the perturbed model is a better response to the opponent model than the original model. In an (approximate) local Nash equilibrium, this should almost never be the case.

In all but one of our successful runs (the ones converging to partial cooperation), none of 10,000 random perturbations of each of the models after alternating best response training led to a better response to the opponent model. In one run, three perturbations of one of the models decreased loss.

When applying the test after CCDR pretraining but before alternating best response training, close to half of random perturbations improve utility.

F.6 Compute costs

We here provide details on how computationally costly our experiments are. To do so, we ran the experiment for a single random seed on an AMD Ryzen 7 PRO 4750U, a laptop CPU launched in 2020. (Note that we used the CPU not GPU (CUDA).) The CCDR pretraining took about 30 seconds per model. The ABR phase took about 3h. (Note that we ran most of our experiments as reported in the paper via remote computing on different hardware.)

G Experiments with LOLA

Wehave tried to learn SBC using Foerster et al.'s (2018) Learning with Opponent-Learning Awareness (LOLA). We here specifically report on an experiment in which we tried a broad range of parameters. The results are in line with prior exploratory experiments.

LOLA sometimes succeeds in learning SBC (without CCDR pretraining). It finds similar policies as CCDR pretraining. Some of the SBC models found by LOLA remain cooperative throughout a subsequent ABR phase.

Unfortunately, none of our LOLA results are nearly as robust as the CCDR results reported in the main text and in Appendix F. Specifically, they are not even robust to changing the random seed. One reason for this is that LOLA is capable of 'unlearning' LOLA-learned SBC.

G.1 Experimental setup

We ran Foerster et al.'s (2018) Learning with Opponent-Learning Awareness (LOLA). We tried every combination setting the learning rate and lookahead parameter in LOLA to values in 0.0003, 0.001, 0.003, 0.01, 0.03, 0.1, 0.3, 1. For each combination we tested both exact and Taylor LOLA as described by Willi et al. (2022, Sect. 3.2). We tested each of these 8 · 8 · 2 = 128 combinations with 5 different random seeds. In each of these experiments we initialized the nets randomly and then ran LOLA for both agents for 30,000 steps. We then ran ABR training as described in Appendix F.4 for at most 500 steps, to test whether the resulting policies are in equilibrium. To save compute, we generally aborted ABR when the loss hit the Defect-Defect utility of 5 or when it was clear that ABR learning had converged.

We then labeled each individual run as a 'weak LOLA success' if the loss after the LOLA phase was smaller for both players than the loss of mutual defection and as a LOLA-ABR success if the loss after the ABR phase was smaller for both players than the loss of mutual defection.

For each parameter configuration in which at least 3 out of 5 runs were a weak LOLA success we ran another 20 runs (with different random seeds) without the ABR phase to determine how robust the success is. Similarly, for each parameter configuration in which at least 3 out of 5 runs were a LOLA-ABR success we ran another 20 runs with the ABR phase.

0

5000

10000

15000

20000

25000

30000

Time step

1

2

3

4

5

6

7

8

9

Loss

G.2 Qualitative analysis

We start with a qualitative discussion of our results, because the experiments with LOLA exhibit a much wider range of complex phenomena than the CCDR-based results.

G.2.1 A few clear-cut successful runs

Out of all the runs in our experiment, a few are as clear-cut successes as most of the CCDR runs. Figure 7 shows the learning curve of one of these runs across both the LOLA and at the end the ABR phase. It uses a lookahead of 0 . 001 , learning rate of 0 . 001 and exact LOLA. The players learn to cooperate with each other in the LOLA phase. When the switch to ABR is made after 30k steps of LOLA, cooperation deteriorates slightly, but ultimately the models converge in the ABR phase to an outcome with a loss well below the loss of mutual defection. This shows that LOLA is to some extent capable of solving the given problem.

Unfortunately, both in the present experiment and in preliminary experiments we have found that these results are not even robust to changing the random seed. For example, the four other runs with the same parameters as the run from Figure 7 (but different random seeds) all failed to produce significant positive results. (One resulted in marginal cooperation at the end of the LOLA phase (losses 4.986 and 4.928) that disappeared immediately when switching to ABR learning. Another partially cooperated for some of the LOLA phase but failed to retain cooperation even until the end of the LOLA phase.)

G.2.2 What models LOLA learns

Generally, when LOLA succeeds, it learns similar models as CCDR pretraining. The main difference is that for high diff values, LOLA models typically do not exactly defect. (Of course, it also does not cooperate. Instead if performs an action far from both f C and f D .) This is to be expected, since these agents are not trained on randomly generated opponents. An example model after 30,000 steps of LOLA is shown in Figure 8.

When LOLA does not succeeed, the learned models vary greatly. Some are clearly set up to impose some incentives - others just constantly defect.

G.2.3 Instability in the LOLA phase

The learning in the LOLA phase is often highly unstable. That is, the during the learning phase, the change in losses in a single learning step is often very large. This occurs even in successful runs. As an example, consider the learning curve in Figure 9 (lookahead 0 . 003 , learning rate 0 . 01 , exact LOLA). While it is unclear whether cooperation would have survived further ABR learning, this run is relatively successful. However, the utilities vary by large amounts during the LOLA phase. In fact for parts of the LOLA learning phase, both players' losses are much higher than the loss of mutual defection. Most successful runs with LOLA look like this. Of course, successes based on such runs cannot be very robust: if we had ceased LOLA learning somewhat earlier, the run would not have succeeded. It is unclear what would happen if the run had run for more than 30k LOLA steps.

0.0

0.2

0.4

0.6

0.8

1.0

y

0.0

0.5

1.0

1.5

2.0

2.5

Distance of

( ,

y

) to

fC

+ distance of

( ,

y

) to

fD

Distance of

fD

to

fC

, i.e.,

x

[

d

(

fC

(

x

),

fD

(

x

))]

Distance of

( ,

y

) to

fC

, i.e.,

x

[

d

(

fC

(

x

)),

(

x

,

y

))]

Distance of

( ,

y

) to

fD

0

5000

10000

15000

20000

25000

30000

Time step

1

2

3

4

5

6

7

8

9

10

Loss

G.2.4 Cooperation in the LOLA phase versus cooperation in the ABR phase

While CCDR is explicitly a pre training method, LOLA can also be used as a standalone learning procedure. So for each run, we can ask both whether the players cooperate, say, at the end of the LOLA phase and whether the players cooperate at the end of the ABR phase. It turns out that all four combinations of answers are possible! In particular, there are runs in which LOLA versus LOLA fails to converge to a pair of policies that give both players a higher utility than mutual defection; but in which ABR then converges to an outcome that is marginally better for both players than mutual defection.

G.3 Quantitative analysis

Bearing in mind the different qualitative phenomena, we here give a more quantitative analysis.

Table 5 lists the parameters in which at least three out of five runs exhibited weak LOLA success as defined earlier. They also summarize the result of 20 further runs with these parameters. Specifically, we categorized runs as stable successes if the loss was below the loss of mutual defection for both players for the final 5,000 LOLA steps. We categorized them as unstable successes if the loss was below the loss of mutual defection for both players for the final 50 but not the final 5,000 LOLA steps. The fourth and fifth columns of Table 5 show the numbers of these different kinds of successes out of 20 runs. The sixth colum shows the average loss across both players and all successful runs. The final column shows the standard deviation across the losses of the successful runs (also across players).

Table 6 lists the set of parameter configurations that achieved a LOLA-ABR success in at least three out of five runs. They also summarize the result of 20 further runs with each of these parameter configurations. Specifically, we categorized runs again as stable successes if the loss was below the loss of mutual defection for both players for the final 5,000 LOLA steps and at the end of the ABR phase. We labeled them as unstable successes if the loss was below the loss of mutual defection for both players at the end of the ABR phase but not for the final 5,000 LOLA steps. Again, the

fourth and fifth columns of Table 6 show the numbers of these different kinds of successes out of 20 runs. The sixth column shows the average loss across both players and all successful runs. The final column shows the standard deviation across the losses of the successful runs (also across players).

From these results we conclude that LOLA can succeed under various parameter configurations, but none of the parameter configurations succeed nearly as reliably as CCDR pretraining.

G.4 Compute costs

We again provide details on how computationally costly our experiments are. To do so, we ran the experiment for a single random seed on an AMD Ryzen 7 PRO 4750U. The LOLA phase took about 20 minutes. The ABR phase took about 1h and 30min. Note again we ran most of our experiments as reported in the paper via remote computing on different hardware. Note also that we ran ABR for half as many steps compared to the experiments for the main text (500 instead of 1000). So, the cost per ABR step is roughly the same between the two experiments.

H Distantly related work

H.1 Learning in Newcomb-like decision problems

There is some existing work on learning in Newcomb-like environments that therefore also applies to the Prisoner's Dilemma against a copy. Whether cooperation against a copy is learned generally depends on the learning scheme. Bell et al. (2021) show that Q -learning with a softmax policy learns to defect. Regret minimization also learns to defect. Other learning schemes do converge to cooperating against exact copies (Albert & Heiner, 2001; Mayer et al., 2016; Oesterheld, 2019a; Oesterheld et al., 2023). All schemes in prior work differ from the present setup, however, and to our knowledge none offer a model of cooperation between similar but non-equal agents.

H.2 Cooperation via iterated play, reputation, etc.

Perhaps the best-known way to achieve cooperation in the Prisoner's Dilemma is to play the Prisoner's Dilemma repeatedly (e.g., Axelrod 1984; Osborne 2004, Ch. 14, 15). Clearly, the underlying mechanism (repeated play) is very different from the mechanism underlying SBC, in which the game is played one-shot. That said, our folk theorem (Theorem 3) is similar to the well-known folk theorem for repeated games. (As noted in Section 7, the folk theorem for program equilibrium is also similar.) A number of variants of iterated play have been considered to study, for example, reputation effects, effects of allowing players to choose with whom to play based on reputation, and so on (e.g., Nowak &Sigmund 1998).

While cooperation via repeated play is very different from similarity-based cooperation as studied in this paper, some variants of the former are easy to confuse with the latter. As a simplistic example, consider a variant of the infinitely repeated Prisoner's Dilemma. Typically, it is imagined that players observe the entire history of past play. But now imagine that instead the players in each round only observe a single bit: 0 if they have taken the same action in each round so far and 1 otherwise.

Then it is a (subgame-perfect) Nash equilibrium for both players to follow the following strategy: cooperate upon observing 0 and defect upon observing 1 . This is somewhat similar to similarity-based cooperation, but the underlying mechanism is still the one of repeated play: the reason it is rational for each of the players to cooperate is that if they defect, their opponent will then defect in all subsequent rounds. In fact, the strategies played are equivalent to the grim trigger strategies for the iterated Prisoner's Dilemma.

As another example of cooperation via repeated play that can be confused with SBC, consider assortative matchmaking as proposed by Wang et al. (2018, Sect. 2.4). Imagine that agents in a large population are repeatedly paired up to play a Prisoner's Dilemma. In each round, each agent is matched up with a new opponent who is similarly cooperative as they are. Again, similarity plays a role, but the underlying mechanism is very different from the ones studied in our paper. For one, similarity is used by the environment (the matching procedure) not the agent itself. Second, the reason why cooperation is rational is again its impact on rewards in subsequent rounds. For instance, imagine that for some reason an agent has, e.g., by accident, defected in the first few rounds and is now matched with uncooperative agents. Then it may be rational for the agent to cooperate in order to be matched with more cooperative agents in the future, even if it knows that it will receive the sucker's payoff in the current round.

H.3 Tag-based cooperation

In the literature the evolution of cooperation and in particular population dynamics, researchers have studied so-called tag-based cooperation Riolo et al. (2001); Cruciani et al. (2017); Traulsen &Claussen (2004); Traulsen (2008), which has sometimes also been referred to as similarity-based cooperation. The underlying mechanism is very different from similarity-based cooperation as studied in the present paper, however.

In models of tag-based cooperation, an agent is defined not only by a policy but also by a tag that has nothing to do with the policy. When choosing whether to cooperate with each other, agents can see each other's tags. Thus, the policies can choose based on the other agent's tag. For example, one policy could be to cooperate if and only if the two agents have the same tag. Notice that in tag-based cooperation, only similarity of tags is considered. In contrast, our paper feeds the similarity of the policies themselves as input to the policy.

Because the tag is not tied to the policy, tags are effectively cheap talk that have no impact on the Nash equilibria of the game. For example, consider the following meta game on the Prisoner's dilemma. Each player i submits a tag τ i ∈ { 1 , ..., 100 } and a policy π i : { 1 , ..., 100 } ⇝ { C, D } that stochastically maps opponent tags onto actions. Then actions are determined by a i = π i ( τ -i ) for i = 1 , 2 . Each player i receives the utility u i ( a 1 , a 2 ) . It is easy to see that in all Nash equilibria of this game, both players submit a policy that always defects. So tag-based cooperation does not help with achieving cooperative equilibrium in a two-player Nash equilibrium. In contrast, similarity-based cooperation as studied in this paper does allow for cooperative equilibria in two-player games, as the main text has shown.

So why might one study tag-based cooperation? Perhaps one would expect that any population of evolving agents would converge to defecting in the above tag-based Prisoner's Dilemma. Interestingly, it turns out that this is not the case! As computational experiments conducted in the above works show, evolving populations sometimes maintain some level of cooperation when they can observe each others' tags. Because cooperation is not an equilibrium, cooperation is maintained dynamically, i.e., by a population whose distribution of tags and policies continually changed. Roughly, the reason seems to be the following. By mere chance (from mutation) there will be tags whose individuals are more cooperative toward each other than the individuals associated with other tags. The cooperative tags and their associated cooperative policies will therefore become more prominent in the population. Of course, this cannot be stable: once an agent is discovered that has the cooperative tag but defects, this type of agent takes over within the cooperative population. But in the meantime a new cooperative tag may have emerged. And so forth. This mechanism for achieving cooperation seems to have no parallel at all in our model of similarity-based cooperation. 1

H.4 The green beard effect

The green beard effect (Hamilton 1964; Dawkins 1976, Ch. 6; Gardner & West 2010) refers to the idea that there could be a gene G that causes individuals to be altruistic to people who also have the gene G without relying on kinship to identify other people with the gene G . In particular, imagine a gene G with the following three properties:

Gene G causes people with the gene to have some perceptible trait. For concreteness let the trait consist in growing a green beard.

Gene G causes people to be altruistic toward people who have a green beard, e.g., to cooperate with them in a one-shot Prisoner's Dilemma.

People without gene G cannot develop a green beard.

Then such a gene could spread and be dominant in a population.

Hamilton and Dawkins discuss the green beard effect as a theoretical idea. To what extent the green beard effect is a real biological phenomenon is subject to debate. Some candidates were pointed out by Keller & Ross (1998), Queller et al. (2003), Smukalla et al. (2008) (cf. Sinervo et al., 2006).

On first sight, similarity-based cooperation as studied in this paper and the green beard effect might seem similar. For example, if you observe a population of individuals, some of whom have the green beard and some of whom do not, you will see cooperation between agents that are similar (in that they both have the green beard). A connection between similarity-based cooperation and the green beard effect has been noted before by Howard (1988), Štˇ epán Veselý (2011), and Martens (2019).

On second sight, however, we think that similarity-based cooperation as studied in our model is very different from the the green beard effect as an evolutionary phenomenon. To understand why, we take the gene-centered view of evolution (e.g. Dawkins, 1976): we view evolution as a process that produces genes that take measures to spread themselves (as opposed to a process that creates organism with particular properties). Now imagine that two individuals sharing the gene G play a Prisoner's Dilemma where they can either take 1 unit of a resource for themselves or give G > 1 units of the resource to the other player. Then from the perspective of trying to spread G , this is not a Prisoner's Dilemma at all! (This is assuming that both players can make similarly good use of the resources.) From the gene's perspective, it is a fully cooperative game wherein cooperation simply dominates defecting. This is the essence of how the green-beard effect works. Our model of SBC has no analogous mechanism or perspective.

As a result, the green beard effect allows for a very different set of outcomes than our folk theorem (Theorem 3). For example, imagine that organisms 1 and 2 both share G , that organism 1 can either take 1 unit of a resource for itself or give G > 1 units of the resource to organism 2, and that organism faces no choice that concerns organism 1. Then G would still have the organism give, because doing so increases the spread of the gene. Meanwhile, the unique outcome allowed by our folk theorem has organism 1 not give. In other games, our folk theorem allows for many different equilibria of the meta game. In contrast, in a game played by two organisms who share G , there will typically be a unique outcome that best spreads G .

H.5 Studies of human psychology on similarity-based cooperation

In this section, we review empirical and theoretical work in psychology related to the phenomenon of similarity-based cooperation.

Studies have shown that humans often cooperate in the one-shot Prisoner's Dilemma, contrary to what standard game theory recommends. Many explanations have been given for this phenomenon. A few authors have proposed explanations based on Newcomb's problem as discussed in Section 7. That is, they have proposed that people cooperate in the Prisoner's Dilemma because cooperating gives them evidence that their opponent will defect (Krueger & Acevedo, 2005; Krueger et al., 2012). To test this hypothesis, a few studies have tried to measure the correlation in subjects' choice in Newcomb's problem and the Prisoner's Dilemma. The results have been mixed. Goldberg et al. (2005, Sect. 4.3), Tversky & Shafir (1992) have found strong correlations, while Toplak & Stanovich (2002) and Goldberg et al. (2005, Sect. 4.2) have failed to find such correlations. Another prediction of the Newcomb's problem/SBC explanation for human cooperation in the one-shot Prisoner's Dilemma is that individuals are more likely to cooperate with similar than with dissimilar individuals. Indeed, a number of studies have found this to be the case (e.g. Acevedo & Krueger, 2005; Fischer, 2009). (Note that some other theories have been proposed for this phenomenon as well (e.g. Aksoy, 2015).)

Another related idea in social psychology is homophily , which refers to the observed tendency of humans to be more likely to engage in interactions with similar individuals. (Homophily cannot always be cleanly differentiated from the tendency to be more altruistic/cooperative toward similar individuals as discussed in the previous paragraph.) Homophily could also be explained by SBC, because SBC allows for better outcomes (e.g., mutual cooperation over mutual defection) when playing against similar individuals. We are not aware of any discussion connecting homophily with Newcomb's problem or the idea that it could be rational to cooperate in a Prisoner's Dilemma against a copy. The theory of the evolution of homophily seems to be that individuals with a similar cultural background can more effectively communicate (and coordinate) (Fu et al., 2012). This is different from SBC as studied in the present paper.

Peer reviews:
comments: SUMMARY:
The paper proposes similarity-based cooperative equilibrium, which extends program equilibrium to a setting of partial transparency. This modification is more practical (as full transparency is much less realistic and is hard to work with) and has useful theoretical properties (e.g., the folk theorem states that by choosing an appropriate similarity measure, one can implement the same cooperative equilibria as if given full transparency). A simple (deep) RL algorithm is proposed to find the proposed equilibrium, which is tested in a newly proposed high-dimensional variant of PD.

STRENGTHS:
- The paper builds on existing concepts like program equilibrium to propose a logical extension that is more practical and relevant for AI.
- The paper presents a strong theory followed by limited but representative experiments.
- The paper is well-written and well-structured. I appreciate the examples in the introduction and the explanations throughout the paper.
- HDPD is a cool independent contribution.
- Experiments with a relevant baseline, LOLA, are provided.

WEAKNESSES:
The only potential reason to not accept this paper is if other reviewers find problems with the theory (e.g., proofs), as I have only glanced over it. Otherwise, I do not see reasons to not accept this paper.

QUESTIONS:
- There is some very recent work about cooperative equilibria + MARL that could be discussed/mentioned in the camera ready. https://dl.acm.org/doi/10.5555/3545946.3598670 https://arxiv.org/abs/2305.06807 https://dl.acm.org/doi/10.5555/3545946.3598618

LIMITATIONS:
- The theoretical analysis is restricted to a certain class of diff-based policies (lines 161-163).
- As mentioned in section 6.4, the proposed empirical method may have limited applicability in more complex games.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-06-11

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents 2-player “difference meta games,” which augment standard one-shot games with similarity information about the other player.  The paper shows that such games have Pareto optimal cooperative equilibria that cause (seemingly naive) “ML algorithms” to cooperate rather than defect.  Results are established primarily theoretically.  Additionally, the CCDR algorithm is presented that cooperates in self play in a high-dimensional one-shot prisoner’s dilemma (formulated as a difference meta game).\n\nSTRENGTHS:\n- The paper presents an interesting: difference meta games.  These games are simpler than program meta games, and are in important ways more realistic.\n\n- The paper has a high-level or rigor and is, for the most part, easy to read and understand.\n\nWEAKNESSES:\n- While difference meta games overcome the issue of full transparency, coming up with a reliable similarity metric (that can’t be duped) seems unrealistic.  At the very least, there seems to be very limited situations in which a difference meta game could be used.\n\n- The CCDR algorithm seems to rely on the idea of their being “cooperate” and “defect” actions.  This seems problematic to me.  First, for an arbitrary game, I didn’t see in the paper where the high-level actions of “cooperate” and “defect” were defined.  Thus, it seems unclear what CCDR actually does.  (Perhaps I missed something obvious, for which I apologize).  Second, It seems that many games would not have such high-level actions (e.g., chicken or battle of the sexes).  From my perspective, this further limitation seems rather confounding.\n\nQUESTIONS:\nPractically speaking, how often in the world would an AI agent actually encounter a scenario in which a different meta game could be used?\n\nLIMITATIONS:\nThe paper points out some limitations as “future work,” including that the best response can perform poorly against random agents and that the CCDR approach  However, it seems there are many more issues that could be made more explicit in the conclusions of the paper.  First, the CCDR algorithm appears to rely on the fact that there are “cooperative” and “defect” higher-level actions.  However, many games don’t have such actions.  Second, the approach is also limited to two players.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nTraining AI agents that will reliably cooperate, both with humans and with other AI, is one of the principal goals of AI alignment research, and the prisoner's dilemma (PD) is simple game that has long been used to study cooperation.  PD is interesting because it has only one Nash equilibrium: to defect (i.e. not cooperate), which is a somewhat paradoxical result because mutual cooperation would be better for all participants.  The usual modification to PD is the iterated prisoner's dilemma, in which each agent can observe a history of past interactions.  This history opens up new equilibria, and allows agents to learn to cooperate.   \n\nThis paper proposes another potential modification to PD with cooperative equilibria, namely supplying each agent with a measure of the similarity between its policy, and the policy of other agents.  Agents can learn to cooperate with similar agents, and defect against dissimilar ones.  The paper proves some basic theorems about equilibria in such games.\n\nThe paper also proposes a variant -- high-dimensional PD (HDPD), which has a more complex notion of \"cooperate\" and \"defect\" than simple PD.  Naive training of an agent in HDPD does not result in cooperation.  However, the authors introduce a pre-training mechanism called Cooperate against Copies and Defect against Random (CCDR), in which agents alternate self-play (i.e. play against perfectly similar copies) and play against random opponents.  When using this pre-training mechanism, agents learn to cooperate.\n\nSTRENGTHS:\nAlthough it is not mentioned by the authors in the paper, I would like to note that \"perceived similarity\" does, in fact, appear frequently in the natural world.  Kin selection (genetic similarity) is the basis for cooperation among social insects like ants and bees, and shared language, culture, and religion have historically been the basis for cooperation among humans.  IMO, this is thus a potentially important avenue of research, which has been overlooked in the literature.\n\nThe paper is well written, and the results seem solid.  The authors have an excellent discussion of the limitations of CCDR -- in particular, after learning to cooperate during pre-training, some agents partially unlearn it during the rest of training.   Further exploration in this area could have potential applications on AI alignment research.\n\nWEAKNESSES:\nMy main concern is this is mostly a paper on game theory, so I question whether NeurIPS is really the right venue for it.  The actual experiment also seems somewhat simplistic to me.  I believe that the primary application of this work relates to AI alignment, so I would have preferred to see more discussion about how these ideas could be applied to the real world.\n\nQUESTIONS:\nFrom an AI alignment perspective, can a meaningful measure of \"similarity\" can be derived for more complex agents, operating in more complex environments?  In more complicated cases, the only reliable information that an agent can observe about other agents is the history of past actions, in which case the notion of \"similarity\" becomes much the same as the iterated PD, or other reputation-based systems.  \n\nIs it possible for agents to game the system, and hack the similarity function?  This is probably relevant only for more complex agents than those considered in this paper, but is extremely import wrt. to AI alignment.\n\nIs it possible to derive a notion of similarity that would enable otherwise dissimilar agents to cooperate?  E.g. among humans, people with very different goals and backgrounds can cooperate if both parties agree to follow the same \"moral code\", or \"rule of law\".\n\nLIMITATIONS:\nAlthough this paper has potential applications wrt. to AI alignment, the authors focus primarily on detailed mathematical analysis, and fail to discuss the larger social implications.   \n\nHowever, I applaud the authors for their frank and open discussion of the limitations and failures of CCDR.  IMO, that discussion is one of the most valuable parts of the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the problem of two agents trained by machine learning who interact in a social dilemma.  The agents are able to observe a numeric measure of the similarity of their learned policies, and condition on this similarity when choosing their actions (in contrast to the \"full transparency\" case, in which the agents can each observe the other's full source code).  When policies can condition on similarity, cooperation can be supported in equilibrium (similarly to various full transparency results in the literature).\n\nWhen partially-transparent policies are naively trained, they tend not to learn to play toward cooperative equilibria; however, the paper demonstrates that pre-training by averaging between playing against the same policy and a randomly-sampled policy can enable machine learned models to find partially cooperative equilibria.\n\nSTRENGTHS:\nThis paper correctly observes that partial transparency supports the same equilibrium outcomes as full transparency, while being dramatically more plausible.  The results are carefully derived and clearly presented.  The paper studies a possible approach to solving an important problem; social dilemmas succinctly describe a fundamental problem in many situation where agents would all gain by cooperating.\n\nThis paper takes a step beyond proving the existence of equilibria, and demonstrates that plausible training procedures can actually find them.\n\nThe notion of similarity is based on the similarity of the policies considered as functions, rather than being computed on weights or some other parametric representation.  This is very natural, and I think it's an improvement over the \"source code\" based approach that earlier full-transparency literature tends to appeal to.\n\nWEAKNESSES:\nMy main concern is that the results really only apply to a very specific class of games (player-symmetric additively decomposable games).  It makes sense that the equilibrium uniqueness results of theorem 4 should be narrowly targeted (it's hard to prove things about large classes of games), but I would have liked to see some experimental examination of whether the proposed approach works well on games that don't satisfy these restrictions.\n\nThe procedure described still tends not to find fully cooperative equilibria, which seems like a shortcoming.\n\n\n#### Minor issues (did not affect rating)\n\n- p.1: \"While partial transparency is the norm\": This is a pretty strong claim, and one which is important for the main claims of the paper.  Could you back it up with a citation or example or something else beyond a bare assertion?\n- p.2: \"In particular, full transparency can make the problem of equilibrium selection harder\": Isn't this also true of partial transparency?\n\nQUESTIONS:\nHave you evaluated your approach on non-additively-decomposable games?\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}