Similarity-Based Cooperation
Authors: Caspar Oesterheld
Johannes Treutlein
Roger Baker Grosse
Vincent Conitzer
Jakob Nicolaus Foerster
Venue: Submitted to ICLR 2023 (2023)
Keywords: multi-agent reinforcement learning
cooperative AI
program equilibrium

Abstract: As ML agents act more autonomously in the world, they will increasingly interact with each other. Unfortunately, in many social dilemmas like the one-shot Prisoner’s Dilemma, standard game theory predicts that ML agents will fail to cooperate with each other. Prior work has shown that one way to enable cooperative outcomes in the one-shot Prisoner’s Dilemma is to make the agents mutually transparent to each other, i.e., to allow them to access one another’s source code (Rubinstein 1997; Tennenholtz 2004) or weights in the case of ML agents. However, full transparency is often unrealistic, whereas partial transparency is commonplace. Moreover, it is difficult to machine-learn to cooperate in the full transparency setting. In this paper, we introduce a more realistic setting in which agents only observe a single number indicating how similar they are to each other. We prove that this allows for the same set of cooperative outcomes as the full transparency setting. We also demonstrate experimentally that cooperation can be learned using simple ML methods.

SIMILARITY-BASED COOPERATION
Anonymous authors Paper under double-blind review
ABSTRACT
As machine learning agents act more autonomously in the world, they will in- creasingly interact with each other. Unfortunately, in many social dilemmas like the one-shot Prisoner's Dilemma, standard game theory predicts that ML agents will fail to cooperate with each other. Prior work has shown that one way to enable cooperative outcomes in the one-shot Prisoner's Dilemma is to make the agents mutually transparent to each other, i.e., to allow them to access one another's s source code (Rubinstein, 1998; Tennenholtz, 2004) - or weights in the case of ML agents. However, full transparency is often unrealistic, whereas partial trans- parency is commonplace. Moreover, is challenging for agents to learn their way cooperation in the full transparency setting. In this paper, we introduce more realistic setting in which agents only observe single number indicating how simi- lar they each other. We prove that this allows for the same set of cooperative outcomes as the full transparency setting. We also demonstrate experimentally that cooperation can be learned using simple ML methods.
INTRODUCTION
As Al systems start to autonomously interact with the world, they will also increasingly interact with each other. We already see this in contexts such as trading agents (CFTC & SEC, 2010), but the number of domains where separate AI agents interact with each other in the world is sure to grow; for example, consider autonomous vehicles. In the language of game theory, Al systems will play general-sum games with each other. For example, autonomous vehicles may find themselves in Game-of- Br-CChicken-iin dynamics with each other (cf. Fox et al., 2018). In many of these interactions, cooperative or even peaceful outcomes are not a given. For example, standard game theory famously predicts and recommends defecting in the one-shot Prisoner's 's Dilemma. Even when cooperative equilibria exist, there are typically many equilibria, including uncooperative and asymmetrie ones. For instance, in the infinitely repeated Prisoner's Dilemma, mutual cooperation is played in some equílibria, but so is mutual defection, and so is the strategy profile in which one player cooperates 70% of the time while the other cooperates 100% of the time. Moreover, the strategies from different equilibria typically do not cooperate with each other. A recent line of work at the intersection of Al/(multi- agent) ML and game theory aims to increase AI/ML systems' ability to cooperate with each other (Stastny et al., 2021; Dafoe et al., 2021; Conitzer & Oesterheld, 2022).
Prior work has proposed make Al agents mutually transparent to allow for cooperation in equilib- rium (MeAfee 1984; Howard 1 1988; Rubinstein 1998, Section 10.4; Tennenholtz 2004; Barasz et al. 2014: Critch 2019; Oesterheld 2019b). Roughly, this literature considers for any given 2-player normal-form game I the following program meta game: Both players submit a computer program, e.g., some some neural net, to choose actions I on their behalf. The computer program then receives as input the computer program submitted by the other player. Prior work has shown that the program meta game has cooperative equilibria in the Prisoner's 's Dilemma.
Unfortunately, there are multiple obstacles to cooperation based on full mutual transparency. 1) While partial transparency is the norm, settings of full transparency are rare. For example, while GPT-3's architecture and training regime are public knowledge, the exact learned model is not. 2) Games played with full transparency in general have many equilibria, including ones that are much worse for some or all players than the Nash equilibria of the underlying game (see the folk theorems given by Rubinstein 1998, Section 10.4, and Tennenholtz 2004). In particular, full mutual transparency can make the problem of equilibrium selection harder. 3) The full transparency setting poses challenges to modern ML methods. In particular, it requires at least one of the models to receive as input model that has at least many parameters itself. Meanwhile, most modern

---

Principal 1
Principal 2
diff: A(A1)* R x x(22) + R
Policy
Policy n22 IR A(A2)
Ty IR ((A))
cerect
(diff( x2)1
(diff( z1. n2))
6.5
00
2,2
0,8
3.0
1,1
Mixed strategy 1 61 A(A1)
Mixed strategy 2
O2 E A(A2)
(a)
Figure 1: (a) Illustration of the diff meta game of a Prisoner's Dilemma. (b) graphical representa- tion of diff meta games (Definition 1). Nodes with incoming nodes are determined by applying one of the parent nodes to the other.
successes of ML use models that are orders of magnitudes larger than the input. Consequently, are not aware of successful projects on learning general-purposs models such as neural nets in the full transparency setting.
Contributions In this paper introduce novel variant of program meta games called difference (diff) meta games that enables cooperation equilibrium while also addressing obstacles 1-3. As in the program meta game, imagine that two players each submit program or policy to instruct agen! to play a given game, such as the Prisoner's Dilemma. The main idea is that before choosing an action, the agents are given information about how similar the two players' policies are each w.r.t. how they make the present decision. We formally introduce this setup in Section 3. For an informal illustration, see Figure Because requires a much lower degree of mutual transparency, we find the diff meta game setup more realistic than the full mutual transparency setting. Thus, it addresses Obstacle to cooperation based on full mutual transparency.
Diff meta games can still have cooperative equilibria when the underlying base game does not. Specifically, in Prisoner's Dilemma-like games, there are equilibria in which both players submit policies that cooperate with similar policies and thus with each other. We call this phenomenon similarity-based cooperation (SBC). For example, consider the Prisoner's Dilemma as given in Ta- ble 1 for G = 3. (We study such examples in more detail in Section 3.) Imagine that the players can only submit threshold policies that cooperate if and only if the perceived difference to the opponent is at most 0, As measure of difference, the policies observe diff 0, 02) |8 021 +N, where N is sampled independently for each player according to the uniform distribution over [o, 1]. For instance, if Player submits threshold of 1/2 and Player 2 submits threshold of 3/4, then the per- ceived difference is + N. Hence, Player I cooperates with probability P(1/+
1/2) 1/4 and Player 2 cooperates with probability P(1/++N < 3/4) 1/2. It turns out that (0) 1, 02 = 1). which leads to mutual cooperation with probability 1, is Nash equilibrium of the meta game. Intuitively,
the only way for either player to defect more to lower their threshold. But then @ O₂1 will increase, which will cause the opponent to defect more (at a rate of 1/2). This outweighs the benefit of defecting more oneself.
In Section 4, we prove folk theorem for diff meta games. Roughly speaking, this result shows that merely observing diff value is sufficient for enabling all the cooperative outcomes that full mutual transparency enables. Specifically,
show that for every individually rational strategy profile o (i.e., every strategy profile that is better for each player than their minimax payoff), there is a diff function such that played in equilibrium of the resulting diff meta game.
Next, we address Obstacle to full mutual transparency - the multiplicity of equilibria. First, note that any given measure of similarity will typically only enable specific set of equilibria. much smaller than the set of individually rational strategy profiles. For instance, in the above example, all equilibria symmetric. In general, would hope that similarity-based cooperation will
2

---

Player 2
Cooperate
Defect
Cooperate Defect
G,G
0,
Player
G+1,0
1,
Table I: The Prisoner's Dilemma, parameterized by some number G > 1.
generally result in symmetric outcomes in symmetric games. After all, the new equilibria of the diff game are based on submitting similar policies and if two policies play different strategies against each other, they cannot be similar. In Section 5, substantiate this intuition. Specifically, prove, roughly speaking, that in symmetric, additively decomposable games, the Pareto-optimal equilibrium of the meta game unique and gives both players the same utility, if the measure of difference between the agents satisfies few intuitive requirements (Section 5). For example, in the Prisoner's Dilemma, the unique Pareto-optimal equilibrium of the meta game must be one which both players cooperate with the same probability.
Finally we show that diff meta games address Obstacle 3: we demonstrate that in games with higher- dimensional action spaces, we can find cooperative equilibria of diff meta games with ML. methods. In Section 8, we show that, if we initialize the two policies randomly and then let each of them learn to be best response to the other, they generally converge to the Defect-Defect equilibrium. This is expected based on results in similar contexts, such as in the Iterated Prisoner's Dilemma. However, in Section 6, we introduce novel pretraining method that trains policies to cooperate against copies and defect against randomly generated policies. Our experiments show that policies pretrained in this way find partially cooperative equilibria of the diff game when trained against each other.
We discuss how the present paper relates to prior work in Section 9, We conclude in Section 10 with some ideas for further work.
BACKGROUND
Elementary game theory definitions. We assume familiarity with game theory. For an introduc- tion, see Osborne (2004). A (two-player, normal-form) game I"= (A1, u) consists of sets of actions or pure strategies A₁ and A2 for the two players and a utility function u: A x A + R2. Table gives the Prisoner's Dilemma as classic example of game. mixed strategy for Player i is distribution over A;. We denote the set of such distributions by A(A,). We can extend to mixed strategies by taking expectations, i.e., u(O), O2) == Eaj ax oi (()))))(()))(()))))))) For any player i, we use -i to denote the other player. We call G, best response to strategy E A(A-i, if supp(a;) S argmax,,es 0,EA, (,,a,a-)), where supp denotes the support. A strategy profile o A(Ai) x A(A2) is a vector of strategies, one for each player. We call a strategy pro- file (2) (strict) Nash equilibrium if o is (unique) best response O2 and vice versa. As first noted by Nash (1950), each game has at least one Nash equilibrium. We say that a strat- egy profile is individually rational if each player's payoff 2 max mina_,eA_.. ui(o,,a-i) for i = 1,2. We say that o is Pareto-optimal if there exists no o' s.t. u/(a") 2 (,')") for i = 1,2 and (1(o") (oo))) for at least one i.
least its minimax payoff, i.e. if
Symmetric games and additively decomposable games. We say that a game is (player) symmetric if A₁ = A2 and for all aı,a2 for = 1,2, we have that u;(a).a2) = (a, The Prisoner's Dilemma in Table
symmetric. We say that a game additively decomposes into (u.j Aj -> R)( if 4;(a),42) = (;,((a) (1,2 for i- {1,2} and all a E AA,a2 E A2. Intuitively, this means that each action aj of Player j generates some amount of utility u.,(az) for Player i independently of what Player -j Player-j plays. For example, the Prisoner's Dilemma in Table is additively decomposable, where uyy Cooperate + O, Defect 1 and ui--i i Cooperate - G, Defect - 0 for = 1,2. Intuitively, Cooperate generates for the opponent and for oneself, while Defect generates for oneself and for the opponent.
Alternating best response learning. The orthodox approach to learning in games is to learn to best respond to the opponent, essentially ignoring that the opponent is also a learning agent. In this paper, we specifically consider alternating best response (ABR) learning. In ABR, the players take turns. In each turn, of the two players updates the parameters 0 of her strategy optimize un 0-)),
3

---

Le., updates her model to be best response the opponent's current model (Brown cf. 1951; Hein- rich et al. 2021: Zhang et al. 2022). Since learning an exact best response is generally intractable, will specifically consider the use of gradient ascent in each turn to optimize u;{ (0 0-)) over 0, In continuous games if ABR with exact (locally) best response updates converges to (01, 02). then (0) 02) is (local) Nash equilibrium. Note, however, that ABR may fail to converge (e.g.. in dynamics). Moreover, if the best response updates of 0 are only
the face of Slock-Paper-Sciii approximated, ABR may converge to non-equilibria (Mazumdar et al., 2020, Proposition 6).
DIFF META GAMES
We now formally introduce diff meta games, the novel setup consider throughout this paper. Given some base game r, we consider a new meta game played by two players whom we will call principals. Each principal i submits policy. The two players' policies each observe real-valued measure of how similar they are to each other. Based on this, the policies then output a (potentially mixed) strategy for the base game. Finally, the utility is realized as per the base game. Below we define this new game formally. This model is illustrated in Figure 1.
Definition 1. Let r = uA.u) be game. A (diff-based policy for Player i for I is function R- + A(A)) mapping the perceived real-valued difference between the policies to a mixed strategy of the game, For == 1.2 let o C ((A,) be a set of difference-based policies for Player Then a policy difference (diff) function for (5)) a stochastic function diff: d X sh us RR... For any two policies Ti, T we say thut (T1,Tz) T2) plays the stralegy profile O E A(A1) x A(A2) of if G 6==Bi =R. (7((fff ==]]in(diff [T, (T,7)))) for i= 1,2 For sets of policies d ,4, and difference function diff we then define the diff meta game 12))) for all z E Mh, T2 E ah.
diff) to be the game (6,VV where (Z) ==
E (T((f
Note that Definition I does not put any restrictions on diff. For example, the above definition allows (diff( TT:))) to be a real number whose binary representation uniquely specifies π- This paper is dedicated to situations in which diff specifically represents some intuitive notion of how different the policies are, thus excluding such diff functions. Unfortunately, there are many different ways in which one could formalize this constraint, especially in asymmetric games. In Section 5 we will impose some restrictions along these lines, including symmetry. Our folk theorem (Theorem 3 in Section 4) will similarly impose constraints on diff to avoid diff functions like the above.
The rest of this section will study concrete examples of Definition 1. First, we define a particularly simple type of diff-based policy. Almost all of our theoretical analysis will be based on this class of policies.
Definition 2. Let E TRU and o ,0," E(Aii) be strategies for Player 1,2. Then we define (a - 0, O to be the policy π s.t. x(d)= of if d S o and of otherwise. We call policies of this form threshold policies. Let o denote the set of such threshold policies.
Throughout the rest of this section, we analyze the Prisoner's Dilemma as a specific example. We limit attention to threshold agents of the form (C, DD.. i.e., policies that cooperate against similar opponents (diff below threshold 9) and defect against dissimilar opponents. This is because such policies can be used to form cooperative equilibria, while policies that always cooperate ((C, I,C) or policies that are more cooperative against lesy similar opponent policies (e.g.. (D, 1,)) cannot be used form cooperative equilibria in the PD with a natural diff function. Policies of the form (C, ()) are uniquely specified by single real number e. A natural measure of the similarity between two policies 0, 0₂ is then the absolute difference |0, - |0l- We allow diff to be noisy, however. We summarize this in the following.
Example 1. Let I be the Prisoner's Dilemma as per Table 1. Then consider the (T,A,, A2,fff diff)
meta game where Å = [[C -(C | 0 E TR.} and diff, D), (C, D)) = 10: Oz + N₂ for i 1= 1,2 where N; is some real-valued random variable.
The only open parameters of Example 1 are G (the parameter used in our definition of the Prisoner's Dilemma) and the noise distribution. Nevertheless, Example is a rich setting that allows for non- trivial results. We leave detailed analysis for Appendix and only give two specific results about equilibria here.
4

---

Proposition 1. Consider Example with N, ~ (niform((OOe]] El) i.i.d. for some E = 0 and with G G23 2. Then ((C, D), (C, D)) is a Nash equilibrium if and only if 0, 02 - 0 or 0 < 8 = 02 - E. In case of the latter, the equilibrium is strict if G 2.
Another natural distribution to use for N, is the normal distribution. The following result shows for G what the Nash equilibria of the diff meta game are.
Proposition 2. Consider Example with G = 2. Assume N; is i..dd fori 1,2 according some uni- modal distribution with mode with positive measure on every interval. Then ((C, D), DC, is a Nash equilibrium if and only if 0 02 S.V. V.
4 A FOLK THEOREM FOR DIFF META GAMES
What are the Nash equilibria of a diff meta game on I? Of course, the answer depends on what diff function we use. A first answer is that Nash equilibria of T carry over to the diff meta game regardless of what diff function is used (assuming that at least all constant policies are available); see Proposition 15 in Appendix C.I. Any other equilibria of the diff meta game hinge on the use of the right diff function. In fact, if diff is constant and thus uninformative, the Nash equilibria of the diff meta game are exactly the Nash equilibria of I; see Proposition 16 in Appendix C.I. The more interesting question is for what strategy profiles o there exists some diff function s.t. is played in an equilibrium of the resulting diff meta game. The following result answers this question.
Theorem (folk theorem for diff meta games). Let be a game and be a strategy profile for I. Let o 2 o for i = 1,2. Then the following two statements are equivalent:
1. There is a diff fanction such that there is a Nash equilibrium (T) π2) of the diff meta game (T. diff, A. ob) s.t. (T1 R2) play G.
2. The strategy profile a is individually rational (i.e., better than everyone's 's minimax payoff).
The result continues to hold true if we restrict attention to deterministic diff functions with diff; diff, and (7), x2) E {0. 1} 1,,
We leave the full proof to Appendix C.2. but give short sketch of the construction for 2-1 1 here. For any O, we construct the desired equilibrium from policies z (o,, 1/2, â;) for i == 1,2, where O; is Player i's minimax strategy against Player -i. We then take any diff function s.t. diff (T/ ii (0,0) if x= = n,, and diff( (T x² = (1, 1) otherwise.
A UNIQUENESS THEOREM
Theorem allows for highly asymmetric similarity- -base cooperation. For example, in the PD with, say, G G=22 2, Theorem 3 shows that with the right diff function, the strategy profile (C,3/3*C++/4) is played in an equilibrium of the diff meta game of the PD. This seems odd, as one would expect similarity-based cooperation to result in playing symmetric strategy profiles. Note that, for example, all equilibria of Propositions and 2 are symmetric. In this section, we show that under some restrictions on diff and the base game I, we can recover the symmetry intuition.
D)
We first need a few definitions of properties of diff. Let I' be a symmetric game. We say that diff is minimized by copies if for all policies π, x², all y and i= 1,2, P(diff, xx,,'f((zxx')))) x')x)))))))) P(diff, (I. X) x)))). For example, the diff function in Example 1 is minimized by copies. The diff functions in the proof of Theorem 3 are not in general minimized by copies when the given base game is symmetric. For example, to achieve (C,2/3* 1/333 x D) in equilibrium, the proof of Theorem 3 (as sketched above) uses the policies πं = -C, D) and πιj = C 1/3* D, 1/2,,, D) and a diff function with diff( ,T)) (0,0) 0) but diff( , T = (1. If the base game is symmetrie, we call diff symmetric if for all ni N2, diff( 1, n22) is distributed the same as diff( T2, 21) ) and (diff, (TI, diffz X2)) is distributed the same (diffz (T)) diff, (2).
Finally, we need a more complicated but nonetheless intuitive property of diff functions. In this paper, we generally imagine that low values of diff are informative about the other player's policy. In contrast, we will her assume that high values of diff are uninformative. That is, for any o and π will assume that there is a policy πy that plays G; against π_; and triggers the above-threshold policy of π with the highest-possible probability. Appendix D. shows why this assumption is necessary. Formally, let n = (o o2,) be any threshold policy. Let p be the highest number such that there is n4 s.t. in (z, ;), Player -i plays arbitrarily close (1 p)o + ~a and
5

---

Intuitively, Oz is the strategy played by ズ against the most different opponent policies. For the examples of Section we have p and thus simply Ox; o But if diff is bounded (as in the proof of Theorem 3), then might even have p = 0 or anything in between.
Definition 3. We call diff: A x 1 Ny R2 high value uninformative if for each threshold policy nh O: and & there is threshold policy such that in (T4, TT-) a strategy profile within of (o,, xx)) is played.
We are now ready to state uniqueness result for the Nash equilibria of diff meta games.
Theorem 4. Let I be a player-symmetrey additively decomposable game. Let diff be symmetric, high-value uninformative, and minimized by copies. Then if (T1,77 T2) is a Nash equilibrium that not Pareto-dominated by another Nash equilibrium, we have that (1 72) , Hence, if there exists a Pareto-optimal Nash equilibrium, its payoffs are unique, Pareto-dominant among Nash equilibria and equal across the two players.
We prove Theorem 4 in Appendix D.3. Roughly, we prove that under the given assumptions, equi- librium policies are more beneficial to the opponent when observing a diff value below the threshold than if they observe diff value above the threshold. Second, we show (using the first fact) that in a given strategy profile Principal receives a lower utility than Principal -i, then Principal i can increase her utility by submitting copy of Principal -i's policy. Appendix D.1 shows why the assumptions (additive decomposability of the game and and high-value uninformativenes. and symmetry of diff) are necessary.
A NOVEL PRETRAINING METHOD FOR SIMILARITY-BASER COOPERATION
We now describe a simple machine learning method that use to find cooperative equilibria in more complex games. To use this method, we consider neural net policies Hh parameterized by real vector O.
We call this procedure Cooperate against Copies and Defect against Random (CCDR) pretraining based its intended effect in Prisoner's Dilemma-like games. First, for any given diff game, let vd: (IRT) (B**1) x (IR")) (1*T))((*))) + 2 be the utility of a version of the game in which diff' is non- noisy. Then we will pretrain each model πæ, to maximize vd (To,, Te,) VV Tig for randomly sampled O',,- That is, each player pretrains their policy Te, do well in both of the following scenarios: principal -i copies principal i's model; and principal -i generates a random model.
CCDR pretraining motivated by two considerations. First, games like the HDPD, it can be shown that there exist cooperative equilibria between policies that cooperate at a diff' value of 0 and defect as the perceived diff value increases. We give a toy model of this in Appendix E. CCDR puts in place the rudimentary structure of these equilibria. Note, however, that CCDR does not directly optimize for the model's ability to form a cooperative equilibrium. Second, CCDR can be thought of as form of curriculum training. Before trying to play diff games against other (different but similar) learned agents, we might first train a policy to solve two (conceptually and technically) easier related problems.
A HIGH-DIMENSIONAL ONE-SHOT PRISONER'S S DILEMMA
To study similarity -bssed cooperation in an ML context, we need more complex version of the Prisoner's Dilemma. The complex Prisoner's Dilemma-like games studied by the multi-agent learn- ing community generally offer other mechanisms that establish cooperative equilibria (e.g., playing a game repeatedly). For our experiments, however, we specifically need SBC to be the only mecha- nism to establish cooperation.
We therefore introduce a new game, the High- Highh-Dimnssioall (one-shot) Prisoner's Dilemma (HDPD). The goal is to give variant of the one-shot Prisoner's Dilemma that is conceptually simple but in- troduces scalable complexity that makes finding, for example, exact best responses in the diff meta game intractable. In addition to G. the HDPD is parameterized by two functions fC: IR" - R representing the two actions Cooperate and Defect, respectively, as well as a probability measure over R². Each player's action is also function fi: IR" TR. This is illustrated in Figure 2 for

---

Figure 2: The figure describes how utilities are calculated in the HDPD. Here the functions fi and f2 the actions chosen by the players. First sample point (red). We then calculate the distance of fi(x) to fc(x) and fn to determine both players' losses.
the case of и = and m = 2. For any pair of actions f2. payoffs are then determined as follows. First, we sample some x according to μ from R" In Figure 2, this is represented by a red dot on the axis. Then to determine how much Player cooperates, we consider the distance (x, fx(x) to determine, roughly speaking, how much Player cooperates. Here, denotes the Euclidean dis- tance. The larger the distance the less cooperative is ft. This distance is visualized in Figure by the arrow from (x, fi (x) to (x. fx))). We analogously determine how much the players defect. For- mally, we define Hi (2) Thus, the action fi fD corresponds to defecting and the action fc corresponds to cooperating, e.g., u(fc.fc) (-1,-1) and u(fD. fb) (-G, -G). The unique equilibrium of this game is (fD, fD)- In our experiments, we specifically used G = 5. If we further let be uniform over [0, IJ. the utilities for Figure 2 come
- Ex-s [d(fi(xx, fx))
/ /Ex-u [x(fcxx). (x())
about 55500 for Player and 2.599 for Player 2.
We consider a diff meta game on the HDPD. Formally, a diff- base- policy for the HDPD is a function R - (IRT)) (R*), For notational convenience, we will instead write policies as functions IR++1 > R We then define our diff function by ( π²) = Eyyx)-v [d x2 + +N, where v is
some probability distribution over I and N; is some real-valued noise.
EXPERIMENTS
Our results far demonstrate the theoretical viability of similarity-based cooperation, but leave open questions regarding its practicality. In this section, we address one these questions: In complex environments, where cooperating and defecting are by themselves complex operations, can we find the cooperative equilibria for given diff function with machine learning methods?
Experimental setup. We trained the environment from section 7. We selected fixed set of hyperparameters based on prior exploratory experiments and the theoretical considerations in Ap- pendix We then randomly initialized 0 and 6₂ and CCDR-pretrainee them. Finally, we trained the 0 and 0₂ against each other using ABR. We repeated the experiment with 28 different seeds. As control, we also ran the experiment without CCDR pretraining (on 26 seeds). We also ran experi- ments with Learning with Opponent-Learming Awareness (LOLA) (Foerster et al., 2018), which we report in Appendix G.
Results. First, observe that in the runs without CCDR pretraining, the players generally converge to mutual defection during alternating best response learning. particular, in all 26 runs, at least one player's s utility was below -5. Only two runs had a utility above -5 for one of the players (-4.997 and -4.554). The average utility across the 26 runs and across the two players was -5.257 with standard deviation of 0. 1978. Anecdotally, these results robust ABR without pretraining practically never finds cooperative equilibria in the HDPD.
Second, observe that in all 28 runs, CCDR pretraining qualitatively yields the desired policy models, i.c., policy that cooperates at low values of diff and gradually comes closer to defecting at high values of diff. Figure 3a shows representative example.

---

Jedurco of cnterce of
Sutance Adon/an
12
Solunce af
3.5
3.0
2.5
2.0
02
00 02 0.1 0.8
03
13
200
400
600
800
1000
Time
(a)
(h)
Figure 3: (a) The behavior of a CCDR pretrained policy. For each perceived diff to the opponent y, the graph shows the expected distance of the learned policy's ' choice to fc and fp. (b) Losses of Player 1 in 10 runs through the ABR phase.
Our main positive experimental result is that after CCDR pretraining, the models converged in al- ternating best response learning to a partially cooperative equilibrium in 26 out of 28 runs. Thus, the cooperative equilibria postulated in general by Theorem and in simplified examples by Propo- sitions and (as well as Proposition 24), do indeed exist and can be found with simple methods. The minimum utility of either player across the successful runs was -4.854. The average utility than to
across all runs and the two players was about -2.77 and thus a little closer to u(fc.fc) f(fn:fp) = 5. The standard deviation was about 1.19. Figure shows the losses (i.e., the negated utilities) across ABR learning. Generally, the policies also converge receiving approximately the same utility (cf. Section 5). The average the absolute differences in utility between the two play- ers the end of the 28 runs is about 0.04 with a standard deviation of 0.05. We see that in line with Theorem 4, we tend to learn egalitarian equilibria in this symmetric, additively decomposable setting. After alternating best response learning, the models generally have similar structure as the model in Figure 3a, though often they cooperate only a little at low diff values. Based on prior exploratory experiments, CCDR's success moderately robust.
Discussion. Without pretraining, ABR learning unsurprisingly converges to mutual defection. This due to a bootstrapping problem. Submitting a policy of the form "cooperate with similar policies, defect against different policies" is unique best response if the opponent submits model of this form as well. If the opponent model π is not this form, then any policy πΙ that defects, i.c., that satisfies n; (diiff( i)) fD. is best response. Because is complex, learning model that cooperates at all unlikely. (Even if fc was simple, the appropriate use of the perceived diff value would still be specific and thus unlikely to be found by chance.) Similar failures find the more complicated cooperative equilibria by default have also been observed in the iterated PD (Sandholm & Crites 1996; Foerster al. 2018; Letcher al. 2019) and in the open-source PD (Hutter, 2020) (cf. Section 9.1). Opponent shaping methods have been used successfully to learn to cooperate both in the iterated Prisoner's Dilemma (Foerster al. 2018: Letcher al. 2019) and the open-source Prisoner's Dilemma (Hutter, 2020). Our experiments in Appendix G show that LOLA can also learn SBC, but unfortunately not as robustly as CCDR pretraining.
CCDR pretraining reliably finds models that cooperate with each other and that continue to partially cooperate with each other throughout ABR training. This shows that when given some guidance, ABR can find uumilarit-based cooperative equilibria. We conclude from our experiments that SBC a suitable means of establishing cooperation between modern MI. agents.
That said, CCDR also has some limitations that we hope can be addressed in future work. For one, many games optimal play against randomly generated opponents is unreasonable when facing a rational opponent. Second, our experiments show that while the two policies almost fully cooperate after CCDR pretraining, they quickly partially unlearn to cooperate in the ABR phase. We would prefer method that preserves closer full cooperation throughout ABR-style training. Third, while CCDR seems to often work, it can certainly fail games which SBC is possible. For instance, CCDR may sometimes result in insufficiently steep incentive curves. We suspect that to
8

---

make progress the latter issues need training procedures that more explicitly reason about incentives à la opponent shaping (cf. our experiments with LOLA Appendix G).
RELATED WORK
9.1 PROGRAM EQUILIBRIUM
We already discussed in Section the literature on program meta games in which players submit computer programs policies and the programs fully observe each other's code (McAfee 1984; Howard 1988; Rubinstein 1998, Section 10.4; Tennenholtz 2004). Interestingly, some constructions for equilibria in program meta games are similarity based. For example, the earliest cooperative program equilibrium for the Prisoner's Dilemma, described all four of the above-cited papers, is the program "Cooperate if the opponent's program equal to this program; else Defect". Other approaches to program equilibrium cannot be interpreted as similarity based, however (see, e.g., Barasz al., 2014; Critch, 2019; Oesterheld, 2019b). To our knowledge, the only published work on ML in program equilibrium is due to Hutter (2020). It assumes the programs to have the structure proposed by Oesterheld (2019b) on simple normal-form games, thus leaving only a few parameters open. Similar to our experiments, Hutter shows that best response learning fails to converge to the cooperative equilibria. Hutter's experiments, the opponent shaping methods LOLA (Foerster et al., 2018) and SOS (Letcher et al., 2019) converge mutual cooperation.
9.2 DECISION THEORY AND NEWCOMB'S 'S PROBLEM
Brams (1975) and Lewis (1979) have pointed out that the Prisoner's Dilemma against a similar op- ponent closely resembles Newcomb's problem, a problem first introduced to the decision-theoretical literature by Nozick (1969). Most of the literature on Newcomb's 's problem is about the norma- tive, philosophical question of whether one should cooperate or defect when playing a Prisoner's Dilemma against an exact copy. Our work is inspired by the idea that in some circumstances one should cooperate with similar opponents. However, this literature only informally discusses the question of whether to also cooperate with agents other than exact copies (Hofstadter e.g., 1983; Drescher 2006, Ch. 7; Ahmed 2014, Sect. 4.6.3). We address this question formally.
One idea behind the present project, as well as the program game literature, is to analyze decision situation from the perspective (actual or hypothetical) principals who design policies. The princi- pals find themselves in an ordinary strategic situation. This is how our analysis avoids the philosoph- ical issues arising in the agent's 's perspective. Similar changes in perspective have been discussed in the literature on Newcomb's 's problem (Gauthier e.g., 1989; Oesterheld & Conitzer 2022).
9.3 LEARNING IN CEWCOLB-LIKE DECISION PROBLEMS
There is some existing work learning in Newcomb-like environments that therefore also applies to the Prisoner's s Dilemma against a copy. Whether cooperation against copy is learned generally depends on the learning scheme. Bell et al. (2021) show that Q-learning with a softmax policy learns to defect. Regret minimization also learns to defect. Other learning schemes do converge to cooperating against exact copies (Albert & Heiner, 2001; Mayer et al., 2016; Oesterheld, 2019a; Oesterheld et al., 2021). All schemes in prior work differ from the present setup, however, and to our knowleds mowledge none offer a model of cooperation between similar but non-equal agents.
10 CONCLUSION AND FUTURE WORK
We make a strong case for the promise of similarity-based cooperation as a means of improving outcomes from interactions between ML agents. the same time, there are many avenues for future work. On the thcoretical side, we would be especially interested in generalizations of Theorem 4, that is, theorems that tell us what outcomes we should expect in diff meta games. Is it true more generally that under reasonable assumptions about the diff function, we can expect similarity-based based cooperation to result in fairly specific, symmetric, Pareto-optimal outcomes? We are also interested in further experimental investigations of SBC. We hope that future work can improve our results in the HDPD in terms of robustness and degree of cooperation. Besides that, we think a natural next step is study settings in which the agents observe their similarity to one another in more realistic fashion. For example, we conjecture that similarity- based cooperation can occur when the agents can determine that their policies were generated by similar learning procedures.

---

REFERENCES
Arif Ahmed. Evidence, Decision and Causality. Cambridge University Press, 2014.
Max Albert and Ronald Asher Heiner. An indirect-evolution approach to newcomb's problem, 2001. URL https: /www. econstor we/bbiiiiiite wwt.tttttttteataee.aaaaaaaaatttt (11r pew..pdf. CSLE Discussion Paper, No. 2001-01.
Mihaly Barasz, Paul Christiano, Benja Fallenstein, Marcello Herreshoff, Patrick VaVittoire, and Eliezer Yudkowsky. Robust cooperation in the prisoner's dilemma: Program equilibrium via provability logic, 2014. URL https::/aaxx.../////////
/140115507
James Bell, Linda Linsefors, Caspar Oesterheld, and Joar Skalse. Reinforcement learning in newcomblike environments. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 2212221666222 221577 Curran Associates, Inc., 2021. URL https: https::///ccceeeeeeeeeeeoo wtw.rcceeeiingg.eeiiisss.es cc/ httppppper/////1l////eaee /Ftp:///.o22...eeeeeeeeeeee11111 83938 fal Paa.ddd-aaprrpdedee
Steven J. Brams. Newcomb's problem and prisoners' dilemma. The Journal of Conflict Resolution, 19(4):596-66) 612, 12 1975.
Gordon W. Brown. Iterative solutions of games by fictitious play. In Tjalling C. Koopmans (ed.), Activity Analysis of Production and Allocation, chapter XXIV, pp. 371-376. John Wiley & Sons and Chapman & Hall, 1951. URL https:
2015. 39951/.
CFTC and SEC. Findings regarding the market events of may 6, 2010. report of the staffs of the efte and sec to the joint advisory committee on emerging regulatory issues., 2010. URL https: / /www sec. gww.goo/o/oooooooo
retesrreeorttt.. pdf.
Vincent Conitzer and Caspar Oesterheld. Foundations of cooperative AI In AAAI-23 Senior Member Blue Sky Ideas track. 2022. URL https: www. cs. cmu. edm (a1/5//////aaa////// focal_paper: pdf.
Andrew Critch. parametric, sssurcce-boudded generalization of löb's theorem, and robust co- operation criterion for open-source game theory. Journal of Symbolic Logic, 84(4)113681138, 12 2019. doi: 10.1017/js1.2017.44
Allan Dafoe, Edward Hughes, Yoram Bachrach, Tantum Collins, Kevin McKee, Joel Leibo, Kate Larson, and Thore Graepel. Open problems in cooperative AI, 2021. URL https:/ // arxiv. org. 2012 08 6308600.000 .dff.
Richard Dawkins. The Selfish Gene. Oxford University Press, 1976.
Gary L. Drescher. Good and Real. Demystifying Paradoxes from Physics to Ethics. The MIT Press, 2006.
Jakob Foerster, Richard Chen, Maruan Al-Shedivat, Shimon Whiteson, Pieter Abbeel, and Igor Mordatch. Learning with opponent-leaming awareness. In Proceedings of the 17th International Conference on Autonomous Agents and MulliAgent Systems, pp. 122-130. 2018
C. W. Fox, F. Camara, G. Markkula, R. A. Romano, R. Madigan, and N. Merat. When should the chicken cross the road?: Game theory for autonomous vehicle human interactions. In Proceed- ings of the 4th International Conference on Vehicle Technology and Intelligent Transport Systems, volume pp. 431-434444 431-439. 2018. doi: 10.52000067754443000939
Andy Gardner and Stuart A. West. Greenbeards. Evolution: International Journal of Organic
Evolution, 64(1):25-38 2010.
David Gauthier. In the neighbourhood of the newcomb-predictor (reflections rationality). In Proceedings of the Aristotelian Society, New Series, 1988-1989, volume 89, pp. 179-194. 1989.
William D. Hamilton. The genetical evolution of social behaviour. Journal of theoretical biology, (1):1-52, 1964.
10

---

Torsten Heinrich, Yoojin Jang, Luca Mungo, Marco Pangallo, Alex Scott, Bassel Tarbush, and Samuel Wiese. Best-response dynamics, playing sequences, and convergence to equilibrium in random games. Technical report. 2021. INET Oxford Working Paper No. 2021-23.
Douglas Hofstadter. Dilemmas for superrational thinkers, leading up luring lottery. Scientific America, 248(6), 6 1983.
J. V. Howard. Cooperation in the prisoner's dilemma. Theory and Decision, 24:203-213, 5 1988. doi: 10. 10.1007/BF00148954.
Adrian Hutter. Learning in two-player games between transparent opponents, 2020. URL https: ///axxiv.rggaas//////.....
02671.
Laurent Keller and Kenneth G Ross. Selfish genes: green beard in the red fire ant. Nature, 394 (6693):573--777 1998.
Alistair Letcher, Jakob Foerster, David Balduzzi, Tim Rocktäschel, and Shimon Whiteson. Stable opponent shaping in differentiable games. In ICLR 2019. 2019.
David Lewis. Prisoners' dilemma is newcomb problem. Philosophy & Public Affairs, 8(3):235-
240, 1979.
Johannes Martens. bttps://ottanneemttttnsbggog files. wordpress com/2019/C1. Seeenloooo-meets--------e-ee-ee-
Hamilton meets causal decision theory, 2019.
URL
Dominik Mayer, Johannes Feldmaier, and Hao Shen. Reinforcement learning in conflicting envi-
ronments for autonomous vehicles. International Workshop on Robotics in the 2 century:
Challenges and Promises, 2016. URL https: /arxivv.vvvv///////////a////aiia org/abs..s....ooooo..ee...e...e. /110/.6.0001 07089.
Eric Mazumdar, Lillian J. Ratliff, and S. Shankar Sastry. On gradient-based learning in continuous games. SIAM Journal on Mathematics of Data Science, 2(1):103-11 131, 2020.
R. Preston McAfee. Effective computaillly in economic decisions. 1984. URL BtLhttps://ee mcafee.
pdf.
John F. Nash. Equilibrium points in n-person games. Proceedings of the National Academy of Sciences of the United States of America, 36(1):48-49)
1950.
Robert Nozick. Newcomb's problem and two principles of choice. In Nicholas Rescher et al. (ed.),
Essays Honor of Carl G. Hempel, pp- 114-146. Springer, 1969. URL http://t//// httttt/::/aaacullt arts. ubc. ca/ / r jocns///oos http:/oocccccnccccmmeeem pdf.
Caspar Oesterheld. Approval-directed agency and the decision theory of newcomb-like problems. Synthese, Feb 2019a. ISSN 1573-0964. doi: 10.1007/s 120007//11229-1--0224444
Caspar Oesterheld. Robust program equilibrium. Theory and Decision, 86(1):143 159, 2019b.
Caspar Oesterheld and Vincent Conitzer. Can de se choice be ex ante reasonable games of imperfect recall? 2022. URL htpp:://wwww.. https::/www.anddrv/ www.aawwww..eeeeeeee..e..e.eeeee
DeSeVsExAnte..pe .pf.
Caspar Oesterheld, Abram Demski, and Vincent Conitzer A theory of bounded inductive rationality, 2021. URL https: /tte:://seeeeeeeeeeeeeeeeeeeeeee cs duke. edu/ -ocaspr /RIA. pdf..pdf.
Martin J. Osborne. An Introduction to Game Theory. Oxford University Press, 2004.
David Queller, Eleonora Ponte, Salvatore Bozzaro, and Joan Strassmann. Single-gene green-
beard effects in the social amoeba dictyostelium discoideum. Science, 299(5603): 2003.
Ariel Rubinstein. Modeling Bounded Rationality. Zeuthen Lecture Book Series. The MIT Press, 1998.
Tuomas W Sandholm and Robert H Crites. Multiagent reinforcement learning in the iterated pris- oner's dilemma. Biosystems, 37(1-2):1447166 1996.

---

Barry Sinervo, Alexis Chaine, Jean Clobert, Ryan Calsbeek, Lisa Hazard, Lesley Lancaster, An- drew G. McAdam, Suzanne Alonzo, Gwynne Corrigan, and Michael E. Hochberg. Self- recognition, color signals, and eycles of greenbeard mutualism and altruism. Proceedings of the National Academy of Sciences, 103(19):372--337 2006.
Scott Smukalla, Marina Caldara, Nathalie Pochet, Anne Beauvais, Stephanie Guadagnini, Chen Yan, Marcelo D. Vinces, An Jansen, Marie Christine Prevost, Jean-Paul Latgé, Gerald R. Fink, Kevin R. Foster, and Kevin J. Verstrepen. Flol is variable green beard gene that drives biofilm- like cooperation budding yeast. Cell, 135(4):726-737, 2008.
Julian Stastny, Maxime Riché, Alexander Lyzhov, Johannes Treutein, Allan Dafoe, and Jesse Clifton. Normative disagreement challenge for cooperative ai. In Cooperative Al workshop, 2021.
Moshe Tennenholtz. Program equilibrium. Games and Economic Behavior, 49(2):333-37,, 11 2004.
Timon Willi, Alistair Hp Letcher, Johannes Treutlein, and Jakob Foerster. Cola: consistent learning with opponent-leariin awareness. In International Conference on Machine Learning, pp. 23804 23831. PMLR, 2022.
Guodong Zhang, Yuanhao Wang, Laurent Lessard, and Roger Grosse. Near-optimal local conver- gence of alternating gradient descent-ascent for minimax optimization. In International Confer- ence on Artificial Intelligence and Statistics, pp. 7659-7679. 2022.
Stěpán Vesely. Psychology of decision making: Effect of learning, end-effect, cheap talk, and other variables influencing decision making in the prisoner's dilemma game, 2011. URL https: //s muni, hts/thhyzz000 http://///daallaaaaappaaa
12


Reviews:
[{"review_id": "b9m0DY_REn0", "reviewer": "Reviewer_ATts", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Overall, this paper provides a unique perspective for the ICLR community -- developing theoretical insights into how cooperative agents might function. The paper focuses on develop the theory to support the sufficiency of agents sharing minimal information to achieve cooperation in a Prisoner's Dilemma setup. Theoretical and empirical results are presented.", "summary_of_the_review": "This paper provides theoretical and empirical results investigating Prisoner Dilemma-like games and whether agents might be cooperative under minimal information sharing. The theoretical analysis is a strong positive for the paper (plus the empirical results); however, the paper is difficult to parse. Without a more approachable presentation, this paper might have limited impact. The reviewer acknowledges limited expertise in game theory (while being an expert in ML).", "strength_and_weaknesses": "Strengths:\n+ The paper clearly states its contributions. These contributions include the introduction a specific game called \"diff meta games,\" and theoretical analysis regarding these games. This analysis leads to the claimed contribution that cooperative equilibria can be found with ML methods.\n+ The paper gives an appropriate if dense set of preliminaries in Section 2.\n+ The intuition/motivation added in the second to last paragraph of Section 6 is helpful.\n+ Section 8's discussion section is helpful.\n\nWeaknesses:\n-Figure 1 is, unfortunately, not beneficial. It appears to setup a typical prisoner's dilemma (Figure 1a). Figure 1b is not explained in the text. These figures should be removed or better explained (or illustrated differently).\n-Table 1 does not show the only formulation of a Prisoner's Dilemma -- \"G+1\" could be any number k > G.\n- The definitions (e.g., Definition 1 and 2) are not explained from an intuitive perspective or with a set of working examples until Example 1. This reviewer recommends each definition add intuition to unpack what is going on rather than waiting for Example 1. Even example 1, however, is more of an illustrative set of cases rather than a grounded example with actual numbers (e.g., \"N_i is some real-valued random variable\"). \n-\"Appendix C.2, but give\" should not have a comma.\n-The pretraining procedure in Section 6 is only briefly described. More details could/should be given.\n-The value of Figure 2 is unclear.\n-Figure 3 is poorly formatted and difficult to parse.\n-More discussion of limitations would be helpful. For example, how large can games be (rows and columns of the normal form game formulation) before the method breaks down?\n-The result that a single number being shared can achieve cooperation is perhaps not as impressive if one considers that the number could be a real-valued number with an infinite number of digits. Can the nature of this number be restricted?\n-What happens if agent(s) are byzantine?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "_g532bPezjv", "reviewer": "Reviewer_sbJz", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a new way to allow cooperation between ML agents using an approach based on a similarity measurement. Authors introduce a program called diff meta games, a novel variant of program meta games, where agents observe the similarity/dissimilarity between the policies using a single number, thus the similarity/ dissimilarity between each other, allowing the agents to learn to cooperate/defect. Authors theoretically and experimentally show the validity of their method of similarity-based cooperation. They prove that this approach allows the same cooperative outcomes as in the full mutual transparency setting. Moreover, they experimentally show that cooperative learning through this approach can be achieved with simple ML methods. ", "summary_of_the_review": "The authors study an interesting problem with simulating motivation. Interesting theoretical and experimental results are obtained. \n\nExplaining the results more in a more accessible way to broader audience and elaborating connections to representation learning could improve the quality of the paper.", "strength_and_weaknesses": "Mathematically rigorous results are obtained and the experiments are done to back up the claims. The paper presents contributions for enhancing similarity based cooperative learning between ML agents and clearly explains how their approach presents solutions to difficulties with existing methods. \nPaper compares full transparency setting vs their approach and explains how the proposed method addresses problems of existing methods in great detail. The authors could consider explaining the results more in a more accessible way to broader audience and elaborate connections to representation learning. \nThe paper is dense and needs more clarifications.\n\nMinor: You have used a distance measure as the similarity measurement. Does the direction matter? i.e. is diff(θ1,θ2) = diff(θ2,θ1) in general or should the measurement change according to the perceiver? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "9MKEU1-3APf", "reviewer": "Reviewer_cK7h", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "One way to enable cooperative outcomes is to make the agents mutually transparent to each other, however, full transparency is often unrealistic. In this paper, authors introduce a more realistic setting in which agents only observe how similar they are to each other. It is proved that this allows for the same set of cooperative outcomes as the full transparency setting.", "summary_of_the_review": "One way to enable cooperative outcomes is to make the agents mutually transparent to each other, however, full transparency is often unrealistic. In this paper, authors introduce a more realistic setting in which agents only observe how similar they are to each other. It is proved that this allows for the same set of cooperative outcomes as the full transparency setting.\nStrengths:\n1.\tA similarity-based cooperation mechanism is proposed.\n2.\tEfficient theorems have been proposed to analyze the Nash equilibrium property of diff meta games.\nWeaknesses:\n1.\tFull transparency are rare in reality, however, similarity is also not an explicit property which can be obtained by observation, how to evaluate the similarity between players in fact? \n2.\tIt is a basic question, why players prefer to cooperate with similar players? Which lacks sufficient explanations.\n3.\tA similarity-based cooperation mechanism can promote cooperation, what will happen if the other player do not adopt the similarity-based cooperation mechanism, for example, the other player uses the TFT strategy. On the other word, I am interested in the robustness of the similarity-based cooperation mechanism. \n4.\tThe words in Fig. 3(b) is too small to read, which should be improved.", "strength_and_weaknesses": "Strengths:\n1.\tA similarity-based cooperation mechanism is proposed.\n2.\tEfficient theorems have been proposed to analyze the Nash equilibrium property of diff meta games.\nWeaknesses:\n1.\tFull transparency are rare in reality, however, similarity is also not an explicit property which can be obtained by observation, how to evaluate the similarity between players in fact? \n2.\tIt is a basic question, why players prefer to cooperate with similar players? Which lacks sufficient explanations.\n3.\tA similarity-based cooperation mechanism can promote cooperation, what will happen if the other player do not adopt the similarity-based cooperation mechanism, for example, the other player uses the TFT strategy. On the other word, I am interested in the robustness of the similarity-based cooperation mechanism. \n4.\tThe words in Fig. 3(b) is too small to read, which should be improved.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "2dQz3t9YvO", "reviewer": "Reviewer_diwd", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies promoting cooperation in games. The authors provide a method where similarity information can achieve similar cooperation levels compared to previous method that requires agent's policies to be transparent.", "summary_of_the_review": "The reviewer generally agree this paper makes some contribution to the cooperation learning in social dilemmas, but has concerns about its position in the literature.", "strength_and_weaknesses": "The paper provides a clear discussion of the motivation and details of the proposed method.\n\nThe reviewer has a major concern about the relationship between the proposed method and the large body of literature on homophily and assortative matchmaking (see the section below).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject