SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We first need a few definitions of properties of diff. Let  $\Gamma$  be a symmetric game. We say that diff is minimized by copies if for all policies  $\pi, \pi'$ , all  $y$  and  $i = 1,2$ ,  $P(\text{diff}_i(\pi, \pi') < y) \leq P(\text{diff}_i(\pi, \pi) < y)$ . For example, the diff function in Example 1 is minimized by copies. The diff functions in the proof of Theorem 3 are not in general minimized by copies when the given base game is symmetric. For example, to achieve  $(C, 2/3 * C + 1/3 * D)$  in equilibrium, the proof of Theorem 3 (as sketched above) uses the policies  $\pi_1^* = (C, 1/2, D)$  and  $\pi_2^* = (2/3 * C + 1/3 * D, 1/2, D)$  and a diff function with  $\text{diff}(\pi_1^*, \pi_2^*) = (0,0)$  but  $\text{diff}(\pi_1^*, \pi_1^*) = (1,1)$ . If the base game is symmetric, we call diff symmetric if for all  $\pi_1, \pi_2$ ,  $\text{diff}(\pi_1, \pi_2)$  is distributed the same as  $\text{diff}(\pi_2, \pi_1)$  and  $(\text{diff}_1(\pi_1, \pi_2), \text{diff}_2(\pi_1, \pi_2))$  is distributed the same as  $(\text{diff}_2(\pi_1, \pi_2), \text{diff}_1(\pi_1, \pi_2))$ .
Finally, we need a more complicated but nonetheless intuitive property of diff functions. In this paper, we generally imagine that low values of diff are informative about the other player's policy. In contrast, we will her assume that high values of diff are uninformative. That is, for any  $\sigma_{i}$  and  $\pi_{-i}$ , we will assume that there is a policy  $\pi_{i}$  that plays  $\sigma_{i}$  against  $\pi_{-i}$  and triggers the above-threshold policy of  $\pi_{-i}$  with the highest-possible probability. Formally, let  $\pi_{-i} = (\sigma_{-i}^{\leq},\theta_{-i},\sigma_{-i}^{>})$  be any threshold policy. Let  $p$  be the supremum of numbers  $p'$  for which there is  $\pi_{i}$  s.t. in  $(\pi_{i},\pi_{-i})$ , Player  $-i$  plays  $(1 - p')\sigma_{-i}^{\leq} + p'\sigma_{-i}^{>}$ . Let  $\sigma_{\pi_{-i}}^{\max} = (1 - p)\sigma_{-i}^{\leq} + p\sigma_{-i}^{>}$ . Intuitively,  $\sigma_{\pi_{-i}}^{\max}$  is the strategy played by  $\pi_{-i}$  against the most different opponent policies. For the examples of Section 3 we have  $p = 1$  and thus simply  $\sigma_{\pi_{-i}}^{\max} = \sigma_{-i}^{>}$ . But if diff is bounded, then we might even have  $p = 0$  or anything in between.
Definition 3. We call diff:  $\bar{\mathcal{A}}_1\times \bar{\mathcal{A}}_2\rightsquigarrow \mathbb{R}^2$  high value uninformative if for each threshold policy  $\pi_{-i}$ ,  $\sigma_{i}$  and  $\epsilon >0$  there is a threshold policy  $\pi_{i}$  such that in  $(\pi_i,\pi_{-i})$ , a strategy profile within  $\epsilon$  of  $(\sigma_{i},\sigma_{\pi_{-i}}^{\max})$  is played.
We are now ready to state a uniqueness result for the Nash equilibria of diff meta games.
Theorem 4. Let  $\Gamma$  be a player-symmetric, additively decomposable game. Let diff be symmetric, high-value uninformative, and minimized by copies. Then if  $(\pi_1,\pi_2)$  is a Nash equilibrium that is not Pareto-dominated by another Nash equilibrium, we have that  $V_{1}(\pi_{1},\pi_{2}) = V_{2}(\pi_{1},\pi_{2})$ . Hence, if there exists a Pareto-optimal Nash equilibrium, its payoffs are unique, Pareto-dominant among Nash equilibria and equal across the two players.
We prove Theorem 4 in Appendix D.3. Roughly, we prove that under the given assumptions, equilibrium policies are more beneficial to the opponent when observing a diff value below the threshold than if they observe a diff value above the threshold. Second, we show that if in a given strategy profile Principal  $i$  receives a lower utility than Principal  $-i$ , then Principal  $i$  can increase her utility by submitting a copy of Principal  $-i$ 's policy. Appendix D.1 shows why the assumptions (additive decomposability of the game and and high-value uninformativeness and symmetry of diff) are necessary.
6 Machine learning for similarity-based cooperation in complex games
Our results so far demonstrate the theoretical viability of similarity-based cooperation, but leave open questions regarding its practicality. In complex environments, where cooperating and defecting are by themselves complex operations, can we find the cooperative equilibria for a given diff function with machine learning methods?
6.1 A novel pretraining method for similarity-based cooperation
We now describe Cooperate against Copies and Defect against Random (CCDR), a simple ML method to find cooperative equilibria in complex games. To use this method, we consider neural net policies  $\pi_{\theta}$  parameterized by a real vector  $\theta$ . First, for any given diff game, let  $V^{d} \colon (\mathbb{R}^{m})^{\left(\mathbb{R}^{n+1}\right)} \times (\mathbb{R}^{m})^{\left(\mathbb{R}^{n+1}\right)} \to \mathbb{R}^{2}$  be the utility of a version of the game in which diff is non-noisy. CCDR trains a model  $\pi_{\theta_{i}}$  to maximize  $V^{d}(\pi_{\theta_{i}}, \pi_{\theta_{i}}) + V^{d}(\pi_{\theta_{i}}, \pi_{\theta_{-i}^{\prime}})$  for randomly sampled  $\theta_{-i}^{\prime}$ . That is, each player  $i$  pretrains their policy  $\pi_{\theta_{i}}$  to do well in both of the following scenarios: principal  $-i$  copies principal  $i$ 's model; and principal  $-i$  generates a random model. The method is named for its intended effect in Prisoner's Dilemma-like games. Note, however, that it is well-defined in all symmetric games, not just Prisoner's Dilemma-like games.
CCDR pretraining is motivated by two considerations. First, in games like the Prisoner's Dilemma, there exist cooperative equilibria of policies that cooperate at a diff value of 0 and defect as the
Figure 2: The figure illustrates how utilities are calculated in the HDPD. The function  $f_{i}$  is an action chosen by Player  $i$ . The area between the curves  $f_{i}$  and  $f_{D}$  determines, intuitively, how much the agent defects; the area between the curves  $f_{i}$  and  $f_{C}$  determines how much it cooperates. The  $f_{i}$  shown in the figure is much closer to defecting than to cooperation.
perceived diff value increases. We give a toy model of this in Appendix E. CCDR puts in place the rudimentary structure of these equilibria. Note, however, that CCDR does not directly optimize for the model's ability to form an equilibrium. Second, CCDR can be thought of as a form of curriculum training. Before trying to play diff games against other (different but similar) learned agents, we might first train a policy to solve two (conceptually and technically) easier related problems.
6.2 A high-dimensional one-shot Prisoner's Dilemma
To study similarity-based cooperation in an ML context, we need a more complex version of the Prisoner's Dilemma. The complex Prisoner's Dilemma-like games studied in the multi-agent learning community generally offer other mechanisms that establish cooperative equilibria (e.g., playing a game repeatedly). For our experiments, however, we specifically need SBC to be the only mechanism to establish cooperation.
We therefore introduce a new game, the High-Dimensional (one-shot) Prisoner's Dilemma (HDPD). The goal is to give a variant of the one-shot Prisoner's Dilemma that is conceptually simple but introduces scalable complexity that makes finding, for example, exact best responses in the diff meta game intractable. In addition to  $G$ , the HDPD is parameterized by two functions  $f_{C}, f_{D} \colon \mathbb{R}^{n} \to \mathbb{R}^{m}$  representing the two actions Cooperate and Defect, respectively, as well as a probability measure  $\mu$  over  $\mathbb{R}^n$ . Each player's action is also a function  $f_{i} \colon \mathbb{R}^{n} \to \mathbb{R}^{m}$ . This is illustrated in Figure 2 for the case of  $n = 1$  and  $m = 1$ . For any pair of actions  $f_{1}, f_{2}$ , payoffs are then determined as follows. First, we sample some  $\mathbf{x}$  according to  $\mu$  from  $\mathbb{R}^n$ . Then to determine how much Player  $i$  cooperates, we consider the distance  $d(f_{i}(\mathbf{x}), f_{C}(\mathbf{x}))$  to determine, roughly speaking, how much Player 1 cooperates. The larger the distance the less cooperative is  $f_{i}$ . In the case of  $n = m = 1$  and  $\mu$  uniform, the expected distance between  $f_{i}(x)$  and  $f_{D}(x)$  is simply the area between the curves of  $f_{i}$  and  $f_{D}$ , as visualized in Figure 2. We analogously determine how much the players defect. Formally, we define  $u_{i}(f_{1}, f_{2}) = -\mathbb{E}_{\mathbf{x} \sim \mu}[d(f_{i}(\mathbf{x}), f_{D}(\mathbf{x})) + Gd(f_{-i}(\mathbf{x}), f_{C}(\mathbf{x}))] / \mathbb{E}_{\mathbf{x} \sim \mu}[d(f_{C}(\mathbf{x}), f_{D}(\mathbf{x}))]$ . Thus, the action  $f_{i} = f_{D}$  corresponds to defecting and the action  $f_{C}$  corresponds to cooperating, e.g.,  $\mathbf{u}(f_{C}, f_{C}) = (-1, -1)$  and  $\mathbf{u}(f_{D}, f_{D}) = (-G, -G)$ . The unique equilibrium of this game is  $(f_{D}, f_{D})$ . In our experiments, we specifically used  $G = 5$ .
We consider a diff meta game on the HDPD. Formally, a diff-based policy for the HDPD is a function  $\mathbb{R} \to (\mathbb{R}^m)^{(\mathbb{R}^n)}$ . For notational convenience, we will instead write policies as functions  $\mathbb{R}^{n+1} \to \mathbb{R}^m$ . We then define our diff function by  $\mathrm{diff}_i(\pi_1, \pi_2) = \mathbb{E}_{(y,\mathbf{x}) \sim \nu}[d(\pi_1(y,\mathbf{x}), \pi_2(y,\mathbf{x}))] + Z_i$ , where  $\nu$  is some probability distribution over  $\mathbb{R}^{n+1}$  and  $Z_i$  is some real-valued noise.
Experimental setup. We trained on the environment from Section 6.2. We selected a fixed set of hyperparameters based on prior exploratory experiments and the theoretical considerations in Appendix E. We then randomly initialized  $\theta_{1}$  and  $\theta_{2}$ , CCDR-pretrained them (independently), and then trained  $\theta_{1}$  and  $\theta_{2}$  against each other using ABR. We repeated the experiment with 28 random seeds. As control, we also ran the experiment without CCDR on 26 seeds. We also ran experiments with Learning with Opponent-Learning Awareness (LOLA) (Foerster et al., 2018), which we report in Appendix G.
Results. First, we observe that in the runs without CCDR pretraining, the players generally converge to mutual defection during alternating best response learning. In particular, in all 26 runs, at least one
Figure 3: (a) The behavior of a CCDR-pretrained policy. For each perceived value of diff to the opponent  $y$ , the green line shows the expected distance of the learned policy's choice to  $f_{C}$  (smaller means more cooperative) and the read line shows the expected distance to  $f_{D}$ . (b) Losses of Player 1 in 10 runs through the ABR phase.
player's utility was below  $-5$ . Only two runs had a utility above  $-5$  for one of the players  $(-4.997$  and  $-4.554)$ . The average utility across the 26 runs and across the two players was  $-5.257$  with a standard deviation of 0.1978. Anecdotally, these results are robust - ABR without pretraining practically never finds cooperative equilibria in the HDPD.
Second, we observe that in all 28 runs, CCDR pretraining qualitatively yields the desired policy models, i.e., a policy that cooperates at low values of diff and gradually comes closer to defecting at high values of diff. Figure 3a shows a representative example.
Our main positive experimental result is that after CCDR pretraining, the models converged in alternating best response learning to a partially cooperative equilibrium in 26 out of 28 runs. Thus, the cooperative equilibria postulated in general by Theorem 3 and in simplified examples by Propositions 1 and 2 (as well as Proposition 25), do indeed exist and can be found with simple methods. The minimum utility of either player across the 26 successful runs was -4.854. The average utility across all runs and the two players was about -2.77 and thus a little closer to  $u(f_{C}, f_{C}) = -1$  than to  $u(f_{D}, f_{D}) = -5$ . The standard deviation was about 1.19. Figure 3b shows the losses (i.e., the negated utilities) across ABR learning. Generally, the policies also converge to receiving approximately the same utility (cf. Section 5). The average of the absolute differences in utility between the two players at the end of the 28 runs is about 0.04 with a standard deviation of 0.05. We see that in line with Theorem 4, we tend to learn egalitarian equilibria in this symmetric, additively decomposable setting. After alternating best response learning, the models generally have a similar structure as the model in Figure 3a, though often they cooperate only a little at low diff values. Based on prior exploratory experiments, CCDR's success is moderately robust.
6.4 Discussion
Without pretraining, ABR learning unsurprisingly converges to mutual defection. This is due to a bootstrapping problem. Submitting a policy of the form "cooperate with similar policies, defect against different policies" is a unique best response against itself. If the opponent model  $\pi_{-i}$  is not of this form, then any policy  $\pi_i$  that defects, i.e., that satisfies  $\pi_i(\text{diff}(\pi_i, \pi_{-i})) = f_D$ , is a best response. Because  $f_C$  is complex, learning a model that cooperates at all is unlikely. (Even if  $f_C$  was simple, the appropriate use of the perceived diff value would still be specific and thus unlikely to be found by chance.) Similar failures to find the more complicated cooperative equilibria by default have also been observed in the iterated PD (Sandholm & Crites 1996; Foerster et al. 2018; Letcher et al. 2019) and in the open-source PD (Hutter, 2020). Opponent shaping methods have been used successfully to learn to cooperate both in the iterated Prisoner's Dilemma (Foerster et al. 2018; Letcher et al. 2019) and the open-source Prisoner's Dilemma (Hutter, 2020). Our experiments in Appendix G show that LOLA can also learn SBC, but unfortunately not as robustly as CCDR pretraining.
CCDR pretraining reliably finds models that cooperate with each other and that continue to partially cooperate with each other throughout ABR training. This shows that when given some guidance, ABR can find SBC equilibria – SBC equilibria have at least some “basin of attraction”. Our experiments therefore suggest that SBC is a promising means of establishing cooperation between ML agents.
That said, CCDR has many limitations that we hope can be addressed in future work. For one, in many games the best response against a randomly generated opponents does poorly against a rational opponent. Second, our experiments show that while the two policies almost fully cooperate after CCDR pretraining, they quickly partially unlearn to cooperate in the ABR phase. We would prefer a method that preserves closer to full cooperation throughout ABR-style training. Third, while CCDR seems to often work, it can certainly fail in games in which SBC is possible. Learning to distinguish randomly sampled opponent policies from copies will in many settings not prepare an agent to distinguish cooperative/SBC opponents from uncooperative but trained (not randomly sampled) opponents. Consequently, CCDR may sometimes result in insufficiently steep incentive curves, cooperating with too dissimilar opponents. We suspect that to make progress on the latter issues we need training procedures that more explicitly reason about incentives à la opponent shaping (cf. our experiments with LOLA Appendix G).
We here relate our project to the two most closely related lines of work. In Appendix H we discuss more distantly related lines of work.
Program equilibrium. We already discussed in Section 1 the literature on program meta games in which players submit computer programs as policies and the programs fully observe each other's code (McAfee 1984; Howard 1988; Rubinstein 1998, Section 10.4; Tennenholtz 2004). Interestingly, some constructions for equilibria in program meta games are similarity based. For example, the earliest cooperative program equilibrium for the Prisoner's Dilemma, described in all four of the above-cited papers, is the program "Cooperate if the opponent's program is equal to this program; else Defect". The program "cooperate if my cooperation implies cooperation from the opponent" proposed by Critch et al. (2022) is also similarity-based. Other approaches to program equilibrium cannot be interpreted as similarity based, however (see, e.g., Barasz et al., 2014; Critch, 2019; Oesterheld, 2019b). To our knowledge, the only published work on ML in program equilibrium is due to Hutter (2020). It assumes the programs to have the structure proposed by Oesterheld (2019b) on simple normal-form games, thus leaving only a few parameters open. Similar to our experiments, Hutter shows that best response learning fails to converge to the cooperative equilibria. In Hutter's experiments, the opponent shaping methods LOLA (Foerster et al., 2018) and SOS (Letcher et al., 2019) converge to mutual cooperation.
Decision theory and Newcomb's problem. Brams (1975) and Lewis (1979) have pointed out that the Prisoner's Dilemma against a similar opponent closely resembles Newcomb's problem, a problem first introduced to the decision-theoretical literature by Nozick (1969). Most of the literature on Newcomb's problem is about the normative, philosophical question of whether one should cooperate or defect in a Prisoner's Dilemma against an exact copy. Our work is inspired by the idea that in some circumstances one should cooperate with similar opponents. However, this literature only informally discusses the question of whether to also cooperate with agents other than exact copies (Hofstadter e.g., 1983; Drescher 2006, Ch. 7; Ahmed 2014, Sect. 4.6.3). We address this question formally.
One idea behind the present project, as well as the program game literature, is to analyze a decision situation from the perspective of (actual or hypothetical) principals who design policies. The principals find themselves in an ordinary strategic situation. This is how our analysis avoids the philosophical issues arising in the agent's perspective. Similar changes in perspective have been discussed in the literature on Newcomb's problem (e.g., Gauthier 1989; Oesterheld & Conitzer 2022).
We make a strong case for the promise of similarity-based cooperation as a means of improving outcomes from interactions between ML agents. At the same time, there are many avenues for future work. On the theoretical side, we would be especially interested in generalizations of Theorem 4, that is, theorems that tell us what outcomes we should expect in diff meta games. Is it true more generally that under reasonable assumptions about the diff function, we can expect SBC to result in fairly specific, symmetric, Pareto-optimal outcomes? We are also interested in further experimental investigations of SBC. We hope that future work can improve on our results in the HDPD in terms of robustness and degree of cooperation. Besides that, we think a natural next step is to study settings in which the agents observe their similarity to one another in a more realistic fashion. For example, we conjecture that SBC can occur when the agents can determine that their policies were generated by similar learning procedures.
We thank Stephen McAleer, Emery Cooper, Daniel Filan, John Mori, and our anonymous reviewers helpful discussions and comments. We thank Maxime Riché and the Center on Long-Term Risk for compute support. Caspar Oesterheld and Vincent Conitzer would like to thank the Cooperative AI Foundation, Polaris Ventures (formerly the Center for Emerging Risk Research) and Jaan Tallinn's donor-advised fund at Founders Pledge for financial support. Roger Grosse acknowledges financial support from Open Philanthropy. Caspar Oesterheld and Johannes Treutlein are grateful for support by FLI PhD Fellowships. Johannes Treutlein was additionally supported by an OpenPhil AI PhD Fellowship.
Melissa Acevedo and Joachim I Krueger. Evidential reasoning in the prisoner's dilemma. The American Journal of Psychology, 118(3):431-457, 2005.
Arif Ahmed. Evidence, Decision and Causality. Cambridge University Press, 2014.
Ozan Aksoy. Effects of heterogeneity and homophily on cooperation. Social Psychology Quarterly, 78(4):324-344, 2015.
Max Albert and Ronald Asher Heiner. An indirect-evolution approach to Newcomb's problem, 2001. URL https://www.econstor.eu/bitstream/10419/23110/1/2001-01_newc.pdf. CSLE Discussion Paper, No. 2001-01.
Robert Axelrod. The Evolution of Cooperation. Basic Books, 1984.
Mihaly Barasz, Paul Christiano, Benja Fallenstein, Marcello Herreshoff, Patrick LaVictoire, and Eliezer Yudkowsky. Robust cooperation in the prisoner's dilemma: Program equilibrium via provability logic, 1 2014. URL https://arxiv.org/abs/1401.5577.
James Bell, Linda Linsefors, Caspar Oesterheld, and Joar Skalse. Reinforcement learning in Newcomblike environments. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 22146-22157. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper/2021/file/b9ed18a301c9f3d183938c451fa183df-Paper.pdf.
Steven J. Brams. Newcomb's problem and prisoners' dilemma. The Journal of Conflict Resolution, 19(4):596-612, 12 1975.
Gordon W. Brown. Iterative solutions of games by fictitious play. In Tjalling C. Koopmans (ed.), Activity Analysis of Production and Allocation, chapter XXIV, pp. 371-376. John Wiley & Sons and Chapman & Hall, 1951. URL https://archive.org/details/in.ernet.dli.2015.39951/.
CFTC and SEC. Findings regarding the market events of may 6, 2010. report of the staffs of the cftc and sec to the joint advisory committee on emerging regulatory issues., 9 2010. URL https://www.sec.gov/files/marketevents-report.pdf.
Phillip J.K. Christoffersen, Andreas A. Haupt, and Dylan Hadfield-Menell. Get it in writing: Formal contracts mitigate social dilemmas in multi-agent rl. In Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems, AAMAS '23, pp. 448-456. International Foundation for Autonomous Agents and Multiagent Systems, Richland, SC, 2023. ISBN 9781450394321.
Vincent Conitzer and Caspar Oesterheld. Foundations of cooperative AI. Proceedings of the AAAI Conference on Artificial Intelligence, 37(13):15359-15367, Sep. 2023. doi: 10.1609/aaai.v37i13.26791. URL https://ojs.aaii.org/index.php/AAAI/article/view/26791.
Andrew Critch. A parametric, resource-bounded generalization of Löb's theorem, and a robust cooperation criterion for open-source game theory. Journal of Symbolic Logic, 84(4):1368-1381, 12 2019. doi: 10.1017/jsl.2017.42.