SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Caspar Oesterheld $^{1*}$  Johannes Treutlein $^{2}$  Roger Grosse $^{3,4,5}$
Vincent Conitzer<sup>1,6</sup> Jakob Foerster<sup>7</sup>
$^{a}$ oesterheld@cmu.edu  $^{b}$ FOCAL, Carnegie Mellon University  $^{c}$ CHAI, UC Berkeley
$^{3}$ Anthropic  $^{4}$ Vector Institute  $^{5}$ University of Toronto
$^{6}$ Institute for Ethics in AI, University of Oxford  $^{7}$ FLAIR, University of Oxford
As machine learning agents act more autonomously in the world, they will increasingly interact with each other. Unfortunately, in many social dilemmas like the one-shot Prisoner's Dilemma, standard game theory predicts that ML agents will fail to cooperate with each other. Prior work has shown that one way to enable cooperative outcomes in the one-shot Prisoner's Dilemma is to make the agents mutually transparent to each other, i.e., to allow them to access one another's source code (Rubinstein, 1998; Tennenholtz, 2004) – or weights in the case of ML agents. However, full transparency is often unrealistic, whereas partial transparency is commonplace. Moreover, it is challenging for agents to learn their way to cooperation in the full transparency setting. In this paper, we introduce a more realistic setting in which agents only observe a single number indicating how similar they are to each other. We prove that this allows for the same set of cooperative outcomes as the full transparency setting. We also demonstrate experimentally that cooperation can be learned using simple ML methods.
As AI systems start to autonomously interact with the world, they will also increasingly interact with each other. We already see this in contexts such as trading agents (CFTC & SEC, 2010), but the number of domains where separate AI agents interact with each other in the world is sure to grow; for example, consider autonomous vehicles. In the language of game theory, AI systems will play general-sum games with each other. For example, autonomous vehicles may find themselves in Game-of-Chicken-like dynamics with each other (cf. Fox et al., 2018). In many of these interactions, cooperative or even peaceful outcomes are not a given. For example, standard game theory famously predicts and recommends defecting in the one-shot Prisoner's Dilemma. Even when cooperative equilibria exist, there are typically many equilibria, including uncooperative and asymmetric ones. For instance, in the infinitely repeated Prisoner's Dilemma, mutual cooperation is played in some equilibria, but so is mutual defection, and so is the strategy profile in which one player cooperates  $70\%$  of the time while the other cooperates  $100\%$  of the time. Moreover, the strategies from different equilibria typically do not cooperate with each other. A recent line of work at the intersection of AI/(multi-agent) ML and game theory aims to increase AI/ML systems' ability to cooperate with each other (Stastny et al., 2021; Dafoe et al., 2020; Conitzer & Oesterheld, 2023).
Prior work has proposed to make AI agents mutually transparent to allow for cooperation in equilibrium (McAfee 1984; Howard 1988; Rubinstein 1998, Section 10.4; Tennenholtz 2004; Barasz et al. 2014; Critch 2019; Oesterheld 2019b). Roughly, this literature considers for any given 2-player normal-form game  $\Gamma$  the following program meta game: Both players submit a computer program, e.g., some neural net, to choose actions in  $\Gamma$  on their behalf. The computer program then receives as input the computer program submitted by the other player. The aforementioned works have shown that the program meta game has cooperative equilibria in the Prisoner's Dilemma.
Unfortunately, there are multiple obstacles to cooperation based on full mutual transparency. 1) Settings of full transparency are rare in the real world. 2) Games played with full transparency in 37th Conference on Neural Information Processing Systems (NeurIPS 2023).
Figure 1: A graphical representation of diff meta games (Definition 1). Nodes with two incoming nodes are determined by applying one of the parent nodes to the other.
general have many equilibria, including ones that are much worse for some or all players than the Nash equilibria of the underlying game (see the folk theorems given by Rubinstein 1998, Section 10.4, and Tennenholtz 2004). In particular, full mutual transparency can make the problem of equilibrium selection very difficult. 3) The full transparency setting poses challenges to modern ML methods. In particular, it requires at least one of the models to receive as input a model that has at least as many parameters as itself. Meanwhile, most modern successes of ML use models that are orders of magnitudes larger than the input. Consequently, we are not aware of successful projects on learning general-purpose models such as neural nets in the full transparency setting.
Contributions. In this paper we introduce a novel variant of program meta games called difference (diff) meta games that enables cooperation in equilibrium while also addressing obstacles 1-3. As in the program meta game, we imagine that two players each submit a program or policy to instruct an agent to play a given game, such as the Prisoner's Dilemma. The main idea is that before choosing an action, the agents receive credible information about how similar the two players' policies are to each w.r.t. how they make the present decision. In the real world, we might imagine that this information is provided by a mediator (cf. Monderer & Tennenholtz, 2009; Ivanov et al., 2023; Christoffersen et al., 2023) who wants to enable cooperation. We may also imagine that this signal is obtained more organically. For example, we might imagine that the agents can see that their policies were generated using the same code base. We formally introduce this setup in Section 3. Because it requires a much lower degree of mutual transparency, we find the diff meta game setup more realistic than the full mutual transparency setting. Thus, it addresses Obstacle 1 to cooperation based on full mutual transparency.
Diff meta games can still have cooperative equilibria when the underlying base game does not. Specifically, in Prisoner's Dilemma-like games, there are equilibria in which both players submit policies that cooperate with similar policies and thus with each other. We call this phenomenon similarity-based cooperation (SBC). For example, consider the Prisoner's Dilemma as given in Table 1 for  $G = 3$ . (We study such examples in more detail in Section 3.) Imagine that the players can only submit threshold policies that are parameterized only by a single real-valued threshold  $\theta_{i}$  and cooperate if and only if the perceived difference to the opponent is at most  $\theta_{i}$ . As a measure of difference, the policies observe  $\text{diff}(\theta_{1}, \theta_{2}) = |\theta_{1} - \theta_{2}| + Z$ , where  $Z$  is sampled independently for each player according to the uniform distribution over [0, 1]. For instance, if Player 1 submits a threshold of  $1/2$  and Player 2 submits a threshold of  $3/4$ , then the perceived difference is  $1/4 + Z$ . Hence, Player 1 cooperates with probability  $P(1/4 + Z \leq 1/2) = 1/4$  and Player 2 cooperates with probability  $P(1/4 + Z \leq 3/4) = 1/2$ . It turns out that  $(\theta_{1} = 1, \theta_{2} = 1)$ , which leads to mutual cooperation with probability 1, is a Nash equilibrium of the meta game. Intuitively, the only way for either player to defect more is to lower their threshold. But then  $|\theta_{1} - \theta_{2}|$  will increase, which will cause the opponent to defect more (at a rate of  $1/2$ ). This outweighs the benefit of defecting more oneself.
In Section 4, we prove a folk theorem for diff meta games. Roughly speaking, this result shows that observing a diff value is sufficient for enabling all the cooperative outcomes that full mutual
Cooperate Defect
Cooperate Defect
G,G	0,G+1
G+1,0	1,1
Table 1: The Prisoner's Dilemma, parameterized by some number  $G > 1$ .
transparency enables. Specifically, we show that for every individually rational strategy profile  $\sigma$  (i.e., every strategy profile that is better for each player than their minimax payoff), there is a function diff such that  $\sigma$  is played in an equilibrium of the resulting diff meta game.
Next, we address Obstacle 2 to full mutual transparency – the multiplicity of equilibria. First, note that any given measure of similarity will typically only enable a specific set of equilibria, much smaller than the set of individually rational strategy profiles. For instance, in the above example, all equilibria are symmetric. In general, one would hope that similarity-based cooperation will result in symmetric outcomes in symmetric games. After all, the new equilibria of the diff game are based on submitting similar policies and if two policies play different strategies against each other, they cannot be similar. In Section 5, we substantiate this intuition. Specifically, we prove, roughly speaking, that in symmetric, additively decomposable games, the Pareto-optimal equilibrium of the meta game is unique and gives both players the same utility, if the measure of difference between the agents satisfies a few intuitive requirements (Section 5). For example, in the Prisoner's Dilemma, the unique Pareto-optimal equilibrium of the meta game must be one in which both players cooperate with the same probability.
Finally we show that diff meta games address Obstacle 3: we demonstrate that in games with higher-dimensional action spaces, we can find cooperative equilibria of diff meta games with ML methods. In Section 6.4, we show that, if we initialize the two policies randomly and then let each of them learn to be a best response to the other, they generally converge to the Defect-Defect equilibrium. This is expected based on results in similar contexts, such as in the Iterated Prisoner's Dilemma. However, in Section 6.1, we introduce a novel, general pretraining method that trains policies to cooperate against copies and defect (i.e., best respond) against randomly generated policies. Our experiments show that policies pretrained in this way find partially cooperative equilibria of the diff game when trained against each other via alternating best response training.
We discuss how the present paper relates to prior work in Section 7. We conclude in Section 8 with some ideas for further work.
Elementary game theory definitions. We assume familiarity with game theory. For an introduction, see Osborne (2004). A (two-player, normal-form) game  $\Gamma = (A_{1}, A_{2}, \mathbf{u})$  consists of sets of actions or pure strategies  $A_{1}$  and  $A_{2}$  for the two players and a utility function  $\mathbf{u} \colon A_{1} \times A_{2} \to \mathbb{R}^{2}$ . Table 1 gives the Prisoner's Dilemma as a classic example of a game. A mixed strategy for Player  $i$  is a distribution over  $A_{i}$ . We denote the set of such distributions by  $\Delta(A_{i})$ . We can extend  $\mathbf{u}$  to mixed strategies by taking expectations, i.e.,  $\mathbf{u}(\sigma_{1}, \sigma_{2}) \coloneqq \sum_{a_{1} \in A_{1}, a_{2} \in A_{2}} \sigma_{1}(a_{1}) \sigma_{2}(a_{2}) \mathbf{u}(a_{1}, a_{2})$ . For any player  $i$ , we use  $-i$  to denote the other player. We call  $\sigma_{i}$  a best response to a strategy  $\sigma_{-i} \in \Delta(A_{-i})$ , if  $\mathrm{supp}(\sigma_{i}) \subseteq \arg \max_{a_{i} \in A_{i}} u_{i}(a_{i}, \sigma_{-i})$ , where  $\mathrm{supp}$  denotes the support. A strategy profile  $\sigma \in \Delta(A_{1}) \times \Delta(A_{2})$  is a vector of strategies, one for each player. We call a strategy profile  $(\sigma_{1}, \sigma_{2})$  a (strict) Nash equilibrium if  $\sigma_{1}$  is a (unique) best response to  $\sigma_{2}$  and vice versa. As first noted by Nash (1950), each game has at least one Nash equilibrium. We say that a strategy profile  $\sigma$  is individually rational if each player's payoff is at least her minimax payoff, i.e., if  $u_{i}(\sigma) \geq \min_{\sigma_{-i} \in \Delta(A_{-i})} \max_{a_{i} \in A_{i}} u_{i}(a_{i}, \sigma_{-i})$  for  $i = 1, 2$ . We say that  $\sigma$  is Pareto-optimal if there exists no  $\sigma'$  s.t.  $u_{i}(\sigma') \geq u_{i}(\sigma)$  for  $i = 1, 2$  and  $u_{i}(\sigma') > u_{i}(\sigma)$  for at least one  $i$ .
Symmetric games and additively decomposable games. We say that a game is (player) symmetric if  $A_{1} = A_{2}$  and for all  $a_{1}, a_{2}$  for  $i = 1, 2$ , we have that  $u_{i}(a_{1}, a_{2}) = u_{-i}(a_{2}, a_{1})$ . The Prisoner's Dilemma in Table 1 is symmetric. We say that a game additively decomposes into  $(u_{i,j} \colon A_{j} \to \mathbb{R})_{i,j \in \{1,2\}}$  if  $u_{i}(a_{1}, a_{2}) = u_{i,1}(a_{1}) + u_{i,2}(a_{2})$  for all  $i = \{1,2\}$  and all  $a_{1} \in A_{1}, a_{2} \in A_{2}$ . Intuitively, this means that each action  $a_{j}$  of Player  $j$  generates some amount of utility  $u_{i,j}(a_{j})$  for Player  $i$  independently of what Player  $-j$  plays. For example, the Prisoner's Dilemma in Table 1 is additively decomposable, where  $u_{i,i} \colon \text{Cooperate} \mapsto 0$ , Defect  $\mapsto 1$  and  $u_{i,-i} \colon \text{Cooperate} \mapsto$
$G$ , Defect  $\mapsto 0$  for  $i = 1,2$ . Intuitively, Cooperate generates  $G$  for the opponent and 0 for oneself, while Defect generates 1 for oneself and 0 for the opponent.
Alternating best response learning. The orthodox approach to learning in games is to learn to best respond to the opponent, essentially ignoring that the opponent is also a learning agent. In this paper, we specifically consider alternating best response (ABR) learning. In ABR, the players take turns. In each turn, one of the two players updates the parameters  $\theta_{i}$  of her strategy to optimize  $u_{i}(\pmb{\theta}_{i},\pmb{\theta}_{-i})$  i.e., updates her model to be a best response to the opponent's current model (Brown cf. 1951; Zhang et al. 2022; Heinrich et al. 2023). Since learning an exact best response is generally intractable, we will specifically consider the use of gradient ascent in each turn to optimize  $u_{i}(\pmb{\theta}_{i},\pmb{\theta}_{-i})$  over  $\theta_{i}$ . In continuous games if ABR with exact (locally) best response updates converges to  $(\pmb{\theta}_1,\pmb{\theta}_2)$ , then  $(\pmb{\theta}_1,\pmb{\theta}_2)$  is a (local) Nash equilibrium. Note, however, that ABR may fail to converge (e.g., in the face of Rock-Paper-Scissors dynamics). Moreover, if the best response updates of  $\theta_{i}$  are only approximated, ABR may converge to non-equilibria (Mazumdar et al., 2020, Proposition 6).
3 Diff Meta Games
We now formally introduce diff meta games, the novel setup we consider throughout this paper. Given some base game  $\Gamma$ , we consider a new meta game played by two players whom we will call principals. Each principal  $i$  submits a policy. The two players' policies each observe a real-valued measure of how similar they are to each other. Based on this, the policies then output a (potentially mixed) strategy for the base game. Finally, the utility is realized as per the base game. Below we define this new game formally. This model is illustrated in Figure 1.
Definition 1. Let  $\Gamma = (A_1, A_2, \mathbf{u})$  be a game. A (diff-based) policy for Player  $i$  for  $\Gamma$  is a function  $\pi: \mathbb{R} \to \Delta(A_i)$  mapping the perceived real-valued difference between the diff-based policies to a mixed strategy of  $\Gamma$ . For  $i = 1, 2$  let  $\mathcal{A}_i \subseteq \Delta(A_i)^\mathbb{R}$  be a set of difference-based policies for Player  $i$ . Then a policy difference (diff) function for  $(\mathcal{A}_1, \mathcal{A}_2)$  is a stochastic function  $\text{diff}: \mathcal{A}_1 \times \mathcal{A}_2 \rightsquigarrow \mathbb{R}^2$ . For any two policies  $\pi_1, \pi_2$  and difference function  $\text{diff}$ , we say that  $(\pi_1, \pi_2)$  plays the strategy profile  $\sigma \in \Delta(A_1) \times \Delta(A_2)$  of  $\Gamma$  if  $\sigma_i = \mathbb{E}[\pi_i(\text{diff}_i(\pi_1, \pi_2))]$  for  $i = 1, 2$ . For sets of policies  $\mathcal{A}_1, \mathcal{A}_2$  and difference function  $\text{diff}$  we then define the diff meta game  $(\Gamma, \mathcal{A}_1, \mathcal{A}_2, \text{diff})$  to be the normal-form game  $(\mathcal{A}_1, \mathcal{A}_2, V)$ , where  $V(\pi_1, \pi_2) := \mathbb{E}[\textbf{u}((\pi_i(\text{diff}_i(\pi_1, \pi_2))))_{i=1,2})]$  for all  $\pi_1 \in \mathcal{A}_1$ ,  $\pi_2 \in \mathcal{A}_2$ .
Note that Definition 1 does not put any restrictions on diff. For example, the above definition allows  $(\mathrm{diff}(\pi_i,\pi_{-i}))_i$  to be a real number whose binary representation uniquely specifies  $\pi_{-i}$ . This paper is dedicated to situations in which diff specifically represents some intuitive notion of how different the policies are, thus excluding such diff functions. Unfortunately, there are many different ways in which one could formalize this constraint, especially in asymmetric games. In Section 5 we will impose some restrictions along these lines, including symmetry. Our folk theorem (Theorem 3 in Section 4) will similarly impose constraints on diff to avoid diff functions like the above.
The rest of this section will study concrete examples of Definition 1. First, we define a particularly simple type of diff-based policy. Almost all of our theoretical analysis will be based on this class of policies.
Definition 2. Let  $\theta \in \mathbb{R} \cup \{-\infty, \infty\}$  and  $\sigma_i^{\leqslant}, \sigma_i^> \in \Delta(A_i)$  be strategies for Player  $i$  for  $i = 1, 2$ . Then we define  $(\sigma_i^{\leqslant}, \theta, \sigma_i^>)$  to be the policy  $\pi$  s.t.  $\pi(d) = \sigma_i^{\leqslant}$  if  $d \leq \theta$  and  $\pi(d) = \sigma_i^>$  otherwise. We call policies of this form threshold policies. Let  $\bar{A}_i$  denote the set of such threshold policies.
Throughout the rest of this section, we analyze the Prisoner's Dilemma as a specific example. We limit attention to threshold agents of the form  $(C,\theta ,D)$ , i.e., policies that cooperate against similar opponents (diff below threshold  $\theta$ ) and defect against dissimilar opponents. This is because such policies can be used to form cooperative equilibria, while policies that always cooperate (say,  $(C,1,C)$ ) or policies that are more cooperative against less similar opponent policies (e.g.,  $(D,1,C)$ ) cannot be used to form cooperative equilibria in the PD with a natural diff function. Policies of the form  $(C,\theta ,D)$  are uniquely specified by a single real number  $\theta$ . A natural measure of the similarity between two policies  $\theta_{1},\theta_{2}$  is then the absolute difference  $|\theta_1 - \theta_2|$ . We allow diff to be noisy, however. We summarize this in the following.
Example 1. Let  $\Gamma$  be the Prisoner's Dilemma as per Table 1. Then consider the  $(\Gamma, \hat{A}_1, \hat{A}_2, \text{diff})$  meta game where  $\hat{A}_i = \{(C, \theta_i, D) \mid \theta_i \in \mathbb{R}\}$  and  $\text{diff}_i((C, \theta_1, D), (C, \theta_2, D))) = |\theta_1 - \theta_2| + Z_i$  for  $i = 1, 2$  where  $Z_i$  is some real-valued random variable.
The only open parameters of Example 1 are  $G$  (the parameter used in our definition of the Prisoner's Dilemma) and the noise distribution. Nevertheless, Example 1 is a rich setting that allows for nontrivial results. We leave a detailed analysis for Appendix B and only give two specific results about equilibria here.
In the first result, we imagine that the noise  $Z_{i}$  is distributed uniformly between 0 and  $\epsilon > 0$  and that  $G$  is at least 2. Then, roughly, there are two kinds of equilibria. First, there are equilibria in which both players always defect, because their threshold for cooperation is at most 0 (such that they defect with probability 1 even against exact copies). Second, and more interestingly, there are equilibria in which both players submit the same threshold strictly between 0 and  $\epsilon$ . Note that this means that if both players submit a threshold of  $\epsilon$ , they both cooperate with probability 1.
Proposition 1. Consider Example 1 with  $Z_{i} \sim \mathrm{Uniform}([0,\epsilon])$  i.i.d. for some  $\epsilon > 0$  and with  $G \geq 2$ . Then  $((C, \theta_1, D), (C, \theta_2, D))$  is a Nash equilibrium if and only if  $\theta_1, \theta_2 \leq 0$  or  $0 < \theta_1 = \theta_2 \leq \epsilon$ . In case of the latter, the equilibrium is strict if  $G > 2$ .
What happens if, instead of the uniform distribution, we let the  $Z_{i}$  be, say, normally distributed? It turns out that for all unimodal distributions (which includes the normal distribution) and  $G = 2$ , we get an especially simple result: in equilibrium, both players submit the same threshold and that threshold must be left of the mode.
Proposition 2. Consider Example 1 with  $G = 2$ . Assume  $Z_{i}$  is i.i.d. for  $i = 1, 2$  according some unimodal distribution with mode  $\nu$  with positive measure on every interval. Then  $((C, \theta_{1}, D), (C, \theta_{2}, D))$  is a Nash equilibrium if and only if  $\theta_{1} = \theta_{2} \leq \nu$ .
4 A folk theorem for diff meta games
What are the Nash equilibria of a diff meta game on  $\Gamma$ ? A first answer is that Nash equilibria of  $\Gamma$  carry over to the diff meta game regardless of what diff function is used (assuming that at least all constant policies are available); see Proposition 16 in Appendix C.1. Any other equilibria of the diff meta game hinge on the use of the right diff function. In fact, if diff is constant and thus uninformative, the Nash equilibria of the diff meta game are exactly the Nash equilibria of  $\Gamma$ ; see Proposition 17 in Appendix C.1. So the next question to ask is for what strategy profiles  $\sigma$  there exists some diff function s.t.  $\sigma$  is played in an equilibrium of the resulting diff meta game. The following result answers this question. In particular, a folk theorem similar to the folk theorems for infinitely repeated games (e.g., Osborne 2004, Ch. 15) and for program equilibrium (see Section 7).
Theorem 3 (folk theorem for diff meta games). Let  $\Gamma$  be a game and  $\sigma$  be a strategy profile for  $\Gamma$ . Let  $\mathcal{A}_i \supseteq \bar{\mathcal{A}}_i$  for  $i = 1,2$ . Then the following two statements are equivalent:
1. There is a diff function such that there is a Nash equilibrium  $(\pi_1, \pi_2)$  of the diff meta game  $(\Gamma, \text{diff}, \mathcal{A}_1, \mathcal{A}_2)$  s.t.  $(\pi_1, \pi_2)$  play  $\sigma$ .
2. The strategy profile  $\sigma$  is individually rational (i.e., better than everyone's minimax payoff).
The result continues to hold true if we restrict attention to deterministic diff functions with  $\mathrm{diff}_1 = \mathrm{diff}_2$  and  $\mathrm{diff}_i(\pi_1, \pi_2) \in \{0, 1\}$  for  $i = 1, 2$ .
We leave the full proof to Appendix C.2, but give a short sketch of the construction for  $2 \Rightarrow 1$  here. For any  $\sigma$ , we construct the desired equilibrium from policies  $\pi_i^* = (\sigma_i, 1/2, \hat{\sigma}_i)$  for  $i = 1, 2$ , where  $\hat{\sigma}_i$  is Player  $i$ 's minimax strategy against Player  $-i$ . We then take any diff function s.t.  $\text{diff}(\pi_i^*, \pi_{-i}) = (0, 0)$  if  $\pi_{-i} = \pi_{-i}^*$  and  $\text{diff}(\pi_i^*, \pi_{-i}) = (1, 1)$  otherwise.
5 A uniqueness theorem
Theorem 3 allows for highly asymmetric similarity-based cooperation. For example, in the PD with, say,  $G = 2$ , Theorem 3 shows that with the right diff function, the strategy profile  $(C, 2/3 * C + 1/3 * D)$  is played in an equilibrium of the diff meta game of the PD. This seems odd, as one would expect SBC to result in playing symmetric strategy profiles. Note that, for example, all equilibria of Propositions 1 and 2 are symmetric. In this section, we show that under some restrictions on diff and the base game  $\Gamma$ , we can recover the symmetry intuition. This is good because in symmetric games the symmetric outcomes are the fair and otherwise desirable ones (Harsanyi et al., 1988, Sect. 3.4) and because SBC thus avoids equilibrium selection problems of other forms of cooperation (including cooperation based on full mutual transparency and cooperation in the iterated Prisoner's Dilemma).