SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Intuitively, the policy  $(a_{2},1 / 2,a_{1})$  with the described diff function implements the following idea: "I want to play  $a_1$  (which is good for me and moderately bad for you). I don't want you to also play  $a_1$ . If you are similar to me (which you are if you give weight to the same action I give weight to), I'll play  $a_2$ , which is very bad for you." Assuming that Player 1 submits such a policy, Player 2 optimizes her utility by always playing  $a_0$ . Player 1 thus obtains her favorite outcome.
D.1.2 Why the game must be additively decomposable in Theorem 4
The following example shows why we need to restrict attention to additively decomposable games. Intuitively, the game is a Prisoner's Dilemma, except that if the players cooperate, they also play a Game of Chicken for an additional payoff. Then (with a natural diff function) similarity-based cooperation takes care of the cooperate versus defect part, but leaves open the Dare versus Swerve part. In particular, there are multiple Pareto-optimal equilibria.
Proposition (Example) 19. Let  $\Gamma$  be the game of Table 3. Define  $\mathrm{diff}(\pi_1, \pi_2) = 0$  if  $a_0 \notin \operatorname{supp}(\pi_i(0))$  for  $i = 1, 2$  and  $\mathrm{diff}(\pi_1, \pi_2) = 1$  otherwise. Then for  $i = 1, 2$ ,
$$
\left(\pi_ {i} = \left(a _ {1}, ^ {1} / 2, a _ {0}\right), \pi_ {- i} = \left(a _ {2}, ^ {1} / 2, a _ {0}\right)\right)
$$
is a Pareto-optimal Nash equilibrium of the diff meta game on  $\Gamma$ .
Player 2
Player 1	a0	a1	a2
0,0	5,-5	5,-5
-5,5	0,0	3,1
-5,5	1,3	1,1
Table 3: A game to show the need for the additive decomposability assumption in Theorem 4.
Player 2
C1	C2	D
Player 1	C1	3,3	2,3	0,4
C2	3,2	2,2	0,3
D	4,0	3,0	1,1
Table 4: A game to show why we need diff to be policy-symmetric in Theorem 4.
D.1.3 Why diff must be observer-symmetric for Theorem 4
We here give an example to show why the diff function in Theorem 4 needs to have  $(\mathrm{diff}_1(\pi_1,\pi_2),\mathrm{diff}_2(\pi_1,\pi_2))$  and  $(\mathrm{diff}_2(\pi_1,\pi_2),\mathrm{diff}_1(\pi_1,\pi_2))$  have the same distribution. One might call this observer symmetry.
Proposition (Example) 20. Let  $\Gamma$  be the Prisoner's Dilemma. Let  $\mathrm{diff}_1(\pi_1, \pi_2) = 0$  if  $\pi_1 = \pi_2$  and  $\mathrm{diff}_1(\pi_1, \pi_2) = 1$  otherwise. Let  $\mathrm{diff}_2(\pi_1, \pi_2)$  be defined in the same way, except that if  $\pi_1 = \pi_2$ , then there is still an  $\epsilon$  probability of  $\mathrm{diff}_2(\pi_1, \pi_2) = 1 - \epsilon$ . Note that this diff meta game satisfies the other conditions of Theorem 4, i.e.,  $\Gamma$  is symmetric and additively decomposable,  $\mathrm{diff}(\pi_1, \pi_2)$  and  $\mathrm{diff}(\pi_2, \pi_1)$  are equally distributed for all  $\pi_1, \pi_2$  and diff is high-value-uninformative and minimized by copies. Then  $((C, ^1/2, D), (C, ^1/2, D))$  has asymmetric payoffs but is a Pareto-optimal Nash equilibrium of the diff meta game on  $\Gamma$ .
In this example, the Pareto-optimal Nash equilibrium is still unique, but it is easy to come up with examples in which there are multiple Pareto-optimal Nash equilibria.
Note also that in this example the player who has less information about the other does better in the cooperative equilibria.
D.1.4 Why diff must be policy-symmetric for Theorem 4
Finally we give an example to show why the diff function in Theorem 4 needs to have  $\mathrm{diff}(\pi_1,\pi_2)$  and  $\mathrm{diff}(\pi_2,\pi_1)$  have the same distribution. One might call this policy symmetry.
Proposition (Example) 21. Let  $\Gamma$  be the game of Table 4. Let  $\mathrm{diff}(\pi_1, \pi_2) = (0, 0)$  if  $(\pi_1, \pi_2)$  equals one of the following
$$
\begin{array}{l} \left(\left(C _ {2}, ^ {1} / _ {2}, D\right), \left(C _ {1}, ^ {1} / _ {2}, D\right)\right) \\ \left(\left(C _ {1}, ^ {1 / 2}, D\right), \left(D, ^ {1 / 2}, D\right)\right) \\ \left(\left(C _ {2}, 1 / 2, D\right), \left(C _ {2}, 1 / 2, D\right)\right) \\ \left(\left(C _ {1}, ^ {1} / _ {2}, D\right), \left(C _ {1}, ^ {1} / _ {2}, D\right)\right) \\ \left(\left(D, 1 / 2, D\right), \left(D, 1 / 2, D\right)\right) \\ \end{array}
$$
and  $\mathrm{diff}(\pi_1,\pi_2) = (1,1)$  otherwise. Note that this diff meta game satisfies the other conditions of Theorem 4, i.e.,  $\Gamma$  is symmetric and additively decomposable,  $\mathrm{diff}_1(\pi_1,\pi_2) = \mathrm{diff}_2(\pi_1,\pi_2)$  for all  $\pi_1,\pi_2$  and diff is high-value-uninformative and minimized by copies. However, the only Pareto-optimal (pure) Nash equilibrium of the diff meta game is  $((C_2,1 / 2,D),(C_1,1 / 2,D))$ .
As in the previous example, the Pareto-optimal Nash equilibrium is still unique, but it is easy to come up with examples in which there are multiple Pareto-optimal Nash equilibria.
D.2 Results on the structure of equilibria
We have various intuitions about similarity-based cooperation. For example, we have the intuition that  $(C,\theta ,D)$  is a sensible policy but  $(D,\theta ,C)$  is not. In this section we prove results of this type
under appropriate assumptions. We find these results interesting independently, but we also need all results here to prove Theorem 4.
The following two lemmas capture the idea that under some assumptions it is rational to be more cooperative if the opponent is similar, i.e., if the observed difference is below the threshold.
Lemma 22. Let  $(\pi_1, \pi_2)$  be a Nash equilibrium of the meta game and diff be high-value uninformative. Then  $\max_{\sigma_i} u_i(\sigma_i, \sigma_{\pi_{-i}}^{\max}) \leq u_i(\pi_1, \pi_2)$  for  $i = 1, 2$ .
Proof. Assume for contradiction that there is some strategy  $\sigma_{i}^{\prime}$  s.t.  $u_{i}(\sigma_{i}^{\prime},\sigma_{\pi_{-i}}^{\max}) > u_{i}(\pi_{1},\pi_{2})$ . Because diff is high-value uninformative, there exists a policy  $\pi_i^\prime$  s.t.  $(\pi_i^\prime ,\pi_{-i})$  resolves to a strategy profile arbitrarily close to  $\sigma_i^\prime ,\sigma_{\pi_{-i}}^{\mathrm{max}}$ . Hence,  $u_{i}(\pi_{i}^{\prime},\pi_{-i}) > u_{i}(\pi_{1},\pi_{2})$  and so  $(\pi_1,\pi_2)$  cannot be a Nash equilibrium after all.
Lemma 23. Let  $\Gamma$  additively decompose into  $(u_{i,j}\colon A_i\to \mathbb{R})_{i,j\in \{1,2\}}$  and let diff be high-value uninformative. Let  $(\sigma_{i}^{\leqslant},\theta_{i},\sigma_{i}^{>})_{i = 1,2}$  be a Nash equilibrium. Then  $u_{i, - i}(\sigma_{-i}^{>})\leq u_{i, - i}(\sigma_{-i}^{\leqslant})$  for  $i = 1,2$
Proof. Follows directly from Lemma 22.  $\square$
Lemma 24. Let  $\Gamma$  be symmetric and additively decomposable. Let diff be symmetric, high-value uninformative and minimized by copies. Let  $(\pi_i = (\sigma_i^{\leqslant}, *, \sigma_i^{>}))_{i=1,2}$  be a Nash equilibrium of the diff meta game that induces strategies  $\sigma_1, \sigma_2$ . If  $\sigma_i = \sigma_i^>$  for some  $i$ , then  $(\sigma_1, \sigma_2)$  is a Nash equilibrium of  $\Gamma$ .
Proof. With high-value uninformativeness, it follows immediately that  $\sigma_{-i}$  is a best response to  $\sigma_{i}$ . The case that  $\sigma_{-i} = \sigma_{-i}^{>}$  is trivial, so we focus on the case where  $\sigma_{-i}$  gives positive probability to  $\sigma_{-i}^{\leqslant}$ . It then follows that  $\sigma_{-i}^{\leqslant}$  is a best response to  $\sigma_{i}$ . Because the game is additively decomposable, this means that  $\sigma_{-i}^{\leqslant}$  (independent of the opponent's strategy) maximizes  $-i$ 's utility (i.e.,  $\sigma_{-i} \in \arg \max_{\sigma_{-i}^{\prime} \in \Delta(A_{-i})} u_{-i, -i}(\sigma_{-i}^{\prime})$ ). So in particular,  $\sigma_{-i}$  is a best response to  $\sigma_{i}$ .
It is left to show that  $\sigma_{i}$  is a best response to  $\sigma_{-i}$ . We will argue that if  $\sigma_{i}$  is not optimizing Player  $i$ 's utility (i.e., if  $\sigma_{i} \notin \arg \max_{\sigma_{i}' \in \Delta(A_{i})} u_{i,i}(\sigma_{i}')$ ), then Player  $i$  could better respond to  $\pi_{-i}$  by also playing  $\pi_{-i}$  instead of  $\pi_{i}$ . Let  $\sigma_{-i}'$  be the strategy induced for both players by  $(\pi_{-i}, \pi_{-i})$ . Because diff is minimized by copies,  $\sigma_{-i}'$  gives weakly more weight to  $\sigma_{-i}^{\leqslant}$  than  $\sigma_{-i}$ . By Lemma 23, this means that  $u_{i,-i}(\sigma_{-i}') \geq u_{i,-i}(\sigma_{-i})$ . Second, by the assumption that  $\sigma_{i}$  doesn't optimize  $i$ 's utility but  $\sigma_{-i}$  and  $\sigma_{-i}^{\leqslant}$  do, it follows that  $u_{i,i}(\sigma_{i}) < u_{i,i}(\sigma_{-i}')$ .
Putting it all together we obtain that
$$
\begin{array}{l} u _ {i} \left(\pi_ {- i}, \pi_ {- i}\right) = u _ {i} \left(\sigma_ {- i} ^ {\prime}, \sigma_ {- i} ^ {\prime}\right) \\ = u _ {i, i} \left(\sigma_ {- i} ^ {\prime}\right) + u _ {i, - i} \left(\sigma_ {- i} ^ {\prime}\right) \\ > u _ {i, i} \left(\sigma_ {i}\right) + u _ {i, - i} \left(\sigma_ {- i}\right) \\ = u _ {i} \left(\sigma_ {i}, \sigma_ {- i}\right) \\ = u _ {i} \left(\pi_ {i}, \pi_ {- i}\right), \\ \end{array}
$$
D.3 Proof of Theorem 4
For the proof we define for additively decomposable games,  $u_{\Sigma,j} \coloneqq u_{1,j} + u_{2,j} \colon A_j \to \mathbb{R}$ . Intuitively,  $u_{\Sigma,j}$  denotes the utilitarian welfare generated by Player  $j$ 's actions. In symmetric games,  $u_{\Sigma,1} = u_{\Sigma,2}$
so that we can simply write  $u_{\Sigma}$ . For example, in the Prisoner's Dilemma  $u_{\Sigma}$ : Cooperate  $\mapsto G$ , Defect  $\mapsto 1$ .
Proof. We will prove that if  $(\pi_i = (\sigma_i^{\leq},\theta_i,\sigma_i^{>}))$  is a Pareto-optimal equilibrium of the meta game, then both players receive the same utility. The uniqueness of the Pareto-optimal equilibrium follows immediately.
We prove this in turn by contradiction. So assume that  $(\pi_1, \pi_2)$  is a Pareto-optimal equilibrium of the meta game and that the two players receive different utilities.
Assume WLOG that in  $(\pi_1, \pi_2)$  Player 1 receives higher utility. Let  $\sigma_1, \sigma_2$  be the strategies played in  $(\pi_1, \pi_2)$ . Then we distinguish two cases:
A) Player 1 "takes" more than Player 2, i.e.,
$$
u _ {1, 1} \left(\sigma_ {1}\right) > u _ {2, 2} \left(\sigma_ {2}\right). \tag {1}
$$
B) Player 1 does not take more but Player 2 "gives" more than Player 1, i.e.,
$$
u _ {1, 1} (\sigma_ {1}) \leq u _ {2, 2} (\sigma_ {2}) \tag {2}
$$
$$
u _ {2, 1} \left(\sigma_ {1}\right) <   u _ {1, 2} \left(\sigma_ {2}\right). \tag {3}
$$
It is easy to see that one of these cases must obtain.
A) We in turn distinguish two cases:
A.1) First consider the case where
$$
u _ {\Sigma} \left(\sigma_ {1} ^ {>} \right) \geq u _ {\Sigma} \left(\sigma_ {1} ^ {\leqslant}\right). \tag {4}
$$
We will show that in this case  $(\pi_1, \pi_2)$  cannot be a Nash equilibrium. Player 2 can better respond by playing the policy  $\tilde{\pi}_1$  that plays  $\sigma_1^>$  and maximizes (as per the high value uninformativeness condition) Player 1's probability of playing  $\sigma_1^>$ . This can be seen as follows:
$$
\begin{array}{l} u _ {2} \left(\pi_ {1}, \pi_ {2}\right) \quad = \quad u _ {2} \left(\sigma_ {1}, \sigma_ {2}\right) \\ \begin{array}{c c} <   & u _ {2} (\sigma_ {1}, \sigma_ {1}) \\ \mathrm {I n e q .} 1 \end{array} \\ \begin{array}{c c} \mathbf {\Pi} _ {\mathrm {I n e q . 4}} ^ {<  } & u _ {2} (\sigma_ {1} ^ {>}, \sigma_ {1} ^ {>}) \end{array} \\ \underset {\text {L e m m a 2 3}} {\leq} u _ {2} \left(\sigma_ {1} ^ {\max }, \sigma_ {1} ^ {>}\right) \\ = \quad u _ {2} \left(\tilde {\pi} _ {1}, \pi_ {2}\right) \\ \end{array}
$$
A.2) Now consider the case where
$$
u _ {\Sigma} \left(\sigma_ {1} ^ {>} \right) \leq u _ {\Sigma} \left(\sigma_ {1} ^ {\leqslant}\right). \tag {5}
$$
We will show that in this case Player 2 can better respond by also playing  $\pi_1$  instead of  $\pi_2$ , such that  $(\pi_1, \pi_2)$  (again) cannot be a Nash equilibrium.
Let  $\sigma_1^\prime$  be the strategy played by both players in  $(\pi_1,\pi_1)$ . Note that because diff is minimized by copies,  $\sigma_1^\prime$  gives at least as much weight to  $\sigma_{1}^{\leqslant}$  as  $\sigma_{1}$ .
$$
\begin{array}{l} u _ {2} \left(\pi_ {1}, \pi_ {1}\right) \quad = \quad u _ {2} \left(\sigma_ {1} ^ {\prime}, \sigma_ {1} ^ {\prime}\right) \\ \geq_ {\text {I n e q .} 5} u _ {2} \left(\sigma_ {1}, \sigma_ {1}\right) \\ \begin{array}{c c} > & u _ {2} \big (\sigma_ {1}, \sigma_ {2} \big) \\ \mathrm {I n e q . 1} & \end{array} \\ = \quad u _ {2} \left(\pi_ {1}, \pi_ {2}\right) \\ \end{array}
$$
B) We again distinguish two cases.
B.1) First consider the case where
$$
u _ {\Sigma} \left(\sigma_ {2} ^ {>} \right) \leq u _ {\Sigma} \left(\sigma_ {2} ^ {\leqslant}\right). \tag {6}
$$
We will show that in this case  $(\pi_2, \pi_2)$  is also a Nash equilibrium and that  $(\pi_2, \pi_2)$  Pareto-dominates  $(\pi_1, \pi_2)$ .
First, we show Pareto dominance. Let  $\sigma_2^\prime$  be the strategy played by both players in  $(\pi_{2},\pi_{2})$ . Because diff is minimized by copies,  $\sigma_2^\prime$  gives at least as much weight to  $\sigma_2^{\leqslant}$  as  $\sigma_{2}$ . Then for Player 1 we have that
$$
u _ {1} \left(\pi_ {1}, \pi_ {2}\right) = u _ {1} \left(\sigma_ {1}, \sigma_ {2}\right) \leq u _ {1} \left(\sigma_ {2}, \sigma_ {2}\right) \tag {7}
$$
$$
\underset {\mathrm {I n e q .} 6} {\leq} u _ {1} (\sigma_ {2} ^ {\prime}, \sigma_ {2} ^ {\prime}) = u _ {1} (\pi_ {2}, \pi_ {2}).
$$
Player 2's utility is strictly higher in  $(\pi_2, \pi_2)$ , which we can see as follows:
$$
u _ {2} \left(\pi_ {1}, \pi_ {2}\right) \quad = \quad u _ {2} \left(\sigma_ {1}, \sigma_ {2}\right)
$$
$$
\begin{array}{c c} <   & u _ {2} (\sigma_ {2}, \sigma_ {2}) \\ \text {I n e q .} 3 \end{array}
$$
$$
\underset { \begin{array}{c} \operatorname {I n e q .} 6 \end{array} } {\leq} u _ {2} (\sigma_ {2} ^ {\prime}, \sigma_ {2} ^ {\prime})
$$
$$
= \quad u _ {2} \left(\pi_ {2}, \pi_ {2}\right).
$$
It is left to show that  $(\pi_2, \pi_2)$  is a Nash equilibrium. By assumption,  $\pi_1$  is a best response to  $\pi_2$ . Line 7 therefore implies that  $\pi_2$  is also a best response to  $\pi_2$ . Because of symmetry, this is true for both players. We conclude that  $(\pi_2, \pi_2)$  is a Nash equilibrium.
B.2) Let
$$
u _ {\Sigma} \left(\sigma_ {2} ^ {>} \right) > u _ {\Sigma} \left(\sigma_ {2} ^ {\leqslant}\right). \tag {8}
$$
We now must make one more distinction. Consider first the case where  $\sigma_2 = \sigma_2^>$ . By Lemma 24,  $(\sigma_1, \sigma_2)$  must be a Nash equilibrium of  $\Gamma$ . The contradiction follows immediately from Proposition 5.
Now consider the case where  $\sigma_2 \neq \sigma_2^>$ . With Ineq. 8 it follows that
$$
u _ {1} \left(\sigma_ {2}, \sigma_ {2}\right) <   u _ {1} \left(\sigma_ {2} ^ {>} , \sigma_ {2} ^ {>}\right). \tag {9}
$$
We will show that  $(\pi_1, \pi_2)$  is not an equilibrium because Player 1 can better-respect by playing the policy  $\tilde{\pi}_1$  that plays  $\sigma_2^>$  and maximizes as per the definition of high-value uninformativeness Player 2's probability of playing  $\sigma_2^>$ . Then
$$
\begin{array}{l} u _ {1} \left(\pi_ {1}, \pi_ {2}\right) \quad = \quad u _ {1} \left(\sigma_ {1}, \sigma_ {2}\right) \\ \begin{array}{c c} \leq & u _ {1} (\sigma_ {2}, \sigma_ {2}) \\ \text {I n e q . 2} \end{array} \\ \begin{array}{c c} <   & u _ {1} (\sigma_ {2} ^ {>,}, \sigma_ {2} ^ {>} \\ \text {I n e q .} 9 \end{array} \\ \underset {\text {L e m m a 2 3}} {\leq} u _ {1} \left(\sigma_ {2} ^ {>}, \sigma_ {2} ^ {\max }\right) \\ = u _ {1} (\tilde {\pi} _ {1}, \pi_ {2}). \\ \end{array}
$$
E Theoretical analysis beyond threshold policies
We here analyze a meta game in which players can not only submit threshold policies but continuous functions. The goal is to show an equilibrium based on linear functions, similar to the equilibria found by CCDR pretraining.
A policy now is a function  $\pi \colon \mathbb{R}_{\geq 0} \to [0,1]$ , where  $\pi(x)$  denotes  $\pi$ 's probability of cooperation. For differences  $x_1, x_2$ , the payoff of Player  $i$  is given by
$$
\left(1 - \pi_ {i} (x _ {i})\right) + G \cdot \pi_ {- i} (x _ {- i}).
$$
Figure 6: Visualization the continuous policy in Proposition 25 for  $a = 1$ . The probability of cooperation is plotted against the perceived difference to the opponent.
It is left to specify the difference function. Let  $\epsilon > 0$  and  $K > \epsilon$ . Then define the function difference
$$
d \left(\pi_ {1}, \pi_ {2}\right) = \frac {1}{K} \int_ {0} ^ {K} \left| \pi_ {1} (x) - \pi_ {2} (x) \right| d x
$$
Further, define the probabilistic difference mapping  $\mathrm{diff}(\pi_1,\pi_2) = d(\pi_1,\pi_2) + Z_i$ , where  $Z_{i}$  is drawn uniformly from  $[0,\epsilon ]$ . As the set of policies for each player consider the set of integrable functions.
Proposition 25. Let  $\pi(x) = \max(1 - x / a, 0)$  for some  $a > \epsilon$ . Then  $(\pi, \pi)$  is a Nash equilibrium if and only if  $1 + aK / \epsilon \leq G$ .
Note that  $\pi$  decreases cooperation linearly in  $x$  down to 0 (which is hit at  $x = a$ ). This function is shown in Figure 6 for  $a = 1$ . Note that our policies look roughly like this after Step 2.
Proof. Consider  $(\pi, \pi)$  and imagine that, WLOG, Player 1 moves away from  $\pi$  by  $\Delta$ , i.e., deviates to play some  $\pi'$  s.t.  $d(\pi, \pi') = \Delta$ . It is easy to see that it is enough to consider small deviations. Specifically, we assume  $\Delta \leq a - \epsilon$ . First, if the difference between the policies increases by  $\Delta$ , what happens to Player 2's expected amount of cooperation? It is easy to see that this decreases by  $\Delta / a$ .
Next, we need to ask: by increasing the difference by  $\Delta$ , how much can Player 1 increase her probability of defection? We need to consider two effects. First, if the difference increases by  $\Delta$ , then automatically Player 1 defects more by the same effect as Player 2. So this gives Player 1 an extra  $\Delta / a$  probability of defection. Moreover, Player 1 can decrease the probability of cooperation on the relevant interval  $[\Delta, \Delta + \epsilon]$ . This decreases the probability of cooperation by (at most)  $K\Delta / \epsilon$ .
Taking stock, Player 1 can increase her probability of defecting by  $\Delta / a + K\Delta/\epsilon$  at the cost of Player 2 increasing her probability of defecting by  $\Delta / a$ . This is good for Player 1 if and only if  $1 + aK/\epsilon > G$ .
F Details on our experiments
F.1 Software
We used pytorch for implementing CCDR and ABR and functorch for implementing LOLA. We used floats with double precision (by running torch.set_defaultdtype(torch.float64)), because preliminary experiments had shown numerical issues as ABR converged. We used Weights and Biases (wandb.ai) for tracking.
F.2 Game and meta game
We here give some details on the game and diff meta game we consider throughout our experiments.
Constructing  $f_{D}, f_{C}$  In our experiments  $f_{D}, f_{C}$  have input dimension 10 and output dimension 3. (Thus, including one dimension for the similarity value, our policies have input dimension 11.) First we generate  $\mathbf{s}_{C,i}$  and  $\mathbf{s}_{D,i}$  for  $i = 1,2,3$  from  $\{0,1\}^{10}$  uniformly at random. Then we define
$$
f _ {C} (\mathbf {x}) = \left(\sin (\mathbf {s} _ {C, i} \cdot \mathbf {x})\right) _ {i = 1, 2, 3}.
$$
We define  $f_{D}$  analogously based on  $\mathbf{s}_D$
We chose this function because it is very simple to understand and implement and at the same time requires using larger nets. The only other approach we tried in preliminary experiments is to generate  $f_{C}, f_{D}$  by randomly generating neural nets. The problem is that large randomly generated fully connected neural nets are close to constant functions.
Constructing  $\mu$  Recall that for any pair of actions  $f_{1}, f_{2}$ , the payoffs of the HDPD are given by  $u_{i}(f_{1}, f_{2}) = -\mathbb{E}_{\mathbf{x} \sim \mu}[d(f_{i}(\mathbf{x}), f_{D}(\mathbf{x})) + Gd(f_{i}(\mathbf{x}), f_{C}(\mathbf{x}))]$ , where  $d$  is the Euclidean distance and  $\mu$  is some measure of  $\mathbb{R}^{10}$ . Thus, to construct a specific instance of the HDPD we need to also construct  $\mu$ . We do this by generating 50 vectors uniformly from  $[0, 1]^{10}$  and then taking the uniform distribution over these 50 vectors.
Constructing the noisy diff mapping Recall that  $\mathrm{diff}_i(\pi_1,\pi_2) = \mathbb{E}_{(y,\mathbf{x})\sim \nu}[d(\pi_1(y,\mathbf{x}),\pi_2(y,\mathbf{x}))] + Z_i$ , where  $\nu$  is some probability distribution over  $\mathbb{R}^{n+1}$  and  $Z_i$  is some real-valued noise. We need to specify  $\nu$  and the distribution  $Z_i$ . For  $\nu$  we first generate 50 reals from  $[0,0.1]$  uniformly at random as our test diffs. We increment each of these by a random draw from the underlying noise distribution, i.e., by a number drawn uniformly at random from  $[0,0.1]$ . We then define  $\nu$  to be the uniform distribution over 50 values that result from pairing the support of  $\mu$  with these 50 values.
For the noise we generate for each player 50 values uniformly from  $[0, 0.1]$  and then use the uniform distribution over these 50 points.
Note that by using the uniform distribution over a finite support, we can compute expected utilities in the meta game exactly.
F.3 Neural net policies
Throughout our experiments, our policies  $\pi_{\theta}$  are represented by neural networks with three fully connected hidden layers of dimensions 100, 50 and 50 with biases and LeakyReLU activation functions. Thus, these networks overall have  $11 + 100 + 50 + 50 + 3 + 11\cdot 100 + 100\cdot 50 + 50\cdot 50 + 50\cdot 3 =$  8964 parameters.
F.4 Methods as used in our experiments
CCDR pretraining We here describe in more detail the CCDR pretraining step in our results. Recall that CCDR pretraining consists in maximizing  $V^{d}(\pi_{\theta_{i}}, \pi_{\theta_{i}}) + V^{d}(\pi_{\theta_{i}}, \pi_{\theta_{-i}}^{\prime})$  for randomly generated opponents  $\pi_{\theta_{-i}}^{\prime}$ . Call  $-V^{d}(\pi_{\theta_{i}}, \pi_{\theta_{i}}) + V^{d}(\pi_{\theta_{i}}, \pi_{\theta_{-i}}^{\prime})$  the CCDR loss.
In our experiments, we maximized this by running Adam for 100 steps. In each step, the CCDR loss is calculated by averaging over 100 randomly generated opponents. The learning rate is 0.02.
Alternating best response training We ran alternating best response training for  $T = 1000$  turns. We need to specify how we updated  $\theta_{i}$  to maximize  $V(\theta_{i},\theta_{-i})$  (holding  $\theta_{-i}$  fixed) in each turn. For this we run gradient descent for  $T^{\prime} = 1000$  steps. However, we only take gradient steps that are successful, i.e., that in fact reduce loss. The learning rate  $\gamma^\prime$  is sampled uniformly from  $[0,\gamma = 0.00003]$  in each step. Note that by randomly sampling the learning rate, the algorithms avoids getting stuck when a gradient step is unsuccessful. We summarize this in Algorithm 1.
F.5 Convergence to spurious stable points?
In theory, alternating best response learning could converge to a "very local Nash equilibrium", i.e., a pair of models that are best responses only within a very small neighborhood of these models. One might also worry about convergence to other stable points (Mazumdar et al., 2020).
However, as far as we can tell, the limit points of our learning procedure are not spurious. To confirm this, we performed the following test (implemented by the function best_response_test of our code). For a given pair of models, we pick either model and perturb each of its parameters a little. We then see whether the perturbed model is a better response to the opponent model than the original model. In an (approximate) local Nash equilibrium, this should almost never be the case.
In all but one of our successful runs (the ones converging to partial cooperation), none of 10,000 random perturbations of each of the models after alternating best response training led to a better response to the opponent model. In one run, three perturbations of one of the models decreased loss.