SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Andrew Critch, Michael Dennis, and Stuart Russell. Cooperative and uncooperative institution designs: Surprises and problems in open-source game theory, 2022. URL https://arxiv.org/pdf/2208.07006.pdf.
Caterina Cruciani, Anna Moretti, and Paolo Pellizzari. Dynamic patterns in similarity-based cooperation: An agent-based investigation. Journal of Economic Interaction and Coordination, 12: 121-141, 2017.
Allan Dafoe, Edward Hughes, Yoram Bachrach, Tantum Collins, Kevin R. McKee, Joel Z. Leibo, Kate Larson, and Thore Graepel. Open problems in cooperative AI, 2020. URL https://arxiv.org/pdf/2012.08630.pdf.
Richard Dawkins. The Selfish Gene. Oxford University Press, 1976.
Gary L. Drescher. Good and Real. Demystifying Paradoxes from Physics to Ethics. The MIT Press, 2006.
Ilan Fischer. Friend or foe: subjective expected relative similarity as a determinant of cooperation. Journal of Experimental Psychology: General, 138(3):341, 2009.
Jakob Foerster, Richard Y Chen, Maruan Al-Shedivat, Shimon Whiteson, Pieter Abbeel, and Igor Mordatch. Learning with opponent-learning awareness. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, pp. 122-130. International Foundation for Autonomous Agents and Multiagent Systems, 2018.
C. W. Fox, F. Camara, G. Markkula, R. A. Romano, R. Madigan, and N. Merat. When should the chicken cross the road?: Game theory for autonomous vehicle - human interactions. In Proceedings of the 4th International Conference on Vehicle Technology and Intelligent Transport Systems, volume 1, pp. 431-439. SciTePress, 2018. doi: 10.5220/0006765404310439.
Feng Fu, Martin A Nowak, Nicholas A Christakis, and James H Fowler. The evolution of homophily. Scientific reports, 2(1):1-6, 2012.
Andy Gardner and Stuart A. West. Greenbeards. Evolution: International Journal of Organic Evolution, 64(1):25-38, 2010.
David Gauthier. In the neighbourhood of the Newcomb-predictor (reflections on rationality). In Proceedings of the Aristotelian Society, New Series, 1988-1989, volume 89, pp. 179-194. [Aristotelian Society, Wiley], 1989.
Jeffrey Goldberg, Lívia Markóczy, and G Lawrence Zahn. Symmetry and the illusion of control as bases for cooperative behavior. Rationality and Society, 17(2):243-270, 2005.
William D. Hamilton. The genetical evolution of social behaviour. Journal of theoretical biology, 7 (1):1-52, 1964.
John C Harsanyi, Reinhard Selten, et al. A general theory of equilibrium selection in games. MIT Press Books, 1, 1988.
Torsten Heinrich, Yoojin Jang, Luca Mungo, Marco Pangallo, Alex Scott, Bassel Tarbush, and Samuel Wiese. Best-response dynamics, playing sequences, and convergence to equilibrium in random games. International Journal of Game Theory, 52:703-735, 9 2023. doi: 10.1007/s00182-023-00837-4.
Douglas Hofstadter. Dilemmas for superrational thinkers, leading up to a luring lottery. Scientific America, 248(6), 6 1983.
J. V. Howard. Cooperation in the prisoner's dilemma. Theory and Decision, 24:203-213, 5 1988. doi: 10.1007/BF00148954.
Adrian Hutter. Learning in two-player games between transparent opponents, 2020. URL https://arxiv.org/abs/2012.02671.
Figure 4: Visualization of outcomes as a function of the thresholds in Example 1 without noise  $(Z_{1} = Z_{2} = 0)$ . For each pair of thresholds (x and y axis), the graph shows whether both players cooperate (yellow), both players defect (blue), or one player cooperates and the other defects (orange).
A Preliminary game theory results
We say that  $\pmb{\sigma}$  very weakly Pareto-dominates  $\hat{\pmb{\sigma}}$  if for all  $i$ , we have that  $u_{i}(\pmb{\sigma}) \geq u_{i}(\hat{\pmb{\sigma}})$ .
Proposition 5. Let  $\Gamma$  be a two-player additively decomposable normal-form game. Then  $\Gamma$  has a Nash equilibrium that very weakly Pareto-dominates all other Nash equilibria. If  $\Gamma$  is furthermore symmetric, then in the Pareto-dominant equilibrium, both players receive the same utility.
Proof. First note that for all  $\sigma_{-i}$ , Player  $i$ 's best responses are given by
$$
\operatorname * {a r g   \max} _ {a _ {i}} u _ {i} (a _ {i}, \sigma_ {- i}) = \operatorname * {a r g   \max} _ {a _ {i}} u _ {i, i} (a _ {i}) + u _ {i, - i} (\sigma_ {- i}) = \operatorname * {a r g   \max} _ {a _ {i}} u _ {i, i} (a _ {i}).
$$
Now among this set of universal best responses, let let  $a_{i}^{*}$  be one that maximizes  $u_{-i,i}(a_i)$  for  $i = 1,2$ . Clearly  $\mathbf{a}^*$  is a Nash equilibrium.
Now let  $\sigma$  be any Nash equilibrium. Note that for  $i = 1,2$  the support of  $\sigma_{i}$  must be in the above argmax. It follows that for  $i = 1,2$ ,
$$
u _ {i} (\mathbf {a} ^ {*}) = u _ {i, i} (a _ {i} ^ {*}) + u _ {i, - i} (a _ {- i} ^ {*}) = u _ {i, i} (\sigma_ {i}) + u _ {i, - i} (a _ {- i} ^ {*}) \geq u _ {i, i} (\sigma_ {i}) + u _ {i, - i} (\sigma_ {- i}) = u _ {i} (\pmb {\sigma}).
$$
B A detailed analysis of Example 1
Example 1 is already surprisingly rich. We here provide a detailed analysis.
Example 1. Let  $\Gamma$  be the Prisoner's Dilemma as per Table 1. Then consider the  $(\Gamma, \hat{A}_1, \hat{A}_2, \text{diff})$  meta game where  $\hat{A}_i = \{(C, \theta_i, D) \mid \theta_i \in \mathbb{R}\}$  and  $\text{diff}_i((C, \theta_1, D), (C, \theta_2, D)) = |\theta_1 - \theta_2| + Z_i$  for  $i = 1, 2$  where  $Z_i$  is some real-valued random variable.
Figures 4, 5a and 5b illustrate Example 1. Specifically, Figure 4 considers the case without noise  $(Z_{i} = 0)$  and shows for each pair of thresholds  $\theta_{1},\theta_{2}$  whether both agents cooperate (yellow), only one agent (the one with the lower threshold) cooperates (orange), or both defect (blue). Figure 5a and Figure 5b consider the case  $Z_{i}\sim$  Uniform([0, 1]), i.e., the case where noise is drawn uniformly from [0, 1]. Figure 5a shows for each pair of thresholds the minimum probability of cooperation across the two players. For instance, if Player 1 submits 0.5 and Player 2 submits 1, then Player 1 cooperates with probability 0 and Player 2 cooperates with probability  $1 / 2$ , so the plot in Figure 5a is 0 (blue) at (0.5, 1) (and symmetrically at (1, 0.5)). Figure 5b shows the action of the agent whose threshold is given by the x axis.
Because we here restrict attention to policies of type  $(C, \theta, D)$ , policies are uniquely specified by a single real number  $\theta$ . So we will denote them as such.
In addition to threshold policies that correspond to real numbers, we will here consider the agents  $-\infty$  by which we mean the agent that always defects, and the agent  $\infty$ , by which we mean the agent that always cooperates.
(b)
Figure 5: Visualization of the probabilities of cooperation as a function of the thresholds in Example 1 with uniform noise  $Z_{1}, Z_{2} \sim \mathrm{Uniform}([0,1])$ . For each pair of thresholds (x and y axis), the left graph shows the minimum probability of cooperation across the two players. The right-hand graph shows the probability of cooperation of the player corresponding to the x axis.
One might suspect that if there is too much noise, there can be no cooperative equilibria. But it's easy to see that the setting of Example 1 is scale-invariant.
Proposition 6 (Scale invariance of noise). Let  $(\Gamma, \mathcal{A}_1, \mathcal{A}_2, \mathrm{diff})$  be a diff-based meta game with utility  $V$ , where  $\mathrm{diff}(\theta_1, \theta_2) = |\theta_1 - \theta_2| + Z_i$ . Further, let  $(\Gamma, \mathcal{A}_1, \mathcal{A}_2, \mathrm{diff}')$  be a diff-based meta game with utility  $V'$ , where  $\mathrm{diff}'(\theta_1, \theta_2) = |\theta_1 - \theta_2| + \alpha Z_i$  for some  $\alpha > 0$ . Then for all  $\theta_1, \theta_2, V(\theta_1, \theta_2) = V'(\alpha \theta_1, \alpha \theta_2)$ . It follows that for all  $\theta_1, \theta_2$ ,  $(\theta_1, \theta_2)$  is a Nash equilibrium in the diff meta game if and only if  $(\alpha \theta_1, \alpha \theta_2)$  is a Nash equilibrium in the diff' meta game.
B.1 Best responses
In the regular Prisoner's Dilemma, defecting strictly dominates cooperating. Similarly, in the diff meta game of Example 1, always defecting strictly dominates always cooperating (without looking at the difference to the opponent).
Definition 4. Let  $(A_{1}, A_{2}, \mathbf{u})$  be a normal-form game. Let  $a_{1}, a_{1}^{\prime} \in A_{1}$  be strategies for Player 1. We say that  $a_{1}$  very weakly dominates  $a_{1}^{\prime}$  if for all  $a_{2} \in A_{2}$  we have that  $u_{1}(a_{1}, a_{2}) \geq u_{1}(a_{1}, a_{2})$ . We further say that  $a_{1}$  weakly dominates  $a_{1}^{\prime}$  if the inequality is strict for at least one  $a_{2}$  and that  $a_{1}$  strictly dominates  $a_{1}$  if the inequality is strict for all  $a_{2}$ .
Proposition 7. The threshold policy  $-\infty$  strictly dominates the threshold policy  $\infty$ .
Intuitively, in our model there is never a reason to submit a policy that defects when it can be sure that it faces an exact copy. If the noise is lower-bounded, this puts a lower bound on what kind of agent it makes sense to submit, as we now show.
Proposition 8. Let  $Z_{i} \geq \theta_{i}$  with certainty. Let  $\theta_{i}' < \theta_{i}$ . Then  $\theta_{i}$  very weakly dominates  $\theta_{i}'$ . If  $P(Z = \theta_{i}) > 0$ ,  $\theta_{i}$  weakly dominates  $\theta_{i}'$ .
The next result shows that if the player who submits the higher threshold decreases her threshold while still staying above the other player's threshold, she cooperates with the same probability. Conversely, one cannot (in the setting of Example 1) decrease one's probability of threshold by increasing one's threshold.
Lemma 9. Let  $\theta_i, \theta_i'$ ,  $\theta_{-i} \in \mathbb{R}$  with  $\theta_i \geq \theta_i' \geq \theta_{-i}$ . Then in  $(\theta_i, \theta_{-i})$ , Player  $i$  cooperates with equal probability as in  $(\theta_i', \theta_{-i})$ .
$$
\begin{array}{l} P \left(\text {P l .} i \mathrm {C} ^ {\prime} \mathrm {s} \mid \theta_ {i}, \theta_ {- i}\right) = P \left(Z _ {i} + \theta_ {i} - \theta_ {- i} \leq \theta_ {i}\right) \\ = P (Z \leq \theta_ {- i}) \\ = P \left(Z _ {i} + \theta_ {i} ^ {\prime} - \theta_ {- i} \leq \theta_ {i} ^ {\prime}\right) \\ = P (\mathrm {P l}. i \mathrm {C} ^ {\prime} \mathrm {s} \mid \theta_ {i} ^ {\prime}, \theta_ {- i}) \\ \end{array}
$$
Theorem 10. Let  $\theta_{i},\theta_{-i}\in \mathbb{R}$  with  $\theta_{i} > \theta_{-i}$ . Then  $u_{i}(\theta_{-i},\theta_{-i})\geq u_{i}(\theta_{i},\theta_{-i})$ . The inequality is strict if and only if  $P(2\theta_{-i} - \theta_{i}\leq Z_{-i}\leq \theta_{-i}) > 0$
Intuitively, there's never a reason to submit a higher threshold than the opponent.
Proof. With Lemma 9, we need only prove that by decreasing  $\theta_{i}$  to  $\theta_{-i}$  the probability that Player 2 cooperates (weakly) increases. This is easy to see, though for the strictness condition, we need the details:
$$
P \left(\text {P l .} - i \mathrm {C} ^ {\prime} \mathrm {s} \mid \theta_ {i}, \theta_ {- i}\right) = P \left(Z _ {- i} + \theta_ {i} - \theta_ {- i} \leq \theta_ {- i}\right)
$$
$$
= P \left(Z _ {- i} \leq 2 \theta_ {- i} - \theta_ {i}\right)
$$
$$
P \left(\mathrm {P l .} - i \mathrm {C} ^ {\prime} \mathrm {s} \mid \theta_ {- i}, \theta_ {- i}\right) = P \left(Z _ {- i} \leq \theta_ {- i}\right)
$$
Clearly,  $P(Z_{-i} \leq \theta_{-i}) \geq P(Z_{-i} \leq 2\theta_{-i} - \theta_i)$ . Moreover, the inequality is strict if and only if  $P(2\theta_{-i} - \theta_i \leq Z_{-i} \leq \theta_{-i}) > 0$ .
B.2 (Pure) Nash equilibria
We now give some results on the Nash equilibria of Example 1. We start with two simple results to warm up.
Proposition 11. For all distributions of the noise:
1.  $(-\infty, -\infty)$  is a Nash equilibrium.
2.  $(\infty ,\infty)$  is not a Nash equilibrium.
Proposition 12. If there is no upper bound to noise, then there is no fully cooperative equilibrium.
Proof. If there is no upper bound to noise, then the only policy profile with universal cooperation is  $(\infty, \infty)$ . But by Proposition 11.2, this is not a Nash equilibrium.
Next we use our results on best responses to show that to form a Nash equilibrium it is never necessary for the two players to submit different thresholds.
Theorem 13. Let  $(\theta_{i},\theta_{-i})$  be a Nash equilibrium with  $\theta_{i} > \theta_{-i}$ . Then  $(\theta_{-i},\theta_{-i})$  is also a Nash equilibrium.
Proof. WLOG assume  $i = 1$  for notational clarity. Assume for contradiction that  $(\theta_{2},\theta_{2})$  is not a Nash equilibrium. First, notice that by Theorem 10 and the assumption that  $\theta_{1}$  is a best response for Player 1 to  $\theta_{2}$ , it follows that for Player 1  $\theta_{2}$  is a best response to  $\theta_{2}$ . So if  $(\theta_{2},\theta_{2})$  is not a Nash equilibrium, then it must be because for Player 2  $\theta_{2}$  is not a best response to  $\theta_{2}$  as submitted by Player 1. So there must be  $\theta_2^\prime$  such that  $u_{2}(\theta_{2},\theta_{2}^{\prime}) > u_{2}(\theta_{2},\theta_{2})$ . By Theorem 10,  $\theta_2^\prime < \theta_2$ .
We now show that we would then also have that  $u_{2}(\theta_{1},\theta_{2}^{\prime}) > u_{2}(\theta_{1},\theta_{2})$  in contradiction with the assumption that  $(\theta_{1},\theta_{2})$  is a Nash equilibrium. We do this via the following sequence of (in)equalities:
$$
u _ {2} (\theta_ {1}, \theta_ {2} ^ {\prime}) \underset {(1)} {\geq} u _ {2} (\theta_ {2}, \theta_ {2} ^ {\prime}) > u _ {2} (\theta_ {2}, \theta_ {2})   =   u _ {2} (\theta_ {1}, \theta_ {2}).
$$
(1) By Lemma 9, Player 1 cooperates with equal probability in  $(\theta_{1},\theta_{2}^{\prime})$  and  $(\theta_{2},\theta_{2}^{\prime})$ . It is easy to see that Player 2's probability of cooperating is weakly lower in  $(\theta_{1},\theta_{2}^{\prime})$ . It follows that  $u_{2}(\theta_{1},\theta_{2}^{\prime}) \geq u_{2}(\theta_{2},\theta_{2}^{\prime})$ .
(2) (A) By Lemma 9, Player 1 cooperates with equal probability in  $(\theta_{1},\theta_{2})$  and  $(\theta_{2},\theta_{2})$ . (B) From A and the assumption that  $\theta_{1}\in \mathrm{BR}_{1}(\theta_{2})$  it follows that Player 2 cooperates with equal probability in  $(\theta_{1},\theta_{2})$  and  $(\theta_{2},\theta_{2})$ . (Because if this were not the case, then  $\theta_{2}$  would be a strictly better response for Player 1 to  $\theta_{2}$ .) From A and B it follows that the distributions over actions are the same in  $(\theta_{1},\theta_{2})$  and  $(\theta_{2},\theta_{2})$  and thus that  $u_{2}(\theta_{2},\theta_{2}) = u_{2}(\theta_{1},\theta_{2})$  as claimed.
We are now ready to show the first of our two results about the main text.
Proof. "  $\Leftarrow$  " : First we show that the given strategy profiles really are equilibria.
1.  $\theta_{1},\theta_{2}\leq 0$  is just the like the earlier  $(-\infty , - \infty)$  equilibrium. If one player plays  $\theta_{-i}\leq 0$  , then clearly the unique best response is to also always defect.
2. By Theorem 10 we only need to consider whether one of the players, WLOG Player 1, can increase her utility by decreasing their threshold. So for the following consider  $\theta_{1} < \theta_{2}$
$$
\begin{array}{l} P \left(\mathrm {P l .} 1 \mathrm {C} ^ {\prime} \mathrm {s} \mid \theta_ {1}, \theta_ {2}\right) = P \left(\theta_ {2} - \theta_ {1} + Z _ {1} <   \theta_ {1}\right) \\ = P \left(Z _ {1} <   2 \theta_ {1} - \theta_ {2}\right). \\ \end{array}
$$
This is equal to  $\max (0, (2\theta_1 - \theta_2) / \epsilon)$ . Clearly, if Player 1 can profitably deviate to some  $\theta_{1}$ , then she can profitably deviate to some  $\theta_{1}$  s.t.  $(2\theta_{1} - \theta_{2}) / \epsilon$  is nonnegative. After all, Player 1 wants to maximize Player 2's probability of cooperation.
$$
\begin{array}{l} P \left(\mathrm {P l}. 2 \mathrm {C} ^ {\prime} \mathrm {s} \mid \theta_ {1}, \theta_ {2}\right) = P \left(\theta_ {2} - \theta_ {1} + Z <   \theta_ {2}\right) \\ = P (Z <   \theta_ {1}) \\ = \theta_ {1} / \epsilon . \\ \end{array}
$$
Now  $\theta_{1} = \theta_{2}$  is a best response to  $\theta_{2}$  if and only if the rate at which  $P(\mathrm{Pl.} 1\mathrm{C's} \mid \theta_{1}, \theta_{2})$  decreases is at most  $G$  times as high as the rate at which  $P(\mathrm{Pl.} 2\mathrm{C's} \mid \theta_{1}, \theta_{2})$  decreases. Now the rates of change / derivatives are  $2/\epsilon$  and  $1/\epsilon$ . So this condition is satisfied (for our payoff matrix).
"  $\Rightarrow$  "It is left to show that no other profile is a Nash equilibrium.
First, notice that for all  $\theta_{-i} > \epsilon$ , the unique best response is  $\theta_i = \epsilon$ , which minimizes the probability of  $i$  cooperating, while ensuring that Player  $-i$  cooperates with probability 1. For this, use part 2 of " $\Leftarrow$ ". From this it follows directly that there is no equilibrium in which both players play  $> \epsilon$ . By the strictness part of  $\Leftarrow$ , all equilibria in which one player plays  $\leq \epsilon$  are as described in the result.
We now prove a lemma in preparation for proving our second result for the main text.
Lemma 14. Assume  $G = 2$  and assume that the two players have the same noise distribution. Then  $(\theta, \theta)$  is a Nash equilibrium if and only if for all  $\Delta > 0$ ,  $P(\theta - 2\Delta < Z < \theta - \Delta) \leq P(\theta - \Delta < Z < \theta)$ . It is a strict Nash equilibrium if all of these inequalities are strict.
Proof. By Theorem 10 we only need to consider deviations to a lower threshold. So consider WLOG the case where Player 1 deviates from  $\theta$  to submit  $\theta - \Delta$ . First, we calculate the probabilities of cooperation under  $(\theta, \theta)$  and  $(\theta - \Delta, \theta)$ :
$$
P \left(\mathrm {P l .} 1 \mathrm {C} ^ {\prime} \mathrm {s} \mid \theta , \theta\right) = P (Z \leq \theta)
$$
$$
\begin{array}{l} P \left(\mathrm {P l .} 1 \mathrm {C} ^ {\prime} \mathrm {s} \mid \theta - \Delta , \theta\right) = P (Z + \Delta \leq \theta - \Delta) \\ = P (Z \leq \theta - 2 \Delta) \\ \end{array}
$$
$$
P \left(\mathrm {P l}. 2 \mathrm {C} ^ {\prime} \mathrm {s} \mid \theta , \theta\right) = P (Z \leq \theta)
$$
$$
\begin{array}{l} P \left(\mathrm {P l .} 2 \mathrm {C} ^ {\prime} \mathrm {s} \mid \theta - \Delta , \theta\right) = P (Z + \Delta \leq \theta) \\ = P (Z \leq \theta - \Delta) \\ \end{array}
$$
Thus by Player 1 switching from  $\theta$  to  $\theta -\Delta$  , Player 1's probability of cooperating decreases by
$$
P (Z \leq \theta) - P (Z \leq \theta - 2 \Delta) = P (\theta - 2 \Delta \leq Z \leq \theta).
$$
Meanwhile, Player 2's probability of cooperating decreases by
$$
P (Z \leq \theta) - P (Z \leq \theta - \Delta) = P (\theta - \Delta \leq Z \leq \theta).
$$
Thus, for this switch to not be profitable for player 1, it needs to be the case that
$$
P (\theta - 2 \Delta \leq Z \leq \theta) \leq 2 P (\theta - \Delta \leq Z \leq \theta),
$$
$$
P (\theta - 2 \Delta \leq Z \leq \theta - \Delta) \leq P (\theta - \Delta \leq Z \leq \theta)
$$
Proposition 2. Consider Example 1 with  $G = 2$ . Assume  $Z_{i}$  is i.i.d. for  $i = 1,2$  according some unimodal distribution with mode  $\nu$  with positive measure on every interval. Then  $((C,\theta_1,D),(C,\theta_2,D))$  is a Nash equilibrium if and only if  $\theta_{1} = \theta_{2}\leq \nu$
Proof. By Theorem 10 and the assumption of positive measure on any interval, all Nash equilibria have the form  $\theta_{1} = \theta_{2}$ . The second part follows directly from Lemma 14 and the fact that the noise distribution is unimodal with mode  $\nu$ .
B.3 A different type of noise
Intuitively, we might expect that more noise is an obstacle to similarity-based cooperation. The above results do not vindicate this intuition (see Proposition 6). We here give an alternative setup with a different kind of noise in which more noise is an obstacle to cooperation.
Example 2. Consider a variant of Example 1 where for  $i = 1,2$  we have with probability  $p_i$  that  $\text{diff}_i((C,\theta_1,D),(C,\theta_2,D)) = |\theta_1 - \theta_2| + Z_i$  with  $Z_i \sim \text{Unif}([0,\epsilon])$  for some  $\epsilon > 0$ ; and with the remaining probability  $\text{diff}_i((C,\theta_1,D),(C,\theta_2,D)) = 0$ .
Note that for  $p_1 = p_2 = 1$  the setting is exactly the setting of Proposition 1.
Intuitively, this models a scenario in which each player can try to manipulate the diff value to 0 and the manipulation succeeds with probability  $1 - p_{i}$ . (It is further implicitly assumed, that if manipulation fails, the other player never learns of the attempt to manipulate. Instead, the diff value is observed normally if manipulation fails. That way we can assume that each player always attempts to manipulate.)
We can generalize Proposition 1 to this new setting as follows:
Proposition 15. In Example 2,  $((C,\theta_1,D),(C,\theta_2,D))$  is a Nash equilibrium if and only if
-  $\theta_{1},\theta_{2}\leq 0$  or
-  $0 < \theta_{1} = \theta_{2} \leq \epsilon$  and  $Gp_{i} \geq 0$  for  $i = 1, 2$ .
The proof works the same as the proof of Proposition 1.
C.1 Nash equilibria of the base game as Nash equilibria of the meta game
We first note two simple results. The first is that every Nash equilibrium of the base game is also a Nash equilibrium of the diff meta game in which both players submit a policy that simply ignores the diff value.
Proposition 16. Let  $\Gamma$  be a game and  $\sigma$  be a Nash equilibrium of  $\Gamma$ . For  $i = 1,2$ , let  $\mathcal{A}_i$  be any set of policies that contains the policy  $\pi_i$ :  $d \mapsto \sigma_i$ . Then  $(\pi_1,\pi_2)$  is a Nash equilibrium of the  $(\Gamma, \text{diff}, \mathcal{A}_1, \mathcal{A}_2)$  meta game.
If the diff function is uninformative, then the Nash equilibria are in fact the only Nash equilibria of the diff meta game, as we now state.
Proposition 17. Let  $\Gamma$  be a game and  $(\Gamma, \text{diff}, \mathcal{A}_1, \mathcal{A}_2)$  be a meta game on  $\Gamma$  where  $\text{diff}(\cdot, \cdot) = \mathbf{y}$  for some  $\mathbf{y}^2$ . Then  $(\pi_1, \pi_2)$  is a Nash equilibrium of the meta game if and only if  $(\pi_1(y_1), \pi_2(y_2))$  is a Nash equilibrium of  $\Gamma$ .
Player 2
Player 1	a0	a1	a2
0,0	-2,1	-5,-1
1,-2	-1,-1	-4,-3
-1,-5	-3,-4	-6,-6
Table 2: A game to show the need for assuming high-value uninformativeness in Theorem 4.
Proof. "  $1 \Rightarrow 2$ ": We show the contrapositive, i.e., that if  $\sigma$  is not individually rational, it is not implemented by any equilibrium of any diff game. Let  $\hat{\sigma}_i$  be the strategy of Player  $i$  that guarantees her minimax utility. Then submitting  $(\hat{\sigma}_i, \theta, \hat{\sigma}_i)$  for any threshold  $\theta$  guarantees minimax utility regardless of what diff function is used. Thus, anything that gives  $i$  less than threat point utility cannot be an equilibrium.
$2 \Rightarrow 1$ : Let  $\tilde{\sigma}_i$  be Player  $i$ 's minimax strategy against Player  $-i$ . Then consider the strategy profile  $(\pi_1 = (\tilde{\sigma}_1, 1/2, \sigma_1), \pi_2 = (\tilde{\sigma}_2, 1/2, \sigma_2))$  and any diff function s.t.  $\text{diff}(\pi_1, \pi_2) = 0$  and for  $i = 1, 2$  and  $\pi_{-i}' \neq \pi_{-i}$ ,  $\text{diff}(\pi_i, \pi_{-i}') = 1$ . Clearly, in  $(\pi_1, \pi_2)$ , the players play  $(\sigma_1, \sigma_2)$ . Finally,  $(\pi_1, \pi_2)$  is a Nash equilibrium of the resulting diff meta game, because if either player deviates they will receive their minimax utility, which is by assumption no larger than their utility in  $\sigma$  and thus in  $(\pi_1, \pi_2)$ .
D On the uniqueness theorem
D.1 Examples to show the need for the assumptions of Theorem 4
D.1.1 Why the diff function must be high-value uninformative in Theorem 4
We now give an example for why we need diff to be high-value uninformative, both for Lemma 22 and for our uniqueness theorem below.
Proposition (Example) 18. Consider the game of Table 2. Note that the game is symmetric and additively decomposable. Consider the diff function defined by  $\mathrm{diff}((\sigma_1^{\leq},*,\sigma_1^>))_{i = 1,2}) = 1$  if  $\mathrm{supp}(\sigma_1^{\leq})\cup \mathrm{supp}(\sigma_1^>)$  and  $\mathrm{supp}(\sigma_2^{\leq})\cup \mathrm{supp}(\sigma_2^>)$  are disjoint and  $\mathrm{diff}((\sigma_1^{\leq},*,\sigma_1^>)_{i = 1,2}) = 0$  otherwise. Then  $((a_2,1 / 2,a_1),(a_0,*,a_0))$  is an equilibrium of the diff meta game.