SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Algorithm 1 Alternating best response learning
Input: Initial model parameters  $\theta_{1},\theta_{2},T,T^{\prime}\in \mathbb{N}$  , learning rate  $\gamma$    
Output: New model parameters  $\theta_1^\prime ,\theta_2^\prime$    
for  $t = 1,\dots,T$  do   
for  $i = 1,2$  do   
for  $t^{\prime} = 1,\ldots ,T^{\prime}$  do   
 $\gamma^\prime \sim \mathrm{Uniform}([0,\gamma])$ $\theta_i^{\prime \prime}\gets \theta_i^{\prime} + \gamma '\nabla_{\theta_i'}V(\theta_i',\theta_{-i}')$    
if  $V(\theta_i^{\prime \prime},\theta_{-i}^{\prime})\geq V(\theta_i^{\prime},\theta_{-i}^{\prime})$  then   
 $\theta_i^\prime \leftarrow \theta_i^{\prime \prime}$    
end if   
end for   
end for   
end for   
return  $\theta_1^\prime ,\theta_2^\prime$
When applying the test after CCDR pretraining but before alternating best response training, close to half of random perturbations improve utility.
F.6 Compute costs
We here provide details on how computationally costly our experiments are. To do so, we ran the experiment for a single random seed on an AMD Ryzen 7 PRO 4750U, a laptop CPU launched in 2020. (Note that we used the CPU not GPU (CUDA).) The CCDR pretraining took about 30 seconds per model. The ABR phase took about 3h. (Note that we ran most of our experiments as reported in the paper via remote computing on different hardware.)
G Experiments with LOLA
We have tried to learn SBC using Foerster et al.'s (2018) Learning with Opponent-Learning Awareness (LOLA). We here specifically report on an experiment in which we tried a broad range of parameters. The results are in line with prior exploratory experiments.
LOLA sometimes succeeds in learning SBC (without CCDR pretraining). It finds similar policies as CCDR pretraining. Some of the SBC models found by LOLA remain cooperative throughout a subsequent ABR phase.
Unfortunately, none of our LOLA results are nearly as robust as the CCDR results reported in the main text and in Appendix F. Specifically, they are not even robust to changing the random seed. One reason for this is that LOLA is capable of "unlearning" LOLA-learned SBC.
G.1 Experimental setup
We ran Foerster et al.'s (2018) Learning with Opponent-Learning Awareness (LOLA). We tried every combination setting the learning rate and lookahead parameter in LOLA to values in 0.0003, 0.001, 0.003, 0.01, 0.03, 0.1, 0.3, 1. For each combination we tested both exact and Taylor LOLA as described by Willi et al. (2022, Sect. 3.2). We tested each of these  $8 \cdot 8 \cdot 2 = 128$  combinations with 5 different random seeds. In each of these experiments we initialized the nets randomly and then ran LOLA for both agents for 30,000 steps. We then ran ABR training as described in Appendix F.4 for at most 500 steps, to test whether the resulting policies are in equilibrium. To save compute, we generally aborted ABR when the loss hit the Defect-Defect utility of 5 or when it was clear that ABR learning had converged.
We then labeled each individual run as a "weak LOLA success" if the loss after the LOLA phase was smaller for both players than the loss of mutual defection and as a LOLA-ABR success if the loss after the ABR phase was smaller for both players than the loss of mutual defection.
For each parameter configuration in which at least 3 out of 5 runs were a weak LOLA success we ran another 20 runs (with different random seeds) without the ABR phase to determine how robust the success is. Similarly, for each parameter configuration in which at least 3 out of 5 runs were a LOLA-ABR success we ran another 20 runs with the ABR phase.
Figure 7: Loss curve of a relatively successful run between two LOLA agents.
G.2 Qualitative analysis
We start with a qualitative discussion of our results, because the experiments with LOLA exhibit a much wider range of complex phenomena than the CCDR-based results.
G.2.1 A few clear-cut successful runs
Out of all the runs in our experiment, a few are as clear-cut successes as most of the CCDR runs. Figure 7 shows the learning curve of one of these runs across both the LOLA and at the end the ABR phase. It uses a lookahead of 0.001, learning rate of 0.001 and exact LOLA. The players learn to cooperate with each other in the LOLA phase. When the switch to ABR is made after 30k steps of LOLA, cooperation deteriorates slightly, but ultimately the models converge in the ABR phase to an outcome with a loss well below the loss of mutual defection. This shows that LOLA is to some extent capable of solving the given problem.
Unfortunately, both in the present experiment and in preliminary experiments we have found that these results are not even robust to changing the random seed. For example, the four other runs with the same parameters as the run from Figure 7 (but different random seeds) all failed to produce significant positive results. (One resulted in marginal cooperation at the end of the LOLA phase (losses 4.986 and 4.928) that disappeared immediately when switching to ABR learning. Another partially cooperated for some of the LOLA phase but failed to retain cooperation even until the end of the LOLA phase.)
G.2.2 What models LOLA learns
Generally, when LOLA succeeds, it learns similar models as CCDR pretraining. The main difference is that for high diff values, LOLA models typically do not exactly defect. (Of course, it also does not cooperate. Instead if performs an action far from both  $f_{C}$  and  $f_{D}$ .) This is to be expected, since these agents are not trained on randomly generated opponents. An example model after 30,000 steps of LOLA is shown in Figure 8.
When LOLA does not succeed, the learned models vary greatly. Some are clearly set up to impose some incentives – others just constantly defect.
G.2.3 Instability in the LOLA phase
The learning in the LOLA phase is often highly unstable. That is, the during the learning phase, the change in losses in a single learning step is often very large. This occurs even in successful runs. As an example, consider the learning curve in Figure 9 (lookahead 0.003, learning rate 0.01, exact LOLA). While it is unclear whether cooperation would have survived further ABR learning, this run is relatively successful. However, the utilities vary by large amounts during the LOLA phase. In fact for parts of the LOLA learning phase, both players' losses are much higher than the loss of mutual defection. Most successful runs with LOLA look like this. Of course, successes based on such runs cannot be very robust: if we had ceased LOLA learning somewhat earlier, the run would not have succeeded. It is unclear what would happen if the run had run for more than  $30\mathrm{k}$  LOLA steps.
Figure 8: An example of a model found in a successful LOLA run.
Figure 9: An example of the loss curve from a highly unstable LOLA run.
G.2.4 Cooperation in the LOLA phase versus cooperation in the ABR phase
While CCDR is explicitly a pretraining method, LOLA can also be used as a standalone learning procedure. So for each run, we can ask both whether the players cooperate, say, at the end of the LOLA phase and whether the players cooperate at the end of the ABR phase. It turns out that all four combinations of answers are possible! In particular, there are runs in which LOLA versus LOLA fails to converge to a pair of policies that give both players a higher utility than mutual defection; but in which ABR then converges to an outcome that is marginally better for both players than mutual defection.
G.3 Quantitative analysis
Bearing in mind the different qualitative phenomena, we here give a more quantitative analysis.
Table 5 lists the parameters in which at least three out of five runs exhibited weak LOLA success as defined earlier. They also summarize the result of 20 further runs with these parameters. Specifically, we categorized runs as stable successes if the loss was below the loss of mutual defection for both players for the final 5,000 LOLA steps. We categorized them as unstable successes if the loss was below the loss of mutual defection for both players for the final 50 but not the final 5,000 LOLA steps. The fourth and fifth columns of Table 5 show the numbers of these different kinds of successes out of 20 runs. The sixth column shows the average loss across both players and all successful runs. The final column shows the standard deviation across the losses of the successful runs (also across players).
Table 6 lists the set of parameter configurations that achieved a LOLA-ABR success in at least three out of five runs. They also summarize the result of 20 further runs with each of these parameter configurations. Specifically, we categorized runs again as stable successes if the loss was below the loss of mutual defection for both players for the final 5,000 LOLA steps and at the end of the ABR phase. We labeled them as unstable successes if the loss was below the loss of mutual defection for both players at the end of the ABR phase but not for the final 5,000 LOLA steps. Again, the
LOLA LA	LOLA LR	Taylor LOLA	Stable Successes	Unstable Successes	average success loss	SD
0.003	0.0003	false	5	1	4.518	0.2857
0.001	0.01	false	1	0	4.666	0.002601
0.001	0.003	false	5	7	3.579	0.6096
0.0003	0.003	true	3	7	4.299	0.1482
0.0003	0.003	false	4	3	4.506	0.07975
0.0003	0.001	true	4	1	4.344	0.2466
Table 5: Parameters for which at least three of five runs resulted in partial cooperation at the end of the LOLA phase along with results of further 20 runs per parameter configuration..
LOLA LA	LOLA LR	Taylor LOLA	Stable Successes	Unstable Successes	average success loss	SD
0.0003	0.003	true	1	12	4.279	0.2830
0.0003	0.003	false	3	9	4.344	0.2566
Table 6: Parameters for which at least three of five runs resulted in partial cooperation at the end of the ABR phase along with results of further 20 runs per parameter configuration.
fourth and fifth columns of Table 6 show the numbers of these different kinds of successes out of 20 runs. The sixth column shows the average loss across both players and all successful runs. The final column shows the standard deviation across the losses of the successful runs (also across players).
From these results we conclude that LOLA can succeed under various parameter configurations, but none of the parameter configurations succeed nearly as reliably as CCDR pretraining.
G.4 Compute costs
We again provide details on how computationally costly our experiments are. To do so, we ran the experiment for a single random seed on an AMD Ryzen 7 PRO 4750U. The LOLA phase took about 20 minutes. The ABR phase took about 1h and 30min. Note again we ran most of our experiments as reported in the paper via remote computing on different hardware. Note also that we ran ABR for half as many steps compared to the experiments for the main text (500 instead of 1000). So, the cost per ABR step is roughly the same between the two experiments.
H Distantly related work
H.1 Learning in Newcomb-like decision problems
There is some existing work on learning in Newcomb-like environments that therefore also applies to the Prisoner's Dilemma against a copy. Whether cooperation against a copy is learned generally depends on the learning scheme. Bell et al. (2021) show that  $Q$ -learning with a softmax policy learns to defect. Regret minimization also learns to defect. Other learning schemes do converge to cooperating against exact copies (Albert & Heiner, 2001; Mayer et al., 2016; Oesterheld, 2019a; Oesterheld et al., 2023). All schemes in prior work differ from the present setup, however, and to our knowledge none offer a model of cooperation between similar but non-equal agents.
H.2 Cooperation via iterated play, reputation, etc.
Perhaps the best-known way to achieve cooperation in the Prisoner's Dilemma is to play the Prisoner's Dilemma repeatedly (e.g., Axelrod 1984; Osborne 2004, Ch. 14, 15). Clearly, the underlying mechanism (repeated play) is very different from the mechanism underlying SBC, in which the game is played one-shot. That said, our folk theorem (Theorem 3) is similar to the well-known folk theorem for repeated games. (As noted in Section 7, the folk theorem for program equilibrium is also similar.) A number of variants of iterated play have been considered to study, for example, reputation effects, effects of allowing players to choose with whom to play based on reputation, and so on (e.g., Nowak & Sigmund 1998).
While cooperation via repeated play is very different from similarity-based cooperation as studied in this paper, some variants of the former are easy to confuse with the latter. As a simplistic example, consider a variant of the infinitely repeated Prisoner's Dilemma. Typically, it is imagined that players observe the entire history of past play. But now imagine that instead the players in each round only observe a single bit: 0 if they have taken the same action in each round so far and 1 otherwise.
Then it is a (subgame-perfect) Nash equilibrium for both players to follow the following strategy: cooperate upon observing 0 and defect upon observing 1. This is somewhat similar to similarity-based cooperation, but the underlying mechanism is still the one of repeated play: the reason it is rational for each of the players to cooperate is that if they defect, their opponent will then defect in all subsequent rounds. In fact, the strategies played are equivalent to the grim trigger strategies for the iterated Prisoner's Dilemma.
As another example of cooperation via repeated play that can be confused with SBC, consider assortative matchmaking as proposed by Wang et al. (2018, Sect. 2.4). Imagine that agents in a large population are repeatedly paired up to play a Prisoner's Dilemma. In each round, each agent is matched up with a new opponent who is similarly cooperative as they are. Again, similarity plays a role, but the underlying mechanism is very different from the ones studied in our paper. For one, similarity is used by the environment (the matching procedure) not the agent itself. Second, the reason why cooperation is rational is again its impact on rewards in subsequent rounds. For instance, imagine that for some reason an agent has, e.g., by accident, defected in the first few rounds and is now matched with uncooperative agents. Then it may be rational for the agent to cooperate in order to be matched with more cooperative agents in the future, even if it knows that it will receive the sucker's payoff in the current round.
H.3 Tag-based cooperation
In the literature the evolution of cooperation and in particular population dynamics, researchers have studied so-called tag-based cooperation Riolo et al. (2001); Cruciani et al. (2017); Traulsen & Claussen (2004); Traulsen (2008), which has sometimes also been referred to as similarity-based cooperation. The underlying mechanism is very different from similarity-based cooperation as studied in the present paper, however.
In models of tag-based cooperation, an agent is defined not only by a policy but also by a tag that has nothing to do with the policy. When choosing whether to cooperate with each other, agents can see each other's tags. Thus, the policies can choose based on the other agent's tag. For example, one policy could be to cooperate if and only if the two agents have the same tag. Notice that in tag-based cooperation, only similarity of tags is considered. In contrast, our paper feeds the similarity of the policies themselves as input to the policy.
Because the tag is not tied to the policy, tags are effectively cheap talk that have no impact on the Nash equilibria of the game. For example, consider the following meta game on the Prisoner's dilemma. Each player  $i$  submits a tag  $\tau_{i} \in \{1, \dots, 100\}$  and a policy  $\pi_{i} : \{1, \dots, 100\} \rightsquigarrow \{C, D\}$  that stochastically maps opponent tags onto actions. Then actions are determined by  $a_{i} = \pi_{i}(\tau_{-i})$  for  $i = 1, 2$ . Each player  $i$  receives the utility  $u_{i}(a_{1}, a_{2})$ . It is easy to see that in all Nash equilibria of this game, both players submit a policy that always defects. So tag-based cooperation does not help with achieving cooperative equilibrium in a two-player Nash equilibrium. In contrast, similarity-based cooperation as studied in this paper does allow for cooperative equilibria in two-player games, as the main text has shown.
So why might one study tag-based cooperation? Perhaps one would expect that any population of evolving agents would converge to defecting in the above tag-based Prisoner's Dilemma. Interestingly, it turns out that this is not the case! As computational experiments conducted in the above works show, evolving populations sometimes maintain some level of cooperation when they can observe each others' tags. Because cooperation is not an equilibrium, cooperation is maintained dynamically, i.e., by a population whose distribution of tags and policies continually changed. Roughly, the reason seems to be the following. By mere chance (from mutation) there will be tags whose individuals are more cooperative toward each other than the individuals associated with other tags. The cooperative tags and their associated cooperative policies will therefore become more prominent in the population. Of course, this cannot be stable: once an agent is discovered that has the cooperative tag but defects, this type of agent takes over within the cooperative population. But in the meantime a new cooperative tag may have emerged. And so forth. This mechanism for achieving cooperation seems to have no parallel at all in our model of similarity-based cooperation.<sup>1</sup>
<sup>1</sup>See also Nowak & May (1992) for another line of work that is even more different from the present paper, but in which unstable cooperation survives dynamically by similar mechanisms.
H.4 The green beard effect
The green beard effect (Hamilton 1964; Dawkins 1976, Ch. 6; Gardner & West 2010) refers to the idea that there could be a gene  $\mathcal{G}$  that causes individuals to be altruistic to people who also have the gene  $\mathcal{G}$  without relying on kinship to identify other people with the gene  $\mathcal{G}$ . In particular, imagine a gene  $\mathcal{G}$  with the following three properties:
1. Gene  $\mathcal{G}$  causes people with the gene to have some perceptible trait. For concreteness let the trait consist in growing a green beard.
2. Gene  $\mathcal{G}$  causes people to be altruistic toward people who have a green beard, e.g., to cooperate with them in a one-shot Prisoner's Dilemma.
3. People without gene  $\mathcal{G}$  cannot develop a green beard.
Then such a gene could spread and be dominant in a population.
Hamilton and Dawkins discuss the green beard effect as a theoretical idea. To what extent the green beard effect is a real biological phenomenon is subject to debate. Some candidates were pointed out by Keller & Ross (1998), Queller et al. (2003), Smukalla et al. (2008) (cf. Sinervo et al., 2006).
On first sight, similarity-based cooperation as studied in this paper and the green beard effect might seem similar. For example, if you observe a population of individuals, some of whom have the green beard and some of whom do not, you will see cooperation between agents that are similar (in that they both have the green beard). A connection between similarity-based cooperation and the green beard effect has been noted before by Howard (1988), Štepan Vesely (2011), and Martens (2019).
On second sight, however, we think that similarity-based cooperation as studied in our model is very different from the green beard effect as an evolutionary phenomenon. To understand why, we take the gene-centered view of evolution (e.g. Dawkins, 1976): we view evolution as a process that produces genes that take measures to spread themselves (as opposed to a process that creates organism with particular properties). Now imagine that two individuals sharing the gene  $\mathcal{G}$  play a Prisoner's Dilemma where they can either take 1 unit of a resource for themselves or give  $G > 1$  units of the resource to the other player. Then from the perspective of trying to spread  $\mathcal{G}$ , this is not a Prisoner's Dilemma at all! (This is assuming that both players can make similarly good use of the resources.) From the gene's perspective, it is a fully cooperative game wherein cooperation simply dominates defecting. This is the essence of how the green-beard effect works. Our model of SBC has no analogous mechanism or perspective.
As a result, the green beard effect allows for a very different set of outcomes than our folk theorem (Theorem 3). For example, imagine that organisms 1 and 2 both share  $\mathcal{G}$ , that organism 1 can either take 1 unit of a resource for itself or give  $G > 1$  units of the resource to organism 2, and that organism faces no choice that concerns organism 1. Then  $\mathcal{G}$  would still have the organism give, because doing so increases the spread of the gene. Meanwhile, the unique outcome allowed by our folk theorem has organism 1 not give. In other games, our folk theorem allows for many different equilibria of the meta game. In contrast, in a game played by two organisms who share  $\mathcal{G}$ , there will typically be a unique outcome that best spreads  $\mathcal{G}$ .
H.5 Studies of human psychology on similarity-based cooperation
In this section, we review empirical and theoretical work in psychology related to the phenomenon of similarity-based cooperation.
Studies have shown that humans often cooperate in the one-shot Prisoner's Dilemma, contrary to what standard game theory recommends. Many explanations have been given for this phenomenon. A few authors have proposed explanations based on Newcomb's problem as discussed in Section 7. That is, they have proposed that people cooperate in the Prisoner's Dilemma because cooperating gives them evidence that their opponent will defect (Krueger & Acevedo, 2005; Krueger et al., 2012). To test this hypothesis, a few studies have tried to measure the correlation in subjects' choice in Newcomb's problem and the Prisoner's Dilemma. The results have been mixed. Goldberg et al. (2005, Sect. 4.3), Tversky & Shafir (1992) have found strong correlations, while Toplak & Stanovich (2002) and Goldberg et al. (2005, Sect. 4.2) have failed to find such correlations. Another prediction of the Newcomb's problem/SBC explanation for human cooperation in the one-shot Prisoner's Dilemma is that individuals are more likely to cooperate with similar than with dissimilar individuals. Indeed, a number of studies have found this to be the case (e.g. Acevedo & Krueger, 2005; Fischer, 2009). (Note that some other theories have been proposed for this phenomenon as well (e.g. Aksoy, 2015).)
Another related idea in social psychology is homophily, which refers to the observed tendency of humans to be more likely to engage in interactions with similar individuals. (Homophily cannot always be cleanly differentiated from the tendency to be more altruistic/cooperative toward similar individuals as discussed in the previous paragraph.) Homophily could also be explained by SBC, because SBC allows for better outcomes (e.g., mutual cooperation over mutual defection) when playing against similar individuals. We are not aware of any discussion connecting homophily with Newcomb's problem or the idea that it could be rational to cooperate in a Prisoner's Dilemma against a copy. The theory of the evolution of homophily seems to be that individuals with a similar cultural background can more effectively communicate (and coordinate) (Fu et al., 2012). This is different from SBC as studied in the present paper.