Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: One way to enable cooperative outcomes is to make the agents mutually transparent to each other, however, full transparency is often unrealistic. In this paper, authors introduce a more realistic setting in which agents only observe how similar they are to each other. It is proved that this allows for the same set of cooperative outcomes as the full transparency setting.
strength_and_weaknesses: Strengths:
1.	A similarity-based cooperation mechanism is proposed.
2.	Efficient theorems have been proposed to analyze the Nash equilibrium property of diff meta games.
Weaknesses:
1.	Full transparency are rare in reality, however, similarity is also not an explicit property which can be obtained by observation, how to evaluate the similarity between players in fact? 
2.	It is a basic question, why players prefer to cooperate with similar players? Which lacks sufficient explanations.
3.	A similarity-based cooperation mechanism can promote cooperation, what will happen if the other player do not adopt the similarity-based cooperation mechanism, for example, the other player uses the TFT strategy. On the other word, I am interested in the robustness of the similarity-based cooperation mechanism. 
4.	The words in Fig. 3(b) is too small to read, which should be improved.
clarity,_quality,_novelty_and_reproducibility: Original work.
summary_of_the_review: One way to enable cooperative outcomes is to make the agents mutually transparent to each other, however, full transparency is often unrealistic. In this paper, authors introduce a more realistic setting in which agents only observe how similar they are to each other. It is proved that this allows for the same set of cooperative outcomes as the full transparency setting.
Strengths:
1.	A similarity-based cooperation mechanism is proposed.
2.	Efficient theorems have been proposed to analyze the Nash equilibrium property of diff meta games.
Weaknesses:
1.	Full transparency are rare in reality, however, similarity is also not an explicit property which can be obtained by observation, how to evaluate the similarity between players in fact? 
2.	It is a basic question, why players prefer to cooperate with similar players? Which lacks sufficient explanations.
3.	A similarity-based cooperation mechanism can promote cooperation, what will happen if the other player do not adopt the similarity-based cooperation mechanism, for example, the other player uses the TFT strategy. On the other word, I am interested in the robustness of the similarity-based cooperation mechanism. 
4.	The words in Fig. 3(b) is too small to read, which should be improved.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Overall, this paper provides a unique perspective for the ICLR community -- developing theoretical insights into how cooperative agents might function. The paper focuses on develop the theory to support the sufficiency of agents sharing minimal information to achieve cooperation in a Prisoner's Dilemma setup. Theoretical and empirical results are presented.
strength_and_weaknesses: Strengths:
+ The paper clearly states its contributions. These contributions include the introduction a specific game called "diff meta games," and theoretical analysis regarding these games. This analysis leads to the claimed contribution that cooperative equilibria can be found with ML methods.
+ The paper gives an appropriate if dense set of preliminaries in Section 2.
+ The intuition/motivation added in the second to last paragraph of Section 6 is helpful.
+ Section 8's discussion section is helpful.

Weaknesses:
-Figure 1 is, unfortunately, not beneficial. It appears to setup a typical prisoner's dilemma (Figure 1a). Figure 1b is not explained in the text. These figures should be removed or better explained (or illustrated differently).
-Table 1 does not show the only formulation of a Prisoner's Dilemma -- "G+1" could be any number k > G.
- The definitions (e.g., Definition 1 and 2) are not explained from an intuitive perspective or with a set of working examples until Example 1. This reviewer recommends each definition add intuition to unpack what is going on rather than waiting for Example 1. Even example 1, however, is more of an illustrative set of cases rather than a grounded example with actual numbers (e.g., "N_i is some real-valued random variable"). 
-"Appendix C.2, but give" should not have a comma.
-The pretraining procedure in Section 6 is only briefly described. More details could/should be given.
-The value of Figure 2 is unclear.
-Figure 3 is poorly formatted and difficult to parse.
-More discussion of limitations would be helpful. For example, how large can games be (rows and columns of the normal form game formulation) before the method breaks down?
-The result that a single number being shared can achieve cooperation is perhaps not as impressive if one considers that the number could be a real-valued number with an infinite number of digits. Can the nature of this number be restricted?
-What happens if agent(s) are byzantine?
clarity,_quality,_novelty_and_reproducibility: The paper is dense and, while technically precise, lacks clarity. Much detail is left for the appendix and the reader's effort. Such is typical of game theoretic papers, but, this paper would have a greater impact if the details were better explained.

The results roughly and empirically support the theoretical analysis, but a definitive "proof positive" result in the empirical section is missing. The results are also difficult to parse.

The work appears novel. 

Code is provided in the supplementary materials.


summary_of_the_review: This paper provides theoretical and empirical results investigating Prisoner Dilemma-like games and whether agents might be cooperative under minimal information sharing. The theoretical analysis is a strong positive for the paper (plus the empirical results); however, the paper is difficult to parse. Without a more approachable presentation, this paper might have limited impact. The reviewer acknowledges limited expertise in game theory (while being an expert in ML).
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies promoting cooperation in games. The authors provide a method where similarity information can achieve similar cooperation levels compared to previous method that requires agent's policies to be transparent.
strength_and_weaknesses: The paper provides a clear discussion of the motivation and details of the proposed method.

The reviewer has a major concern about the relationship between the proposed method and the large body of literature on homophily and assortative matchmaking (see the section below).
clarity,_quality,_novelty_and_reproducibility: Could the authors provide a discussion about how their work is related to works in the following line of research?

__Assortative matchmaking__

Some papers (e.g. Wang et al 2018, https://arxiv.org/pdf/1811.05931.pdf) shows that the mechanism of matching individuals who have similar behaviour can encourage cooperation in social dilemmas (and even sequential social dilemmas). Wang et al (2018) gave a matchmaking system in which before an episode, agents measure their similarity (with respect to the rewards) to each other, and similar agents will be matched together. Pro-social behaviour can be observed under this mechanism. The authors also discussed the theoretical ground of this method. The authors can also see many citations to previous methods on matchmaking in this paper.

The reviewer noticed the some differences between the proposed method and matchmaking. However, the reviewer also believes that the two ideas are similar in spirit.

__Homophily__

Homophily is a very well-studied approach in the field of cooperation emergence. It has shown that if agents are encouraged to be similar to each other, cooperation is more likely to emerge. One insight is that homophily can solve the issue of second order social dilemma. The reviewer just lists some related areas where the concept of homophily has been investigated. For example: 
- [1] Di Stefano, Alessandro, et al. "Quantifying the role of homophily in human cooperation using multiplex evolutionary game theory." PloS one 10.10 (2015): e0140646. 
- [2] Aksoy, Ozan. "Effects of heterogeneity and homophily on cooperation." Social Psychology Quarterly 78.4 (2015): 324-344. 


The authors may also want to discuss the prior findings in evolutionary game theory. For example, 

-Christian Hilbe, Štěpán Šimsa, Krishnendu Chatterjee, Martin A Nowak, “Evolution of cooperation in stochastic games.”
summary_of_the_review: The reviewer generally agree this paper makes some contribution to the cooperation learning in social dilemmas, but has concerns about its position in the literature.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents a new way to allow cooperation between ML agents using an approach based on a similarity measurement. Authors introduce a program called diff meta games, a novel variant of program meta games, where agents observe the similarity/dissimilarity between the policies using a single number, thus the similarity/ dissimilarity between each other, allowing the agents to learn to cooperate/defect. Authors theoretically and experimentally show the validity of their method of similarity-based cooperation. They prove that this approach allows the same cooperative outcomes as in the full mutual transparency setting. Moreover, they experimentally show that cooperative learning through this approach can be achieved with simple ML methods. 
strength_and_weaknesses: Mathematically rigorous results are obtained and the experiments are done to back up the claims. The paper presents contributions for enhancing similarity based cooperative learning between ML agents and clearly explains how their approach presents solutions to difficulties with existing methods. 
Paper compares full transparency setting vs their approach and explains how the proposed method addresses problems of existing methods in great detail. The authors could consider explaining the results more in a more accessible way to broader audience and elaborate connections to representation learning. 
The paper is dense and needs more clarifications.

Minor: You have used a distance measure as the similarity measurement. Does the direction matter? i.e. is diff(θ1,θ2) = diff(θ2,θ1) in general or should the measurement change according to the perceiver? 
clarity,_quality,_novelty_and_reproducibility: Paper is well written. Clarity could be improved
summary_of_the_review: The authors study an interesting problem with simulating motivation. Interesting theoretical and experimental results are obtained. 

Explaining the results more in a more accessible way to broader audience and elaborating connections to representation learning could improve the quality of the paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper considers one-shot games with transparency. In this previously-studied setting, there is a two-player game (e.g., Prisoner's Dilemma) to be played by two algorithms. Before playing the game, the algorithms submit their code to a trusted third party. Algorithm A's (mixed) policy can depend on Algorithm B's code, and vice versa. Past work shows that this setting allows many more equilibria (and better equilibria that support cooperation) than without transparency. Unfortunately, it is hard to make this setting practical, e.g., because of the high-dimensional nature of code as a description and the difficulty of processing this input.

This paper proposes a *potentially* more practical formulation in which Algorithm A only receives a signal diff(Algo A, Algo B) describing the similarity between its algorithm and that of Algorithm B. Algorithm B receives the same signal. Much of the paper is focused on threshold policies, where one policy is followed when the similarity score is above a threshold, and another when it is below.

The paper then shows that this setting supports cooperation. In particular, any individually rational policy (i.e., one that is better than minimax optimal for all players, and that players might wish to follow given the opportunity to commit) is an equilibrium under an appropriate method for computing differences (Theorem 3). 

The paper then shows that, under a set of natural properties on the method for computing differences, Pareto-optimal Nash equilibria have unique payoffs that are symmetric across players (Theorem 4).

The paper then proposes a method for finding cooperative equilibria, i.e., policies such that the best response results in an equilibrium whose payoffs are good for both players. This consists of optimization of parameters in a neural net (which generates the policy's actions)  that, intuitively, seeks to have a large reward when playing both against copies of itself and against randomly generated policies. This novel method is called CCDR. Two policies generated in this way are then supplied as input to alternating best response to find an equilibrium.

The paper then demonstrates this method on a high-dimensional prisoner's dilemma. It finds that using its novel CCDR method allows the players to find cooperative equilibria often (though not all of the time) while doing alternating best response on random policies does not find cooperative equilibria.

Strengths
- Novel approach. While cooperation supported by full-transparency has been studied, and there is existing work on homophily, this paper seems to be the first to consider the power of similarity signals in this way.
- The idea of passing only a similarity rather than the full source code is appealing --- as the paper shows, it offers the same power as the full transparency setting, but is much simpler.

Weaknesses
- The paper is not written in a way that make it widely accessible to an ICLR audience. It is mathematically precise but important definitions and statements are sometimes not accompanied by clear intuitive explanations. Reviewer ATts gives a good summary and examples of these weaknesses.
- The choice of diff function seems critical to creating good cooperative equilibria that can be practically computed. Yet, the construction of such diff functions is not explored well. An existence result for diff functions with cooperative equilibria is given in Theorem 3, but it does not seem practical.  A specific diff function is given for the high-dimensional prisoner's dilemma but this is not general, nor is this choice explored.
- The empirical demonstration of the method is limited to the high-dimensional prisoner's dilemma. While interesting, this game does not seem practically important.
- The paper does not articulate its contributions in the context of the literature on homophily and assortative matchmaking. One of the reviewers felt quite strongly that this was a significant weakness of the paper. The AC feels that this literature is related and should be discussed, but that the approach taken in the current paper is different, as articulated by the authors in the rebuttal.
justification_for_why_not_higher_score: - Weakness of empirical justification
- Sensitivity of the proposed method to the choice of diff function, along with weak evidence that an appropriate diff function can be created for problems beyond the high-dimensional prisoner's dilemma
- Lack of clarity and accessibility to an ICLR audience
Decision: Reject