Venue: ICLR 2018 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper proposes a novel adaptive learning mechanism to improve results in ergodic cooperation games. The algorithm, tagged 'Consequentialist Conditional Cooperation', uses outcome-based accumulative rewards of different strategies established during prior training. Its core benefit is its adaptiveness towards diverse opposing player strategies (e.g. selfish, prosocial, CCC) while maintaining maximum reward.

While the contribution is explored in all its technical complexity, fundamentally this algorithm exploits policies for selfish and prosocial strategies to determine expected rewards in a training phase. During operation it then switches its strategy depending on a dynamically-calculated threshold reward value (considering variation in agent-specific policies, initial game states and stochasticity of rewards) relative to the total reward of the played game instance. The work is contrasted to tit-for-tat approaches that require complete observability and operate based on expected future rewards. In addition to the observability, approximate Markov TFT (amTFT) methods are more processing-intense, since they fall back on a game's Q-function, as opposed to learned policies, making CCC a lightweight alternative. 

Comments:

The findings suggest the effectiveness of that approach. In all experiments CCC-based agents fare better than agents operating based on a specific strategy. While performing worse than the amTFT approach and only working well for larger number of iterations, the outcome-based evaluation shows benefits. Specifically in the PPD game, the use of CCC produces interesting results; when paired with cooperate agents in the PPD game, CCC-based players produce higher overall reward than pairing cooperative players (see Figure 2, (d) & (e)). This should be explained. To improve the understanding of the CCC-based operation, it would further be worthwhile to provide an additional graph that shows the action choices of CCC agents over time to clarify behavioural characteristics and convergence performance.

However, when paired with non-cooperative players in the risky PPD game, CCC players lead to an improvement of pay-offs by around 50 percent (see Figure 2, (e)), compared to payoff received between non-cooperative players (-28.4 vs. -18, relative to -5 for defection). This leads to the question: How much CCC perform compared to random policy selection? Given its reduction of processing-intensive and need for larger number of iterations, how much worse is the random choice (no processing, independent of iterations)? This is would be worthwhile to appreciate the benefit of the proposed approach.

Another point relates to the fishing game. The game is parameterized with the rewards of +1 and +3. What is the bases for these parameter choices? What would happen if the higher reward was +2, or more interestingly, if the game was extended to allow agents to fish medium-sized fish (+2), in addition to small and large fish. Here it would be interesting to see how CCC fares (in all combinations with cooperators and defectors).

Overall, the paper is well-written and explores the technical details of the presented approach. The authors position the approach well within contemporary literature, both conceptually and using experimental evaluation, and are explicit about its strengths and limitations.

Presentation aspects:
- Minor typo: Page 2, last paragraph of Introduction: `... will act act identically.'
- Figure 2 should be shifted to the next page, since it is not self-explanatory and requires more context.

review_title: Review on 'Consequentialist conditional cooperation in social dilemmas with imperfect information'

final_score: rating: 6: Marginally above acceptance threshold
review_content: The main result specifies a (trigger) strategy (CCC) and corresponding algorithm that leads to an efficient outcome in social dilemmas, the theoretical basis of which is provided by theorem 1. This underscores an algorithm that uses a prosocial adjustment of the agents rewards to encourage efficient behaviour. The paper makes a useful contribution in demonstrating that convergence to efficient outcomes in social dilemmas without the need for agents to observe each other's actions. The paper is also clearly written and the theoretical result is accompanied by some supporting experiments. The numerical experiments show that using CCC strategy leads to an increase in the proportion of efficient equilibrium outcomes. However, in order to solidify the experimental validation, the authors could consider a broader range of experimental evaluations. There are also a number of items that could be added that I believe would strengthen the contribution and novelty, in particular:

Some highly relevant references on (prosocial) reward shaping in social dilemmas are missing, such as Babes, Munoz de cote and Littman, 2008 and for the (iterated) prisoner's dilemma; Vassiliades and Christodoulou, 2010 which all provide important background material on the subject. In addition, it would be useful to see how the method put forward in the paper compares with other (reward-shaping) techniques within MARL (especially in the perfect information case in the pong players' dilemma (PPD) experiment) such as those already mentioned. The authors could, therefore, provide more detail in relating the contribution to these papers and other relevant past work and existing algorithms. 

The paper also omits any formal discussion on the equilibrium concepts being used in the Markov game setting (e.g. Markov Perfect Equilibrium or Markov-Nash equilibrium) which leaves a notable gap in the theoretical analysis.  

There are also some questions that to me, remain unaddressed namely:

i. the model of the experiments, particularly a description of the structure of the pong players' dilemma in terms of the elements of the partially observed Markov game described in definition 1. In particular, what are the state space and transitions?

ii. the equilibrium concepts being considered i.e. does the paper consider Markov perfect equilibria. Some analysis on the conditions that under which the continuation equilibria e.g. cooperation in the social dilemma is expected to arise would also be beneficial.

iii. Although the formal discussion is concerned with Markov games (i.e. repeated games with stochastic transitions with multiple states) the experiments (particularly the PPD) appear to apply to repeated games (this could very much be cleared up with a formal description of the games in the experimental sections and the equilibrium concept being used). 

iv. In part 1 of the proof of the main theorem, it seems unclear why the sign of the main inequality has changed after application of Cauchy convergence in probability (equation at the top of the page). As this is an important component of the proof of the main result, the paper would benefit from an explanation of this step?

review_title: The paper makes a contribution to a challenging problem within multi-agent reinforcement learning. The paper is written clearly and it is easy to follow both the theoretical details and the general line of arguments. The paper would however, in my opinion, benefit from developing a number of areas of both the theoretical analysis and experimental section to solidify the contribution and validity. 

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper studies learning to play two-player general-sum games with state (Markov games) with imperfect information. The idea is to learn to cooperate (think prisoner's dilemma) but in more complex domains. Generally, in repeated prisoner's dilemma, one can punish one's opponent for noncooperation. In this paper, they design an apporach to learn to cooperate in a more complex game, like a hybrid pong meets prisoner's dilemma game. This is fun but I did not find it particularly surprising from a game-theoretic or from a deep learning point of view. 

From a game-theoretic point of view, the paper begins with a game-theoretic analysis of a cooperative strategy for these markov games with imperfect information. It is basically a straightforward generalization of the idea of punishing, which is common in "folk theorems" from game theory, to give a particular equilibrium for cooperating in Markov games. Many Markov games do not have a cooperative equilibrium, so this paper restricts attention to those that do. Even in games where there is a cooperative solution that maximizes the total welfare, it is not clear why players would choose to do so. When the game is symmetric, this might be "the natural" solution but in general it is far from clear why all players would want to maximize the total payoff. 

The paper follows with some fun experiments implementing these new game theory notions. Unfortunately, since the game theory was not particularly well-motivated, I did not find the overall story compelling. It is perhaps interesting that one can make deep learning learn to cooperate with imperfect information, but one could have illustrated the game theory equally well with other techniques.

In contrast, the paper "Coco-Q: Learning in Stochastic Games with Side Payments" by Sodomka et. al. is an example where they took a well-motivated game theoretic cooperative solution concept and explored how to implement that with reinforcement learning. I would think that generalizing such solution concepts to stochastic games and/or deep learning might be more interesting.

It should also be noted that I was asked to review another ICLR submission entitled "MAINTAINING COOPERATION IN COMPLEX SOCIAL DILEMMAS USING DEEP REINFORCEMENT LEARNING" which amazingly introduced the same "Pong Player’s Dilemma" game as in this paper. 

Notice the following suspiciously similar paragraphs from the two papers:

From "MAINTAINING COOPERATION IN COMPLEX SOCIAL DILEMMAS USING DEEP REINFORCEMENT LEARNING":
We also look at an environment where strategies must be learned from raw pixels. We use the method
of Tampuu et al. (2017) to alter the reward structure of Atari Pong so that whenever an agent scores a
point they receive a reward of 1 and the other player receives −2. We refer to this game as the Pong
Player’s Dilemma (PPD). In the PPD the only (jointly) winning move is not to play. However, a fully
cooperative agent can be exploited by a defector.

From "CONSEQUENTIALIST CONDITIONAL COOPERATION IN SOCIAL DILEMMAS WITH IMPERFECT INFORMATION":
To demonstrate this we follow the method of Tampuu et al. (2017) to construct a version of Atari Pong 
which makes the game into a social dilemma. In what we call the Pong Player’s Dilemma (PPD) when an agent 
scores they gain a reward of 1 but the partner receives a reward of −2. Thus, in the PPD the only (jointly) winning
move is not to play, but selfish agents are again tempted to defect and try to score points even though
this decreases total social reward. We see that CCC is a successful, robust, and simple strategy in this
game.
review_title: Learning to cooperate with incomplete information

Meta-review: The reviewer reactions to the initial manuscript were generally positive.  They considered the paper to be well written and clear, providing an original contribution to learning to cooperate in multi-agent deep RL in imperfect domains.  The reviewers raised a number of specific issues to address, including improved definitions and descriptions, and proper citations of related work.  The authors have substantially  revised the manuscript to address most or all of these issues.  At this point, the only knock on this paper is that the findings seemed unsurprising from a game-theoretic or deep learning point of view.

Pros: algorithmic contribution, technical quality, clarity
Cons: no real surprises 
Decision: Accept (Poster)