Maintaining cooperation in complex social dilemmas using deep reinforcement learning

Abstract: Social dilemmas are situations where individuals face a temptation to increase their payoffs at a cost to total welfare. Building artificially intelligent agents that achieve good outcomes in these situations is important because many real world interactions include a tension between selfish interests and the welfare of others. We show how to modify modern reinforcement learning methods to construct agents that act in ways that are simple to understand, nice (begin by cooperating), provokable (try to avoid being exploited), and forgiving (try to return to mutual cooperation). We show both theoretically and experimentally that such agents can maintain cooperation in Markov social dilemmas. Our construction does not require training methods beyond a modification of self-play, thus if an environment is such that good strategies can be constructed in the zero-sum case (eg. Atari) then we can construct agents that solve social dilemmas in this environment. 

Reviews and author responses:

I feel like things are becoming more convoluted as we go along. Surely, agents 

"remain on the equilibrium path because of what they anticipate would happen if
they were to deviate" -- Binmore (1992)

But this is a statement that holds for general games, I don't see how this helps define a "social dilemma"? Are you not just trying to say "always cooperating is not an equilibrium" ?


And if this is the case, the response to the second  "bullet" (>>>) is a bit confusing. Where I had interpreted social dilemmas as a broad class including non-exchangeable problems, the response now seems to say that (in other literatures) "social dilemmas" exclude the coordination problem. However, it is not excluded by your own definition, only by the additional assumption, if I understand correctly.

In any case, all this of course is a matter of definitions. Bottom line is that the results in this paper are only for a much narrower class than what I had imagined when reading the title.

I feel like we could iterate on this quite a bit further, but in the end I feel that all this should have been clear in the submitted paper which I assessed. This rebuttal phase is for clarifying minor misconceptions. For performing iterations of in-depth analysis of what is really going on and clarifying the message, I usually get my name on a paper.

"We point out that all RL with function approximation has this same limitation (if your function approximator is bad, it won’t work), "

-> yes, but we don't say that that "converges to the optimal value function up to function approximation". This is precisely why I think the statement is somewhat misleading.


"In practice, we don’t necessarily need (Pi_D, Pi_D) to be an equilibrium, we just need it to be difficult for the other player to find the better strategy. "

-> I understand the sentiment, but defining 'difficult' here is a bit tricky. In any case "equilibrium" means "impossible" not "difficult".

My bad, I had not seen that sentence, I had started reading after the line break. This now makes sense.


>>> “The rollouts procedure seems very complex, and frankly, I cannot understand much of it. (why this procedure? why is it unbiased?) I think this might actually be an important contribution, but clearly it needs much better treatment.”
 
We apologize for any lack of clarity, we are happy to add more of this discussion to the appendix.  
 
Let pi1, pi2, be policies for players 1 and 2 respectively, and let s be an initial state. The job of the rollouts is to compute V1(s, pi1, pi2) which is the expected sum of discounted rewards for player 1 for starting in s and both behaving according to pi1, pi2. 

Rollouts are just the straightforward application of Monte Carlo to compute this expectation, by repeatedly sampling trajectories (starting from s, behaving according to pi1, pi2) and computing their expected discounted sum of rewards. If we do this K times and then average those batches and if K is large, we will approximate V1(s, pi1, pi2) arbitrarily well. 

Our only additional approximation is to compute the discounted reward for finite-length trajectories. Given that there is discounting the potential bias shrinks exponentially (after t periods the bias is at most delta^t / (1-delta) * r_max) where r_max is the biggest possible reward from one period. Therefore, this MC estimate converges to the correct value in the limit of large batch size and trajectory length.
 
The way this is used in the paper is: let a be an action that can be taken today by player 1. We want to find the advantage of the one shot deviation taken by choosing a today and p1 forever after tomorrow relative to just playing pi1 all the time. Call d1 this modified deviation policy.
 
We can approximate V1(s, d1, pi2) the same way as above and subtract it from V1(s, pi1, pi2) to get the advantage.
 
>>> “If you agree that there are many problems with function approximation, I don't understand why the formulation "is a Markov equilibrium (up to function approximation)." is not adapted. It just seems quite misleading... as in general it simply will not be an equilibrium.”
 
We meant “Up to function approximation” in the sense of “will converge to the equilibrium in the limit of low function approximation error”. Perhaps we should be more precise in our language here. 
 
We point out that all RL with function approximation has this same limitation (if your function approximator is bad, it won’t work), so it is the assumption in all function-approximation RL work. These methods have nevertheless been quite successful.
 
We do this because for the computation of the D phase length we need to compute the payoffs lost to the other player from the D phase. 
 
In order to do this, we bound this number using the payoffs of joint defection. 
 
In order for this bound to work, we need that the other player can’t exceed their (Pi_D, Pi_D) payoffs by choosing some other clever policy to follow during the D phase (ie. that PiD, PiD is a finite time equilibrium). 
 
This isn’t as strong as it seems. In practice, we don’t necessarily need (Pi_D, Pi_D) to be an equilibrium, we just need it to be difficult for the other player to find the better strategy. 
 
If the other agent has the same computational capacity as our agent, we can be relatively sure that if we couldn’t find a much better response they probably won’t be able to either. This is because Pi_D was computed with self play, so if there was an obvious better response we wouldn’t have stopped at our current one during training. However, this notion is difficult to formalize. 
 
We are happy to add this discussion to the paper.
 
>>> “This side-steps my question: it seems that the presented experiments somehow had variance that was so large that they were not representative? Clearly this is important to clear up: if the paper reported non-significant results that is a reason to further question the other results too.”
 
Could you clarify which question you feel was not addressed? Assuming you are referring to “why the tables in Figure 1 are not symmetric”, the answer as stated in our reply is: The tables in Figure 1 show the payoff of the row player against the column player – thus there is no reason to expect them to be symmetric (eg. the box for (D,C) corresponds to the payoff that D gets when C is their partner, which is not the same payoff that C gets when D is their partner).
 
In addition, we can check that there is no variance issue by computing the standard errors of the mean in our tournament payoffs. They are on the order of ~1 point in Coins where score differences between strategies are on the order of 40 points. The PPD results are similarly extremely statistically significant. We are happy to add the standard errors to the figures in the appendix. 
 


We thank the reviewer for the constructive discussion. We believe this has fundamentally improved the clarity of the paper. Our replies to the referee's concerns are in-line below.
 
>>> “Sorry, I cannot understand this sentence. I am not sure what is meant with "after every state". I also am not certain how cooperation along the path of play is influenced by what actions are taken off the path off play. The path of play is the path of play because those alternatives paths will lead to lower payoffs (independent of whether those are 'cooperative' on 'noncooperative' actions), right?”
 
You’re right, we could have made this sentence clearer. We mean that a social dilemma is a game where there is no equilibrium between two ***strategies*** that cooperate in every possible state (i.e. unconditional cooperators). E.g. in the repeated PD (where the state is, eg. the history of play) always cooperating can be exploited by always defecting.

However, there may exist strategies (such as grim trigger) that don't unconditionally cooperate (eg. Grim Trigger defects if you defect). However, if you look at the realized trajectory of play against each other, they always cooperate along the path. This is what we mean by “maintaining cooperation”.
 
We argue this is the key property of a social dilemma: **always cooperating** leaves one open to being cheated. We solve the social dilemma by constructing a policy which maintains cooperation on the path by off-path threats. The key difficulty is how to detect that a defection has been made (value rather than action space) and how to compute the proper “threat” (use rollouts).
 
>>> “Alright, but I do think this is a very severe assumption, and the paper ought to be very up front about it. As is, the paper claims to MAINTAIN COOPERATION IN COMPLEX SOCIAL DILEMMAS, but truth is that it does not seem to do this for many settings, such as deciding to which movie (romance or comedy) we should go to?”
 
We agree that our method only attacks one aspect of sociality: creating the incentives to not cheat. It does not fix the coordination problem. We tried to be quite clear on this in the paper.

However, while we agree the coordination problem is very important (and exchangeability is a crucial assumption) we also would like to point out that it’s not as strong as it looks. Exchangeability mostly affects games where we require coordination within a single timestep (eg. the 2 action, 2 player, matrix, Romance or Comedy game). 
 
For example, Pong requires coordination about where on the screen to bounce the ball back and forth. However, strategies of the form “if the other player hits it to any vertical location, hit it gently back in a straight line from that location” form exchangeable cooperative strategies (this is because it takes more than 1 timestep for the ball to get across the screen).
 
We note that in other literatures (eg. evolutionary biology, behavioral economics) cooperation refers specifically only to the instance of social dilemmas, not to the coordination problem. See eg. the well known review by Nowak *Science* 2006 which defines cooperation as: “A cooperator is someone who pays a cost, c, for another individual to receive a benefit, b. A defector has no cost and does not deal out benefits.”
 

I think these clarifications are helpful, but I don't think that they are made sufficiently clear in the updated paper. (I had a really hard time decyphering these statements). I would advise to actually make the " value space as opposed to action space" the primary hypothesis of a revised paper, since this seems to get to the core of the novelty. 

If you agree that there are many problems with function approximation, I don't understand why the formulation "is a Markov equilibrium (up to function approximation)." is not adapted. It just seems quite misleading... as in general it simply will not be an equilibrium.

The rollouts procedure seems very complex, and frankly, I cannot understand much of it. (why this procedure? why is it unbiased?) I think this might actually be an important contribution, but clearly it needs much better treatment.

"From the comments given by the review team we see [...]"

This side-steps my question: it seems that the presented experiments somehow had variance that was so large that they were not representative? Clearly this is important to clear up: if the paper reported non-significant results that is a reason to further question the other results too.

"Thus we define a social dilemma to be one where cooperation after EVERY STATE is impossible in equilibrium – that is, if there is cooperation along the path of play it must be because OFF THE PATH of play cooperation stops."

Sorry, I cannot understand this sentence. I am not sure what is meant with "after every state". I also am not certain how cooperation along the path of play is influenced by what actions are taken off the path off play. The path of play is the path of play because those alternatives paths will lead to lower payoffs (independent of whether those are 'cooperative' on 'noncooperative' actions), right?

"The main assumption used is the exchangeability assumption that all strategies form an equivalence class in that any two pairs of cooperative strategies (C1, C1), (C2, C2) are compatible with each other in the sense that (C1, C2) generates the same stream of payoffs. [...] there is no good zero-shot solution to those issues in the literature as well."

Alright, but I do think this is a very severe assumption, and the paper ought to be very up front about it. As is, the paper claims to MAINTAIN COOPERATION IN COMPLEX SOCIAL DILEMMAS, but truth is that it does not seem to do this for many settings, such as deciding to which movie (romance or comedy) we should go to?

This paper addresses multiagent learning problems in which there is a social dilemma: settings where there are no 'cooperative polices' that form an equilibrium. The paper proposes a way of dealing with these problems via amTFT, a variation of the well-known tit-for-that strategy, and presents some empirical results.

My main problem with this paper is clarity and I am afraid that not everything might be technically correct. Let me just list my main concerns in the below.

The definition of social dilemma, is unclear:
"A social dilemma is a game where there are no cooperative policies which form equilibria. In other
words, if one player commits to play a cooperative policy at every state, there is a way for their
partner to exploit them and earn higher rewards at their expense."
does this mean to say "there are no cooperative *Markov* policies" ? It seems to me that the paper precisely intents to show that by resorting to history-dependent policies (such as both using amTFT), there is a cooperative equilibrium. 

I don't understand:
"Note that in a social dilemma there may be policies which achieve the payoffs of cooperative policies because they cooperate on the trajectory of play and prevent exploitation by threatening non-cooperation on states which are never reached by the trajectory. If such policies exist, we call the social dilemma solvable."
is this now talking about non-Markov policies? If not, there seems to be a contradiction?

The work focuses on TFT-like policies, motivated by 
"if one can commit to them, create incentives for a partner to behave cooperatively"
however it seems that, as made clear below definition 4, we can only create such incentives for sufficiently powerful agents, that remember and learn from their failures to cooperate in the past?

Why is the method called "approximate Markov"? As soon as one introduces history dependence, the Markov property stops to hold?

On page 4, I have problems following the text due to inconsistent use of notation: subscripts and superscripts seem random, it is not clear which symbols denote strategy profiles (rather than individual strategies), there seems mix-ups between 'i' and '1' / '2', there is sudden use of \hat{}, and other undefined symbols (Q_CC?).

For all practical purposes, it seems that the made assumptions imply uniqueness of the cooperative joint strategy. I fully appreciate that the coordination question is difficult and important, so if the proposed method is not compatible with dealing with that important question, that strikes me as a large drawback.

I have problems understanding how it is possible to guarantee "If they start in a D phase, they eventually return to a C phase." without making more assumptions on the domain. The clear example being the typical 'heaven or hell' type of problems: what if after one defect, we are trapped in the 'hell' state where no cooperation is even possible? 

"If policies converge with this training then πˆ is a Markov equilibrium (up to function approximation)." There are two problems here:
1) A problem is that very typically things will not converge... E.g., 
Wunder, Michael, Michael L. Littman, and Monica Babes. "Classes of multiagent q-learning dynamics with epsilon-greedy exploration." Proceedings of the 27th International Conference on Machine Learning (ICML-10). 2010.
2) "Up to function approximation" could be arbitrary large?


Another significant problem seems to be with this statement:
"while in the cooperative reward schedule the standard RL convergence guarantees apply. The latter is because cooperative training is equivalent to one super-agent controlling both players and trying to optimize for a single scalar reward." The training of individual learners is quite different from "joint action learners" [Claus & Boutilier 98], and this in turn is different from a 'super-agent' which would also control the exploration. In absence of the super-agent, I believe that the only guarantee is that one will, in the limit, converge to a Nash equilibrum, which might be arbitrary far from the optimal joint policy. And this only holds for the tabular case. See the discussion in 
A concise introduction to multiagent systems and distributed artificial intelligence. N Vlassis. Synthesis Lectures on Artificial Intelligence and Machine Learning 1 (1), 1-71

Also, the approach used in the experiments "Cooperative (self play with both agents receiving sum of rewards) training for both games", would be insufficient for many settings where a cooperative joint policy would be asymmetric.

The entire approach hinges on using rollouts (the commented lines in Algo. 1). However, it is completely not clear to me how this works. The one paragraph is insufficient to get across these crucial parts of the proposed approach.

It is not clear why the tables in Figure 1 are not symmetric; this strikes me as extremely problematic. It is not clear what the colors encode either.

It also seems that "grim" is better against all, except against amTFT, why should we not use that? In general, the explanation of this closely related paper by De Cote & Littman (which was published at UAI'08), is insufficient. It is not quite clear to me what the proposed approach offers over the previous method.











This paper studies learning to play two-player general-sum games with state (Markov games). The idea is to learn to cooperate (think prisoner's dilemma) but in more complex domains. Generally, in repeated prisoner's dilemma, one can punish one's opponent for noncooperation. In this paper, they design an apporach to learn to cooperate in a more complex game, like a hybrid pong meets prisoner's dilemma game. This is fun but I did not find it particularly surprising from a game-theoretic or from a deep learning point of view. 

From a game-theoretic point of view, the paper begins with somewhat sloppy definitions followed by a theorem that is not very surprising. It is basically a straightforward generalization of the idea of punishing, which is common in "folk theorems" from game theory, to give a particular equilibrium for cooperating in Markov games. Many Markov games do not have a cooperative equilibrium, so this paper restricts attention to those that do. Even in games where there is a cooperative solution that maximizes the total welfare, it is not clear why players would choose to do so. When the game is symmetric, this might be "the natural" solution but in general it is far from clear why all players would want to maximize the total payoff. 

The paper follows with some fun experiments implementing these new game theory notions. Unfortunately, since the game theory was not particularly well-motivated, I did not find the overall story compelling. It is perhaps interesting that one can make deep learning learn to cooperate, but one could have illustrated the game theory equally well with other techniques.

In contrast, the paper "Coco-Q: Learning in Stochastic Games with Side Payments" by Sodomka et. al. is an example where they took a well-motivated game theoretic cooperative solution concept and explored how to implement that with reinforcement learning. I would think that generalizing such solution concepts to stochastic games and/or deep learning might be more interesting.

It should also be noted that I was asked to review another ICLR submission entitled "CONSEQUENTIALIST CONDITIONAL COOPERATION IN
SOCIAL DILEMMAS WITH IMPERFECT INFORMATION
" which amazingly introduced the same "Pong Player’s Dilemma" game as in this paper. 

Notice the following suspiciously similar paragraphs from the two papers:

From "MAINTAINING COOPERATION IN COMPLEX SOCIAL DILEMMAS USING DEEP REINFORCEMENT LEARNING":
We also look at an environment where strategies must be learned from raw pixels. We use the method
of Tampuu et al. (2017) to alter the reward structure of Atari Pong so that whenever an agent scores a
point they receive a reward of 1 and the other player receives −2. We refer to this game as the Pong
Player’s Dilemma (PPD). In the PPD the only (jointly) winning move is not to play. However, a fully
cooperative agent can be exploited by a defector.

From "CONSEQUENTIALIST CONDITIONAL COOPERATION IN SOCIAL DILEMMAS WITH IMPERFECT INFORMATION":
To demonstrate this we follow the method of Tampuu et al. (2017) to construct a version of Atari Pong 
which makes the game into a social dilemma. In what we call the Pong Player’s Dilemma (PPD) when an agent 
scores they gain a reward of 1 but the partner receives a reward of −2. Thus, in the PPD the only (jointly) winning
move is not to play, but selfish agents are again tempted to defect and try to score points even though
this decreases total social reward. We see that CCC is a successful, robust, and simple strategy in this
game.

About the first point, it does not present a clear problem definition. The paper continues stating what it should do (e.g.  "our agents only live once at at test time and must maintain cooperation by behaving intelligently within the confines of a single game rather than threats across games.") without any support for these desiderata. It then continues explaining how to achieve these desiderata, but at this point it is impossible to follow a coherent argument without understanding why are the authors making these strong assumptions about the problem they are trying to solve, and why. Without this problem description and a good motivation, it is impossible to assess why such desiderata (which look awkward to me) are important. The paper continues defining some joint behavior (e.g. cooperative policies), but then construct arguments for individual policy deviations, including elements like \pi_A and \Pi_2^{A_k} that, as you see, A is used sometimes as subindex and sometimes as supperindex. Could not follow this part, as such elements lack definition. D_k is also not defined. 

Experiments are uninteresting and show same results as many other RL algorithms that have been proposed in the past. No comparison with such other approaches is presented, nor even recognized. The paper should include a related work section that explain such similar approaches and their difference with this approach. The paper should continue the experimental section making explicit comparisons with such related work.

**Detailed suggestions**
- On page 2 you say "This methodology cannot be directly applied to our problem" without first defining what the problem is.
- When authors talk about the agent, it is unclear what agent they refer to
- \delta undefined
- You say selfish reward schedule each agent i treats the other agent just as a part of their environment. However, you need to make some assumption about its behavior (e.g. adversarial, cooperative, etc.) and this disregarded. 

We thank the reviewer for their comments. We believe that many of the reviewer's issues are actually addressed in the paper already, though we were unclear in our presentation. 

We have made major revisions to the motivating  text and the presentation of the main results in the newly uploaded version.
 
>>> The reviewer argues there is a lack of clear problem definition
We apologize if the problem definition is unclear, we have edited the text to be clearer. In addition, we have reformulated some of the results presentations to more clearly align with our problem defintion.

The goal of the paper is to begin with a question discussed by *The Evolution of Cooperation *(Axelrod (1984)): suppose that we are going to enter into a repeated Prisoner's Dilemma with an unknown partner, how should we behave?

Axelrod (and much follow up work) comes up with strategies which seek to work well against mixed populations where some individuals are cooperators, some are pure defectors but most are conditional cooperators (often this is justified by the idea that this is a good approximation of the distribution of people). This literature seeks to construct strategies (eg. Tit-for-Tat, or Win-Stay-Lose-Shift/Pavlov) which
1) cooperate with cooperators
2) aren't exploited by defectors
3) incentivize conditional cooperators to cooperate
4) are simple to explain

These are fine desiderata for what it means to "solve" a social dilemma. However, a weakness of this literature is that it mostly works with simple 2 player repeated Prisoner's Dilemma games.

Our goal is to expand the Axelrod ideas from the repeated PD case (where there are 2 actions that are clearly labeled) to some perfect information Markov game G which is not repeated (we only play G once), has a social dilemma structure, and is too complex to be solved in a tabular format (so requires deep RL).

Our question is related to, but actually quite different from, the literatures on:

* The folk theorem in game theory (Fudenberg & Maskin 1986, Fudenberg, Levine & Maskin 1996) – this literature asks “given a repeated game G, does an efficient equilibrium exist?”
* The work on “computational folk theorem” (De Cote & Littman 2008, Littman & Stone 2005) – this literature asks: “can I compute the efficient equilibrium strategies in a repeated game or Markov game?”
* Alternative solution concepts (eg. Sodomka et al. 2013) – this literature asks: “can we define solution concepts beyond Nash and under what conditions will learning converge to them?”
* The learning in (Markov) games literature (Fudenberg & Levine 1998, Sandholm & Crites 1996, Leibo et. al. 2017) – this literature asks: “which equilibrium will learners converge to as a function of game parameters/learning rules?”
* The shaping in learning in games literature (Babes et al 2008) – this literature asks: “if I can change the reward functions of agents, can I guide them to a good equilibrium?”
* Friend-or-Foe learning (Littman 2001) – this paper asks “what kind of learning rule should I use in positive sum games?” This is quite related to our work though again requires multiple plays of G with the same partner rather than self play training and then a SINGLE play of G.
* How do humans behave in these kinds of situations? (eg. Rand et al. 2012, Kleinman-Weiner 2016)

Again, our situation is that we have access to the game and we can do whatever we want at training time, but at test time we play G once and we want to achieve good performance in the Axelrod sense: sometimes we face pure cooperators, sometimes pure defectors, but mostly we face conditional cooperators. As we can see, this question is related to but not the same as the literatures above (though they all provide valuable tools and context).

Note also that we are not looking for equilibria in the game, in the PD tit-for-tat is not an equilibrium strategy (the best response is to always cooperate), however it is a very good commitment strategy if we seek to design an agent.

We can see from the reviews that the relationship between our work and prior work was unclear from the text, we have edited the introduction and main text significantly to address these comments.


>>> The reviewer would like to see more baselines
Given the discussion above, we argue that there are 2 potential baselines to amTFT. Both of these are already studied in the paper:

**Baseline 1: Apply standard self play at training time, save that strategy, use it at play time **
We find that self-play finds the defect policies and thus while it can exploit pure cooperators and not be exploited by pure defectors it isn’t able to realize the gains of cooperation when the partner is a conditional cooperator (eg. amTFT).

**Baseline 2: De Cote & Littman (2008). **
Note that the De Cote & Littman algorithm works ACROSS multiple iterations of a repeated Markov game (playing a Markov game G multiple times) rather than WITHIN a single game (which is what our agent faces). However, we can amend the DeCote & Littman algorithm as follows: compute a cooperative (C) policy (De Cote & Littman actually compute the equitable policies, but in our games they are identical), compute the defect policy, if our partner chooses an ACTION that is inconsistent with the C policy, use the D policy forever after.

We show that this approach does not work well because working in ACTION space is not very robust to function approximation or any existence of multiple ways to cooperate.

amTFT uses a very similar rule but works in value space rather than action space which makes it robust to multiple policies that have the same or similar values. We see in our experiments that this is an important property.

If the reviewer has other baselines in mind that we have missed, we are happy to compare our approach to them.

****Other Responses****
>> The paper continues defining some joint behavior (e.g. cooperative policies), but then construct arguments for individual policy deviations, including elements like \pi_A and \Pi_2^{A_k} that, as you see, A is used sometimes as subindex and sometimes as supperindex. Could not follow this part, as such elements lack definition. D_k is also not defined.

We apologize for the flipping of indices, we thought that we had caught all of the sub/super flips but some managed to get away from us. We have fixed many of the flips.

We do note that D_k is defined on page 4: “we first introduce the notation of a compound policy πXk Z which is a policy that behaves according to X for k turns and then Z afterwards.”

>>> Experiments are uninteresting and show same results as many other RL algorithms that have been proposed in the past. No comparison with such other approaches is presented, nor even recognized.

We discuss above why we believe that our work does indeed consider, discuss, properly cite, and compare to prior work on this problem. 

If there is work that the reviewer believes we have left out, we are happy to discuss it in the paper.

>> “\delta undefined”
Delta is defined in definition 2: “We assume agents discount the future with rate *δ *which we subsume into the value function.”

>> You say selfish reward schedule each agent i treats the other agent just as a part of their environment. However, you need to make some assumption about its behavior (e.g. adversarial, cooperative, etc.) and this disregarded.

We apologize if this is unclear. The “selfish reward” schedule is simply standard self-play where each agent treats the other agent as stationary (this is exactly the assumption made in other learning rules eg. fictitious play). While this assumption is incorrect in finite time it is correct in the limit if agents converge to a Nash equilibrium. We are not trying to study this assumption (it is beyond the scope of this paper), rather we use it because it is what is done in standard self-play/standard learning in games (see eg. Fudenberg & Levine 1998 for more discussion).

>> I have problems understanding how it is possible to guarantee "If they start in a D phase, they eventually return to a C phase." without making more assumptions on the domain. The clear example being the typical 'heaven or hell' type of problems: what if after one defect, we are trapped in the 'hell' state where no cooperation is even possible? **

The referee is correct that there exist domains where a single deviation by a player can simply never be made up in the D phase. Note that Theorem 1 specifically rules out this scenario, it says that for *any* state, the gain in value of cooperating from it forever (vs. defecting forever) is bigger than any one-shot deviation possible in the game. Thus, any debit a partner earns can eventually be made up by playing D for only k periods and then playing C forever.

This doesn't mean we rule out all types of “heaven or hell” scenarios. For example: suppose that cooperation earns a payoff of 10 every turn unless someone has defected once, in which case it earns 5, defection earns the defector 100 points and causes the other to lose 200 and mutual defection is worth -101. However, in this case, after a defection by a partner amTFT will return to cooperation after 1 turn of mutual defection but payoffs will be permanently lower.
 

>>>R3 discusses many issues with convergence guarantees of the deep RL methods.**
We agree that a weakness of any deep RL approach is that it is often hard to make statements about convergence guarantees / issues with function approximation. 

One way to see whether a amTFT is exploitable is to directly train an RL agent to try to exploit the amTFT agent. We see that in Coins learners fail to learn to exploit (we had issues doing this in the PPD due to instability of training Atari policies wiht low discount rates). Nevertheless the Coins result gives us confidence that this at least works empirically in some simple environments. Importantly, this method also gives us a possible way to stress test amTFT in any practical application.
 
We are happy to add this discussion to the main text as a direction for future results.
 
>> The entire approach hinges on using rollouts (the commented lines in Algo. 1). However, it is completely not clear to me how this works. The one paragraph is insufficient to get across these crucial parts of the proposed approach. 

The rollouts work as follows:

1) The amTFT agent has policy pairs (C,C), (D,D) saved from training
2) At time t, suppose the partner takes a' when the amTFT agent expected a (according to C(s)). 
3) The amTFT agent simulates 2B replicas of the game for M turns. 
4) In B of the replicates their partner starts with a’ and continues with C - “true path”
5) In B of the replicates their partner starts with a and continues with C - “counterfactual path”
6) The amTFT agent takes the difference in the average total reward to the partner from the two paths and uses that as the per period debit

In the limit of large M and B this is an unbiased estimator of the partner's Q function.

There is also the option to append the continuation value V(s) to the end of the rollout, we elide it. Note that in games where an action today can only affect payoffs up to M periods from now it suffices to use rollouts of length M and elide the continuation value
 
We have changed the text to make this clearer.
 
>> It is not clear why the tables in Figure 1 are not symmetric; this strikes me as extremely problematic. It is not clear what the colors encode either. 
The tables in Figure 1 show the payoff of the row player against the column player – thus there is no reason to expect them to be symmetric (eg. the box for (D,C) corresponds to the payoff that D gets when C is their partner, which is not the same payoff that C gets when D is their partner).

From the comments given by the review team we see that those figures were not the best way to present our main results. Rather, we have specifically measured the exploitability of a strategy as well as whether it incentivizes cooperation from a partner and have added those numbers as our main results.


We thank the reviewer for pointing out several important issues. We believe these are mostly issues of clarity in exposition/notation. We have edited the text to address these issues.

>> The definition of social dilemma, is unclear: "A social dilemma is a game where there are no cooperative policies which form equilibria…. does this mean to say "there are no cooperative *Markov* policies" ? **
 
The referee is correct, this should mean that there are no Markov policies. Note that when we refer to cooperative policies we specifically refer to ones which cooperate at ALL states. 

Thus we define a social dilemma to be one where cooperation after EVERY STATE is impossible in equilibrium – that is, if there is cooperation along the path of play it must be because OFF THE PATH of play cooperation stops.
 
This is identical to the logic in the standard repeated Prisoner’s Dilemma where policies which always cooperate are not equilibria, rather, in order to maintain cooperation along the path of play there must be defection off the path of play (eg. Grim Trigger). 
 
We have edited the text to make this point clearer.
 
**>> Why is the method called "approximate Markov"? As soon as one introduces history dependence, the Markov property stops to hold? **
 
We call the method approximate Markov because we use function approximation (approximate) and because amTFT only uses Markov policies from the original game (only using the augmented memory to switch between them).
 
We have made this clearer in the paper.
 
**>> On page 4, I have problems following the text due to inconsistent use of notation: subscripts and superscripts seem random, it is not clear which symbols denote strategy profiles (rather than individual strategies), there seems mix-ups between 'i' and '1' / '2', there is sudden use of \hat{}, and other undefined symbols (Q_CC?).**

We apologize if the mixup between sub/superscripts caused any confusion, we have fixed these typos. In addition, we now clarify the hat/no hat notation - as in statistics we use the no hat symbol to refer to a "real" policy whereas \hat{} objects refer to approximations (eg. the output of the deep RL training).
 
We note that the Q function is introduced in Definition 2 but the notation Q_CC is introduced in section 4 (“we call the converged policies under the selfish reward schedule πˆiD and the associated Q function approximations QˆiDD.”). We apologize for this confusion and will edit Defintion 2 notation to match the Section 4 notation.
 
** **
**>> For all practical purposes, it seems that the made assumptions imply uniqueness of the cooperative joint strategy. I fully appreciate that the coordination question is difficult and important, so if the proposed method is not compatible with dealing with that important question, that strikes me as a large drawback. **
** **
The main assumption used is the exchangeability assumption that all strategies form an equivalence class in that any two pairs of cooperative strategies (C1, C1), (C2, C2) are compatible with each other in the sense that (C1, C2) generates the same stream of payoffs. 

This is much weaker than a uniqueness assumption. Indeed, working in value space is one of the innovations of am TFT.  

As an example of this, consider the Pong Player’s Dilemma.  The exchangeability assumption it allows both players to do whatever they want as long as they “softly” hit the ball over to the other player in some way. 

For example, our partner can move the paddle around however they like while the ball is in flight, and, importantly, it allows for a partner (eg. a human) who hits the ball slightly too fast sometimes (but not so fast that our agent can't get to it). In both of these situations a strategy like the Grim Trigger (a direct application of De Cote & Littman 2008) will assume the partner is not cooperating and defect.

We agree that there are situations where this assumption fails (for example if we need to make simultaneous decisions that may or may not be compatible with one another as in, eg. a coordination games), but there is no good zero-shot solution to those issues in the literature as well.

**>> It also seems that "grim" is better against all, except against amTFT, why should we not use that? In general, the explanation of this closely related paper by De Cote & Littman (which was published at UAI'08), is insufficient. It is not quite clear to me what the proposed approach offers over the previous method. **

The main result of the experiments is that Grim, in practice, behaves almost always like the defect policy (due to its reliance on “wrong action” rather than “wrong value”), and thus gives the same rewards to both players as defect.

What is wrong with playing a policy of pure defection? Indeed it does achieve high payoffs against either pure cooperation or pure defection. One problem is that it incentivizes the partner to defect rather than cooperate, which can be seen in the table. A second problem is that it fails to realize gains of cooperation with conditional cooperators. This can't be shown directly in the table (as we cannot enumerate all possible conditionally cooperative strategies), but as a necessary condition it even fails to cooperate with itself (or amTFT). 

All of this can be extracted from the current figure, but we agree that it needs to be highlighted better, and have added an additional table that measures these desiderata explicitly.  We thank the reviewer for pointing this out. 


In addition, we note that amTFT is different from De Cote & Littman in 4 ways

    * amTFT is usable within a single game rather than across multiple iterations of the same game 
    * amTFT uses self-play and deep RL rather than the tabular computation in De Cote & Littman thus can be applied to more complex games
    * amTFT returns to cooperation following a defection rather than applying a Grim Trigger strategy which stops cooperating after the wrong action is taken
    * The biggest difference: amTFT uses value space as the trigger as opposed to action space. As we see in our experiments this is important in Markov games where there are multiple value-equivalent cooperating strategies that differ on actions (eg. move left then move down vs. move down then move left in Coins).  

 
We have edited the text to make these clearer.

We thank R1 for their thorough comments. We have made several changes to the presentation of the paper to address them.
 
**>>> “The paper follows with some fun experiments implementing these new game theory notions. Unfortunately, since the game theory was not particularly well-motivated, I did not find the overall story compelling…” **
 
Response: 
This may stem from our lack of clarity in our problem definition (see reply to R3 above). The point here is not to “get RL to cooperate.” Rather, we are interested in expanding the ideas proposed by Axelrod (1984) to Markov games. 

Please see our reply to R3 above discussing why our work is related to but also quite different from what is done in other work on cooperative games. 
 
>> Similar paragraphs in 2 papers
We are also the authors of the other paper. 

Could the reviewer please clarify the issue here? Is it that the game is re-used without attribution or is it that we use similar text to describe it? 

We are happy to make it clear that this (the amTFT paper) is the first one to use the PPD as an environment and the CCC paper uses it for robustness checks.

The important differences here are as follows:
 
amTFT (this paper) is a strategy that can only be implemented in Markov perfectly observed games. The CCC paper looks at IMPERFECTLY observed games where amTFT cannot be used. Note that there are other major differences in the guarantees between strategies (eg. CCC only has infinite time limit guarantees).
 
Since any MDP can be trivially written into a POMDP it follows that the CCC strategy introduced in the other paper can also be used whenever amTFT can be used. 

Does this mean that amTFT is completely dominated by CCC? The answer is no, in the CCC paper the PPD is used as an example to show that the CCC algorithm works well in some places (standard PPD) and not others (risky PPD). 
 
 
****Other Comments****
>>> Even in games where there is a cooperative solution that maximizes the total welfare, it is not clear why players would choose to do so. When the game is symmetric, this might be "the natural" solution but in general it is far from clear why all players would want to maximize the total payoff.

We agree with the reviewer on this point. One can view this as a discussion about whether a particular equilibrium is a focal point or not. It is well known that in symmetric games (including bargaining and coordination games) that people view the symmetric sum of payoffs to be a natural focal point while in asymmetric versions of the problem they do not (see eg. the chapter on bargaining in Kagel & Roth Handbook of Experimental Economics or more recent work on inequality in public goods games eg. Hauser, Kraft-Todd, Rand, Nowak & Norton 2016).
 
Figuring out which kinds of payoff distributions are “reasonable” focal points, especially for playing with humans, is an important direction for future research but beyond the scope of this paper (the question is not even settled in behavioral science as there are many debates on whether people are averse to inequality itself, unequal treatment or perhaps something else).

Note that amTFT can be adapted to any focal point that can be expressed in terms of payoffs (for example, pure inequity aversion can be expressed as U_1(payoff1, payoff2) = payoff1 - A*|payoff1-payoff2|, see eg. Charness & Rabin (2002) for a generic utility function that can express many social goals). 

The way one can adapt amTFT to these focal points is to train the D policies as we do in the paper, but now train the C policies using this modified reward at each time step (and use the amTFT switching rule at test time). A full discussion of when particular focal points can be implemented in particular games is beyond the scope of the paper.

We have made this point clear in the both the introduction and conclusion of the paper.


Meta-review: The reviewers found numerous issues in the paper, including unclear problem definitions, lack of motivation, no support for desiderata, clarity issues, points in discussion appearing to be technically incorrect, restrictive setting, sloppy definitions, and uninteresting experiments.  Unfortunately, little note of positive aspects was mentioned.  The authors wrote substantial rebuttals, including an extended exchange with Reviewer2, but this had no effect in terms of score changes. Given the current state of the paper, the committee feels the paper falls short of acceptance in its current form.

Decision: rejected-papers