This paper studies learning to play two-player general-sum games with state (Markov games). The idea is to learn to cooperate (think prisoner's dilemma) but in more complex domains. Generally, in repeated prisoner's dilemma, one can punish one's opponent for noncooperation. In this paper, they design an apporach to learn to cooperate in a more complex game, like a hybrid pong meets prisoner's dilemma game. This is fun but I did not find it particularly surprising from a game-theoretic or from a deep learning point of view. From a game-theoretic point of view, the paper begins with somewhat sloppy definitions followed by a theorem that is not very surprising. It is basically a straightforward generalization of the idea of punishing, which is common in "folk theorems" from game theory, to give a particular equilibrium for cooperating in Markov games. Many Markov games do not have a cooperative equilibrium, so this paper restricts attention to those that do. Even in games where there is a cooperative solution that maximizes the total welfare, it is not clear why players would choose to do so. When the game is symmetric, this might be "the natural" solution but in general it is far from clear why all players would want to maximize the total payoff. The paper follows with some fun experiments implementing these new game theory notions. Unfortunately, since the game theory was not particularly well-motivated, I did not find the overall story compelling. It is perhaps interesting that one can make deep learning learn to cooperate, but one could have illustrated the game theory equally well with other techniques. In contrast, the paper "Coco-Q: Learning in Stochastic Games with Side Payments" by Sodomka et. al. is an example where they took a well-motivated game theoretic cooperative solution concept and explored how to implement that with reinforcement learning. I would think that generalizing such solution concepts to stochastic games and/or deep learning might be more interesting. It should also be noted that I was asked to review another ICLR submission entitled "CONSEQUENTIALIST CONDITIONAL COOPERATION IN SOCIAL DILEMMAS WITH IMPERFECT INFORMATION" which amazingly introduced the same "Pong Player’s Dilemma" game as in this paper. Notice the following suspiciously similar paragraphs from the two papers:From "MAINTAINING COOPERATION IN COMPLEX SOCIAL DILEMMAS USING DEEP REINFORCEMENT LEARNING": We also look at an environment where strategies must be learned from raw pixels. We use the method of Tampuu et al. (2017) to alter the reward structure of Atari Pong so that whenever an agent scores a point they receive a reward of 1 and the other player receives −2. We refer to this game as the Pong Player’s Dilemma (PPD). In the PPD the only (jointly) winning move is not to play. However, a fully cooperative agent can be exploited by a defector. From "CONSEQUENTIALIST CONDITIONAL COOPERATION IN SOCIAL DILEMMAS WITH IMPERFECT INFORMATION": To demonstrate this we follow the method of Tampuu et al. (2017) to construct a version of Atari Pong which makes the game into a social dilemma. In what we call the Pong Player’s Dilemma (PPD) when an agent scores they gain a reward of 1 but the partner receives a reward of −2. Thus, in the PPD the only (jointly) winning move is not to play, but selfish agents are again tempted to defect and try to score points even though this decreases total social reward. We see that CCC is a successful, robust, and simple strategy in this game.