# Adherence Improves Cooperation in Sequential Social Dilemmas
## 2 Preliminaries and Backgrounds
### Sequential Social Dilemmas  
Social dilemmas provide a powerful platform to study the development of behavior and the emergence of cooperation. Multi-person social dilemmas are divided into two categories: public goods dilemmas and commons dilemmas [37]. In public goods dilemmas [38; 39], every individual has to pay the price in exchange for the resources shared by all. That means that public goods are non-exclusive, so there is a temptation to enjoy products without contribution. Although free-riding is rational for individuals, there will be no public good if everyone does so, and all will be worse off. The commons dilemmas [40; 41; 42] is a situation in which an individual's short-term selfish interests are at odds with the long-term group interests. In commons dilemmas, every individual who has open access to resources is tempted to deplete resources shared by all, according to their self-interest. The regeneration rate of resources depends on the number of resources. When the number of resources is reduced to a certain level, the resources are no longer regenerated. In social dilemmas, there are four possible outcomes of each player, namely R (reward arising from mutual cooperation), P (punishment of mutual defection), S (sucker outcome obtained by the player who cooperates with a defecting one), and T (temptation outcome achieved by defecting against a cooperator). The four payoffs must satisfy the following _social dilemma inequalities_[13]:  
* \(R>P\): mutual cooperation is preferred to mutual defection. * \(R>S\): mutual cooperation is preferred to being exploited by a defector. * \(2R>S+T\): mutual cooperation is preferred to an equal probability of unilateral cooperation and defection. * \(T>R\): exploiting a cooperator is preferred over mutual cooperation. * \(P>S\): mutual defection is preferred over being exploited. In order to capture aspects of real-world social dilemmas, [PERSON] et al. proposed sequential social dilemmas (SSDs) [18] which are spatially and temporally extended partially observable Markov games, because practical cooperation and defection are not simple atomic actions. Markov games are environments that simulate the Markov decision process and are convenient for studying reinforcement learning. An N-player SSD is a tuple \((\mathcal{M},\Pi_{c},\Pi_{d})\) of Markov games and satisfies the following properties. \(\Pi_{c}\) and \(\Pi_{d}\) are two disjoint sets of policies which represent cooperation and defection, respectively. The set of all players' policies is \((\pi_{c}^{1},\ldots,\pi_{c}^{l},\pi_{d}^{1},\ldots,\pi_{d}^{m})\in\Pi_{c}^{l} \times\Pi_{d}^{m}\) with \(l+m=N\). Assuming \(N=2\), the empirical payoffs \((R,P,S,T)\) under the initial state \(s\) can be defined as \((R(s),P(s),S(s),T(s))\) through their long-term payoff, where  
\[R(s) :=V_{1}^{\pi_{c},\pi_{c}}(s)=V_{2}^{\pi_{c},\pi_{c}}(s), \tag{2}\] \[P(s) :=V_{1}^{\pi_{d},\pi_{d}}(s)=V_{2}^{\pi_{d},\pi_{d}}(s),\] (3) \[S(s) :=V_{1}^{\pi_{c},\pi_{d}}(s)=V_{2}^{\pi_{d},\pi_{c}}(s),\] (4) \[T(s) :=V_{1}^{\pi_{d},\pi_{c}}(s)=V_{2}^{\pi_{c},\pi_{d}}(s). \tag{5}\]  
Let the empirical payoffs \((R,P,S,T)\) be induced by policies \((\pi_{c}^{1},\ldots,\pi_{c}^{l},\pi_{d}^{1},\ldots,\pi_{d}^{m})\in\Pi_{c}^{l} \times\Pi_{d}^{m}\) via Equations (2)-(5). A Markov game is an SSD game if there exists a state \(s\in S\) for which the induced payoff satisfies the five _social dilemma inequalities_. Sequential social dilemmas show the tension between individual short-term incentives and collective long-term interests. An individual agent needs to balance its short-term benefits and long-term benefits. Each agent will get a higher reward in the long-term if all agents choose to cooperate. However, an agent can obtain a higher reward in the short term by choosing to defect with non-cooperative behavior. Agents are tempted to defect because of the incentive structure of the SSDs.