# Admission-Based Reinforcement-Learning Algorithm in Sequential Social Dilemmas
## 3 Method
### Admission Reward  
The give-or-take-some (GOTS) paradigm, proposed in [29], introduces a more realistic hybrid social dilemma by allowing participants to both contribute and request resources from a shared pool. Considering there are \(N\) players, each of these players \(i\) has a endow ment, \(e_{i}\), (where \(e_{i}\geq 0\)_and_\(i=1,\ldots,n\)). Each player \(i\) can \"contribute\" \(c_{i}\geq 0\), and/or \"request\" \(r_{i}\geq 0\). The individual payoff is formulated as:  
\[P_{i}=(e_{i}-c_{i})+\zeta(r_{i}+b),\text{ where }\zeta=1\text{ if }T\geq 0\text{ and }\zeta=0\text{ if }T<0, \tag{3}\]  
where \(b\) is a positive bonus, and it depends on the sum of net contributions, \(T=\sum(c_{i}-r_{i})\), which we also call the admission boundary. In real dilemmas, the contributed and demanded resources are often not the same resource, and there is no bonus. To this end, we set \(r_{i}\) as the environmental reward obtained by the player \(i\), and then the utility of each player can be simplified as:  
\[U_{i}=\zeta r_{i},\text{ where }\zeta=1\text{ if }T\geq 0\text{ and }\zeta=0\text{ if }T<0, \tag{4}\]  
in which \(T=\sum(\kappa c_{i}-r_{i})\). The parameter \(\kappa\) controls the proportion of contribution and environmental reward, which depends on the specific environment. However, as a computational model, the GOTS model is only suitable for matrix games. Equation (4) can only work in stateless games [42]. Thus, we need to extend this model to temporally extended Markov games. The main issue with redefining Equation (4) for a Markov game is that different players' rewards are obtained at different timesteps. Therefore, the key to extending Equation (4) to this case is to make each player's reward trajectory temporally smooth. Let \(r_{i}(s,a)\) represent the reward the \(i\)-th player receives for taking action \(a\) in state \(s\) at time step \(t\). We define the \(i\)-th player's Admission reward \(u_{i}(s,a)\) and admission boundary at timestep \(t\) as:  
\[ \begin{cases}u_{i}(s_{t}^{i},a_{t}^{i})=\zeta r_{i}(s_{t}^{i},a_{t}^{i}),\text { where }\zeta=1\text{ if }T\geq 0\text{ else }\zeta=0\\ T=\sum[xd_{t}^{i}(s_{t}^{i},a_{t}^{i})-e_{t}^{i}(s_{t}^{i},a_{t}^{i})]\end{cases}  \tag{5}\]  
where the temporal smoothed contributions \(a_{t}^{i}\) and rewards \(e_{t}^{i}\) of each agent \(i\) are updated at each timestep according to:  
\[d_{t}^{i}(s_{t}^{i},a_{t}^{i}) =\gamma\lambda d_{t-1}^{i}(s_{t-1}^{i},a_{t-1}^{i})+c_{t}^{i}(s_{t }^{i},a_{t}^{i}) \tag{6}\] \[e_{t}^{i}(s_{t}^{i},a_{t}^{i}) =\gamma\lambda e_{t-1}^{i}(s_{t-1}^{i},a_{t-1}^{i})+r_{t}^{i}(s_{ t}^{i},a_{t}^{i}), \tag{7}\]  
where \(\lambda\)consists of a high-level policy and several low-level policies, respectively, denoted as \(\pi_{hi}\) and \(\pi_{i}\). The \(\pi_{hi}(z|o)\) selects a \(\pi_{i}\) based on partial observations of the environment, where \(z\in Z\) is the space of possible low-level policies. The selected low-level policy starts to perform specific actions and obtains the corresponding reward \(r\), while \(\pi_{hi}\) obtains Admission rewards \(u_{t}\). Unlike earlier work, the high-level policy does not switch between low-level policies every \(\tau\) timestep. Instead, it executes the selected low-level policy at every timestep, such as the low-level policy. This is necessary because we have no way to ensure that the timesteps required by each strategy are the same. In addition, in order to avoid frequent replacement of low-level policies by high-level policy, the high-level policy will be subject to a minor penalty when each low-level policy is changed. This structure is illustrated in Figure 2.  
To improve efficiency, we specify the low-level policy \(\pi_{1}\) and \(\pi_{2}\) to maximize the environmental reward \(r\) and the contribution reward, respectively. For other low-level policies, since we cannot directly quantify the differences in low-level policies, we utilize information theory to guide the exploration of various possible behaviors.