## 4 Cooperative Reinforcement Learning Algorithm for Resource Allocation
In addition, if the channel gains (\(G_{Bu}\) and \(G_{uv}\)) quality is good then it is possible to achieve higher throughput without excessively increasing the transmit power, i.e., without causing too much interference to others. On the other hand, if the channel gain is too low, higher transmit power is required, which leads to increased interference. Now, the state values of these variables can be either 0 or 1 based on following conditions. If the value of the variables are greater than or equal to a threshold value, then this denotes that their state value is '0'. On the contrary, if the values are less than the threshold value, then their state value is '1'. So, \(\gamma_{r}^{c}\geq\tau_{0}\) means state value '1' and \(\gamma_{r}^{c}<\tau_{0}\) means state value '0'. Similarly, \(G_{Bu}\geq\tau_{1}\) means state value '1' and \(G_{Bu}<\tau_{1}\) means state value '0'. Consequently, \(G_{uv}\geq\tau_{2}\) means state value '1' and \(G_{uv}<\tau_{2}\) means the state value '0'. In this way, based on the combination of the value of these variables, the total number of possible states is eight where \(\tau_{0}\), \(\tau_{1}\) and \(\tau_{2}\) are the minimum SINR and channel gain guaranteeing the QoS performance of the system. Action/Task: The action of each agent consists of a set of transmitting power levels. It is denoted by  
\[A=(a_{r}^{1},a_{r}^{2},\ldots,a_{r}^{pl})\]  
where \(r\) represents the \(r\)th Resource Block (RB), and \(pl\) means that every agent has \(pl\) power levels. In this work, we consider the power levels to assign within the range of 1 to \(P_{max}\) in the interval of 1 dBm. **Reward Function**: The reward function for the reinforcement learning is designed focusing on the throughput of each agent/user which is formulated as follows:  
\[\Re=\log_{2}(1+SINR(u)) \tag{7}\]  
when \(\gamma_{r}^{c}\geq\tau_{0}\), \(G_{Bu}\geq\tau_{1}\) and \(G_{uv}\geq\tau_{2}\). Otherwise, \(\Re=-1\). SINR (\(u\)) denotes the signal to interference plus noise power ratio of user \(u\) (Step 7-10 in Algorithm 1). SARSA(\(\lambda\)) is an on-policy reinforcement learning algorithm that estimates the value of the policy being followed where \(\lambda\) is a parameter such as learning rate [26]. In SARSA learning algorithm, every agent needs to maintain a Q matrix which is initially assigned 0 and the agents may be in any state. Based on performing one particular action, it shifts from one state to another. The basic form of the learning algorithm is \((s_{t},a_{t},\Re,s_{t+1},a_{t+1})\), which means that the agent was in state \(s_{t}\), did action \(a_{t}\), received reward \(\Re\), and ended up in state \(s_{t+1}\), from which it decided to perform action \(a_{t+1}\). This provides a new iteration to update \(Q_{t}(s_{t},a_{t})\). SARSA(\(\lambda\)) helps to find out the appropriate sets of actions for some states. The considered state-action pair's value function \(Q_{t}(s_{t},a_{t})\) as follows:  
\[Q_{t}(s_{t},a_{t})=\Re+\gamma Q_{t+1}(s_{t+1},a_{t+1}) \tag{8}\]  
In Equation (8), \(\gamma\) is a _discount-factor_ which varies from 0 to 1. The higher the value, the more the agent relies on future rewards than on the immediate reward. The objective of applying reinforcement learning is to find the optimal policy \(Q_{t}^{\pi}(s_{t},a_{t})\) which maximizes the value function \(\pi=\max\limits_{\pi}Q_{t}^{\pi}(s_{t},a_{t})\)We consider the cooperative fashion of this algorithm where each agent shares the value function with each other. At each time step, \(Q_{t+1}\) for the iteration \(t+1\), \(Q_{t+1}\) is updated with the temporal difference error \(\delta_{t}\) and the immediate received reward. The \(Q\) value has the following update rules:  
\[Q_{t+1}(s_{t+1},a_{t+1})\gets Q_{t}(s_{t},a_{t})+\alpha\delta_{t}e_{t}(s_ {t},a_{t}) \tag{9}\]  
for all \(s\), \(a\). In Equation (9), \(\alpha\in[0,1]\) is the learning rate which decreases with time.