## 4 Cooperative Reinforcement Learning Algorithm for Resource Allocation
\(\delta_{t}\) is the temporal difference error which is calculated by following rule (Step 7 in Algorithm 2):  
\[\delta_{t}=\Re+\gamma_{1}fQ_{t+1}(s_{t+1},a_{t+1})-Q_{t}(s_{t},a_{t}) \tag{10}\]  
In Equation (10), \(\gamma_{1}\) is a discount-factor which varies from 0 to 1. The higher the value, the more the agent relies on future rewards than on the immediate reward. \(\Re_{t+1}\) represents the reward received for performing an action. \(f\) is the neighboring weight factor of agent \(i\) where this factor consists of the effect of neighbor's Q-value, which helps to update the Q-value of agent \(i\) that is calculated as follows [27]:  
\[f=\frac{1}{ngh(n_{i})} \text{if }ngh(n_{i})\
eq 0 \tag{11}\] \[=1 \text{otherwise}. \tag{12}\]  
where \(ngh(n_{i})\) is the number of neighbors of agent \(i\) within the D2D radius. BS provides the information of number of neighbors for each agent [28]. There is a trade-off between exploration and exploitation in reinforcement learning. Exploration chooses an action randomly in the system to find out the utility of that chosen action. Exploitation deals with the actions which have been chosen based on previously learned utility of the actions. We use a heuristic for exploration probability at any given time such as:  
\[\epsilon=\min(\epsilon_{max},\epsilon_{min}+k*(S_{max}-S)/S_{max}) \tag{13}\]  
where \(\epsilon_{max}\) and \(\epsilon_{min}\) denote upper and lower boundaries for the exploration factor, respectively. \(S_{max}\) represents the maximum number of states which is eight in our work and S represents the current number of states already known [29]. At each time step, the system calculates \(\epsilon\) and generates a random number in the interval \([0,1]\). If the selected random number is less than or equal to \(\epsilon\), the system chooses a uniformly random task (exploration), otherwise it chooses the best task using \(Q\) values (exploitation). \(k\) is a constant which controls the effect of unexplored states (Step 4 in Algorithm 2). SARSA(\(\lambda\)) helps to improve the learning technique by eligibility trace. In Equation (9), \(e_{t}(s,a)\) is the eligibility trace. The eligibility trace is updated by the following rule:  
\[S_{t}^{u,r}=\gamma_{r}^{c}\cup G_{Bu}\cup G_{uv}\]0 \[S_{t}^{u,r}=\gamma_{r}^{c}\cup G_{Bu}\cup G_{uv}\]1  
Here, \(\lambda\) is learning parameter for guaranteed convergence, whereas \(\gamma_{2}\) is the discount factor. In addition, the eligibility trace helps to provide higher impact on revisited states. For example, for a state-action pair \((s_{t},a_{t})\), if \(s_{t}\in s\) and \(a_{t}\in a\), the state-action pair is reinforced. Otherwise, the eligibility trace is removed (Step 8 in Algorithm 2). The learning rate \(\alpha\) is decreased in such a way that it reflects the degree to which a state-action pair has been chosen in the recent past. It is calculated as:  
\[S_{t}^{u,r}=\gamma_{r}^{c}\cup G_{Bu}\cup G_{uv}\]2  
where \(\rho\) is a positive constant and \(visited(s,a)\) represents the visited state-action pairs so far [30] (Step 6 in Algorithm 2). ```
Input :\(P_{max}=23\) dBm, Number of resource blocks = 30, Number of cellular users = 30, Number of D2D user pairs = 12, D2D radius = 20 m, Pathloss parameter = 3.5, Cell radius = 500 m, \(\tau_{0}=0.004\), \(\tau_{1}=0.2512\), \(\tau_{2}=0.2512\) [9] Output :RB-Power level, System Throughput
1 loop
2 Pathloss calculation by \(PL_{dB,B,\mu(.