## 4 Cooperative Reinforcement Learning Algorithm for Resource Allocation  
In this section, we describe the basics of Reinforcement learning (RL), followed by our proposed cooperative reinforcement learning algorithm. After that, we describe the set of states, the set of actions and reward function for our proposed algorithm. Finally, Algorithm 1 shows the overall proposed resource allocation method and Algorithm 2 shows the execution steps of our proposed cooperative reinforcement learning. We apply a Reinforcement learning (RL) algorithm named state action reward state action, SARSA(\(\lambda\)), for adaptive resource in D2D communication for efficient resource allocation. This variant of standard SARSA(\(\lambda\)) [24] algorithm has some important features like cooperation by using a neighboring factor, a heuristic policy for exploration and exploitation, and a varying learning rate considering the visited state-action pair. Currently, we are applying the learning algorithm for the resource allocation of D2D users considering that the allocation of cellular users is performed prior to the allocation of D2D users. We consider the cooperative fashion of this learning algorithm which helps to improve the throughput as explained in Section 1 by sharing the value function and incorporating weight factors for the neighbors of each agent. In reinforcement learning, there is no need for prior knowledge about the environment. Agents learn how to behave with the environment based on the previous experience achieved, which is traced by a parameter, i.e., Q-value and controlled by a reward function. There should be some actions/tasks to perform at every time step. After performing every action, the agents shifts from one state to another and it gets a reward that reflects the impact of that action, which helps to decide about the next action to perform. The basic reinforcement learning is a form of Markov decision process (MDP). Figure 2 depicts the overall model of a reinforcement learning algorithm. Each agent in RL has the following components [25]:  
* Policy: The policy acts as a decision making function for the agents. All other functions/components help to improve the policy for better decision making. * Reward function: The reward function defines the ultimate goal of an agent. This helps to assign a value/number to the performed action, which indicates the intrinsic desirability of the states. The main objective of the agent is to maximize the reward function in the long run. * Value function: The value function determines the suitability of action selection in the long run. The value of a state is accumulated reward over long run when starting from the current state. * Model: The model of the environment mimics the behavior of the environment which consists of a set of states and a set of actions. In our model of the environment, we consider the components of the reinforcement learning algorithm as follows:  
Figure 2: Basic components of a reinforcement learning. The agent performs an action to the environment which gives a reward and helps to shift from one state to another. Agent: All the resource allocators: D2D Transmitters. State: The state of D2D user \(u\) on RB \(r\) at time \(t\) is defined as:  
\[S_{t}^{u,r}=\gamma_{r}^{c}\cup G_{Bu}\cup G_{uv}\]  
We consider three variables \(\gamma_{r}^{c}\), \(G_{Bu}\) and \(G_{uv}\) for defining the states for maintaining the overall quality of the network. \(\gamma_{r}^{c}\) is the SINR of a cellular user on the \(r\)th RB. \(G_{Bu}\) is the channel gain between the BS and an user \(u\). \(G_{uv}\) is the channel gain between two users \(u\) and \(v\). The variables \(\gamma_{r}^{c}\), \(G_{Bu}\) and \(G_{uv}\) are important to consider for the resource allocation. The SINR \(\gamma_{r}^{c}\) is the indicator of the quality of service of the network.