## 4 ML-Based Resource Allocation
### Deep Reinforcement Learning Approach  
Deep reinforcement learning (DRL) is a computational approach to learning from action, and the most common algorithm used for that is Deep Q-learning. DRL is the fusion of DL and RL algorithms where the agent learns the best actions from the environment to reach its goal. To overcome the issues in a large, complex, and dynamic environment, DRL uses a deep Q-network (DQN) algorithm that utilizes neural network models to predict the next appropriate action. Hence, RL can further be classified as model-based methods like deep-Q-Networks and model-free methods such as Q-Learning and state action reward state action (SARSA). In the Q-learning method, for each state-action pair a DNN is used by DQN algorithm to estimate Q(s, a) instead of calculating them. The DQN flowchart is described in Figure 3 where a DNN is used by DQN with weight \(\theta\) for an action-value function network model \(Q(s,a;\theta)\), and for the target network model \(Q(s,a;\theta^{\prime})\) another DNN with weight \(\theta^{\prime}\) is used. With each interaction with the environment, the action-state function is updated with the present iteration parameter \(\theta^{\prime}\). Here DQN algorithm objective is to minimize the loss function \(L(\theta)\), which is given by [46] as follows:  
\[L(\theta)=\mathbb{E}\Big{[}\big{(}y_{t}^{tar}-Q(s,a;\theta)\big{)}^{2}\Big{]}\] (23a) where the target Q-value, \[y_{t}^{tar}\], can be estimated by the following equation: \[y_{t}^{tar}=r_{t+1}+\eta\max_{a}Q(s^{\prime},a;\theta^{\prime})\] (23b) where \[a=argmax_{a}Q(s^{\prime},a;\theta)\] and \[\theta^{\prime}\] will be updated by \[\theta\] after every training step. Like the Q-learning algorithm, to obtain the optimal value of \(\theta^{*}\) and corresponding Q-values, an interactive process is adopted in DRL. In DQN, the agent uses an \(\epsilon\)-greedy strategy to avoid local optimal solutions and receive a new experience when it interacts with the environment. During DQN training period, the agent will use the \(\epsilon\)-greedy strategy to select the next action, \(\mathbf{a}\), from \(Q(s,a;\theta)\), obtain reward \(r_{t}\) immediately and move into the new state \(s_{t+1}\) at time, \(t+1\) and finally store the quadruple information \((s_{t},a_{t},r_{t},s_{t+1})\) in replay memory. During network training, the agent reuses these sample experiences data stored in the replay memory to improve the convergence speed and stability of the algorithm. After every iteration, the main DNN parameters \(\theta\) of the network model in the action-value function are synchronized to the target network, and it is represented by  
\[\theta\leftarrow\theta^{\prime}+\tau\
abla L(\theta) \tag{24}\]  
where \(\tau\in(0,1)\) and \(\
abla\) are the learning rate and gradient descent respectively [47]. Algorithm 2 represents the resource allocation and power control in a D2D-enabled cellular network using a DQN-based DRL algorithm. The optimal policy can be obtained through the DRL agent by repeating the above procedure until the convergence of the weights \(\theta\). Converged DQN is well-trained to learn the patterns of the environment and is capable of managing unknown environmental states. Hence, by using the well-trained DQN model, each agent can make decisions autonomously to keep high performance without further training. Figure 3: DQN framework for D2D-enabled cellular networks. ```
1 Input: D2D-enabled HetNets cellular network environment parameters, and minimum SINR requirements for D2D pairs, SUEs, and CUEs;
2 Output: DQN/RL model training and decision results in the testing phase. ;
3 Output: Effective resource allocation and power control;
4 Initialization: Set \(\theta=\theta^{\prime}\), a random DNN parameters initialization for the main network and target network as \(\theta\) and \(\theta^{\prime}\) respectively.