## 4 ML-Based Resource Allocation
### Reinforcement Learning Approach
In an RL-based scheme for the given network, the D2D pair acts as an agent that interacts with the environment, generates trajectory, and moves into a new state \(s\to s^{\prime}\) by executing an action a. For each action executed by the agent, it will receive either a penalty or reward, r from the environment. In other words, in an RL-based D2D-enabled network, each \(j^{th}\) agent will act as follows: (i) Monitor its current status using the environment state \(s\in S\); (ii) Perform one of the valuable action \(a\in A\); (iii) Receive an instant penalty or reward, \(r\) from the environment; (iv) Move into a new state \(s^{\prime}\in S\). If the reward is based on the current state-action and not affected by any earlier state-action, then the \(j^{th}\) agent will satisfy the Markov property. In that case, MDP can be defined as follows: (i) A discrete set of environment states \(S_{j}\); (ii) A discrete set of possible actions \(A_{ji}\); (iii) The probability of state transition for an environment-specific time slot, \(P_{r}(s^{\prime}|s)=P_{r}(s^{\prime},a,s)\), where \(\forall a_{j}\in A_{j}\) and \(\forall s_{j}\in S_{j}\). Q-learning is an extremely popular algorithm in RL to determine the optimal action of a system. Figure 2 represents the agent-environment interactions in an RL-based network where a reward is represented by a Q-function and the main goal is to maximize it. The agent explores various action states by interacting with the environment and updates the Q-value given by [46; 47]. To solve the above MDPs problem smoothly, the state-value function \(V^{\pi}(s)\) and the action-value function \(Q^{\pi}(s,a)\) for \(a\in A\) and \(s\in S\) are defined as follows:  
\[V^{\pi}(s)= \mathbb{E}^{\pi}\left[\sum_{t=0}^{\infty}\eta^{t}r(t)|s_{0}=s\right]\] \[= \mathbb{E}^{\pi}\left[r(t)+\eta V(s^{\prime})|s^{\prime}=s\right] \tag{20a}\]  
Figure 2: Agent–environment interaction in RL D2D-enabled cellular networks. \[Q^{\pi}(s,\ a)= \mathbb{E}^{\pi\pi}\Bigg{[}\sum_{t=0}^{\infty}\eta^{t}r(t)|s_{0}=s,\ a_{0}=a\Bigg{]}\] \[= \mathbb{E}^{\pi\pi}[r(t)+\eta Q(s^{\prime},a^{\prime}]\] \[= r(s,a)+\eta\sum_{s^{\prime}\in S}p(s^{\prime}|s,a)\sum_{a^{ \prime}}\pi(a^{\prime}|s^{\prime})Q(s^{\prime},a^{\prime}) \tag{20b}\]  
where \(\eta\in(0,1)\) represents the discount factor and \(r(s,a)\) is the immediate reward. \(p(s^{\prime}|s,a)\) and \(\pi\) represent the transition probability and policy for action, \(a\), in environment state, \(s\), respectively. Since expected future rewards are determined by the state-action value functions, they can be obtained for each state-action on every strategy. Therefore, at equilibrium, there will always be a maximum of state-action value functions and the optimal policy \(\pi^{*}\), and hence Equations (20a) and (20b) can be converted as  
\[V^{*}(s)= \text{max}_{\pi}V^{\pi}(s)\] \[= \mathbb{E}^{\pi}\Bigg{[}\sum_{t=0}^{\infty}\eta^{t}r(t)(s,a,s^{ \prime})|s_{0}=s,\pi\Bigg{]} \tag{21a}\] \[Q^{*}(s,\ a)= \text{max}_{\pi}Q^{\pi}(s,a)\] \[= r(t)+\eta\operatorname*{arg\,max}_{a^{\prime}}Q^{*}(s^{\prime}, a^{\prime})\] \[= \mathbb{E}^{\pi}\left[r(t)+\eta V(s^{\prime})|s^{\prime}=s\right] \tag{20a}\]0  
Thus, \(\pi^{*}\) is obtained by \(\pi^{*}(s)=\operatorname*{arg\,max}_{a}Q^{*}(s,a)\). By using Equation (21b) we cannot derive the value of \(Q^{*}\) and \(\pi^{*}\) due to unknown transition probabilities. But to obtain them we can use a Q-learning algorithm where the RL agent will continuously update the Q-values by executing new actions to interact with the environment iteratively. Therefore, the updated Q-value functions can be expressed as  
\[= \mathbb{E}^{\pi}\left[r(t)+\eta V(s^{\prime})|s^{\prime}=s\right] \tag{20a}\]1  
where \(Q^{*}(s,a)\) and \(Q(s,a)\) represent the new and old Q value, respectively, \(\mu\) is the learning rate, and the term \(\max_{a^{\prime}}Q(s^{\prime},a^{\prime})\) is the target value to the new state-action.