## 4 ML-Based Resource Allocation
### Reinforcement Learning Approach  
Reinforcement learning (RL) is an adaptable machine learning algorithm that can obtain the optimal decision-making policy in a dynamic environment without a training dataset [44] by using a trial-and-error process. Thus, it automatically adapts to the new environment. In the RL, the decision maker is called an agent. Markov Decision Processes (MDP) are widely used as optimization tools for wireless communication systems due to the stochastic and dynamic nature of networks like fluctuating signal quality, user mobility and variable traffic loads. In the RL framework, MDP provides a mathematical model in the decision-making process, which is used to interact between the agent and the environment for optimal strategies. Based on the outcome of each action, the agent either receives a reward or penalty, and the process randomly moves into a new state based on state transition probabilities. Figure 2 is a basic RL framework for a D2D-enabled cellular network where basic elements are illustrated. The different components of the reinforcement learning algorithm are described as follows:  
AGENT: In the RL algorithm, D2D pairs act as agents. For each communication link, an agent learns and makes decisions by interacting with the environment. In MDP, an agent, under different network states, decides the proper communication modes and optimum resource blocks (RB) from the available radio resources. each D2D user's transmitter functions as an agent in D2D-enabled cellular HetNets. STATE: To determine the states for a particular time slot, \(t\) a learning agent relies on the environmental conditions. The state observed by any agent, '\(j\)' can be given as:  
\[s_{j}=(A_{RB},A_{CSI},A_{QoS})\in S \tag{17}\]  
where \(A_{CSI}\), \(A_{RB}\), and \(A_{QoS}\) represent the observed channel status information, RB occupancy status among users, and QoS requirements for all users. ACTION: In a D2D-enabled HetNet, the agent has various modes and a certain number of resource blocks available for communication. Based on current state conditions, an agent will decide on appropriate mode selection, transmit power, and RB allocation. So, the action of an agent \(a_{j}\) can be defined as:  
\[a_{j}=(A_{MS},A_{PC},A_{RB})\in A. \tag{18}\]  
where \(A_{MS}\), \(A_{PC}\), and \(A_{RB}\) represent mode selection, transmit power control, and RB assignment for a specific action \(a_{j}\in A\) under the current state \(s_{j}\). TRANSITION PROBABILITY:  
Transition probability describes the changes in the environmental status during the interactions with the agent. For example, an agent executes an action, \(a\), in a time slot, \(t\), and state, \(s\), and gains a reward, \(r\). The probability of that agent will move to the next state, \(s^{\prime}\), at time \(t+1\) is defined by the transition probability, \(P_{r}(s^{\prime}|s,a)\). So, the transition probability, \(P_{r}(s^{\prime}|s,a)\), is the probability that an agent executes the action \(a\in A\) under the state \(s\in S\) and moves into a new state \(s^{\prime}\in S\)[45], and it can be expressed as:  
\[P_{r}(s^{\prime}|s,a)= \begin{cases}1,&s^{\prime}=state(a),\\ 0,&\text{otherwise}.\end{cases}  \tag{19}\]  
REWARD:  
In RL, the reward function is constructed to achieve the goal of the optimization problem. To maximize the overall system sum rate, each agent makes a joint decision for mode selection, power control, and resource allocation, and therefore, the reward of the agent, \(j\), can be defined as follows:  
\[R_{j}(s,a)= \begin{cases}R_{j}^{D},&\text{if conditions C2, C5 and C7 are met,}\\ -1,&\text{otherwise}.\end{cases} \]  
POLICY: Simply put, the policy is a decision-making rule for an agent. In any state, \(s\), the strategy of the agent to execute an action, \(a\), is represented by policy \(\pi(s,a)\), where we have \(a\in A,s\in S\), and \(\sum_{a\in A}\pi(s,a)=1,\forall s\in S\).