## 6 Proposed Joint Mode Selection and Resource Allocation Scheme (SD-Scheme)
### Proposed RL-Based Power Allocation Scheme  
The power allocation optimization can be achieved using a dynamic distribution scheme. A wireless network with real-time communications demands immediate training and learning of the D2D pairs to provide distributed power allocation without imposing a significant load over the gNB. Thus, ML provides a potential solution with a wide range of applications in the execution of dynamic resource allocation and tackles many challenges associated with prospective communications networks. The transmitter of the D2D pair, performing as the intelligent agent in this scenario, has the ability to learn and make the most suitable decision to enhance the network performance. One of the most advanced ML techniques is RL. RL utilizes an approach based on trial and error to determine the best resource allocation decisions. Moreover, RL works efficiently without any previous information about the system environment, in contrast to the traditional techniques. RL may enhance performance by facilitating rapid detection of optimum solutions or decisions in comparison with conventional centralized techniques. SARSA is a reinforcement learning approach utilized to identify the best action in a dynamic resource allocation system. This research presents an SARSA-based approach to solving the issue of power distribution, including the following elements:  
Agent: the agent is the D2D transmitter and serves as a crucial element in the power allocation issue. State: The state of the SARSA algorithm includes essential network information, such as interference levels, channel conditions, and user location. These factors describe the present state of the environment, operating as inputs to an agent's decision-making in power distribution. In this case, the agent indicates the connection of D2D pairs. Action: The action is an activity performed via the agent. The power distribution levels established by the D2D pairs constitute the action, which comprises a range of powers from \(P_{min}\) to \(P_{max}\). Reward: The reward function in the SARSA-based power management approach is defined as the EE of each D2D pair within the system. The agent's interactions with the environment in SARSA are illustrated in Figure 2.  
In the specific state \(\mathit{s_{t}}\), the action \(\mathit{\epsilon_{t}}\) is selected, and the reward \(\mathcal{R}_{t}\) is allocated to the D2D pair (agent) for each action performed. The agent subsequently transitions to the newly created state \(\mathit{s_{t+1}}\) and executes another action, \(\mathit{\alpha_{t+1}}\) for its present state \(\mathit{s_{t+1}}\). Moreover, the pattern \(\mathit{s_{t}}\)-\(\mathit{\alpha_{t}}\)-\(\mathit{\mathcal{R}_{t}}\)-\(\mathit{\alpha_{t+1}}\)-\(\mathit{\alpha_{t+1}}\) defines the sequence of procedures for the suggested SARSA algorithm. The Q-value is firstly set to zero value, then the proposed algorithm modifies the Q-table in accordance with the current policy. The algorithm shows the agent inside the SARSA framework has to accomplish several episodes. At each time step \(t\), the agent in the state \(\mathcal{A}_{t}\) chooses an action \(\mathcal{a}_{t}\) based on greedy strategy. Afterwards, the agent receives the reward \(\mathcal{R}_{t}\) and proceeds to the following state \(\mathcal{a}_{t+1}\), where they choose the action \(\mathcal{a}_{t+1}\) in dependence on the Q-table. The state-action equation can be represented as:  
\[\mathcal{G}(\mathcal{a}_{t},\mathcal{a})=(1-\mathcal{a})\mathcal{G}(\mathcal{ a}_{t},\mathcal{a})+\alpha[\mathcal{R}_{t+1}+\mathcal{Y}\mathcal{G}(\mathcal{a}_{t+ 1},\mathcal{a}_{t+1})], \tag{16}\]  
Here, \(\mathcal{a}\) represents the learning rate of the agent, \(\mathcal{R}_{t+1}\) denotes the reward function of the next state, and \(\mathcal{Y}\) signifies the discount factor. In the framework of RL, the agent aims to optimize the reward by adopting an optimal policy.