## 6 Proposed Joint Mode Selection and Resource Allocation Scheme (SD-Scheme)
### Proposed RL-Based Power Allocation Scheme
The optimum policy can be calculated through the Bellman formula:  
\[\mathcal{Y}^{*}(\mathcal{a})=\underset{\mathcal{a}\in\mathcal{A}}{max} \mathcal{G}^{*}(\mathcal{a}), \tag{17}\]  
Subsequently, the value function is identified by the following equation:  
\[\mathcal{Y}(\mathcal{a})=\underset{\mathcal{a}\in\mathcal{A}}{max}\mathcal{G} (\mathcal{a},\mathcal{a}), \tag{18}\]  
The given equation is utilized to identify the optimal action value in order to optimize \(\mathcal{G}(\mathcal{a}_{t},\mathcal{a}_{t})\) for each state involved. \[\mathcal{a}=\underset{\mathcal{a}\in\mathcal{A}}{max}\mathcal{G}(\mathcal{a}, \mathcal{a}), \tag{19}\]  
To decide what action \(\mathcal{a}\) is going to be chosen during a particular time \(t\), the Exploration and Exploitation Policy (EEP) feature is employed as follows:  
\[\mathcal{a}_{t}= \begin{cases}\underset{\mathcal{a}\in\mathcal{A}}{argmax} \mathcal{G}(\mathcal{a},\mathcal{a})&\text{exploitation}\\ \underset{\mathcal{a}\in\mathcal{A}}{\text{randrand}}(\mathcal{a})&\text{exploration} ^{\prime}\end{cases}  \tag{20}\]  
In Equation (20), the '\(\mathcal{e}-\)greedy' strategy is used while performing EEP, indicating the fact that the probabilities of exploitation and exploration are \(\mathcal{e}\) and \(1-\mathcal{e}\), accordingly. Furthermore, a Markov Decision Process model of D2D communications underlaying cellular networks is provided to allocate power for each D2D pair using SARSA-based RL. Moreover, the actions of the agents, which are represented as \(\mathcal{a}\), include a set of transmission powers, are indicated by \(P_{n}\), and are allocated to the D2D transmitters \(T_{n}\). Figure 2: Agent–environment interactions in SARSA. The reward function is formulated based on EE for each D2D pair \(n\) employing sub-channel \(k\) as follows:  
\[R_{n}= \begin{cases}EE_{n,k}(\sqrt{R_{n,k}(t)-R_{min}^{th}})&if\ R_{n,k}(t)\geq\ R_{ min}^{th}\\ EE_{n,k}(-e^{(R_{n,k}(t)-R_{min}^{th})})&if\ R_{n,k}(t)<R_{min}^{th}\end{cases} , \tag{21}\]  
Exploration involves a comprehensive examination of the network, gathering information, and randomly selecting actions to evaluate their efficiency. However, exploitation exploits the advantage of previous decisions according to the Q-table. Exploration and exploitation have trade-off features in power distribution techniques that utilize RL. SARSA is more appropriate for our case than Q-learning because of its on-policy nature, which updates action values according to the current policy, hence offering a more adaptive response to network changes. This functionality is especially beneficial in D2D communication systems, where interference levels and network topology may vary. The Q-learning method, while efficient in static situations, fails to update according to real-time rules, potentially limiting its flexibility in dynamic contexts. Our investigation demonstrates that SARSA surpasses Q-learning in EE and power consumption, particularly under changing interference levels, highlighting its enhanced capability to handle real-time interference in high-density networks. Algorithm 2 defines the extensive structure of the SARSA strategy.