## 3 Reinforcement Learning-Based Solution
### Reinforcement Learning (RL)  
The interactions between the agent and the environment in RL can be well described by the MDP model. The MDP model is structured as a tuple \((\mathcal{S},\mathcal{A},r,\mathcal{P},\delta)\), where \  
where \(G_{k}^{c,t}\) denotes the instantaneous channel information of the \(k^{th}\) cellular communication link that includes \(g_{b,i}\), \(g_{i,c}\), and \(g_{b,c}\). \(H_{m}^{d,t}\) represents the instant channel information of the \(m^{th}\) D2D communication link that includes \(\mathbf{h}_{t,i}\), \(\mathbf{h}_{r,i}\), and \(h_{tr}\). * _Action:_ The learning agent (D2D pair) does the appropriate action \(a(t)\in\mathcal{A}\) at time \(t\) during the learning process following the current state, \(s(t)\in\mathcal{S}\), based on the policy \(\pi\). Hence, the action is defined by constraints C3 and C4 (a D2D pair shares at most one CU's spectrum and the spectrum of a CU can be shared by at most one D2D pair, respectively), constraints C5 and C6 (maximum transmission power constraints for the CUs and the D2D pairs, respectively), and constraint C7 (practical reflecting coefficients of the IRS). Based on the constraints in the formulated optimization problem, the action in this reinforcement learning framework depends on the transmit power, the resource reuse indicator, and the IRS reflection coefficients. Thus, the action \(a(t)\in\mathcal{A}\) can be defined as follows: \[a(t)=\bigg{\{}\Big{\{}\big{\{}P_{k}^{c,t}\big{\}}_{k\in\mathcal{X}^{\prime}} \big{\{}P_{m}^{d,t}\big{\}}_{m\in\mathcal{M}^{\prime}}\big{\{}x_{k,m}^{t} \big{\}}_{k\in\mathcal{K},m\in\mathcal{M}^{\prime}}\big{\{}\theta_{n}^{t} \big{\}}_{n\in\mathcal{N}}\bigg{\}}.\] (6)
* _Transition probability:_\(\mathcal{P}(s^{\prime}|s(t),a(t))\) is the probability of transitioning from a current state \(s(t)\in\mathcal{S}\) to a new state \(s^{\prime}\in\mathcal{S}\) after executing an action \(a(t)\in\mathcal{A}\). * _Policy:_ The D2D agent's behavior is defined by a policy \(\pi\), which maps states \(s(t)\) to a probability distribution over the actions \(a(t)\), e.g., \(\pi:\mathcal{S}\rightarrow\mathcal{P}(\mathcal{A})\). Note that the policy function needs to satisfy \(\sum_{(a(t)\in\mathcal{A})}\pi(s(t),a(t))=1\). * _Reward:_ The reward \(r\) is the immediate return to the D2D agent after taking the action \(a(t)\in\mathcal{A}\) given the state \(s(t)\in\mathcal{S}\). It is also a performance indicator that indicates how good the action \(a(t)\) is in a given state \(s_{m}^{t}\) at time instant \(t\). Hence, considering the interactions with the environment, each D2D agent takes its decision to maximize its reward. Now, the reward function for the D2D pair can be written as \[r=R-Y_{1}\sum_{k\in\mathcal{K}}(\gamma_{k}^{c}-\gamma_{\text{min}}^{c})-Y_{2 }\sum_{m\in\mathcal{M}}\Big{(}\gamma_{m}^{d}-\gamma_{\text{min}}^{d}\Big{)},\] (7) where the first part is the network's SE at time \(t\) and the second and third parts denote the minimum data rate requirement that is not satisfied for the CUs and the D2D pairs, respectively. The parameters \(Y_{1}\) and \(Y_{2}\) are the weights of the second and third parts, and \[r= \begin{cases}R,&\gamma_{k}^{c}\geq\gamma_{\text{min}}^{c}\ \&\ \gamma_{m}^{d}\geq\gamma_{\text{min}}^{d},\\ \text{Negative},&\text{otherwise}.\end{cases} \] (8)  
The reward function remains positive for the D2D agent if it satisfies both the conditions \(\gamma_{k}^{c}\geq\gamma_{\text{min}}^{c}\) and \(\gamma_{m}^{d}\geq\gamma_{\text{min}}^{d}\), otherwise, it will be negative. The framework for the RL-based MDP model works as follows: the D2D agent observes a state \(s(t)\in\mathcal{S}\), and considering the interaction with the environment, it takes an action \(a(t)\in\mathcal{A}\), selecting the transmit power, the resource reuse indicators, and the passive beamforming based on the policy \(\pi\).