## 3 System Model  
We consider a network that consists of one Base station (BS) and a set of \(\hat{\mathcal{C}}\) Cellular users (CU), i.e., \(\hat{\mathcal{C}}=\{1,2,3,\ldots,C\}\). There are also \(\hat{\mathcal{D}}\) D2D pairs, \(\hat{\mathcal{D}}=\{1,2,3,\ldots,D\}\) coexist with the cellular users within the coverage of BS. In a particular D2D pair, \(d_{T}\) and \(d_{R}\) are the D2D transmitter and D2D receiver respectively. The set of User equipments (UE) in the network is given by UE = \(\{\hat{\mathcal{C}}\cup\hat{\mathcal{D}}\}\). Each D2D transmitter \(d_{T}\) selects an available Resource block (RB) \(r\) from the set \(RB=\{1,2,3,\ldots,R\}\). In addition, underlay D2D transmitters select the transmit power from a finite set of power levels, i.e., \(p_{r}=(p_{r}^{1},p_{r}^{2},\ldots,p_{r}^{R})\). Each D2D transmitter should select resources, i.e., RB-power level combination refers to transmission alignment [21]. For each RB \(r\in R\), there is a predefined threshold \(I_{th}^{(r)}\) for maximum aggregated interference. We consider that the value of \(I_{th}^{(r)}\) is known to the transmitters using the feedback control channels. An underlay transmitter uses a particular transmission alignment in a way that the cross-tier interference should be within the threshold limit. According to our proposed system model, only one CU can be served by one RB where D2D users can reuse the same RB to improve the spectrum efficiency. For each transmitter \(d_{T}\), the transmit power over the RBs is determined by the vector \(p_{r}=[p_{r}^{1},\ p_{r}^{2},\ldots,p_{r}^{R}]^{T}\) where \(p_{r}\geq 0\) denotes the transmit power level of transmitter over resource block \(r\). If RB is not allocated to the transmitter then the power level \(p_{r}=0\). As we assume that each transmitter selects only one RB where only one entity in the power level \(p_{r}\
eq 0\). Signal-to-interference-plus-noise-ratio (SINR) can be treated as an important factor to measure the link quality. The received SINR for any D2D receiver over \(r\)th RB as follows:  
\[\gamma_{r}^{D_{u}}=\frac{p_{r}^{D_{u}}.G_{D_{u}r}^{uu}}{\sigma^{2}+p_{r}^{c}. G_{r}^{cu}+\sum_{v\in D}^{v\
eq u}p_{r}^{d_{u}}.G_{D_{v}r}^{uv}} \tag{1}\]  
where \(p_{r}^{D_{u}}\) and \(p_{r}^{c}\) denote the \(u\)th D2D user and cellular user uplink transmission power on \(r\)th RB, respectively. \(p_{r}^{D_{u}}\leq P_{max}\), \(\forall_{u}\in D\) where \(P_{max}\) is the upper bound of each D2D user's transmit power. \(\sigma^{2}\) is the noise variance [9]. \(G_{D_{u}r}^{uu}\), \(G_{D_{u}r}^{uv}\) and \(G_{r}^{cu}\) are the channel gains in the \(u\)th D2D link, the channel gain from D2D transmitter \(u\) to receiver \(v\), and the channel gain from cellular transmitter \(c\) to receiver \(u\), respectively. \(D_{r}\) is a D2D pairs set sharing the \(r\)th RB. The SINR of a cellular user \(c\in\hat{\mathcal{C}}\) on the \(r\)th RB is  
\[\gamma_{r}^{c}=\frac{p_{r}^{c}.G_{c,r}}{\sigma^{2}+\sum_{v\in D}p_{r}^{d_{v}}. G_{v,r}} \tag{2}\] \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline \multirow{2}{*}{**Methods**} & \multirow{2}{*}{**References**} & **D2D** & **System** & **Transmission** & **Action** & \\  & & **Throughput** & **Throughput** & **Alignment** & **Scheduling** & **Cooperation** \\ \hline \multirow{8}{*}{Classical approaches} & LWFA [7] & Yes & Yes & No & N/A & No \\  & MBS [8] & Yes & Yes & No & N/A & No \\  & MINLP [9] & Yes & No & No & N/A & No \\  & CFG [10] & Yes & No & No & N/A & No \\  & RIR [11] & Yes & No & No & N/A & No \\  & RIR [12] & Yes & No & No & N/A & No \\  & RIR [13] & Yes & No & No & N/A & No \\  & LDD [14] & Yes & No & No & N/A & No \\  & JS [15] & Yes & Yes & No & N/A & No \\  & GTA [16] & Yes & Yes & No & N/A & No \\ \hline \multirow{2}{*}{RL based method} & DIRL [17] & Yes & No & No & No & No \\  & DIRL [18] & No & Yes & No & No & No \\ \hline Proposed method & Cooperative RL & Yes & Yes & Yes & Yes & Yes \\ \hline \hline \end{tabular}
\end{table}