## 2 System Model and Problem Formulation  
### System Model  
We consider a single cellular system supporting D2D communication where \(M\) D2D multicast groups coexist with \(K\) CUs, as depicted in Figure 1. Let \(\mathcal{K}=\{1,2,...,K\}\) and \(\mathcal{M}=\{1,2,...,M\}\) denote the index sets of CUs and D2D groups respectively. The available spectrum is divided into \(K\) orthogonal sub-channels, and each of them has been pre-assigned to a CU. Besides, \(M\) D2D groups are trying to access into the cellular system in non-orthogonal spectrum sharing model. We adopt uplink resource sharing of cellular network, which is more effective than downlink resource sharing in a fully loaded cellular network, as demonstrated in [23]. We assume in this paper a single multicast group can share spectrum resources with multiple CUs and each CU's spectrum resource can only be reused by at most one D2D group in order to avoid cross-interference between different D2D groups. Additionally, we assume the eNB can obtain the perfect channel state information (CSI) of all the links, based on which, the available spectrum resources are allocated effectively to the CUs and the D2D groups.  
Define a set of binary variables \(\omega_{m,k}\) with \(\omega_{m,k}=1\) if the \(m\)-th D2D group reuses channel \(k\) and \(\omega_{m,k}=0\) otherwise. For CU \(k\in\mathcal{K}\), the maximum transmission power is limited to \(p_{k}^{\mathcal{C}}\). The signal to interference plus noise ratio (SINR) of eNB on the sub-channel \(k\) can be expressed as  
Figure 1: System model of Device-to-device (D2D) multicast groups and cellular uses (CUs).  
\[\gamma_{k}^{C}=\frac{p_{k}^{C}H_{k,B}^{CB}}{\sum_{m\in\mathcal{M}}\omega_{m,k}p_{m, k}^{D}\bar{H}_{m,B}^{DB}+\sigma_{0}^{2}}, \tag{1}\]  
where \(H_{k,B}^{CB}\) and \(\bar{H}_{m,B}^{DB}\) represent channel gain of the cellular communication link from CU \(k\) to the eNB and the interference link from D2D transmitter \(m\) to the eNB, respectively. \(p_{k}^{C}\) and \(p_{m,k}^{D}\) represent the transmission powers of the CU \(k\) and the \(m\)-th D2D group transmitter on sub-channel \(k\), respectively. \(\sigma_{0}^{2}\) represents the noise power on the sub-channel \(k\). For simplicity, we assume that the noise power is the same for the CUs and the D2D links on each sub-channel.  
Let \(\mathcal{D}_{m}\) represent the set of D2D receivers in the \(m\)-th multicast group. Then, \(|\mathcal{D}_{m}|\) is the total number of receivers in the group, where \(|\cdot|\) is to derive the cardinality of the corresponding set. The SINR of D2D receiver \(n\in\mathcal{D}_{m}\) in the \(m\)-th multicast group on sub-channel \(k\) can be expressed as  
\[\gamma_{n,m,k}^{D}=\frac{p_{m,k}^{D}H_{m,n}^{DD}}{p_{k}^{C}\bar{H}_{k,m,n}^{CD }+\sigma_{0}^{2}}, \tag{2}\]  
where \(H_{m,n}^{DD}\) and \(\bar{H}_{k,m,n}^{CD}\) represent channel gain of the cellular communication link from the \(m\)-th D2D group transmitter to its desired D2D receiver \(n\) and the interference link from CU \(k\) to D2D receiver \(n\) in group \(m\).  
For the D2D link between multicast transmitter \(m\) and receiver \(n\in\mathcal{D}_{m}\), we define its channel quality on the sub-channel \(k\) by  
\[q_{n,m,k}^{D}=\frac{H_{m,n}^{DD}}{p_{k}^{C}\bar{H}_{k,m,n}^{CD}+\sigma_{0}^{2}}. \tag{3}\]  
In order to minimize the D2D link's power consumption, there is no need to transmit data in minimum rate to ensure all D2D users receive data correctly. Let \(q_{n,m,k}^{D}\) represent the lowest channel quality of D2D links in \(m\)-th group on sub-channel \(k\), the set of admitted D2D receivers in \(m\)-th multicast group on sub-channel \(k\) whose channel quality is higher than user \(n\) can be expressed as  
\[A_{n,m,k}=\{n^{\prime}\in D_{m}|q_{n^{\prime},m,k}^{D}\geq q_{n,m,k}^{D}\}. \tag{4}\]  
Furthermore, \(X_{n,m,k}=\left|A_{n,m,k}\right|\) shows the number of D2D links that are admitted to transmit data correctly in \(m\)-th D2D group on sub-channel \(k\).