## 2 System Model  
Consider the cooperative NOMA system where an AP transmits messages to two NOMA user terminals (\(U_{1}\) and \(U_{2}\)) through two half-duplex relays (HDRs) (\(R_{1}\) and \(R_{2}\)) with multiple antennas, where the two relays successively relay the AP signals to emulate full-duplex relaying. In power domain NOMA transmission, the AP transmits a superposed signal toward multiple UTs, where the individual signals for the UTs are added up and sent in the same spectral resource. The UTs apply the SIC decoding principle to decode the particular message destined to themselves. The two HD relays operate with the DF protocol. The AP and the user terminals are equipped with single antennas. In even time slots, \(R_{1}\) transmits the signal from the AP received in the previous odd time slots, while \(R_{2}\) helps the AP transmission in odd time slots heard from the previous even time slots. The two relays equipped with \(M\) antennas, respectively, use \(M\times 1\) receive beamformer vectors \(\mathbf{w}_{R,i}\), \(i=1,2\), with \(\|\mathbf{w}_{R,i}\|^{2}=1,i=1,2\) and use \(M\times 1\) transmit beamformer vectors \(\mathbf{w}_{T,i}\), \(i=1,2\), with \(\|\mathbf{w}_{T,i}\|^{2}=P_{r},i=1,2\) when \(P_{r}\) is the relay transmit power. The decode and forward (DF) protocol is used for the two relays. The multiple-input single-output (MISO) channel from the AP to the relay \(R_{i}\) is denoted by the \(M\times 1\) vector \(\mathbf{h}_{i}\), the multiple-input multiple-output (MIMO) channel from \(R_{1}\) to \(R_{2}\) is denoted as an \(M\times M\) matrix \(\mathbf{G}_{1}\), and the opposite direction channel is denoted by the \(M\times M\) matrix \(\mathbf{G}_{2}\). Furthermore, the MISO channel from \(R_{i}\) to \(U_{j}\) is denoted by a \(M\times 1\) vector \(\mathbf{g}_{i,j}\). Note that all the elements of these channel matrices and vectors are independent and identically distributed random variables with \(\mathcal{CN}(0,1)\). Therefore, we assume here flat fading narrow band wireless channels with Rayleigh distributed channel parameters. Note that more aggressive channel assumptions such as wide band multi-path model or interference from an outside entity may complicate the problem to be solved and we leave such cases for future research topics. We assume that full channel state information (CSI) is available at the AP and relays so that all the beamformer design is conducted at the AP or at the relays. For the binary variable \(i=1,2\), the notation \(\tilde{i}=2\) when \(i=1\) and \(\tilde{i}=1\) when \(i=2\). The received signals at the relays and at the users at time slot \(n\) are expressed, respectively, as  
\[\mathbf{y}_{R,i}(n) =\mathbf{h}_{i}x_{S}(n)+\mathbf{G}_{i}\mathbf{w}_{T,\tilde{i}}x_{ R}(n)+\mathbf{n}_{R,i}(n),\;i=1,2, \tag{1}\] \[\mathbf{y}_{k}(n) =\mathbf{g}_{i,k}^{T}\mathbf{w}_{T,\tilde{i}}x_{R}(n)+\mathbf{n} _{k}(n),k=1,2, \tag{2}\]  
where \(x_{S}(n)=\sum_{k=1}^{2}x_{k}(n)\) is the composite AP message symbol with \(\mathbb{E}\big{[}|x_{S}(n)|^{2}\big{]}=P_{s}\) when \(P_{s}\) is the source transmit power. The message \(x_{k}(n)\) is destined to the \(k\)-th user with \(\mathbb{E}\big{[}|x_{k}(n)|^{2}\big{]}=\rho_{k}P_{s}\) and \(\sum_{k=1}^{2}\rho_{k}=1\). The cooperative HDRs apply the received beamforming to the signal in (2) as \(\mathbf{w}_{R,i}^{H}\mathbf{y}_{R,i}(n)\), decode two messages with SIC manner from \(x_{1}(n)\) to \(x_{2}(n)\), re-encode these message signals \(x_{R}(n+1)=\sum_{k=1}^{2}x_{k}(n)\sqrt{1/P_{s}}\) and transmits \(x_{R}(n+1)\) with beamforming as \(\mathbf{w}_{T,\tilde{i}}x_{R}(n+1)\) at the time slot \(n+1\). The vector \(\mathbf{n}_{R,i}(n)\)denotes an \(M\times 1\) additive noise vector at the receiver antennas of \(R_{i}\) with \(\mathcal{CN}(\mathbf{0},\mathbf{I}_{M})\) distribution, while \(n_{k}(n)\) denotes the additive noise at \(U_{k}\) antenna with \(\mathcal{CN}(0,1)\) distribution.