## 2 System Model  
We consider a single-user single-carrier SM-MIMO-FD relay system with a source node S, which transmits its signal to a destination node D via a relay node R as shown in Figure 1. Particularly, S and D operate in the half-duplex (HD) mode with \(N_{\text{S}}\) transmitting antennas and \(N_{\text{D}}\) receiving antennas, respectively. The relay node operates in the FD mode with \(N_{\text{RR}}\) transmitting antennas and \(N_{\text{rR}}\) receiving antennas. At time slot \(t\), the received signal at R can be calculated as  
\[\mathbf{y}_{\text{R}}(t)=\sqrt{P_{\text{S}}}\mathbf{h}_{i}^{\text{R}}x_{\text{S }}(t)+\sqrt{P_{\text{R}}}\mathbf{h}_{j}^{\text{R}}x_{\text{R}}(t)+\mathbf{z}_{ \text{R}}(t), \tag{1}\]  
where \(\mathbf{h}_{i}^{\text{R}}\) and \(\mathbf{h}_{j}^{\text{R}}\) are respectively the channel vector from the \(i\)th active antenna of S to \(N_{\text{rR}}\) receiving antennas of R and from the \(j\)th active antenna of R to all reception antennas of R. These channels are assumed to undergo flat Rayleigh fading, which can be modeled by independent and identically distributed complex Gaussian random variables with zero mean and unit variance. \(x_{\text{S}}\) and \(x_{\text{R}}\) are the transmitted signals at the \(i\)th antenna of S and the \(j\)th antenna of R, respectively; \(P_{\text{S}}\) and \(P_{\text{R}}\) are the average transmission power at S and R, respectively; \(\mathbf{z}_{\text{R}}\) is the noise vector whose elements follow a complex Gaussian distribution with variance \(\sigma^{2}\). At the FD relay, we assume that the transmitting and receiving antennas are both directional, thus there will be no direct link which causes the self-interference (SI) from the transmit to the receive antenna. This SI is mainly due to reflections caused by multipath propagation. We also assume that the system can use all SIC techniques in the three domains, that is, propagation, analog and digital, to remove the SI [19; 20]. Specifically, R can use all available isolation techniques to suppress SI. It can use the cross-polarization transmission to isolate the transmitting and receiving antennas [19; 21]. In the analog domain, thanks to the SI awareness of analog circuits, the transmitted signal via the transmit antenna is collected and then subtracted from the received signal. The RSI is then converted to the digital domain for further SIC via digital signal processing. As R knows its transmitted signal, it can subtract the SI from the received signal by using SI channel estimation [20; 22]. Thanks to all these SIC techniques, the relay node can achieve up to 110 dB SI suppression [23]. Moreover, since the SI is canceled from the received signal in the analog and digital domain by reconstructing the SI signal, the RSI is in effect the resulted errors due to the imperfect reconstruction or more correctly, the imperfect SI channel estimation. Moreover, as the digital-domain cancellation is done after a quantization operation, RSI at the relay \(\mathbf{r}_{\text{SI}}\) can be modeled using complex Gaussian random variable [4; 17; 20; 23]  
Figure 1: System model of the considered SM-MIMO-FD relay system. with zero mean and variance of \(\sigma^{2}_{\text{RSI}}\), that is \(\sigma^{2}_{\text{RSI}}=\tilde{\Omega}P_{\text{R}}\) where \(\tilde{\Omega}\) denotes the SIC capability at the relay. Therefore, the received signal at R can be rewritten from (1) as  
\[\mathbf{y}_{\text{R}}(t)=\sqrt{P_{\text{S}}}\mathbf{h}_{i}^{\text{R}}x_{\text{S }}(t)+\mathbf{r}_{\text{SI}}(t)+\mathbf{z}_{\text{R}}(t), \tag{2}\]  
and the received signal at the destination D is then given by  
\[\mathbf{y}_{\text{D}}(t)=\sqrt{P_{\text{R}}}\mathbf{h}_{j}^{\text{D}}x_{\text {R}}(t)+\mathbf{z}_{\text{D}}(t), \tag{3}\]  
where \(\mathbf{h}_{j}^{\text{D}}\) is the channel vector from the \(j\)th antenna of R to \(N_{\text{D}}\) receiving antennas of D; \(\mathbf{z}_{\text{D}}\) is the noise vector at D. Both S and R are assumed to have the same \(N_{t}\) transmitting antennas for the same expected spectral efficiency.