## 1 Introduction  
Full-duplex (FD) communication and spatial modulation (SM) are two promising techniques to increase the spectral efficiency of wireless systems [1; 2; 3]. Theoretically, an FD communication system can double the spectral efficiency as its transceivers can transmit and receive signals at the same time and on the same frequency [4; 5]. Besides increasing the channel capacity, the FD communication systems can also reduce the end-to-end and feedback delay, improve network security and solve the hidden terminal problem. Therefore, the FD mode has found its applications in various wireless systems such as sensor networks, massive MIMO, relaying networks and possibly future wireless networks such as the fifth-generation (5G) and beyond. Unfortunately, the residual self-interference (RSI) due to imperfect self-interference cancellation (SIC) limits the capacity and performance of the FD communication systems [6]. In the literature, numerous solutions such as relay selection scheme and adaptive transmission [7; 8] and optimal power allocation [9; 10; 11] were proposed to reduce the impact of the RSI and improve the performance of the FD relay communication system. These solutions significantly improved the capacity, outage performance and energy efficiency of the single-input single-output (SISO) systems. Meanwhile, the multiple-input multiple-output (MIMO) transmission is another solution that can also achieve high capacity and better performance over the SISO systems [12]. However, the hardware of the MIMO system is more complex, that is, more radio frequency (RF) chains, as both transmitter and receiver user multiple antennas. The MIMO receiver also requires a high-complexity detector to deal with the inter-channel interference (ICI) due to simultaneous transmissions on the same frequency from different antennas. In that context, SM can tackle these issues correctly as it activates only one transmit antenna for transmission and requires only low-complexity iterative maximal-ratio combining (MRC) at the receiver for signal estimation [12; 13; 14; 15]. Therefore, the combined SM and FD system achieves higher spectral efficiency while reducing the complexity requirement at the transceiver, particularly of the relay node. In the literature, using FD relay in SM systems has attracted great interest because this integrated FD-SM system has the advantages of both spectral efficiency improvement and performance enhancement. Numerous works focused on analyzing the system performance in terms of the outage probability (OP) [2], bit error rate (BER) [3], average symbol error probability (SEP), ergodic capacity [16] and average BER [17; 18]. Specifically, in Reference [2], the lower and upper bounds of the OP of the SM-MIMO system with decode-and-forward (DF) FD relay were derived over cascaded \(\alpha-\mu\) fading channels. It also demonstrated that the RSI had a strong impact on the OP performance of the system. The work in Reference [3] considered the SM-MIMO system with amplify-and-forward (AF) FD/half-duplex (HD) relaying. It successfully derived a new unified tight upper-bound for the system BER. The results of the paper indicate that the SM-MIMO-FD relay system can improve the BER and the spectral efficiency if suitable SIC techniques are applied. Under the same assumption of the RSI, References [16; 17; 18] investigated the SM-MIMO-FD relay systems which can exploit the benefits of the FD transmission mode. The approximate expressions of SEP ([16]) and BER ([17; 18]) were also derived for performance evaluation. Although the previous works conducted various performance analyses, their results were limited to either upper and lower bounds or approximate expressions but not the exact closed-form expressions of SEP and BER. Therefore, it is required to have exact mathematical expressions for the performance evaluation rather than the upper bound or approximate ones for better understanding the system behaviors.