###### Abstract  
This paper considers the transceiver design in a multiuser multiple-input multiple-output (MIMO) full-duplex (FD) amplify-and-forward (AF) relay downlink communication system, where users simultaneously transmit data via an FD relay node. The design incorporates an imperfect loop interference (LI) cancellation which results in a residual LI. Linear precoders are employed at the sources and relay, and minimum mean-squared-error (MMSE) combiners are employed at the destinations to mitigate the effect of the residual LI. The corresponding design problem is highly nonconvex, so a closed-form solution is intractable. Thus, an iterative method is developed to solve this optimization problem. Simulation results are presented which show that the proposed iterative algorithm provides better performance than the corresponding half-duplex (HD) solution in terms of the achievable rate under residual LI. amplify and forward (AF); capacity, downlink, full-duplex (FD); loop interference (LI); MIMO; transceiver +
Footnote †: journal:  
cancellation methods such as antenna isolation and analog/digital precancellation have been shown to be effective. However, it is impossible to cancel the LI completely, and the residual LI can still be larger than the noise [12]. Estimating channel state information (CSI) is challenging due to the dynamic nature of wireless environments and the complexity of modern communication systems. However, current estimation algorithms have been shown to provide good accuracy, particularly in cellular systems [13]. Thus, it can be assumed that CSI is available in the proposed system. In [14], deep neural networks were considered for channel estimation with multiuser precoding in the downlink of a frequency-division duplex massive MIMO system. In [15], an efficient transceiver design was examined for full-duplex (FD) multiuser massive MIMO systems operating in millimeter wave frequencies. An iterative solution was developed for the proposed digital MMSE transceiver beamforming. Secrecy in cell-free massive MIMO systems was proposed in [16] through the joint optimization of uplink power control, downlink beamforming, and duplex mode selection. Artificial noise was incorporated, and an iterative method was used to solve the associated nonconcave-convex approximation. In [17], beamforming was investigated for in-band FD multi-cell multiuser MIMO networks. Hardware impairments, channel uncertainty, and limited channel state information (CSI) were considered in the MMSE beamforming design to improve system performance under global and local CSI. However, none of the approaches in the literature considered the practical limitation of residual loop interference due to imperfect CSI cancellation. This interference should be considered in developing robust and effective wireless communication systems. The impact of residual self-interference (SI) and loop interference (LI) in a full-duplex space shift keying communication system was investigated in [18]. Thus, the residual LI due to imperfect interference cancellation is considered in this paper. Unlike the uplink communications in [19], this paper considers a MIMO FD relaying downlink communication system with multiple source-destination pairs. A transceiver design for the sources and relay, and a linear combiner design at the destinations, are investigated considering the residual LI with the goal of minimizing the MSE of the received signals at the destinations. An iterative algorithm is developed to update the source and relay transceiver matrices and the linear combining matrix at the destinations. The remainder of this paper is organized as follows. Section 2 presents the system model, and the optimization problem is solved in Section 3. Results are presented in Section 4, which demonstrate the effectiveness of this solution, and some conclusions are given in Section 5. Notation: Bold uppercase, bold lowercase, and normal letters denote matrices, vectors, and scalars, respectively.