# A Deep Learning Approach for MIMO-NOMA Downlink Signal Detection
## 1 Introduction
Mobile traffic classifiers based DL have also been reported to handle encrypted traffic and reflect their complex traffic patterns [10; 11]. Achievements in blind detection for MIMO systems with deep learning are also continuously reported [12; 13]. Additionally, a scheme integrating DL into an orthogonal frequency division multiplexing (OFDM) system has been put forward [14], and its numerical results revealed the potential performance of DNNs. A fully connected end-to-end DL system including an encoding layer, noise layer, and decoding layer was reported in [15] for MIMO, and [16] designed a long short term memory (LSTM) network for uplink NOMA to realize the end-to-end transmission as well. Their results showed the excellent performance of the autoencoder for jointly learning transmit and receive functions. However, in most realistic scenarios, it is difficult to synthetically train and optimize both send and receive functions of the DL system. Traditionally, the SIC method is affected by the error propagation (EP) and receiver complexity related to the number of UEs. We propose a learning method based on a DNN to efficiently estimate the channel state and decode the expected signal. Its benefits not only lie in the performance gain in the communication system, but in the reduction of reference signal overhead to increase the throughput in the downlink system. The main contributions of this paper are summarized as follows:  
1. To the best of our knowledge, we designed the first downlink MIMO-NOMA detecting system based on DL methods. The proposed system can process the traditional MIMO-NOMA signal directly instead of implementing an SIC receiver. 2. We take full advantage of the ability of DNNs to process high-dimensional data. The detection performance can be improved by the DL-based method. 3. The MIMO-NOMA-DL system can estimate the characteristics of the MIMO Rayleigh fading channel and then decode the signal. The process of channel estimation and signal detection can be considered synthetically instead of separately. 4. We evaluated the performance of the proposed system. Results comparable to traditional MIMO-NOMA based on SIC were obtained. 5. We provide relevant simulation results on the impact of some key parameters. These parameters include the modulation type, power allocation, and mini-batch. The proposed system outperformed the traditional SIC method in all testing situations. The remainder of this paper is structured as follows: In Section 2, the necessary background on MIMO-NOMA system and DL is briefly reviewed. Then, the proposed MIMO-NOMA-DL system is depicted in detail, including the feasibility analysis, overall structure, and DNN design in Section 3, followed by numerical studies in Section 4. Finally, Section 5 concludes the paper and discusses future works. _Notations_: Vectors are denoted by boldface small letters, while matrices are denoted by boldface capital letters. Superscripts \((\cdot)^{*}\), \((\cdot)^{T}\), \((\cdot)^{H}\), \((\cdot)^{-1}\) and \(||\cdot||\) represent the conjugate, transpose, Hermitian transpose, inverse, and Frobenius norm operators, respectively. Also, \(\mathbb{R}^{N\times M\times K}\) denotes the vector space of all \(N\times M\times K\) real matrices and \(\mathbf{H}_{(n)}\) is the node-n matricization.