# A Deep MIMO Detector Based on MQAM Signal Decomposition and Channel Ordering

Zichun Huang, Yuehua Ding, Biyun Ma, Jie Li, Yide Wang

► To cite this version:

Zichun Huang, Yuehua Ding, Biyun Ma, Jie Li, Yide Wang. A Deep MIMO Detector Based on MQAM Signal Decomposition and Channel Ordering. IEEE Access, 2024, 12, pp.53958 - 53964. 10.1109/access.2024.3387532. [hal-04511553](https://hal.science/hal-04511553v1)

HAL Id: hal-04511553

https://hal.science/hal-04511553v1

Submitted on 4 Jun 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Received 7 March 2024, accepted 19 March 2024, date of publication 11 April 2024, date of current version 22 April 2024.

Digital Object Identifier 10.1109/ACCESS.2024.3387532

RESEARCH ARTICLE

# A Deep MIMO Detector Based on MQAM Signal Decomposition and Channel Ordering

ZICHUN HUANG<fnref n="1" />, YUEHUA DING<fnref n="1" />, BIYUN MA<fnref n="1" />, JIE LI<fnref n="1" />, AND YIDE WANG<fnref n="1" />,<fnref n="2" />, (Senior Member, IEEE)

Corresponding author: Biyun Ma (eebyma@scut.edu.cn)

This work was supported by Guangdong Science and Technology Program under Grant 2022A0505050011, Grant 2022A1515011707, Grant 2023A1515011420, and Grant 2022A1515011830.

**ABSTRACT** The recently proposed learning-to-learn iterative search algorithm (LISA) for multiple-input multiple-output (MIMO) detection can achieve excellent BER performance for low-order modulation signals, such as BPSK/QPSK, but its performance is degraded as the modulation order increases. This limits its application in high data rate transmission. Motivated by this observation, this paper proposes a decomposition-ordering-based LISA (DO-LISA) to improve the accuracy of LISA for higher-order QAM signals. Firstly, the detection of high-order QAM signal is transformed into the detection of QPSK signal to exploit the facility of binary classification. To further exploit the sequential features of a bidirectional long short-term memory (BiLSTM) network, channel ordering is adopted to enhance the detection accuracy. Moreover, to make the proposed method more practical, low-rank decomposition and channel pruning are adopted to accelerate and compress the neural network. Numerical experiments show that the proposed method can achieve quasi-ML BER performance. Compared with the full model, the compact model achieves 2.37× compression ratio and 2.38× acceleration ratio with limited accuracy loss. In addition, the proposed DO-LISA shows its good generalization ability by preserving very good BER performance both in the correlated channel condition and imperfect channel condition.

**INDEX TERMS** MIMO detection, deep learning, LSTM, V-BLAST, compression.

## I. INTRODUCTION

Multiple-input multiple-output (MIMO) [1] can provide high spectrum efficiency in wireless data transmission. It plays an important role in 4G, 5G, and future communication systems. MIMO detection is a key process for separating and recovering the overlapped signals from different transmitting antennas, which is extremely important for system performance. Unfortunately, MIMO detection is an NP-hard problem.

The optimal solution to MIMO detection is the maximum likelihood (ML) detection, whose complexity increases exponentially with the number of transmitting antennas. The complexity of ML detection can be reduced by the sphere decoder (SD) [2], [3] without performance loss. However,

The associate editor coordinating the review of this manuscript and approving it for publication was Yuli Yang.

SD still has exponential complexity, which varies depending on random channels and SNR. Even if the complexity of SD is fixed by fixed complexity sphere decoding (FCSD) [4], [5], the complexity of FCSD is exponential. The linear detectors such as zero-forcing (ZF) [1], minimum mean square error (MMSE) [7], [8] can realize the detection with polynomial complexity at the price of significant performance loss. Even if they are improved by the ordered successive interference cancellation (OSIC), such as vertical Bell laboratories layered space-time (V-BLAST) structure, there is still a huge gap between ZF/MMSE-OSIC and ML, which is the reflection of the diversity gap between them, because the diversity order of ZF/MMSE-OSIC is $N_r - N_t + 1$, while ML/SD/FCSD can achieve full diversity $N_r$ [5]. To fill the gap, K-best SD is proposed to reach a trade-off between performance and complexity [5], [6].

<footnote n="1">¹School of Electronics and Information Engineering, South China University of Technology, Guangzhou 510640, China</footnote>

<footnote n="2">²Institut d'Electronique et des Technologies du numeRique (IETR), CNRS UMR6164, Nantes University, 44306 Nantes, France</footnote>
With the rapid development of artificial intelligence, deep learning has successful applications in natural language processing (NLP) and computer vision (CV), which provides a new perspective for signal detection [9]. Under such background, some initial works are proposed for MIMO detection, e.g. DetNet [10], MMNet [11]. Very recently, a MIMO detector based on learning to learn iterative search algorithm (LISA) [15] is developed by using a long short-term memory (LSTM) network to fit the non-linear relationship between the received signal and the transmitted signal. The detector based on LISA performs quite well for the BPSK/QPSK signal, but its performance deteriorates as the modulation order increases, which is also the case of DetNet [10], MMNet [11], WeSNet [13], QLcgNetV and LcgNet [12], [14], this implies that for a given bandwidth, transmission rate should not be too high to adapt the capability of LISA. The neural network is excellent at binary classification (polarity judgment). For the multi-level recognition, its performance degrades.

Inspired by this observation, this paper proposes a decomposition and ordering-based LISA (DO-LISA) MIMO detection method. It has the following contributions:

Firstly, the proposed method converts the detection of high-order QAM signal into the detection of QPSK signal to exploit the advantage of the bidirectional LSTM (BiLSTM) network in binary classification.

Secondly, the channel reordering process is adopted to further exploit the sequential features of the BiLSTM network, which can reduce the error propagation and accelerate the convergence.

Thirdly, to make the proposed method more practical, low-rank decomposition and channel pruning methods are adopted to accelerate and compress the BiLSTM network. Simulation results show that the proposed method can achieve quasi-ML BER performance for high-order QAM signals. Even with an acceleration-compression process, the proposed method still has better BER performance than LISA.

The paper is organized as follows. Section II introduces the MINO system model. Section III describes the model adaptation for the BiLSTM network. Section IV presents the model acceleration and compression. Section V shows and discusses some simulation results in different scenarios, and Section VI concludes the paper.

**Notations:** Lower case letters denote scalars (x). Lowercase letters of boldface denote column vectors (x). Capital letters of boldface are used for matrices (H). (·) denotes a complex-valued symbol, e.g. $\bar{H}$ represents a complex matrix. (·)^T, (·)^H and (·)† denote the transpose, conjugate transpose, and pseudo-inverse, respectively. Re(·) and Im(·) denote the real and imaginary parts of a complex symbol, respectively. ⊗ represents the Kronecker product.

## II. MIMO SYSTEM MODEL

Consider a MIMO system with $N_t$ transmitting antennas and $N_r$ receiving antennas. Let $\bar{x} = (\bar{x}_1, \bar{x}_2, \dots, \bar{x}_{N_t})^T \in \bar{\Omega}^{N_t}$ denote the transmitted vector, with $N_t$ elements

independently drawn from a QAM constellation $\bar{\Omega}$. Let $\tilde{H} \in \mathbb{C}^{N_r \times N_t}$ represent the complex channel matrix. The received signal $\bar{y} \in \mathbb{C}^{N_r}$ is given by

$$ \bar{y} = \tilde{H}\bar{x} + \bar{n} \quad (1) $$

where $\bar{n}$ denotes the complex additive white Gaussian noise (AWGN). In this paper, the following assumptions are used:

1. $E[\bar{x}\bar{x}^H] = I_{N_t}$, the average power of each transmitting antenna is normalized to a unit, and the signals transmitted by different antennas are independent.

2. $E[\bar{n}\bar{n}^H] = \sigma^2 I_{N_t}$, where $\sigma^2$ is the power of noise at each receiving antenna.

3. $E[\bar{n}\bar{x}^H] = 0$, which means that the signal and noise are independent.

4. Constellation points in a QAM plane are equally probable.

## III. MODEL ADAPTATION FOR BILSTM NETWORK

Theoretically, model (1) can be transformed into a real-valued model, a triangular-matrix-based sequential data structure is then obtained by QL (or QR) decomposition, the BiLSTM network can take the data in a sequential way to train the model parameters and detect the signal $\bar{x}$. This is the work proposed in [15]. To further exploit the advantage of the BiLSTM network in binary classification and sequential processing, the following model adaptation is carried out.

### A. QAM SIGNAL DECOMPOSITION

It is shown in [3] that a M-ary QAM signal can be decomposed into a weighted sum of $\alpha$ QPSK signals with $\alpha = \frac{1}{2}\log_2 M$, provided that $\log_2 M$ is an even number, which is a strictly equivalent transformation. Therefore for an M-ary QAM vector $\bar{x}$, it can be represented as follows [3]:

$$ \bar{x} = \Psi\bar{s} \quad (2) $$

with $\bar{s} = [\bar{s}_1^T, \bar{s}_2^T, \dots, \bar{s}_\alpha^T]^T$, and

$$ \Psi = \left[ \frac{\sqrt{2}}{2}, \frac{\sqrt{2}}{2}, \dots, \frac{\sqrt{2}}{2} \right]^{\alpha-1} \otimes I_{N_t} $$

where $\bar{s}_k \in Q^{N_t}$ with $Q$ being the QPSK constellation. Combining (2) and (1), an equivalent system transmitting QPSK signals is obtained as follows:

$$ \bar{y} = \tilde{H}'\bar{s} + \bar{n} \quad (3) $$

where $\tilde{H}' = \tilde{H}\Psi$. To further exploit the noise information, model (3) can be extended as follows:

$$ \bar{y}' = \tilde{H}''\bar{s} + \bar{n}' \quad (4) $$

$$ \text{where } \bar{y}' = \begin{bmatrix} \bar{y} \\ 0 \end{bmatrix}, \tilde{H}'' = \begin{pmatrix} \tilde{H}' \\ \sigma I_{\alpha N_t} \end{pmatrix}, \bar{n}' = \begin{bmatrix} \bar{n} \\ -\sigma \bar{s} \end{bmatrix}. $$

The complex signal model in (4) is further transformed into the following equivalent real-valued model:

$$ y = Hs + n \quad (5) $$
<table><caption>TABLE 1. Search for the optimal order.</caption><thead><tr><th colspan="2">Algorithm 1: Search for the optimal order</th></tr></thead><tbody><tr><td>1.</td><td><b>o</b> = []</td></tr><tr><td>2.</td><td>for j = 1 to αN<sub>t</sub> <b>do</b></td></tr><tr><td>3.</td><td><b>G</b> = <b>H</b><sup>†</sup></td></tr><tr><td>4.</td><td>k<sub>j</sub> = arg min<sub>j∈{o}</sub> ||(<b>G</b>)<sub>j</sub>|| % (<b>G</b>)<sub>j</sub> is the jth row</td></tr><tr><td>5.</td><td><b>o</b> = [<b>o</b>, k<sub>j</sub>]</td></tr><tr><td>6.</td><td><b>h</b><sub>k<sub>j</sub></sub> = <b>0</b></td></tr><tr><td>7.</td><td>%<b>h</b><sub>k<sub>j</sub></sub> is the k<sub>j</sub>th column of <b>H</b></td></tr><tr><td>8.</td><td>return <b>o</b></td></tr></tbody></table>

where

$$ 
\begin{aligned}
y &= \begin{bmatrix} \mathrm{Re}(\tilde{y}') \\ \mathrm{Im}(\tilde{y}') \end{bmatrix}, H = \begin{bmatrix} \mathrm{Re}(\tilde{H}'') & -\mathrm{Im}(\tilde{H}'') \\ \mathrm{Im}(\tilde{H}'') & \mathrm{Re}(\tilde{H}'') \end{bmatrix}, \\
s &= \begin{bmatrix} \mathrm{Re}(\tilde{s}) \\ \mathrm{Im}(\tilde{s}) \end{bmatrix}, n = \begin{bmatrix} \mathrm{Re}(\tilde{n}') \\ \mathrm{Im}(\tilde{n}') \end{bmatrix},
\end{aligned}
$$

One notes that $s$ in (5) is drawn from a $2\alpha N_t$-dimensional BPSK constellation $\mathcal{B}^{2\alpha N_t}$, i.e. $s \in \mathcal{B}^{2\alpha N_t}$. Model (5) can be considered as a system transmitting BPSK signals, facilitating greatly the work of the BiLSTM network.

## B. CHANNEL ORDERING

To reduce the error propagation in the sequential processing, the order of the signals feeding into the BiLSTM network is optimized in this subsection. Generally speaking, firstly the data with the highest reliability should be taken by the network, then the data of the second best quality...etc. To this end, the order of data feeding is optimized using ranking the quality of the sub-channels of the elements of $s$. The effect of noise amplification of the rows of $H^\dagger$ is considered as the criterion. This is exactly the ordering process of the V-BLAST OSIC detection, which is summarized in Table 1 [1].

**Algorithm 1** can return the optimal order $o$, the columns of **H** and the elements in **s** are ordered accordingly. An ordered MIMO model is obtained as follows:

$$ y = H's' + n \quad (6) $$

where $H'$ is the sorted $H$, and $s'$ is the sorted $s$.

## C. BILSTM FITTING

The ML detection searches all possible candidates according to the following criterion:

$$ \min_{\tilde{s} \in \mathcal{B}^{2\alpha N_t}} \|y - H' \hat{s} \|^2. \quad (7) $$

Applying the QL-decomposition on channel matrix $H'$, i.e., $H' = QL$, with $Q$ a unitary matrix, and $L$ a lower triangular matrix. Problem (7) can be converted to the following equivalent form:

$$ \min_{\tilde{s} \in \mathcal{B}^{2\alpha N_t}} \|\tilde{y} - L\hat{s}\|^2, \quad (8) $$

<figure><img src="image_2.png" /><figcaption>FIGURE 1. Architecture of DO-LISA detectors.</figcaption></figure>

where $\tilde{y} = Q^T y$. Assuming $\hat{s} = [\hat{s}_1, \dots, \hat{s}_{2\alpha N_t}]^T$, (8) can be expanded as follows:

$$ \min_{\tilde{s} \in \mathcal{B}^{2\alpha N_t}} \{f_1(\hat{s}_1) + f_2(\hat{s}_1, \hat{s}_2) + \dots + f_{\alpha N_t}(\hat{s}_1, \dots, \hat{s}_{2\alpha N_t})\}, \quad (9) $$

where

$$ f_k(\hat{s}_1, \dots, \hat{s}_k) = \left( \tilde{y}_k - \sum_{l=1}^{k} l_k \hat{s}_l \right)^2, \quad (10) $$

and $l_{k,l}$ is the element of $L$ at $(k, l)$. The proposed detection strategy is to recursively detect the symbols from $s_1$ to $s_{2\alpha N_t}$, that is

$$ 
\left\{
\begin{aligned}
\tilde{s}_1 &= \arg \min_{\hat{s}_1 \in \mathcal{B}} f_1(\tilde{y}_1, l_{1,1}) \\
\tilde{s}_2 &= \arg \min_{\hat{s}_2 \in \mathcal{B}} f_2(\tilde{y}_2, \hat{s}_1, l_{2,1}, l_{2,2}) \\
&\vdots \\
\tilde{s}_{2\alpha N_t} &= \arg \min_{\hat{s}_{2\alpha N_t} \in \mathcal{B}} f_{2\alpha N_t}(\tilde{y}_{2\alpha N_t}, \hat{s}_1, \hat{s}_2, \dots, \hat{s}_{2\alpha N_t-1}, \\
&\qquad l_{2\alpha N_t-1}, l_{2\alpha N_t-2}, \dots, l_{2\alpha N_t})
\end{aligned}
\right.
\quad (11) $$

Due to the noise, the relation between the detected signal $\hat{s}$ and the received signal $\tilde{y}$ is non-linear. BiLSTM neural network is employed to fit the non-linear relationship, for which $\tilde{y}$ and $L$ are the input of BiLSTM network and the probability vectors of detection signal $\hat{p}_k$, $k = 1, \dots, 2\alpha N_t$ are the output of BiLSTM network. $\hat{p}_k$ represents the probabilities of the corresponding constellation points. DO-LISA chooses the constellation point with the highest probability as the prediction value. The detection architecture is shown in Fig. 1.

# IV. COMPLEXITY ANALYSIS, MODEL ACCELERATION AND COMPRESSION

## A. COMPLEXITY ANALYSIS

The structure of an LSTM unit is shown in Fig. 2. With a little abuse of representation, the computation can be formulated as:

$$ \begin{bmatrix} i_t \\ f_t \\ g_t \\ o_t \end{bmatrix} = \begin{bmatrix} \sigma(W_{iz}z_t + W_{ih}h_{t-1} + b_i) \\ \sigma(W_{fz}z_t + W_{fh}h_{t-1} + b_f) \\ \tanh(W_{gz}z_t + W_{gh}h_{t-1} + b_g) \\ \sigma(W_{oz}z_t + W_{oh}h_{t-1} + b_o) \end{bmatrix}, \quad (12) $$

where $\sigma(\cdot)$ and $\tanh(\cdot)$ are the sigmoid activation function and tanh function, respectively. $z_t$ and $h_{t-1}$ represent the input of
<figure><img src="image_3.png" /><figcaption>FIGURE 2. Architecture of an LSTM unit.</figcaption></figure>

the LSTM block, $b_i$, $b_f$, $b_g$, and $b_o$ are the biases, one can refer to [15] for more details.

The number of multiplication flops for $N_t$ symbols is given as follows:

$$C = (2\alpha N_t \times d_h \times 2\alpha N_t \times \frac{1}{2} + 2\alpha N_t \times d_h \times d_h) \times 2 \times 4 \quad (13)$$

where $d_h$ is the dimension of hidden layer. The coefficient $\frac{1}{2}$ is corresponding to the triangular matrix. The last two coefficient 2 and 4 represent the bidirectional LTSM and 4 rows in (12), respectively. Recall that $\alpha = \frac{1}{2}log_2 M$, the complexity is given by

$$C = 4d_h(N_t log_2 M)^2 + 8d_h^2 N_t log_2 M \quad (14)$$

## B. LOW-RANK DECOMPOSITION

Let $W_z = [W_{iz}^T, W_{fz}^T, W_{gz}^T, W_{oz}^T]^T$ denote the input matrix and $W_h = [W_{ih}^T, W_{fh}^T, W_{gh}^T, W_{oh}^T]^T$ the hidden matrix.

The main cost of computation and storage for a LSTM unit lies in $W_z$ and $W_h$. Following [16], we can perform the truncated singular value decomposition (TSVD) on the two matrices for model acceleration and compression. Assuming $W \in \mathbb{R}^{m \times n}$, its SVD is applied as:

$$W = U_{m \times m} \Sigma_{m \times n} V_{n \times n}^T \quad (15)$$

TSVD approximates $W$ by taking only $r$ largest singular values in $\Sigma_{m \times n}$ as follows:

$$W \approx U_{m \times r} \Sigma_{r \times r} V_{r \times n}^T = U_{m \times r} N_{r \times n}, \quad (16)$$

The acceleration ratio $R_a$ and compression ratio $R_c$ are given by:

$$R_a = R_c = \frac{m \times n}{m \times r + r \times n} \quad (17)$$

TSVD can be applied to both $W_z$ and $W_h$, the corresponding $U_z$, $N_z$, $U_h$ and $N_h$ are obtained.

## C. CHANNEL PRUNING

Channel pruning aims to remove the less useful output-input channels of the matrix in the network. It usually applies a mask matrix $M$ with elements being value {0, 1}. The channel pruning of LSTM is quite different from the convolutional

<figure><img src="image_4.png" /><figcaption>FIGURE 3. Channel pruning of LSTM.</figcaption></figure>

neural network (CNN). In CNN, the pruning of the output channel of the current layer will only affect the input channel of the next layer. However, in Eq. (12), the operations between $W_z z_t$ and $W_h h_{t-1}$ are element-wise, so that the mask matrices applied to $W_z z_t$ and $W_h h_{t-1}$ should be the same. Therefore, as shown in Fig. 3, the masks for the decomposed output sub-matrices $U_z$ and $U_h$ should be the same. Since the output of a hidden layer would be considered as the input of the next hidden layer, the input channel of $N_h$ should be pruned with the same mask matrix.

In Eq. (12), the output $i_t, f_t, g_t$, and $o_t$ would be further used for element-wise operations, so that the removed elements in $i_t, f_t, g_t$, and $o_t$ should be in the same locations. In other word, the mask matrix $M$ consists of 4 sub-masks $M_i, M_f, M_g, M_o$, which are identical.

In this paper, we take the input matrix $U_z$ as an example to generate the mask. The sub matrix $U_{gz}$ in $U_z$ is considered. $l_1$-norm pruning [17] is applied on it to get the mask $M_{gz}$. Then the mask $M$ can be obtained by concatenating 4 sub-matrices $M_{gz}$:

$$M = [M_{gz}^T, M_{gz}^T, M_{gz}^T, M_{gz}^T]^T, \quad (18)$$

## V. SIMULATION RESULTS AND DISCUSSIONS

To examine the performance of the proposed method, simulations are carried out under varying channels. According to Eq. (1), 210 million samples are generated in the form of $(\bar{x}, \bar{H}, \bar{y})$, of which 200 million are used for training and 10 million for test. Our experiments are implemented with Pytorch, a BiLSTM block in Fig.1 contains 8 LSTM units, with 4 units for each direction. The computing platform is 40-core 2.20GHz Intel Xeon E5-2630 CPUs under Ubuntu18.04 with GPU NVIDIA RTX2080Ti. The ADAM optimizer is used for training with the following hyper-parameter settings: $\beta_1 = 0.9, \beta_2 = 0.999, l_r = 0.0006$ and $\epsilon = 10^{-8}$.

## A. BER PERFORMANCE

### 1) INDEPENDENT CHANNEL WITH PERFECT CSI

We compare the proposed DO-LISA detector with SD, ZF, MMSE, LISA, DetNet, and MMNet respectively. SD detector can be considered as a simplified ML detector. The
<figure><img src="image_5.png" /><figcaption>FIGURE 4. BER performance (4 × 4 independent channel, 16QAM).</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>FIGURE 5. BER performance (4 × 4 independent channel, 64QAM).</figcaption></figure>

performance of these algorithms is evaluated by simulating an uncoded V-BLAST system. Figs. 4-5 compare the BER performance of these algorithms in a 4 × 4 MIMO system transmitting 16QAM and 64QAM signals respectively. Figs. 4-5 show that the DO-LISA detector can preserve its quasi-optimal performance when the modulation order is increased to 16QAM and 64QAM, for which, LISA has significant performance degradation. For 16QAM modulation, DO-LISA can save 4 dB in power to achieve the BER performance at the level of $10^{-3}$, compared to LISA. For the 64QAM signal, at the BER level of $10^{-2}$, DO-LISA has only 1 dB SNR loss, compared to SD.

## 2) INDEPENDENT CHANNEL WITH IMPERFECT CSI

Fig. 6 investigates the BER performance of DO-LISA with imperfect CSI. A 4 × 4 MIMO system transmitting 16QAM signals over varying wireless channels is simulated. A zero-mean Gaussian process with variance 0.05 models the channel estimation errors. In this experiment, the accurate channel matrix is used to generate the training data, and the

<figure><img src="image_7.png" /><figcaption>FIGURE 6. BER performance (imperfect 4 × 4 channel, 16QAM).</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>FIGURE 7. BER performance (4 × 4 correlated channel $\rho = 0.1$, 16QAM).</figcaption></figure>

polluted channel matrix is used in the test stage. One can observe that DO-LISA has degraded performance compared to that of SD detector, which is due to the channel estimation error. In addition, one notes that it is still superior to the remaining compared detectors. This implies that our model has a certain generalization ability, which can be achieved by the training process with enough channel samples, besides a good point of stop for the training process can avoid overfitting.

## 3) CORRELATED CHANNEL

Fig. 7 simulates a 4 × 4 MIMO system transmitting 16QAM signals over the correlated channels. The realization of the channel matrix is based on the Kronecker model with correlation coefficient $\rho = 0.1$. Fig. 7 shows that DO-LISA exhibits the best BER performance except the SD detector.

## 4) DIFFERENT ANTENNA CONFIGURATIONS

Figs. 8-9 examine the BER performance of DO-LISA in 4 × 3 and 4 × 2 MIMO systems transmitting 16QAM
<figure><img src="image_9.png" /><figcaption>FIGURE 8. BER performance (4 × 3 channel, 16QAM).</figcaption></figure>

<figure><img src="image_10.png" /><figcaption>FIGURE 9. BER performance (4 × 2 channel, 16QAM).</figcaption></figure>

signals. It is shown that DO-LISA can achieve almost the same BER performance as the SD detector.

## B. PERFORMANCE OF MODEL ACCELERATION AND COMPRESSION

The effectiveness of model acceleration and compression is also evaluated. For the first LSTM unit of a BiLSTM block, the sizes of the input matrix and hidden matrix are 17 × 600 and 600 × 600, respectively. From the second to the fourth LSTM unit, the sizes of the input matrices and hidden matrices are 1200 × 600 and 600 × 600, respectively. TSVD is not applied to the input matrix in the first LSTM unit because its size is relatively small. The ranks for the remaining input matrices and hidden matrices are set to 600 and 180 respectively. The pruning ratios are set as 0.3 for both directions. Fig. 10 shows the BER performance of the accelerated and compressed DO-LISA. The compact model achieves 2.38× acceleration and 2.37× compression with limited BER loss. We can observe that the

<figure><img src="image_11.png" /><figcaption>FIGURE 10. BER performance of compact model (4 × 4 channel, 16QAM).</figcaption></figure>

<figure><img src="image_12.png" /><figcaption>FIGURE 11. Computational complexity of SD in 6 × 6 system.</figcaption></figure>

BER performance of the compact model is slightly degraded after the acceleration and compression, but it is still much better than LISA.

## VI. CONCLUSION AND PERSPECTIVE

This paper proposes a deep MIMO detector, called DO-LISA, for high-order QAM signals. Firstly, DO-LISA decomposes MQAM signals into QPSK signals to exploit the advantage of the LSTM network in binary classification, and then DO-LISA orders the signals to be detected to adapt the sequential feature of the BiLSTM network. The ordered signals are input into the BiLSTM network. Simulation results show that DO-LISA can obtain quasi-optimal BER performance, and has certain generalization ability. To simplify the network and accelerate the calculation, a compact MIMO detection network is obtained by compression and pruning. Simulation results show that the compact model can significantly simplify the calculation while preserving a better BER performance than the LISA method.
Deep MIMO detection is an intensive research areas. Considerable progresses have been achieved with excellent BER performance and non exponential complexity. However, one should note that the large dimension of hidden layer of the deep neural network can bring big complexity even higher than that of SD. Fig. 11 shows the flops needed by SD to detect a symbol in a 6 × 6 system. Recall equation (14) to calculate flops per symbol of DO-LISA by using $N_t = 6$ and $M = 64$ (64QAM). The flops/symbol needed is $864d_h + 48d_h^2$. In our work, we take the value used in [15] ($d_h = 600$), the practical flops/symbol will be likely to be higher than SD. Even if GPU can realize fast detection by the computation parallelization, lightweight neural network model will be our future research direction to reduce the energy cost.

REFERENCES

[1] P. W. Wolniansky et al., "V-BLAST: An architecture for realizing very high data rates over the rich-scattering wireless channel," in *Proc. Int. Symp. Signals, Syst., Electron.*, Pisa, Italy, Sep./Oct. 1998.

[2] E. Agrell, T. Eriksson, A. Vardy, and K. Zeger, "Closest point search in lattices," *IEEE Trans. Inf. Theory*, vol. 48, no. 8, pp. 2201–2214, Aug. 2002.

[3] T. Cui and C. Tellambura, "An efficient generalized sphere decoder for rank-deficient MIMO systems," *IEEE Commun. Lett.*, vol. 9, no. 5, pp. 423–425, May 2005.

[4] L. G. Barbero and J. S. Thompson, "Fixing the complexity of the sphere decoder for MIMO detection," *IEEE Trans. Wireless Commun.*, vol. 7, no. 6, pp. 2131–2142, Jun. 2008.

[5] S. Yang and L. Hanzo, "Fifty years of MIMO detection: The road to large-scale MIMOs," *IEEE Commun. Surveys Tuts.*, vol. 17, no. 4, pp. 1941–1988, 4th Quart., 2015.

[6] Z. Guo and P. Nilsson, "Algorithm and implementation of the K-best sphere decoding for MIMO detection," *IEEE J. Sel. Areas Commun.*, vol. 24, no. 3, pp. 491–503, Mar. 2006.

[7] D. Tse, *Fundamentals of Wireless Communication*. Cambridge, U.K., 2007.

[8] D. Wubben, R. Bohnke, V. Kuhn, and K.-D. Kammeyer, "MMSE extension of V-BLAST based on sorted QR decomposition," in *Proc. IEEE 58th Veh. Technol. Conference. VTC -Fall*, Orlando, FL, USA, vol. 1, Oct. 2003, pp. 508–512.

[9] J. Schmidhuber, "Deep learning in neural networks: An overview," *Neural Netw.*, vol. 61, pp. 85–117, Jan. 2015.

[10] N. Samuel, T. Diskin, and A. Wiesel, "Deep MIMO detection," in *Proc. IEEE 18th Int. Workshop Signal Process. Adv. Wireless Commun. (SPAWC)*, Jul. 2017, pp. 1–5.

[11] M. Khani, M. Alizadeh, J. Hoydis, and P. Fleming, "Adaptive neural signal detection for massive MIMO," 2019, arXiv:1906.04610.

[12] Y. Wei, M.-M. Zhao, M. Hong, M.-J. Zhao, and M. Lei, "Learned conjugate gradient descent network for massive MIMO detection," *IEEE Trans. Signal Process.*, vol. 68, pp. 6336–6349, 2020.

[13] A. Mohammad, C. Masouros, and Y. Andreopoulos, "Complexity-scalable neural-network-based MIMO detection with learnable weight scaling," *IEEE Trans. Commun.*, vol. 68, no. 10, pp. 6101–6113, Oct. 2020.

[14] M. A. Albreem, A. H. Alhabbash, S. Shahabuddin, and M. Juntti, "Deep learning for massive MIMO uplink detectors," *IEEE Commun. Surveys Tuts.*, vol. 24, no. 1, pp. 741–766, 1st Quart., 2022.

[15] J. Sun, Y. Zhang, J. Xue, and Z. Xu, "Learning to search for MIMO detection," *IEEE Trans. Wireless Commun.*, vol. 19, no. 11, pp. 7571–7584, Nov. 2020.

[16] Y. Yang, K. Liang, X. Xiao, Z. Xie, L. Jin, J. Sun, and W. Zhou, "Accelerating and compressing LSTM based model for online handwritten Chinese character recognition," in *Proc. 16th Int. Conf. Frontiers Handwriting Recognit. (ICFHR)*, Aug. 2018, pp. 110–115.

[17] H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. P. Graf, "Pruning filters for efficient convnets," in *Proc. Int. Conf. Learn. Represent.*, 2017, pp. 1–13.

<figure><img src="image_13.png" /></figure>

**ZICHUN HUANG** received the B.S. degree in information engineering from South China University of Technology, Guangzhou, China, in 2019, where she is currently pursuing the M.S. degree in electronic information. Her current research interests include array signal processing, spectral estimation, and signal processing for communications.

<figure><img src="image_14.png" /></figure>

**YUEHUA DING** received the M.S. degree in electrical engineering from South China University of Technology (SCUT), Guangzhou, China, in 2008, and the Ph.D. degree in signal processing and telecommunications from Ecole Polytechnique de l'Université de Nantes, Nantes, France, 2011. In June 2012, he joined the School of Electronic and Information Engineering, SCUT. His research interests include array signal processing in communications, cognitive radios, MIMO communication systems, and multi-antenna technologies.

<figure><img src="image_15.png" /></figure>

**BIYUN MA** received the B.S. degree in information engineering from South China University of Technology, Guangzhou, China, in 2004, and the M.S. and Ph.D. degrees in electronic engineering from Nantes University, France, in 2007 and 2010, respectively. She is currently an Associate Professor with the School of Electronic and Information Engineering, South China University of Technology. Her research interests include array signal processing, mobile wireless communication systems, and signal processing for ultrasonic detection and communication.

<figure><img src="image_16.png" /></figure>

**JIE LI** received the B.S. degree in electronic and information engineering from Xidian University, Xi'an, China, in 2008, and the Ph.D. degree in communication and information system from South China University of Technology, Guangzhou, China, in 2013. He is currently an Associate Professor with the School of Electronic and Information Engineering, South China University of Technology. His current research interests include array signal processing and machine learning, with applications to wireless communications and underwater acoustic communications and detection.

<figure><img src="image_17.png" /></figure>

**YIDE WANG** (Senior Member, IEEE) received the B.S. degree in electrical engineering from Beijing University of Post and Telecommunication, Beijing, China, in 1984, and the M.S. and Ph.D. degrees in signal processing and telecommunications from the University of Rennes, France, in 1986 and 1989, respectively. He is currently a Professor with Ecole Polytechnique de Nantes Université. His research interests include array signal processing and signal processing for fault detection and diagnosis in electrical machines.