# Early Neighbor Rejection-Aided Tabu Search Detection for Large MIMO Systems
## 1 Introduction  
Multiple-input multiple-output (MIMO) systems have been widely employed in wireless communication systems, including long-term evolution (LTE), LTE-advanced, and 5G new radio (NR) systems. In these systems, MIMO techniques are leveraged to provide improved link reliability and enhanced channel capacity. The successful use of MIMO techniques to enhance the performance of mobile networks has driven the research community to achieve further benefits using large MIMO systems, which employ a large number of antennas for signal transmission and reception at base stations [1]. However, the use of a large number of antennas results in excessively high computational complexity for optimal signal detection schemes at the receiver. For large MIMO systems, we can consider various signal detection schemes, such as successive interference cancellation (SIC) [2], sphere decoder (SD) [3], \(K\)-best SD (KSD) [4], adaptive threshold-aided KSD (AKSD) [5] and parallel interference cancellation (PIC) [6, 7]. The SD and \(K\)-best SD algorithms begin their search over the noiseless received signals that lie within a hypersphere radius \(R\) around the received signal [8]. In a PIC-based MIMO decoder, all symbols are detected simultaneously. The computational cost of SIC is not high, but its performance is sub-optimal. By contrast, the SD scheme achieves near-optimal bit error-rate (BER) performance, whereas its complexity is typically impractically high in large MIMO systems. Tabu search (TS) is considered to be a near-optimal detection scheme for MIMO systems; it has been shown that in a large MIMO system, TS requires remarkably lower complexity than SD schemes [9, 10, 11]. To reduce the complexity of TS, previous studies have applied several modifications to its basic form. In [9], a restart diversification scheme is employed, which restarts the TS procedure a number of times from randomly generated initial solutions. The authors of [10] considered a layered approach in which the TSalgorithm is applied to each layer by increasing the search space. In [11], groupwise neighbor examination for TS is employed, resulting in complexity reduction with almost no performance loss. In [12], QR decomposition-aided TS (QR-TS) was introduced for large MIMO systems in which the QR-decomposed channel matrix is employed for the neighbor examination of TS, achieving nearly the same performance as the conventional TS while providing significantly reduced computational complexity. In the TS algorithm, we need to compute the maximum likelihood (ML) metric for neighbors in each layer. The number of neighbors increases with the number of antennas, so the complexity of computing the ML metrics of neighbors can be excessively high in large MIMO systems. To resolve this problem, we propose in this work a novel early neighbor rejection (ENR)-aided TS (ENR-TS) scheme that sequentially rejects neighbors in each layer. The main contributions of this study can be summarized as follows:  
1. We develop a novel early neighbor rejection ENR-TS scheme. The TS algorithm computes the cumulative ML metrics of all the neighbors in each layer and rejects \(k\) neighbors whose cumulative metrics are larger than the others. The rejected neighbors are excluded from the computation of ML metrics in further layers, which reduces complexity. 2. To mitigate the risk of excluding the best neighbor in early layers, we apply a layer ordering scheme, which sorts the layers of each neighbor in descending order of their expected metrics. 3. To evaluate the performance and complexity of the proposed scheme, we perform numerical analysis. The simulation results show that when the ERN scheme is incorporated into QR-TS (ENR-QR-TS), it reduces complexity by approximately 74% compared to the original TS scheme, while attaining almost the same BER performance. The rest of this paper is organized as follows: in Section 2, the system model, conventional TS and QR-TS algorithms are described. In Section 3, the proposed ENR with QR-TS and ordering schemes are developed.