## 3 Hybrid GA with Layered Process for MIMO Detection
### Hybrid GA-Based MIMO Detection with Layered Process
Calculate \[\tilde{z}_{k}=\hat{x}_{k}-\sum_{l=o_{k}+1}^{N_{t}}\frac{r_{o_{k}l}}{r_{o_{k}o_{k}} }(\tilde{x}_{l}-\hat{x}_{l})\] (9) which is a cancellation operation that removes the interference due to the symbols detected in the previous layer (i.e., \(\tilde{x}_{l}\)'s). * Signal detection. Find the symbol \(\tilde{a}_{q}\) in the alphabet \(\hat{\mathbf{S}}\) which is closest to \(\tilde{z}_{k}\) in Euclidean distance. * Test : \(\left|\tilde{z}_{k}-\tilde{a}_{q}\right|\lesssim\delta d_{min}\), \(0<\delta\leq 0.5\)
* Decision. If \(\left|\tilde{z}_{k}-\tilde{a}_{q}\right|<\delta d_{min}\), then \(\tilde{x}_{k}=\tilde{a}_{q}\). Make \(k=k-1\) and return to (b)
* Proposed HGA. If \(\left|\tilde{z}_{k}-\tilde{a}_{q}\right|\geq\delta d_{min}\), then set \(\tilde{x}_{k}=\hat{\mathbf{x}}^{\ddagger}\). Perform the proposed HGA by replacing the initial individual \(\hat{\mathbf{x}}^{\ddagger}\) with \(\hat{\mathbf{x}}\), \(\hat{\mathbf{H}}\) with \(\hat{\mathbf{H}}\), \(\hat{\mathbf{y}}\) with \(\hat{\mathbf{y}}\), where \(\hat{\mathbf{x}}\), \(\hat{\mathbf{H}}\), and \(\hat{\mathbf{y}}\) for the \(k\)th layer are taken as \[\hat{\mathbf{x}}= \begin{bmatrix}\tilde{x}_{k}&\tilde{x}_{k+1}&\cdots&\tilde{x} _{2N_{t}}\end{bmatrix} ,\] (10) \[\hat{\mathbf{H}}= \begin{bmatrix}r_{kk}&r_{k(k+1)}&\cdots&r_{2N_{t}}\\ 0&r_{(k-1)(k-1)}&\cdots&r_{(k-1)2N_{t}}\\ \vdots&\vdots&\ddots&\vdots\\ 0&0&\cdots&r_{2N_{t}2N_{t}}\end{bmatrix} ,\] (11) \[\hat{\mathbf{y}}=\hat{\mathbf{H}} \begin{bmatrix}\hat{x}_{k}&\hat{x}_{k+1}& \cdots&\hat{x}_{2N_{t}}\end{bmatrix} ^{\mathcal{T}}.\] (12) The output vector form is made as the update \( \begin{bmatrix}\tilde{x}_{k}&\tilde{x}_{k+1}&\cdots&\tilde{x}_{2N_{t}}\end{bmatrix} \) subvector. Make \(k=k-1\) and return to (b)  
After processing all the \(2N_{t}\) layers, the output of the last layer vector \(\hat{\mathbf{x}}= \begin{bmatrix}\hat{x}_{1}&\tilde{x}_{2}&\cdots&\tilde{x}_{2 N_{t}}\end{bmatrix} \) is declared as the final detected vector. The solution space to explore in the HGA process is very large. For example, in a \(4\times 4\) MIMO system with 1024-QAM, the total number of possible symbol vectors is \(1024^{4}\approx 10^{12}\). The HGA without layering is carried out once on the full system model, which means we have to perform a single HGA to find the final solution. Although the HGA has improved search ability compared to the basic GA, it is very difficult to find an optimal solution in a very wide solution space with a single HGA implementation, and/or the computational complexity to pay for finding a good solution is very high. Whereas, in the LHGA, the HGA is performed multiple times with the IC, at most once on each layer. The size of the symbol space that the HGA has to explore for by each layer is smaller than or equal to the total symbol space, and the searching process is performed after the inter-layer interference is removed, so the probability of finding a better solution increases. Furthermore, since the HGA is selectively performed according to the IC result, the computational complexity is lower than that of the HGA without layering. By embedding the local search mechanism into the GA, the proposed HGA can balance diversification and intensification of the exploration. In addition, by introducing the _tabu_ concept in the search process, it has improved search efficiency compared to the basic GA. Furthermore, the proposed HGA-based MIMO detection scheme can reduce the computational complexity and better improve BER performance through use of a layered process.