## 3 Hybrid GA with Layered Process for MIMO Detection
### Hybrid GA-Based MIMO Detection with Layered Process  
The proposed HGA-based MIMO detection demonstrates improved BER performance over conventional detectors through intensive search for large signal spaces. However, HGA requires more computational complexity to find better solutions. In other words, the computational complexity for applying the proposed HGA to real systems is still very high. For better BER performance and reduced computational complexity, we adopt a strategy of detecting symbols in a layered manner. The layered structure is inspired by the approaches suggested in [14; 20; 27]. In particular, the detection with a layered structure [27] that update the solution vector under consideration within the specific layer is adopted with the proposed HGA. The layered HGA (LHGA) basically performs MMSE successive interference cancellation detection based on QR decomposition, and the proposed HGA in each layer is applied. Let \(\tilde{\mathbf{R}}\) denote the upper triangular matrix obtained from the QR decomposition of \(\tilde{\mathbf{H}}\) and \(r_{ij}\) denote the element in the \(i\)th row and \(j\)th column of the \(\tilde{\mathbf{R}}\) matrix. Then, the objective equivalent to (3) will be to find the transmitted vector \(\tilde{\mathbf{x}}\) that minimizes \(\|\tilde{\mathbf{R}}(\tilde{\mathbf{x}}-\tilde{\mathbf{x}})\|^{2}\), where  
\[\tilde{\mathbf{x}}=((\tilde{\mathbf{H}}^{H}\tilde{\mathbf{H}}+\sigma_{n}^{2}/ \sigma_{x}^{2}\mathbf{I})^{-1}\tilde{\mathbf{H}}^{H})\tilde{\mathbf{y}}. \tag{8}\]  
is the MMSE detection result and \(\tilde{x}_{i}\) denotes the \(i\)th element of the vector \(\tilde{\mathbf{x}}\). The proposed layered HGA processes one layer at a time with the ordering based on the post-detection SNR of the symbols. The optimum order \((\o_{1},\o_{2},\cdots,\o_{2N_{t}})\), which is a permutation of \((1,2,\cdots,2N_{t})\), can be obtained as follows. Perform the following steps for \(i=2N_{t},\cdots,1\) with \(\tilde{\mathbf{H}}_{2N_{t}}=\tilde{\mathbf{H}}\):  
1. Find the \(\tilde{\mathbf{H}}_{i}^{\dagger}\) where \(\tilde{\mathbf{H}}_{i}\) is obtained by zeroing \((\o_{i+1},\o_{i+2},\cdots,\o_{2N_{t}})\) columns of \(\tilde{\mathbf{H}}\). 2. Find \(o_{i}\), the index that corresponds to the row with the lowest norm among all rows of \(\tilde{\mathbf{H}}_{i}^{\dagger}\). Detection is conducted in the following order: \(\o_{2N_{t}},\o_{2N_{t}-1}\), and so on. That is, the layer with highest post-detection SNR should be detected first. In the \(o_{k}\)th layer, \(k=2N_{t},2N_{t}-1\cdots,1\), the algorithm detects the \((2N_{t}-k+1)\)-sized subvector \(\left[\tilde{x}_{k}\quad\tilde{x}_{k+1}\quad\cdots\quad\tilde{x}_{2N_{t}}\right]\). We detect the symbols of this subvector jointly because they interfere with each other. The joint detection method employed in each layer is the proposed HGA described in Section 3.1. If a cancellation of interference due to the already detected symbols in the previous layer results in a good quality output, the LHGA skips the joint detection search in a layer. The resulting LHGA is represented in Figure 5 and stated below. Let \(\hat{\mathbf{x}}=((\tilde{\mathbf{H}}^{H}\tilde{\mathbf{H}}+\sigma_{n}^{2}/ \sigma_{x}^{2}\mathbf{I})^{-1}\tilde{\mathbf{H}}^{H})\tilde{\mathbf{y}}\) be the MMSE solution and \(\hat{\mathbf{x}}^{\ddagger}\) be the quantized version of \(\hat{\mathbf{x}}\), i.e., each element in \(\hat{\mathbf{x}}\) is rounded-off to its nearest symbol in the alphabet to obtain \(\hat{\mathbf{x}}^{\ddagger}\), so that \(\hat{\mathbf{x}}^{\ddagger}\in\hat{\mathbf{S}}\). Let \(d_{min}\) be the minimum Euclidean distance between any two symbols in the alphabet \(\tilde{\mathbf{S}}\). The steps performed in the \(o_{k}\)th layer, \(k=2N_{t},2N_{t}-1\cdots,1\), are as follows:  
1. Initialization. Define the Euclidean distance in (4) as an fitness function, and set the HGA operations such as \(G,C_{r}\) and \(M_{y}\). 2. Initialization. Define the Euclidean distance in (4  
* Interference cancellation.