## 3 Hybrid GA with Layered Process for MIMO Detection
### Proposed Hybrid Genetic Algorithm
#### 3.1.3 Mutation  
The _mutation_ operation of the HGA is modified to act as an intensive search for the areas where the fitness is expected to be good by adopting the concept of neighborhood search of the TS. Because the neighbors of individual with good fitness are also likely to have good fitness, a neighborhood search for an individual with good fitness can find a solution that is closer to the optimal. The concept of neighborhood search is introduced in [31].  
Let \(L\) denote the cardinality of \(\mathbb{A}=\{a_{1},a_{2},\cdots,a_{L}\}\). Define a set \(\mathcal{N}(a_{q})\), \(q\in\{1,\cdots,L\}\), as a fixed subset of \(\mathbb{A}\setminus a_{q}\), which we refer to as the symbol neighborhood of \(a_{q}\). We choose the cardinality of this set to be the same for all \(a_{q},\ q=1,\cdots,L\); i.e., we take \(\left|\mathcal{N}(a_{q})\right|=N,\ \forall q\). Note that the maximum and minimum values of \(N\) are \(L-1\) and \(1\), respectively. We choose the symbol neighborhood based on the Euclidean distance, i.e., for a given symbol, those \(N\) symbols which are the nearest will form its neighborhood; the nearest symbol will be the first neighbor, the next nearest symbol will be the second neighbor, and so on. For, e.g., \(\mathbb{A}=\{-3,-1,1,3\}\) for 4-PAM, and choosing \(N\) to be 2, \(\mathcal{N}(-3)=\{-1,1\}\), \(\mathcal{N}(-1)=\{-3,1\}\), \(\mathcal{N}(1)=\{-1,3\}\), and \(\mathcal{N}(3)=\{1,-1\}\) are possible symbol neighborhoods. Let \(w_{v}(a_{q}),\ v=1,\cdots,N\) denote the \(v\)th element in \(\mathcal{N}(a_{q})\): i.e., we say \(w_{v}(a_{q})\) is the \(v\)th symbol neighbor of \(a_{q}\).  
Let \(\mathbf{x}=\begin{bmatrix}x_{1}&x_{2}&\cdots&x_{N_{t}}\end{bmatrix}^{T}\) denote the data vector belonging to the solution space, where \(x_{i}\in\mathbb{A}\). We refer to the vector  
\[\mathbf{z}(u,v)=\begin{bmatrix}z_{1}(u,v)&z_{2}(u,v)&\cdots&z_{N_{t}}(u,v) \end{bmatrix}, \tag{6}\]  
as \((u,v)\)th vector-neighbor (or simply the \((u,v)\)th neighbor) of \(\mathbf{x}\), \(u=1,\cdots,N_{t}\), \(v=1,\cdots,N\), if (\(i\)) \(\mathbf{x}\) differs from \(\mathbf{z}(u,v)\) in the \(u\)th coordinate only, and (\(ii\)) the \(u\)th element of \(\mathbf{z}(u,v)\) is the \(v\)th symbol-neighbor of \(x_{u}\). That  
the _mutation_ creates a vector-neighbor of selected individuals and inserts them into the population of the next generation. The individuals selected for the neighborhood searches are determined by the roulette wheel selection approach, so an intensive search around the individuals with good fitness can be performed. Furthermore, if the offspring selected using the current _selection_ operation differs from the result of previous _selection_ operation, this means that a new best solution has been searched. Since there is a high probability that the more fit solution exists around the new best solution, we perform the intensive search by performing the proposed _mutation_ on the current _selection_ result.  
To prevent repeated searches of already searched space, the proposed _mutation_ memorize the center of the performed neighborhood search in what is called the '_mutation list_\((\mathbf{MU}_{list})\)', which adopts the memory feature of the TS algorithm. During all generations, the \(\mathbf{MU}_{list}\) accumulates the individuals selected as the center of the neighborhood search in the _mutation_ operation and the newly updated individuals in the _selection_ operation. By using the \(\mathbf{MU}_{list}\), the algorithm attempts to avoid repeated searches by marking the already searched space, which ensures efficient search of the solution space.  
Through the above process, the proposed HGA avoids repeated searches of the same areas and enables the intensive search of areas where fitness is likely to be good.