## 2 Serial Detection
### Neural Network  
In Figure 4, the structure of the neural network is fully connected and includes \(n\) inputs, \(t\) hidden layers having \(h_{t}\) neurons for each layer, and 1 output neuron. Thus, the structure of the neural network can be defined by selecting the parameters \(n\), \(t\), and \(h_{t}\). Let \(\mathbf{h}=[h_{1}\,h_{2}\,h_{3}\,\dots\,h_{t}]\) correspond to the hidden layers. For instance, if \(\mathbf{h}=[90\,42\,53]\), we have 90, 42, and 53 neurons for the first, second, and third hidden layers, respectively. For the  
Figure 3: Trellis for the VA.  
input of the neural network, we consider a block of size \(5\times 5\) and a vector of size \(25\times 1\), as in Equations (6) and (7), respectively [24]. \[\mathbf{Y}_{nn}=\left[ \begin{array}{ccccc}y[j-2,k-2]&y[j-2,k-1]&y[j-2,k]&y[j-2, k+1]&y[j-2,k+2]\\ y[j-1,k-2]&y[j-1,k-1]&y[j-1,k]&y[j-1,k+1]&y[j-1,k+2]\\ y[j,k-2]&y[j,k-1]&y[j,k]&y[j,k+1]&y[j,k+2]\\ y[j+1,k-2]&y[j+1,k-1]&y[j+1,k]&y[j+1,k+1]&y[j+1,k+2]\\ y[j+2,k-2]&y[j+2,k-1]&y[j+2,k]&y[j+2,k+1]&y[j+2,k+2]\end{array} \right], \tag{6}\]  
\[\mathbf{y}_{nn}=\left[ \begin{array}{ccccc}y[j-2,k-2]&\ldots&y[j-2,k+2]&y[j-1, k-2]&\ldots&y[j+2,k+2]\end{array} \right]^{\mathsf{T}}, \tag{7}\]  
Hence, there are 25 inputs (i.e., \(n=25\)) and one output neuron. As regards the structure of the hidden layer, many studies have suggested optimal numbers of hidden layers and neurons; however, no universally accepted method exists for a particular problem [26]. Therefore, using the model in Figure 5 for training, we used the mean square error (MSE) parameter to experimentally evaluate the structure of the hidden layers. Figure 4: Structure of the neural network. Figure 5: Model for training the neural network. The details of the model are described in the next section. In Figure 5, the training process of the neural network is illustrated. In the training process, we implement it as in [22; 24]. We collect a data block size of \(5\times 5\) including two neighboring ISI and ITI bits for all eight directions from \(y[j,k]\) and reshape that into a size of \(25\times 1\) for the input of the neural network. The noise signal \(e[j,k]\) is the label for the output of the neural network, which is the difference between \(z[j,k]\) and \(d[j,k]\) after estimating the parameter of the GPR target. The estimation of the GPR target was performed before training the neural network. In addition, we used the scaled conjugate gradient (SCG) algorithm [27] to optimize the network parameters. Tables 2-4 show the resulting MSEs for each structure. For reasonable complexity, we tested one to three hidden layers and approximately 100 neurons. In addition, when increasing the hidden layers, we still achieve a similar performance with three hidden layers. Based on the results, we chose one, two, and three hidden layers, that is, [100], [60; 40], and [30; 50; 20], respectively. Furthermore, each neuron in the hidden layers uses the activation function, \(\text{tansig}(p)=\frac{2}{1+\exp(-2p)}-1\) and the node in the output layer uses a linear function \(f(p)=p\). To choose \(\text{tansig}(p)\) function, we compared to other possible activation functions, and the results are presented in Table 5.  
After choosing the architecture of the network, we trained the network with 10% data set for the validation and 90% data set for the training. The learning curve for one to three hidden layers is presented in Figures 6-8, respectively. \begin{table}
\begin{tabular}{c c} \hline \hline
**Hidden Layer** & **MSE** \\ \hline
[30 20 50] & 0.0238 \\
[50 50 50] & 0.0171 \\
[50 20 30] & 0.0162 \\
[20 50 30] & 0.0154 \\
**[30 50 20]** & **0.0141** \\
**[20 30 50]** & 0.0169 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: MSE of the neural network with three hidden layers. \begin{table}
\begin{tabular}{c c} \hline \hline
**Hidden Layer** & **MSE** \\ \hline
[5] & 0.0442 \\
[10] & 0.0425 \\
[20] & 0.0302 \\
[50] & 0.0273 \\
**[100]** & **0.0272** \\
[200] & 0.0475 \\ \hline \hline \end{tabular}
\end{table}