## 4 The ANN Design
### Regularization  
To avoid the problem of overfitting, we add regularization to the ANN construction. Selecting the optimal value of the regularization parameter \(\lambda\) will allow us to achieve a balance between the simplicity of the model and the risk of overfitting. The larger \(\lambda\) is, the simpler the model is, but the predictions are worse. There will be a risk of underfitting. If \(\lambda\) is very small, the model will be highly complex and the risk of overfitting will increase. If \(\lambda\) is equal to zero, regularization is canceled and learning occurs only by minimizing the loss, this may lead to overfitting.  
We test the performance of the model with different values of \(\lambda\) in the validation set, and choose the value of \(\lambda\) that gives the best result.  
Appendix A contains a Matlab script that implements the algorithm that checks if the value of \(\lambda\) is found in the range of \([\,0\ldots\,5\cdot 10^{-5}]\).  
As a result, the regularization parameter corresponding to the lowest classification error equal to \(\lambda=5\cdot 10^{-6}\) (Illustrated with a red circle in Figure 18).  
Let us train the network (Network 6) using the obtained value of \(\lambda\) on Dataset 3. Network parameters are given in Table 13.  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Parameter** & **Value** \\ \hline Layer activation function & sigmoid \\ \hline Output layer activation function & softmax \\ \hline Error search method & Cross-Entropy Loss \\ \hline Hidden layers & 1 \\ \hline The number of neurons & 47 \\ \hline \hline \end{tabular}
\end{table}
Table 13: The Network 6 parameters.  
Figure 18: Changing the regularization parameter (**a**) general view; (**b**) enlarged fragment.  
The result of the training is shown in Figure 19. The confusion matrices are shown in Figure 20. The results are summarized in Table 14.  
The result obtained by applying regularization is far superior to previous solutions.  
We can improve the result by increasing dataset, i.e., adding more data to the training sample. This is the easiest and most feasible way to reduce the scatter. Let us prepare new data (Dataset 4) with 1,172,889 total values and 133,068 significant values that are collisions.  
Since the input data have been changed, it is necessary to recalculate the regularization, since it depends on the input data. As before, we train the network with different values of \(\lambda\) (Figure 21). As a result, we found the regularization parameter corresponding to the lowest classification error \(\lambda=1\):\(10^{-6}\).  
\begin{table}
\begin{tabular}{c c c} \hline \hline  & **Parameter** & **Value** \\ \hline \(\lambda\) & \(5\times 10^{-6}\) \\ \hline Training method & holdout validation \\ \hline Input dataset & Dataset3 \\ \hline Learning function & LBFGS \\ \hline Number of epochs & 1000 \\ \hline Number of inputs & 24 \\ \hline Number of output & 1 \\ \hline GradientTolerance: & \(1\times 10^{-6}\) \\ \hline LossTolerance: & \(1\times 10^{-6}\) \\ \hline StepTolerance: & \(1\times 10^{-6}\) \\ \hline \hline \end{tabular}
\end{table}
Table 13: _Cont._  
Figure 19: The result of the training Network 6 (**a**) model accuracy; (**b**) model loss.  
We derive the networks at the best value \(\lambda=1\):\(10^{-6}\) (Network 7) using the obtained value of \(\lambda\) on dataset with 1172889 total values and 133068 collisions being significant values.  
The training result of Network 7 is shown in Figure 22 and Table 15. The confusion matrix is shown in Figure 23.  
Figure 21: Changing the regularization parameter.  
Figure 20: The confusion matrices (**a**) for the training sample; (**b**) for the test sample; (**c**) for the random sample.  
The result obtained by applying regularization and augmented dataset shows a high result, in terms of prediction accuracy.