## 5 Evaluation Protocol and Results
### Ablation Study
#### 5.2.1 Model Selection  
Various model architectures were examined to find the best one. The process was done by changing the hyper-parameters of (CNN) model, e.g., number of layers, number of kernels in each layer, activations, optimizers, brain regions, and pattern's length and overlapping amount. In the following sections, we show the results of our experiments in detail. The hyper-parameters of the layers of a 1D CNN model were determined empirically to select the best model and compare their results for further improvement. Table 1 shows the architectures of the best four models and their results. From Table 1, we notice that Model 1 has the highest results in terms of accuracy, sensitivity, and specificity scores.  
All the models were tested using 10-fold cross-validation. To determine whether there is a significant difference between the performance of the different models, we employed the Wilcoxon signed-rank test with the following null and alternate hypotheses:  
**H0:**: _There is no difference between Model 1 and Model 2;_
**H1:**: _Model 1 is better than Model 2 in performance._  
The results are shown in Table 2. The \(p\)-values in the three cases were less than 0.05, which indicates that Model 1 is better than Model 2, Model 3, and Model 4, and the difference is statistically significant at a 5% significance level. Therefore, Model 1 was selected to examine other design choices. The proposed model has pyramidal architecture, i.e., the low-level Conv layers have more kernels than higher-level layers. To validate the effectiveness of this approach, we investigated its counterpart, in which the number of kernels increases from low-level layers to semantic (high-level layers), i.e., the number of kernels increases from 4 to 16 kernels. This model achieved 89.54%, 91.44%, and 87.37% accuracy, sensitivity and specificity, respectively. This indicates that the pyramidal architecture is better.  
* Activations  
ReLU, Elu, and Selu activation functions were examined to check their effect on the best model. Table 3 shows the results of examining each activation function using 10-fold cross-validation with the best model. As shown in Table 3, ReLU has the best effect on the selected model.  
\begin{table}
\begin{tabular}{c c c} \hline
**Models** & \(p\)**-Value** \\ \hline Model 1 vs. Model 2 & 0.0012 \\ \hline Model 1 vs. Model 3 & 0.0078 \\ \hline Model 1 vs. Model 4 & 0.0039 \\ \hline \end{tabular}
\end{table}
Table 2: Statistically significant test results.  
\begin{table}
\begin{tabular}{c c c c} \hline
**Activation** & **Avg-Accuracy** & **Avg-Sensitivity** & **Avg-Specificity** \\ \hline ReLU & 99.88\% & 100\% & 99.74\% \\ \hline Elu & 97.74\% & 98\% & 97.44\% \\ \hline Selu & 91.31\% & 93.78\% & 88.46\% \\ \hline \end{tabular}
\end{table}
Table 3: Results of activation functions with the best model.  
* Optimizers  
We tested three optimizers with the best model to see which one works better for this task. RMSProb, Adam, and SGD with momentum (SGDM) were examined. The learning rate was set to 0.001 in all optimizers. In SGDM, the momentum was set to 0.9. Table 4 shows the 10-fold cross-validation results of using these optimizers to train the selected model.