## 3 Results
### Results: Part B
#### 3.2.4 Part B, Phase 4  
As a summary metric, the values of _kappa_ were calculated based on the \(5\times 5\) confusion matrix results obtained (Alpha, Beta, Delta, Gamma and Theta) for each of the 12 (3 time (Slot1/Slots 2-5/Slots 6-8) \(\times\) 4 stimulation frequency) models. The results are shown in Table 9.  
The CV for _kappa_ is 0.137, and the mean 0.850 (SD 0.116). Using this approach rather than the MLP-NN algorithm, differences among the EEG bands are again greater for 2.5 pps at baseline and less for 10 pps post-stimulation, but otherwise, there is little agreement between the two methods. _Kappa_ is greatest for the sham and 2.5 pps at baseline, for 10 pps during stimulation, and for 80 pps post-stimulation. According to the guidelines of [PERSON] [PERSON] [53], 8 of the differences are 'perfect' (>0.81) and the remainder'substantial'. The processing time for Phase 1 is again estimated to be around 5 h.  
Figures 6-9 provide examples of outputs (train and test accuracy or ROC plots, and confusion matrices) for the models in each phase with the highest value of _kappa_. Graphs of model accuracy and loss are shown over 100 epochs, where each epoch represents a full pass through the training dataset. Test loss is a measure of how well the model can \begin{table}
\begin{tabular}{c c c c c} \hline
**Band** & **Sham** & **10 pps** & **2.5 pps** & **80 pps** \\ \hline Alpha & 0.563 (Model 1) & 0.165 (Model 2) & **0.763** (Model 3) & **0.906** (Model 4) \\ Beta & 0.180 (Model 5) & 0.281 (Model 6) & 0.474 (Model 7) & 0.496 (Model 8) \\ Delta & ns (Model 9) & 0.352 (Model 10) & 0.316 (Model 11) & 0.296 (Model 12) \\ Gamma & ns (Model 13) & ns (Model 14) & **0.803** (Model 15) & ns (Model 16) \\ Theta & 0.457 (Model 17) & 0.748 (Model 18) & **0.789** (Model 19) & 0.508 (Model 20) \\ \hline \end{tabular}
\end{table} 
Table 7: Confusion matrix results (_kappa_) for the 20 models in Phase 2. Values in **bold** are in the upper quartile of all 16 values, and those in red in the lower quartile (‘ns’ indicates mean overall accuracy < 0.33). \begin{table}
\begin{tabular}{c c c c c} \hline
**Time** & **Sham** & **2.5 pps** & **10 pps** & **80 pps** \\ \hline Baseline (Slot 1) & **0.698** (Model 1) & **0.680** (Model 2) & 0.679 (Model 3) & 0.658 (Model 4) \\ Stim (Slots 2–5) & 0.667 (Model 5) & 0.642 (Model 6) & 0.639 (Model 7) & 0.666 (Model 8) \\ Post (Slots 6–8) & 0.666 (Model 9) & **0.718** (Model 10) & 0.634 (Model 11) & 0.577 (Model 12) \\ \hline \end{tabular}
\end{table} 
Table 8: Confusion matrix results (_kappa_) for the 12 models in Phase 3. Values in **bold** are in the upper quartile of all 16 values, and those in red in the lower quartile. \begin{table}
\begin{tabular}{c c c c c} \hline
**Time** & **Sham** & **2.5 pps** & **10 pps** & **80 pps** \\ \hline Baseline (Slot 1) & 0.868 (Model 1) & **0.969** (Model 2) & 0.847 (Model 3) & 0.912 (Model 4) \\ Stim (Slots 2–5) & 0.765 (Model 5) & 0.739 (Model 6) & **0.990** (Model 7) & 0.945 (Model 8) \\ Post (Slots 6–8) & 0.862 (Model 9) & 0.683 (Model 10) & 0.650 (Model 11) & **0.968** (Model 12) \\ \hline \end{tabular}
\end{table} 
Table 9: Confusion matrix results (_kappa_) for the 12 models in Phase 4. Values in **bold** are in the upper quartile of all 16 values, and those in red in the lower quartile. make predictions on data it has not seen before (i.e., the test set). Ideally, the test loss will decrease over time, and the accuracy will increase, indicating that the model is learning and improving. Figure 6: Phase 1 – Model 13 of 16 models: sham, Slot 1 vs. Slot 5. Graphs of (**a**) model accuracy and (**b**) loss over 100 epochs, with (**c**) the associated confusion matrix and (**d**) the receiver operating characteristic (ROC) curve. For this model, the train-to-test ratio was 9:1, with _kappa_ = 0.488, area under the ROC curve (AUC) = 0.75 and F-measure (harmonic mean of ‘precision’ and ‘recall’) = 0.488.