## 3 Results
### Wearing-Off Prediction Model
#### 3.5.1 Participant 1 Prediction Model  
For participant 1, we analyzed the performance of each pipeline on the validation set. The CI pipeline produced the best balanced accuracy of 74.7% for the 15 min time frame and 73.5% for the 15 s time frame, as shown in Table 9.  
Next, we used the CI pipeline to compare the performance of each learning algorithms using the validation set. In the 15 min time frame, the GB algorithm produced the best balanced accuracy of 74.3% and and an f1-score of 74.0% over other algorithms. The next best learning algorithm was DT, with a 73.9% balanced accuracy and a 74.4% f1-score. Similarly, the GB algorithm had the best balanced accuracy of 72.7% and an f1-score of 69.7% on the 15 s time frame. However, the L. SVM had the second best performance, with 71.4% balanced accuracy and an f1-score of 63.3%. Table 10 shows the performance of the other learning algorithms.  
Thus, we evaluated the GB, DT, and L. SVM algorithms using the CI pipeline for model evaluation. In the end, the CI pipeline with the GB learning algorithm had the best balanced accuracy and f1-score in both time frames. The prediction model had an average balanced accuracy of 71.7% \(\pm\) 0.035 for the 15 min time frame and 70.0% \(\pm\) 0.032 for the 15 s time frame, as shown in Table 11. In addition, the permutation feature importance for the prediction models were computed to identify which feature contributed to the prediction model. The permutation feature importance measured the prediction model error after a feature was permuted [41]. DrugElapsed, Stress, and HR were the top features in both time frames for participant 1 (Table 12).  
\begin{table}
\begin{tabular}{l c c c c c c c c c c} \hline \hline  & \multicolumn{6}{c}{**15 min Interval**} & \multicolumn{6}{c}{**15 s Interval**} \\ \hline
**Metrics** & **LR** & **DT** & **L. SVM** & **RF** & **GB** & **LR** & **DT** & **L. SVM** & **RF** & **GB** \\ \hline Bal. Acc. & 0.712 & **0.739** & 0.655 & 0.691 & **0.743** & 0.698 & 0.659 & **0.714** & 0.668 & **0.727** \\ F1 Score & 0.713 & **0.744** & 0.563 & 0.671 & **0.740** & 0.682 & 0.617 & **0.633** & 0.609 & **0.697** \\ Acc. & 0.712 & **0.739** & 0.655 & 0.691 & **0.742** & 0.697 & 0.664 & **0.731** & 0.678 & **0.731** \\ Precision & 0.704 & **0.723** & 0.757 & 0.709 & **0.744** & 0.655 & 0.645 & **0.839** & 0.684 & **0.720** \\ Recall/Sn & 0.725 & **0.771** & 0.597 & 0.642 & **0.739** & 0.715 & 0.598 & **0.519** & 0.552 & **0.679** \\ Sp & 0.699 & **0.707** & 0.712 & 0.740 & **0.746** & 0.682 & 0.720 & **0.909** & 0.784 & **0.775** \\ \hline \hline \end{tabular}
\end{table}
Table 10: Comparison of learning algorithm performance on the validation set using the CI pipeline for participant 1. DT, L. SVM, and GB learning algorithms produced the best results in 15 min and 15 s interval, as shown in bold.  
\begin{table}
\begin{tabular}{l c c c c c c c c} \hline \hline  & \multicolumn{6}{c}{**15 min Interval**} & \multicolumn{6}{c}{**15 s Interval**} \\ \hline
**Metrics** & **Initial** & **FS** & **CI** & **CI + FS** & **Initial** & **FS** & **CI** & **CI + FS** \\ \hline Bal. Acc. & 0.742 & 0.744 & **0.747** & 0.743 & 0.730 & 0.729 & **0.735** & 0.728 \\ F1 Score & 0.735 & 0.732 & **0.745** & 0.731 & 0.686 & 0.682 & **0.704** & 0.691 \\ Acc. & 0.742 & 0.744 & **0.747** & 0.743 & 0.738 & 0.738 & **0.739** & 0.734 \\ Precision & 0.749 & 0.761 & **0.746** & 0.760 & 0.764 & 0.768 & **0.739** & 0.743 \\ Recall/Sn & 0.722 & 0.706 & **0.747** & 0.713 & 0.633 & 0.624 & **0.687** & 0.656 \\ Sp & 0.761 & 0.781 & **0.746** & 0.773 & 0.827 & 0.833 & **0.784** & 0.800 \\ \hline \hline \end{tabular}
\end{table}
Table 9: Best hyperparameter configuration performance on the validation set for participant 1. CI pipeline resulted as the best pipeline for both time frames, as shown in bold.