## 3 Results
### Wearing-Off Prediction Model
#### 3.5.2 Participant 2 Prediction Model  
The same process was conducted for participant 2's prediction model. First, we compared the performance of each pipeline on the validation set in both time frames. The best balanced accuracy for both time frame was the initial pipeline. However, upon further testing with the pipelines, and due to the lack of wearing-off reports from participant 2, we opted to use the CI pipeline to handle the class imbalance. The 15 min time frame had a balanced accuracy of 75.6%, while the 15 s time frame had a balanced accuracy of 74.8%. The other pipelines' performances on the validation set are presented in Table 13.  
Second, the learning algorithms were compared using the CI pipeline for both time frames. For the 15 min time frame, the LR learning algorithm had the best balanced \begin{table}
\begin{tabular}{l c c c c c c c} \hline \hline  & \multicolumn{4}{c}{**15 min Interval**} & \multicolumn{4}{c}{**15 s Interval**} \\ \hline
**Metrics** & **High** & **High** & **High** & **High** & **High** & **High** \\ \hline Bal. Acc. & 0.770 & 0.763 & **0.756** & 0.758 & 0.745 & 0.734 & **0.748** & 0.754 \\ F1 Score & 0.265 & 0.233 & **0.304** & 0.235 & 0.242 & 0.174 & **0.183** & 0.183 \\ Acc. & 0.658 & 0.597 & **0.573** & 0.562 & 0.776 & 0.711 & **0.567** & 0.555 \\ Precision & 0.273 & 0.191 & **0.275** & 0.182 & 0.236 & 0.132 & **0.147** & 0.147 \\ Recall/Sn & 0.889 & 0.941 & **0.953** & 0.967 & 0.713 & 0.758 & **0.938** & 0.964 \\ Sp & 0.650 & 0.585 & **0.560** & 0.548 & 0.778 & 0.710 & **0.558** & 0.544 \\ \hline \hline \end{tabular}
\end{table} 
Table 13: Best hyperparameter configuration performance on the validation set for participant 2. CI pipeline was the chosen pipeline, as shown in bold. \begin{table}
\begin{tabular}{l c c} \hline \hline  & **15 min** & **15 s** \\ \hline
**Features** & **Feature Importance** & **Feature** & **Feature Importance** \\ \hline DrugElapsed & 0.190 \(\pm\) 0.018 & DrugElapsed & 0.168 \(\pm\) 0.002 \\ Stress & 0.023 \(\pm\) 0.009 & Stress & 0.048 \(\pm\) 0.002 \\ HR & 0.012 \(\pm\) 0.009 & HR & 0.009 \(\pm\) 0.001 \\ Awake & 0.008 \(\pm\) 0.004 & Steps & 0.007 \(\pm\) 0.001 \\ Steps & 0.006 \(\pm\) 0.003 & Deep & 0.002 \(\pm\) 0.000 \\ SleepEfficiency & 0.004 \(\pm\) 0.002 & Awake & 0.000 \(\pm\) 0.000 \\ Total & 0.003 \(\pm\) 0.005 & Total & 0.000 \(\pm\) 0.000 \\ NonREMTotal & 0.002 \(\pm\) 0.001 & NonREMPercentage & \(-\)0.000 \(\pm\) 0.000 \\ Light & \(-\)0.000 \(\pm\) 0.002 & NonREMTotal & \(-\)0.001 \(\pm\) 0.000 \\ Deep & \(-\)0.002 \(\pm\) 0.003 & SleepEfficiency & \(-\)0.001 \(\pm\) 0.000 \\ REM & \(-\)0.004 \(\pm\) 0.003 & Light & \(-\)0.002 \(\pm\) 0.000 \\ NonREMPercentage & \(-\)0.008 \(\pm\) 0.002 & REM & \(-\)0.002 \(\pm\) 0.000 \\ \hline \hline \end{tabular}
\end{table} 
Table 12: The permutation feature importance for the best models for participant 1 using the CI pipeline with GB. Values were balanced accuracy error when a feature was permuted. The larger the error value, the more important the feature was for the model. \begin{table}
\begin{tabular}{l c c c c c c c} \hline \hline  & \multicolumn{4}{c}{**15 min Interval**} & \multicolumn{4}{c}{**15 s Interval**} \\ \hline
**Metrics** & **High** & **High** & **High** & **High** & **High** & **High** & **High** \\ \hline Bal. Acc. & 0.770 & 0.763 & **0.756** & 0.758 & 0.745 & 0.734 & **0.748** & 0.754 \\ F1 Score & 0.265 & 0.233 & **0.304** & 0.235 & 0.242 & 0.174 & **0.183** & 0.183 \\ Acc. & 0.658 & 0.597 & **0.573** & 0.562 & 0.776 & 0.711 & **0.567** & 0.555 \\ Precision & 0.273 & 0.191 & **0.275** & 0.182 & 0.236 & 0.132 & **0.147** & 0.147 \\ Recall/Sn & 0.889 & 0.941 & **0.953** & 0.967 & 0.713 & 0.758 & **0.938** & 0.964 \\ Sp & 0.650 & 0.585 & **0.560** & 0.548 & 0.778 & 0.710 & **0.558** & 0.544 \\ \hline \hline \end{tabular}
\end{table} 
Table 13: Best hyperparameter configuration performance on the validation set for participant 2. CI pipeline was the chosen pipeline, as shown in bold. accuracy of 73.4% on the validation set.