## 3 Results
### Performance of Drowsy State Classification Using Ensemble Machine Learning Algorithms  
The performance values of the classification of alert and drowsy state (alert vs. slightly drowsy, alert vs. moderately drowsy or more) in the case of using full hybrid measures are listed in Table 3. Two ensemble algorithms (MVC, RF) achieved higher values of all performance compared to the DT algorithm. The RF algorithm achieved especially higher values of detection accuracy, precision, and F1 compared to the MVC algorithm when classifying the alert and the slightly drowsy state; its detection accuracy was 82.4%. The MVC algorithm achieved higher values of detection accuracy, precision, and F1 compared to the RF algorithm when classifying the alert and the moderately (or more than moderately) drowsy states; its detection accuracy was 95.4%. The algorithms' performance values in the case of excluding physiological indices are listed in Table 4. The RF algorithm achieved higher values of detection accuracy, precision, recall and F1 compared to the MVC algorithm in all cases. The RF algorithm achieved values of 78.7% and 89.8% detection accuracy in the case of classifying the alert vs. slightly drowsy, and the alert vs. moderately drowsy states, respectively. When physiological measures were excluded, detection accuracy decreased by 3.7%-9.0% compared to the case in which all measures were used. Figure 5: Changes in the mean of the drowsiness level, N = 16, Error bars represent SEM [20]. \begin{table}
\begin{tabular}{c c c c c c} \hline \multirow{2}{*}{**Classifiers**} & \multirow{2}{*}{**Cases (Alert State vs.)**} & \multicolumn{4}{c}{**Performance of Classification**} \\ \cline{3-6}  & & **Acc. (\%)** & **Pre. (\%)** & **Rec. (\%)** & **F1 (\%)** \\ \hline \multirow{3}{*}{DT} & Slightly Drowsy & 73.5 & 77.3 & 67.6 & 71.9 \\  & Moderately Drowsy (or more) & 90.4 & 90.4 & 88.3 & 89.3 \\  & Slightly Drowsy & 82.0 & 84.5 & 78.6 & 81.4 \\  & Moderately Drowsy (or more) & 95.4 & 97.1 & 92.9 & 94.9 \\ \multirow{3}{*}{RF} & Slightly Drowsy & 82.4 & 81.6 & 84.1 & 82.8 \\  & Moderately Drowsy (or more) & 93.7 & 93.0 & 93.5 & 93.2 \\ \hline \end{tabular}
\end{table} 
Table 3: Classification results in the case of using full hybrid measures. The curves of receiver operating characteristic (ROC) and the value of the area under curve (AUC) for each condition are shown in Figure 6. The ROC curve is created by plotting the true positive rate against the false positive rate at various threshold settings [32]. The ROC of two ensemble algorithms (RF, MVC) show points in the upper-left corner compared to that of the DT algorithm in all cases. In addition, the ROC of the RF algorithm show points in the upper-left corner compared to that of the MVC algorithm in the case of excluding the physiological indices. The details of the selected features and the priority of the features used by the RF algorithms are listed in Table 5. When full hybrid measures were used, as shown in Table 5, PERCLOS was selected as the most important feature. RRI and THW was then selected as the second and the third important feature, respectively. On the other hand, when physiological indices were excluded, PERCLOS was still selected as the most important feature (Table 5). The y and x coordinate of the centroid was then selected as the second and the third important feature, respectively.