# A Multimodal Recurrent Model for Driver Distraction Detection
## 3 Results
The recall and F1 scores show that averaging the probabilities of the _CNN_ classes leads to worse results. This degradation could be due to sequences in which the driver frequently swivels his head back and forth to briefly focus on the road while reading the sentences. In such cases, the model tends to classify some frames with high confidence as _normal_driving_, resulting in an incorrect average classification of _normal_driving_ for the whole sequence. For the _looking_to_passenger_ class, the recurrent networks (_CNN-LSTM_ and _CNN-LSTM-S_) achieve the best results. However, even here, the average recall rates are relatively low, 58% and 62%, which may be due to the same problem. When comparing the performance of _CNN-LSTM_ and _CNN-LSTM-S_ across classes, several similarities can be observed. However, there is a notable difference in the _reaching_behind_ class. The _CNN-LSTM_ model achieves a precision of 100%, which means that all sequences classified as _reaching_behind_ really belong to this class. On the other hand, when the additional sensor data are considered, the average precision drops to 91% (with a standard deviation of 8%), which is surprising. \begin{table}
\begin{tabular}{l c c c c} \hline \hline  & **CNN** & **CNN Sequential** & **CNN-LSTM** & **CNN-LSTM-S** \\ \hline Accuracy & 0.89 (0.01) & 0.92 (0.01) & 0.91 (0.02) & 0.91 (0.02) \\ Precision & 0.85 (0.03) & 0.92 (0.02) & 0.88 (0.03) & 0.86 (0.02) \\ Recall & 0.71 (0.03) & 0.75 (0.03) & 0.73 (0.03) & 0.77 (0.05) \\ F1 Score & 0.75 (0.03) & 0.78 (0.04) & 0.77 (0.05) & 0.79 (0.04) \\ Balanced Accuracy & 0.71 (0.03) & 0.75 (0.03) & 0.73 (0.03) & 0.77 (0.05) \\ ROC AUC & 0.94 (0.01) & 0.99 (0.00) & 0.97 (0.00) & 0.97 (0.01) \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Results of different models on the validation set for multi-class classification. Each cell represents the mean and standard deviations of the scores obtained from 5 training runs. The visualization of the confusion matrices for the different models is shown in Figure 7. It is important to note that each confusion matrix represents the results of a single model, as it is difficult to capture the performance across multiple training runs. Therefore, for each model, the confusion matrix is generated using the model from the training run with the median F1 score. Additionally, it should be emphasized that the absolute numbers in the matrices are only comparable within Figure 7b-d as a different validation set was used in Figure 7a. Despite the significant differences in model behavior indicated by the metrics, similar patterns can be seen in the confusion matrices. In each matrix, the majority of samples are correctly classified, resulting in a slightly denser diagonal line from top left to the bottom right. However, due to the class imbalance, there is an outlier in the _normal_driving_ class, which contains by far the largest number of samples. Across all models, the most frequent misclassifications occur in the _looking_to_passenger_ class, with many samples classified as _normal_driving_. As mentioned before, this is due to the constant head-turning of the drivers while reading the sentences. In addition, especially in the _CNN-LSTM_ model, there is a lot of confusion between the _infotainment_ and _drinking_ classes. Differentiating between samples where the driver is reaching for the infotainment system and when they are reaching for a water bottle or putting it down seems to be a challenge for the models. Both the _CNN Sequential_ and _CNN-LSTM-S_ models show 47 misclassifications, where _reaching_behind_ sequences are classified as _infotainment_. These two classes involve manual distractions where the driver moves his right hand, which likely contributed to the difficulty in distinguishing between them for these specific models.