# A Multimodal Recurrent Model for Driver Distraction Detection
## 3 Results  
Several models were trained to evaluate the potential improvement of a vision-based driver distraction detection method by incorporating a recurrent neural network and sensor data. The models used in this study were the vision-based model (referred to as _CNN_), the vision-based recurrent model (referred to as _CNN-LSTM_), and the multimodal recurrent model that processed sequences of images and sensor data (referred to as _CNN-LSTM-S_). To allow a fair comparison with the recurrent networks, an additional approach was used to obtain the _CNN_ predictions for image sequences without training a new model. This involved averaging the class probabilities from the _CNN_ for each image within a sequence, and the resulting approach was named _CNN Sequential_. Consequently, the validation of the _CNN_ was performed on individual images, while the validations for the remaining three approaches were performed on image (and sensor) sequences. Since the initial weights for each model training were chosen randomly, identical results were never obtained. To obtain statistically significant results, each model was trained five times in a row, and the results of each simulation run were averaged. The standard deviation of the corresponding scores was also reported for each result. A high variance in the scores could indicate that the models are stuck in local maxima due to the stochastic nature of the training, or are experiencing overfitting. The results of the different models on the validation set are presented in Table 2. For each model, the metrics of overall accuracy, precision (macro average), recall (macro average), F1 score (macro average), balanced accuracy, and ROC area under the curve (AUC) were computed. The _CNN_ showed an overall accuracy of 89% and a precision of 85%. However, the recall was slightly lower at 71%, indicating that the model had some difficulty in correctly assigning certain classes. This observation is also reflected in the F1 score and the balanced accuracy, which is insensitive to class imbalance, unlike the ordinary accuracy. The high ROC AUC value of 0.94 suggests that the model had a strong discriminative power across classes. Looking at the table, it can be seen that _CNN_ was the weakest in all metrics. In terms of accuracy and precision, the sequential approach with the CNN (_CNN Sequential_) achieved the highest scores of 92% each. The _CNN-LSTM-S_ model achieved the best recall with 77%, but it showed significant variation across training runs, as indicated by a standard deviation of 5%. For the F1 score and balanced accuracy, the _CNN-LSTM-S_ model also showed the highest average results, but with a relatively high standard deviation. Nevertheless, these results suggest that distractions can be detected more accurately when the temporal component is taken into account. However, determining whether the _CNN-LSTM-S_ model reliably outperforms the _CNN-LSTM_ model is challenging due to the considerable variability in scores across multiple training runs. In Table 3, the results of the individual models on the validation set are presented, this time divided by the individual classes. Notably, high standard deviations are often observed for all metrics for all classes except _normal_driving_. These variations suggest that the final trained models behave very differently and may have ended up in different local minima due to initial random weights. However, the table shows that all models show the best classification performance for the _normal_driving_ class. Notably, this class has a particularly high recall, indicating a high number of true positives and minimal misclassification of other classes as _normal_driving_. The _CNN Sequential_ model even achieves a recall of 100%, perfectly classifying all _normal_driving_ sequences. The _CNN Sequential_ model also performs impressively well for the _drinking_, _infotainment_, and _reaching_behind_ classes. However, for the class _looking_to_passenger_, the sequential approach with the _CNN_ model encounters difficulties.