# Driving Fatigue Detection Based on the Combination of Multi-Branch 3D-CNN and Attention Mechanism
## 4 Experiments and Result Analysis
### Evaluation of the Impact of the Attention Mechanism on the Model  
To verify the improvement of the attention mechanism module on the performance of the proposed network, we modified the base 3D-SE-Net by removing the SE module and using a linear classifier to output detection results as mentioned in [13]. After obtaining a feature map of size \(8*8\) and \(28*6\) channels in the Pool2 layer, the feature map is converted into feature vectors using \(8*8\) 2D convolution. These feature vectors are output directly \begin{table}
\begin{tabular}{c c c c c} \hline
**Dataset** & **Training Object** & **Test Data** & **Method** & **Accuracy (\%)** \\ \hline NTHU-DDD & Eyes & 1135 & ResNet-50 & 97.31 \\  & & & MobileNetV3 & 97.47 \\  & & & EfficientNet & 97.72 \\  & & & 3D-SE-Net(T) & **99.02** \\  & & & 3D-SE-Net(D) & **98.53** \\  & & & ResNet-50 & 97.09 \\  & & & MobileNetV3 & 97.43 \\  & & & EfficientNet & 98.27 \\  & & & 3D-SE-Net(T) & **98.84** \\  & & & 3D-SE-Net(D) & **98.31** \\ \hline YawDD & Eyes & 2014 & ResNet-50 & 97.13 \\  & & & MobileNetV3 & 97.26 \\  & & & EfficientNet & 97.48 \\  & & & 3D-SE-Net(T) & **98.67** \\  & & & 3D-SE-Net(D) & **98.04** \\  & & & ResNet-50 & 97.32 \\  & & & MobileNetV3 & 97.47 \\  & & & EfficientNet & 97.72 \\  & & & 3D-SE-Net(T) & **98.55** \\  & & & 3D-SE-Net(D) & **97.81** \\ \hline FDF & Eyes & 5461 & ResNet-50 & 98.45 \\  & & & MobileNetV3 & 98.61 \\  & & & EfficientNet & 98.57 \\  & & & 3D-SE-Net(T) & **99.12** \\  & & & 3D-SE-Net(D) & **98.74** \\  & & & ResNet-50 & 98.32 \\  & & & MobileNetV3 & 98.57 \\  & & & EfficientNet & 98.62 \\  & & & 3D-SE-Net(T) & **99.38** \\  & & & 3D-SE-Net(D) & **98.55** \\ \hline \end{tabular}
\end{table} 
Table 1: Cont. through SVM classifier without going through the attention mechanism module. This model is named '3D-CNN' in Table 2.  
In this experiment, based on the FDF dataset, we retrained the two models on this dataset for 80 epochs each. To ensure objectivity, we used the exact same images for training. It is worth mentioning that we introduced confusion matrices to visualize the detection results and also used precision, recall and F1-score as metrics to judge the accuracy of the models. Precision refers to the proportion of positive samples judged by the classifier as positive cases, recall is the proportion of positive cases predicted to be positive to the total number of positive cases and F1-score is a measure of the classification problem that is the summed average of precision and recall, with a maximum of one and a minimum of zero. A total of 1302 images were used as the test set, including 487 'Normal' images, 283 'Talking' images, 152 'Sleepy-nodding' images and 380 'Xawning' images. The performance of these two models on the test set is shown in Figure 8.  
By analyzing the two confusion matrices in Figure 8, we find that the main diagonal region of the confusion matrix is darker, while the region outside the main diagonal is lighter. This indicates that the proposed 3D convolution operation is effective for the detection of key regions of the driver's face and that the optical flow and gradient encode the original features, bringing good results. However, in Figure 8 (Left), it is clear that the 3D-CNN has a large deficit in the classification of 'Talking' and 'Yawning'. Both behaviors are very similar and both include an act of opening the mouth. Due to the different positions of the people in the dataset, the different mouth sizes and the camera angles, it is inevitable that the two behaviors are defined with a high degree of error. The 3D-CNN achieves good results for the two behaviors 'Normal' and 'Sleepy-nodding'. With the addition of the attention mechanism SE module (3D-SE-Net), the recognition rate of the two driving behaviors 'Normal' and 'Sleepy-nodding' reached almost 100%, as shown in Figure 8 (Right). The recognition of the 'Talking' and 'Yawning' behaviors is also a significant improvement over the original 3D-CNN.