# Dual-Branch Multimodal Fusion Network for Driver Facial Emotion Recognition
## 4 Experiment  
### Dataset and Implementation Details  
Dataset: This article uses the DEFE++ dataset [15] to validate the effectiveness of DMFNet. This dataset and the DEFE dataset [14] were both collected by Chongqing University. DEFE++ was further evaluated and screened based on the DEFE dataset, combined with facial expression records and medical data. The DEFE++ dataset contains facial expression records and related medical data from 40 participants while driving. Score their emotional responses based on the same driving scenarios. This dataset shares the same natural driving emotions as the DEFE dataset. At present, DEFE and DEFE++ are the only publicly available datasets of driver facial expressions, which can be widely applied in fields such as driving emotion recognition, facial action recognition, and driving state estimation.  
This study investigates the recognition of drivers' facial expressions in dynamic natural scenarios, categorizing the emotional states of the dataset into three categories: positive, neutral, and negative, as illustrated in Figure 3. According to practical requirements, drivers may experience personnel changes occasionally due to work schedules. It is necessary to directly apply models trained on known individuals to another group of unknown individuals for application. Therefore, the dataset was divided into the training set \(P\) and the testing set \(T\). For dataset partitioning, it was necessary to meet the following requirements: \(P\cap T=\varnothing\), 25 drivers were selected for the training set, and 15 testers were selected for the training set. The dataset image is shown in Figure 3.  
We used [PERSON] et al. [53] to analyze the data generation process, which includes three steps: sampling, face detection, and face alignment. Firstly, in the sampling stage, complete emotional data of 40 drivers were selected from the original dataset. Each video had a duration of 30 s and a frame rate of 24 frames per second. To avoid invalid samples (such as images with strong emotional interference), we manually sampled from the video, selecting a total of 9 images for each driver's emotion. Secondly, in the face detection stage, open-source libraries were employed to detect facial regions, followed by background removal from the images and adjustment of the facial image resolution to 224 \(\times\) 224. Finally, in the facial alignment stage, by detecting 68 facial feature points, an affine transformation was used to align the drivers' facial image to restore the geometric structure of the face. This experiment requires RGB-IR image pairs, and the IR dataset was processed the same way as the RGB images. Finally, 2160 images (1080 RGB images and 1080 IR images) were captured from the video. These steps provide preparation for model training and testing.  
Implementation Details: During the training process, we applied data augmentation methods, such as horizontal rotation and random flipping, to all training and testing images and normalized the data. We set the batch size to 8, and the optimizer used the Adam algorithm with an initial learning rate of 0.0001. We used PyTorch 1.9.0 as a deep learning framework and trained DMFNet for 30 epochs on a single RTX 2080 Ti GPU.  
Evaluation: We utilized accuracy and F1-score to evaluate the model. Accuracy measures the proportion of correct classifications made by the classification model across the entire dataset. Meanwhile, the F1-score is commonly used to assess the performance  
Figure 3: The dataset used in this article shows (**a**) neutral, (**b**) positive, and (**c**) negative emotions.  
of classification tasks. F1-score comprehensively evaluates the model's performance by combining its precision and recall.  
\[Accuracy=\frac{Correct}{Total} \tag{16}\]  
\[F_{1}=\frac{2\times\Pr\times\Re}{\Pr+\Re} \tag{17}\]  
where _Correct_ represents the correct number. _Total_ represents the total number. Pr represents model precision. Re represents model recall.