# Dual-Branch Multimodal Fusion Network for Driver Facial Emotion Recognition
## 4 Experiment
### Experimental Results  
To verify the effectiveness of DMFNet, we conducted extensive experiments on the DEFE++ dataset. Due to the abundance of noise information in RGB images, we employed URNet to enhance image details and reduce noise, followed by conducting corresponding experiments. As shown in Tables 1 and 2, we selected Resnet [45], tResnet + RAM [65], Resnet + EAC [66], DERNet [53], and CDERNet [53] as baseline models for single-modal driver emotion recognition experiments on RGB images. Among these models, CDERNet demonstrated the best recognition performance. Due to the inspiration of the feature pyramid network (FPN) in the network structure of the URNet in this article, and for single-modal (RGB) driver emotion recognition, we combined it with the model of [PERSON] et al. [53] (i.e., CDERNet + FPN), taking into account the FPN structure and the multiscale features of our model (such as Multiscale = 1 and FPN = 1). Furthermore, considering the excellent performance of the Transformer in the field of image classification (only for single-modal (RGB) driver emotion recognition), we also used ViT [67] and SuiT [68] for comparison in the experiment. Based on the results in Tables 1 and 2, we can see that Svit has the best performance. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Method** & **Encoder** & **Accuracy** & **F1-Score** \\ \hline Vit [67] & Base & 0.5678 & 0.5523 \\ \hline \multirow{3}{*}{Svit [68]} & Tiny & 0.5728 & 0.5643 \\  & Small & 0.543 & 0.5388 \\  & Base & 0.504 & 0.4865 \\ \hline \multirow{3}{*}{Resnet [45]} & Resnet 18 & 0.452 & 0.442 \\  & Resnet 34 & 0.458 & 0.448 \\  & Resnet 50 & 0.465 & 0.4546 \\  & Resnet 101 & 0.4757 & 0.471 \\ \hline \multirow{3}{*}{Resnet + RAM [65]} & Resnet 18 & 0.4632 & 0.4732 \\  & Resnet 34 & 0.4314 & 0.423 \\ \cline{1-1}  & Resnet 50 & 0.442 & 0.427 \\ \cline{1-1}  & Resnet 101 & 0.474 & 0.469 \\ \hline \multirow{3}{*}{Resnet + EAC [66]} & Resnet 18 & 0.5106 & 0.5016 \\ \cline{1-1}  & Resnet 34 & 0.4971 & 0.485 \\ \cline{1-1}  & Resnet 50 & 0.4997 & 0.4903 \\ \cline{1-1}  & Resnet 101 & 0.4999 & 0.5021 \\ \hline \multirow{3}{*}{DERNet [53]} & Resnet 18 & 0.4532  
As shown in Tables 3 and 4, to meet the demands of multimodal feature fusion networks, we have designed all the aforementioned networks as dual-branch structures to achieve multimodal fusion (referred to as Dual-CDERNet and Dual-CDERNet + FPN). Additionally, we combined the original model proposed by [PERSON] et al. [53] with other multimodal fusion network models [69] (i.e., CDERNet MF) and compared it with the DMFNet model. Based on Tables 1 and 2, we can see that our proposed multimodal approach has a significant performance improvement, and our model DMFNet has the best driver emotion recognition performance. In terms of the evaluation metrics accuracy and F1-score, our DMFNet model has demonstrated superior performance compared to the original CDERNet model and other models. Detailed performance comparisons are provided in Tables 1 and 2. The qualitative results of our proposed DMFNet and several other models are depicted in Figure 4. CDERNet yielded 13 incorrect predictions out of 18 predicted emotions. Similarly, other models also exhibited errors in emotion recognition predictions. The proposed DMFNet outperforms other models for each emotion category, particularly achieving higher accuracy in identifying positive emotions.