# Dual-Branch Multimodal Fusion Network for Driver Facial Emotion Recognition
## 4 Experiment
### Discussion  
The DMFNet model proposed in this paper processes the video data through sampling, face detection, and other preprocessing steps, then trains a backbone network to classify the emotions of drivers on the road. Our proposed network model is a novel approach that achieves good recognition accuracy with multimodal data. As shown in Tables 1-4 and Table 8, the experimental results of DMFNet demonstrate that the model achieves a high accuracy rate. Among the various configurations, the model using Resnet 101 as the backbone network performed the best, with an accuracy rate of 70.62%, a precision of 69.89%, a recall of 69.35%, and a maximum F1-score of 69.62%. Furthermore, we analyzed the performance of the DMFNet model in predicting three types of emotions. As illustrated in Figure 5, the model accurately recognizes positive emotions, though there is some deficiency in distinguishing between neutral and negative emotions. Despite the discrepancies in recognizing the three types of emotions, our proposed model still outperforms the unimodal baseline model, proving the effectiveness of the multimodal network. Since RGB images contain more information compared to IR images, we applied detail enhancement and high-frequency noise filtering to the RGB images to improve the image information. We also conducted both individual and cross-testing of the backbone network, and the results indicate that the Resnet network is highly effective. In \begin{table}
\begin{tabular}{c c c c c c c c c c c} \hline \hline \multicolumn{2}{c}{**Method**} & \multicolumn{1}{c}{**Encoder**} & \multicolumn{1}{c}{**Accuracy**} & \multicolumn{1}{c}{**F1-Score**} & \multicolumn{1}{c}{**Accuracy**} & \multicolumn{1}{c}{**F1-Score**} & \multicolumn{1}{c}{**Accuracy**} & \multicolumn{1}{c}{**F1-Score**} \\ \hline
**Dual-branch** & **URNet** & & & & & & & & & \\ \hline \multirow{9}{*}{} & Resnet 18 & 0.5185 & 0.4951 & - & - & - & - & - & - \\  & Resnet 34 & 0.5112 & 0.4944 & - & - & - & - & - & - \\  & Resnet 50 & 0.4938 & 0.4868 & - & - & - & - & - & - \\  & Resnet 101 & 0.5037 & 0.4967 & - & - & - & - & - & - \\ \hline \multirow{9}{*}{} & Multiscale & Scale = 1 & Scale = 2 & - & Scale = 3 & - & Scale = 4 \\  & Resnet 18 & 0.5111 & 0.5054 & 0.5086 & 0.4962 & 0.516 & 0.4987 & 0.5407 & 0.5309 \\  & Resnet 34 & 0.5 & 0.4799 & 0.5606 & 0.5546 & 0.5303 & 0.5324 & 0.5432 & 0.5367 \\  & Resnet 50 & 0.5259 & 0.5097 & 0.5086 & 0.4921 & 0.5506 & 0.5467 & 0.5086 & 0.4943 \\  & Resnet 101 & 0.5037 & 0.4897 & 0.5432 & 0.5345 & 0.5588 & 0.5502 & 0.5062 & 0.4898 \\ \hline \multirow{9}{*}{} & Resnet 18 & 0.5177 & 0.4766 & - & - & - & - & - & - \\  & Resnet 34 & 0.5303 & 0.5178 & - & - & - & - & - & - \\  & Resnet 50 & 0.5631 & 0.5532 & - & - & - & - & - & - \\  & Resnet 101 & 0.4924 & 0.4786 & - & - & - & - & - & - \\ \hline \multirow{9}{*}{} & Resnet 18 & 0.5852 & 0.5765 & 0.6741 & 0.6543 & 0.5253 & 0.5137 & 0.6938 & 0.6739 \\  & Resnet 34 & 0.6272 & 0.6032 & 0.6222 & 0.5969 & 0.6198 & 0.6022 & 0.6074 & 0.597 \\ \cline{1-1}  & Resnet 50 & 0.6222 & 0.6123 & 0.6099 & 0.6543 & 0.6444 & 0.6431 & 0.6395 & 0.6328 \\ \cline{1-1}  & Resnet 101 & 0.6519 & 0.6234 & 0.6815 & 0.6778 & 0.7062 & 0.6896 & 0.637 & 0.6191 \\ \hline \hline \end{tabular} Note: The symbol indicates that the network exists in this method.  
\end{table} 
Table 7: Results of ablation studies on URNet and dual-branch structures.