# Dual-Branch Multimodal Fusion Network for Driver Facial Emotion Recognition
## 1 Introduction  
With the continuous improvement of living standards, people are paying more attention to their quality of life. The number of people choosing private cars as a means of transportation has significantly increased, and the number of vehicles has also increased, leading to an increased probability of traffic accidents. Based on current research [1; 2; 3], human factors emerge as the predominant cause of most traffic accidents. Simultaneously, with the rapid technological advancements, smart cars have evolved into multifunctional mobile platforms, seamlessly integrating a myriad of features and services [4]. Currently, incorporating numerous advanced functions, smart cars play a role in mitigating traffic accidents to some extent [5; 6]. Nonetheless, the extensive implementation of fully autonomous driving technology remains in its nascent stage [7; 8]. The interaction between drivers and vehicle functions remains a prerequisite for completing the entire driving process. Therefore, in this transitional phase, reducing the occurrence of traffic accidents becomes an urgent issue. This issue is closely associated with drivers. [PERSON] et al. [9] conducted a correlation analysis, demonstrating that drivers under nervous tension exhibit the most significant changes in biological behavior when encountering unpleasant events while driving. Thus, the emotional state of the driver holds substantial significance throughout the entirety of the driving process [10]. Driver emotion recognition can effectively provide an intelligent driving system with the basis to determine whether the driver is fatigued through real-time analysis, thus reducing the occurrence oftraffic accidents [11]. For instance, when driver fatigue is detected, the system can proactively provide alerts [12]. Therefore, upon detecting fluctuations in the driver's emotional state, the system model can be promptly adjusted to enhance the driving experience [13]. The emotions of drivers mainly include sadness, anger, happiness, excitement, boredom, etc. [PERSON] et al. [14] classified these emotions of drivers into three categories: happy, neutral, and anger. This paper uses the DEFE++ dataset [15] to classify the main emotions of drivers into three categories: positive emotions, neutral emotions, and negative emotions. During the driving process, variations in the driver's head pose and the diversity of environmental lighting situations significantly affect the accuracy of facial recognition. Relying solely on a convolutional neural network (CNN) for extracting single visible light image (RGB) modal facial features proves challenging for capturing subtle emotional variations. Consequently, we reconstruct RGB facial images to capture detailed contour features reflecting emotional changes, thereby accomplishing the reconstruction of facial details under normal lighting situations to enhance model robustness and generalization capabilities. Furthermore, under ample natural illumination, facial details may differ from those observed under dim infrared situations, influenced by light and shadow. Consequently, while RGB image features proficiently discern driver emotional states, emotions are more concentrated during changes in human expressions. Infrared image (IR) images can indicate regions of heightened emotional concentration or intensity, facilitating a more precise comprehension of the driver's emotional state. To achieve more comprehensive and in-depth feature extraction and to better recognize the facial microexpressions of drivers, we propose a widely used multimodal network [16]. This method can effectively integrate information from different sources and capture features more accurately. It not only improves the robustness and adaptability of the network structure but also enhances the precise recognition of the driver's state, thereby increasing driving safety. This paper focuses on the work of emotion recognition in a natural driving environment and considers the situations of dynamic drivers.