# Dual-Branch Multimodal Fusion Network for Driver Facial Emotion Recognition
## 3 Method
### Single-Modal Feature Extraction
#### 3.2.1 URNet  
URNet decomposes images into different resolutions using Spatial Domain Decomposition (SDD), enhances details of each resolution using Detail Enhancement Block (DEB), and finally reconstructs processed images of different scales using Reconstruction (RC). The structure of URNet is shown in Figure 1b. Spatial Domain Decomposition (SDD): The original image \(x\in R^{H\times W\times 3}\) contains a large amount of high-frequency noise, which can blur out the edges and details of the drivers' face. Therefore, we used Gaussian blur to smooth the image, suppress noise, and reduce aliasing effects. \[DCG(x)=D(FCB(x)) \tag{1}\]  
where \(FCB\) represents a \(5\times 5\) convolution operation for Gaussian filtering, \(D\) represents downsampling. Driver facial microexpressions can exhibit different features and structures at different scales in natural situations, capturing rich information in images, including details, edges, textures, etc. Inspired by the Feature Pyramid Network (FPN) [62], the processed smooth image was downsampled to achieve multiscale. \[x_{i+1}=DCG(x_{i}) \tag{2}\]  
Compared with the original image, multiscale images have a reduced size of \(1/2\) and lower resolution. The multiscale image contains global structure and rough contour features, while driver facial microexpressions extraction requires more detailed features. Therefore, the original image was processed by difference with the multiscale image. The size of the processed multiscale image was inconsistent with the original size, and it is necessary to upsample the multiscale image to restore it to the same level of original size. \[L_{i}=x_{i}-U(DCG(x_{i})) \tag{3}\]  
where \(U\) represents upsampling. The upsampling operation uses bilinear interpolation. Through the above operations, effective image decomposition can be achieved in the spatial domain. Detail Enhancement Block (DEB): The multiscale construction of SDD focuses more on global information, while the microexpression features of the driver's face are mainly reflected in local features. Thus, DEB was used to achieve Contour Enhancement (CE) and High-frequency Filtering (HF) for multiscale images. The structure of DEB is shown in Figure 2.  
Contour Enhancement (CE) focuses on both global information and local information. CE uses residual connections to preserve global information from top to bottom. Use the Sobel operator and Laplacian operator respectively to preserve locally important information. Figure 2: The architecture of the Detail Enhancement Block (DEB). (**a,b**) represent the convolutional layer. (**c**) Residual block. (**d**) Operator 1 uses the Sobel operator in the horizontal and vertical directions, while Operator 2 uses the Laplace operator. To retain both global and local information and enhance the network's representational capacity for input features, this study employs residual connections. The first residual block increases the number of channels to capture more complex features and patterns, thereby enhancing the network's representational capacity for input features and improving model performance and generalization ability. The second residual block adjusts the dimensions back to the original 3 channels to match the target, resulting in better model fitting and output results. Capturing global information of driver facial expressions has been proven effective in natural situations for low-level visual tasks, such as emotion recognition in CDERNet [53]. By utilizing residual connections, the model can enhance the extraction and processing capabilities for complex features while preserving the original feature information, thereby improving the accuracy and robustness of driver facial expression recognition. The local information of driver facial expressions mainly concentrates on edge features. To effectively detect these edge features, this study employs edge detection operators, as illustrated in Figure 2d.