# Facial Expression Recognition Using Dual Path Feature Fusion and Stacked Attention
## 3 Methods  
### Overview  
As shown in Figure 2, first, the face image is segmented using the segmentation method to segment four local image regions. The face key region segmentation method can extract more local details, and the size of each region image is resized to be equal to the size of the global input image via scaling. Then, the global image and the four local image regions are input to the updated ResNet18 for feature extraction in the global path and local path, respectively. The general structure of the updated ResNet18 is shown in the dashed rectangle at the lower left corner, and a more detailed structure is shown in Figure 3. After completing the feature extraction, the global image feature map is fused with the local image feature map. After that, the fused feature maps are fed into the stacked attention module (SA). The SA module can learn features from different regions and complete the weighting of different channels by re-estimating the importance of each feature. The image segmentation method is shown in Figure 4. The details of the structure of the stacked attention module are shown in Figure 5.  
The model consists of two main parts. One is the dual path feature fusion module, which performs segmentation, feature extraction, and feature fusion on the input face image. The other is the SA module, which weights each channel of the feature map according to its contribution to the classification result. The detailed process of the first module is as follows. First, the model segments the local area of the base image, and compresses and scales the size of the local area image and the size of the base image; and then the scaled image is used as the input to the network. The dual path feature fusion network concatenates the features extracted from the four local regions with the features of the complete face image and outputs the fused feature map. The main role of the SA module is to calculate the importance of the feature maps of each layer and compress the irrelevant information. Finally, the classification of expressions is output from the linear connectivity layer. Figure 2 illustrates the DPSAN framework.  
Figure 3: The updated ResNet18. We removed the avg-pool layer and the fully connected (FC) layer to facilitate the next step of feature fusion.  
Figure 2: The framework of DPSAN consists of a dual path feature fusion module and a stacked attention module.  
DPSAN is used with one global image and four local image regions as inputs. These five parts are input to two ResNet18, and the features are extracted and represented as five sets of feature maps. Then, DPSAN merges the global feature maps and the four sets of local feature maps into one set of feature maps. The advantage of this processing is that the fused feature maps have both the global information of the facial image and local information that can be easily ignored. The SA module calculates the weights for each channel of the feature map. The weighted features contain all the key information that we are interested in that is beneficial to FER. The classification layer after the stacked attention module aims to classify the face images into their corresponding facial expression classes. The softmax loss is used to optimize the DPSAN model.  
Figure 4: Example of facial key region segmentation method. (**a**) The 42-point cropping method based on face; (**b**) Crop method based on four regions of the face; (**c**) Four-equivalent cropping method based on face image.  
Figure 5: SA consists of two channel attention networks and a \(1\times 1\) convolutional neural network conv \(1\times 1\). DPSAN accomplishes the weighting of five different regions through the channel attention mechanism.