# MCF-Net: Fusion Network of Facial and Scene Features for Expression Recognition in the Wild
## 1 Introduction
In order to extract more fine-grained facial expression features, current methods generally use manual and mechanical segmentation to crop the face image into a number of regions for extracting local features of facial regions [22; 23]. These methods need a lot of experiments to determine the number of blocks, which are inefficient and lead to a large increase in the number of parameters. Furthermore, typical convolutional networks apply all convolutional filters on the entire image. However, the expression we want to classify or detect is surrounded by background pixels, and the necessary features can be extracted using only a few operations. This spatially sparse characteristic in the images will also impair the effectiveness of existing methods. To deal with the above problems, we propose a novel deep learning model with the two-stream structure named Multi-cues Fusion Net for FER. Inspired by the fact that both facial and scene information is important for FER in the wild [24], our model constructs two networks for the extraction of multi-cue features. In the face coding stream, the facial images are first convoluted and max-pooled. To further study the effect of different encoding network architectures, ResNet-18 [15] is adopted to replace the shallow convolution and pooling layers of the face coding stream. Then, the Sparse Mask Attention Learning (SMAL) module [25] is introduced to extract facial expression features. SMAL can adaptively learn the corresponding key mask region, which defines the vital spatial positions to be processed by convolutions. Due to the fact that the occlusion and pose variation problems lead to a significant change in facial appearance at the spatial level, the Multi-scale Attention module [26] is applied to extract features with different receptive fields for increasing the robustness of our method to deal with the diversity of global features. In addition, channel attention [27] is introduced to make the network pay attention to more discriminative and significant features under the channel dimension. The face coding network can extract more comprehensive and key features of facial expression and eliminate the interference of useless information, so as to enhance the feature extraction ability of the network. Although the facial region is the most informative representation of human emotion, scene information also plays an important role in the understanding of the perceived emotion, especially when the emotions on the face are expressed weakly or indistinguishable. Consequently, in the scene coding stream, the scene images without the facial region are input into the context coding module for collecting contextual cues other than facial expressions. Here, the Relational Attention module [28] is utilized to capture the similarity between the facial features and contextual features of non-facial regions to model the hidden relationship. Finally, the facial expression features and context features are combined by the adaptive fusion network for classification. The contributions of this paper can be summarized as follows:  
1. A novel deep learning model with a two-stream structure, named MCF-Net, is proposed for facial expression recognition. 2. In the face coding stream, the Sparse Mask Attention Learning module can adaptively generate the key sparse mask, which avoids manual and mechanical segmentation to extract facial features. Furthermore, the Multi-scale Attention module is employed to extract multi-scale global features at a fine-grained level and richer channel information. 3. In the scene coding stream, the Relational Attention module captures more important contextual information about non-facial regions and establishes the connection between the facial and contextual features, so as to guide the network to focus on more meaningful regions in the real-world scene. 4. A large number of experiments are carried out on two standard FER datasets to evaluate the effectiveness and feasibility of our proposed model.