# MCF-Net: Fusion Network of Facial and Scene Features for Expression Recognition in the Wild
## 2 Related Work
### FER Methods in the Wild  
Due to various restrictions and problems in the controlled environment, some deep learning-based works have been presented for facial expression recognition in the wild. Besides, techniques such as pretraining, multi-modal data, and GAN have also been applied to solve the problem of overfitting in the training process. [PERSON]. [38] presented FaceNet2 ExpNet to train an expression recognition network based on static images. [PERSON]. [39] proposed recognizing video emotions in an end-to-end manner based on a deep Visual-Audio Attention Network. [PERSON]. [40] introduced a learning-based algorithm for context-aware perceived human emotion recognition by combining three interpretations of context. [PERSON]. [41] designed an Expression Generative Adversarial Network (ExprGAN) for photo-realistic facial expression editing with controllable expression intensity. [PERSON]. [42] proposed a feature separation model exchange-GAN for the FER task, which can realize the separation of expression-related features and expression-independent features. Although fine-tuning and GAN can partially alleviate the issue of small datasets, the performance is still relatively low as the deep features likely contain redundant information. More recently, some FER models based on CNN leveraged auxiliary modules or the combination of multiple networks to further enhance the feature extraction ability of the network. Ref. [43] represented a fusion framework for static facial expression recognition in the wild by varying multiple network architectures, input normalization, weight initialization, and several learning strategies. Ref. [44] introduced a Meta Auxiliary Learning method (MAL) that automatically selected highly related facial expression samples by learning adaptive weights for the training facial expression samples in a meta learning manner. Ref. [45] designed a local-feature extractor and a channel-spatial modulator to improve the robustness of a lightweight network for FER, in which the depthwise convolution was employed for local and global-salient facial features. Ref. [46] used a transformer-based cross-fusion paradigm that enabled effective collaboration of facial landmarks and direct image features to maximize proper attention to salient facial regions. Ref. [47] adopted Facial Landmark Detection (FLD) as the auxiliary task and explored new multi-task learning strategies for FER. Ref. [48] proposed an Adaptive Correlation Loss to guide the network towards generating embedded feature vectors with high correlation for within-class samples and less correlation for between-class samples. The above methods usually improve the performance of the models by increasing the network width or deepening the network depth, which leads to a large number of parameters and high computational complexity. Furthermore, the above methods only take the facial regions into consideration, which may weaken their performances once the human facial regions in images are partly occluded or blurred. Unfortunately, occlusion caused by pose variations or other objects and blur induced by human movement are common problems for the FER task in the wild. Hence, in order to extract more comprehensive and compensatory features, various kinds of cues, such as hands, body posture, and interaction with others, should be exploited. expressions and contextual cues into account, the backbone of our MCF-Net is divided into two streams. The two-stream network architecture is composed of a face coding stream and a scene coding stream, which are used to extract facial expression information and context information, respectively. The images of facial expressions detected by the Dlib algorithm [49] are used as the input of the face coding stream. The face coding stream consists of 3 \(\times\) 3 convolutions with batch Normalization (BN) and the ReLU activation function, max-pooling with the stride of 2, the SMAL module, and the MSA module.