# MCF-Net: Fusion Network of Facial and Scene Features for Expression Recognition in the Wild
## 2 Related Work
### Relational Attention  
The scene coding stream is mainly used to extract the contextual cue features except for facial regions. The context coding module adopts the same backbone network as used in the literature [24], which is composed of a 5-layer 3 \(\times\) 3 convolution with BN and ReLU and 4-layer max-pooling in a stride of 2. In order to construct the relationship between the face and the context, the attention mechanism is utilized in the RA module to capture the similarity between them for building the hidden relationship [28]. Specifically, the facial expression features extracted from the face coding stream and the context features extracted from the scene coding module are taken as inputs of RA. Then, the final output of the scene coding stream is obtained by multiplying the normalized attention map with the extracted context features. The attention map represents the priority of each position in the feature map. The structure of the RA module is shown in Figure 4.  
First, the feature maps \(F_{f}\) of faces extracted from the face coding stream are transformed into the feature vectors \(V_{f}\) by the global ave-pooling operation. The contextual feature map \(F_{c}\in R^{H_{c}\times W_{c}\times D_{c}}\) is a 3D tensor, while \(H_{c}\), \(W_{c}\), and \(D_{c}\) represent the height, width, and channel dimension of the feature map, respectively. \(F_{c}\) is divided into a set of \(H_{c}\times W_{c}\) vectors with \(D_{c}\) dimensions, and each vector in each cell \((i,j)\) represents the embedded features at that location. At each location \((i,j)\) of \(F_{c}\), we have vectors \(F_{c}^{(i,j)}=v_{i,j}v_{i,j}\in R^{D_{c}}\) and \(1\leq i\leq H_{c}\), \(1\leq j\leq W_{c}\). The representation of \(F_{c}\) is shown in Equation (18).  
\[F_{c}=\left\{v_{i,j}\in R^{D_{c}}\left|1\leq i\leq H_{c},1\leq j\leq W_{c}\right.\right\} \tag{18}\]  
where \(v_{i,j}\) represents the feature vector of contextual feature map \(F_{c}\) at the \((i,j)_{th}\) position. A feature matrix containing both the features of the face and context is obtained by concatenating \(V_{f}\) and \(v_{i,j}\) along the channel dimension. Then the concatenated vectors are input into a two-layer 3 \(\times\) 3 convolution to calculate the raw score values \(s_{i,j}\) related to the region. The Softmax function is then used to generate the attention graphs, as shown in  
Figure 4: The structure of Relational Attention module.  
Equation (19). Finally, the attention maps \(\alpha_{i,j}\) are weighted into the corresponding \(v_{i,j}\) to acquire the final representation vectors \(V_{\text{s}}\), as shown in Equation (20).  
\[\alpha_{i,j}=\frac{\exp(s_{i,j})}{\sum_{a}\sum_{b}\exp(S_{a,b})} \tag{19}\] \[V_{\text{s}}=\sum_{i}\sum_{j}\left(\alpha_{i,j}\odot v_{i,j}\right) \tag{20}\]  
where \(\odot\) represents the element-wise multiplication operation. Through the above process, the RA module can guide the network to pay attention to those \(v_{i,j}\) that represent more important and discriminative regions. As a result, the other irrelevant areas in the image are ignored.