## 4 Self-Attention Network for Portrait FER
### Swin Transformer Module  
Swin Transformer: The Swin Transformer [86] module, a variant of the ViT architecture, is tailored to meet the specific demands of portrait Facial Expression Recognition (FER) (Refer to Figure 15). Its objective is to amalgamate local image features, thereby capturing the subtle details and expressions pivotal for precise emotion recognition in portraits. The operation of the Swin Transformer module commences by segmenting the input image into diminutive patches, termed tokens. A hallmark of the Swin Transformer is its shifting window mechanism, which dynamically alters the confines of these tokens throughout the learning phase. This innovation enables the model to assimilate diverse information from various facial regions. By focusing on local and global features, the Swin  
Figure 14: Overview of the architecture of the PST module. The blue dash box shows the location where PST is added in a traditional FER model. A pre-trained backbone model is implemented in the red dash rectangle to generate global features for the following feature fusion with local features output by the Swin Transformer. After feature fusion and a fully connected layer, the 3 Demo will be applied to generate the final results. Figure 15: Swin Transformer structure (the right image is borrowed from article [86]). Transformer adeptly seizes the intricate details and the overarching context of the face, yielding distinctive and resilient features for portrait FER. After the Swin Transformer module, the collated features are channeled through a fully connected layer. This layer condenses the feature vector to a more tractable dimensionality, typically 512. The final step involves the generation of output labels that correspond to the emotions depicted by the portrait faces. ```
0: mini-batch input \(X_{in}\), Dimension of \(X_{in}\) is \(\triangleright\) B\(\times\)3\(\times\)W\(\times\)H
0: Baseline model \(Res50\)
0: Swin Transformer model \(ST\), initialized with pre-trained weights [86]
0: 3 DEmo Module: 3\(DEmo\), initialized with hyper-parameters [52]
1:\(X_{g}=Res50(X_{in})\)
2:\(X_{l}=ST(X_{in})\)\(\triangleright\) B\(\times\)1\(\times\)768
3:\(X_{g}=avgpool(X_{g})\)\(\triangleright\) B\(\times\)1\(\times\)512
4:\(X_{g}=(X_{g}).reshape()\)\(\triangleright\) B\(\times\)1\(\times\)512
5:if The following codes are in the Global-Local Fusion Module
6:\(X_{out}=Concat(X_{l},X_{g})\)\(\triangleright\) B\(\times\)1\(\times\)(512+768)
7:\(X_{out}=Linear(X_{out})\)\(\triangleright\) B\(\times\)1\(\times\)512
8:\(X_{out}=3 DEmo(X_{out})\)\(\triangleright\) B\(\times\)1\(\times\)8
9:return\(X_{out}\)
```  
**Algorithm 1** Forward Algorithm of PST  
The Swin Transformer architecture undergoes end-to-end training, employing a specialized loss function designed for FER tasks. This optimization procedure homes the model's capacity to discern and categorize emotions accurately in portrait imagery. Algorithm 2 is a simplified pseudo-code for the Portrait Swin Transformer (PST) process:  
```
0: mini-batch input \(X_{in}\), Dimension of \(X_{in}\) is \(\triangleright\) B\(\times\)3\(\times\)W\(\times\)H
0: Swin Transformer Blocks Stages SwinStages, initialized with hyper-parameters [86]
0: Linear Embedding Layer \(LE\), initialized with hyper-parameters [86]
1:\(patches=ParattionImage(X_{in})\)\(\triangleright\) B\(\times\)C\(\times\)W/4\(\times\)4\(\times\)4
2:\(tokens=LE(patches)\)\(\triangleright\) B\(\times\)C\(\times\)(W\({}^{*}\)H)
3:for SwinBlock in SwinStagesdo
4:\(tokens=SwinBlock(tokens)\)
5:if SwinBlock not the last then
6:\(tokens=MergeTokens(tokens)\)
7:endif
8:endfor
9:\(X_{out}=Linear(tokens)\)\(\triangleright\) B\(\times\)1\(\times\)(512)
10:return\(X_{out}\)
```  
**Algorithm 2** Forward Algorithm of ST(0)  
By harnessing the strengths of transformers and the specialized Swin Transformer module, neural networks can learn context and understand through the use of sequential data analysis.