# Attention-Based Spatiotemporal-Aware Network for Fine-Grained Visual Recognition
## 4 Experiment and Analysis  
### Datasets and Preprocessing  
Considering the frame rate, emotion type and use frequency of each dataset, we use SMIC [31] and CASME II [32] for the experiments. SMIC contains 164 micro-expression samples of 16 subjects, which are divided into positive, negative and surprised. Among them, the positive emotion is happiness, and negative emotions include sadness, fear and disgust; CASME II contains 255 video samples from 26 participants, which are divided into seven emotions: happiness, surprise, repression, disgust, sadness, fear and others. Table 2 summarizes the characteristics of the two datasets.  
In CASME II, the number of samples of sadness and fear is less than 10. In order to avoid the imbalance of data distribution affecting the recognition effect, we ignore these two emotions in the experiment. Through image operations including mirror, random rotation and clipping, we expand the datasets 20 times. The distribution of original and expanded samples is shown in Table 3.  
\begin{table}
\begin{tabular}{c c} \hline
**Network Layer** & **Our Network** \\ \hline Layer 1 & Conv0\_1, BN1, ReLU, MaxPool \\ Layer 2 & Conv1\_1, BN1, ReLU, Conv1\_2, BN2 \\  & Conv1\_3, BN1, ReLU, Conv1\_4, BN2 \\ Layer 3 & Conv2\_1, BN1, ReLU, Conv2\_2, BN2 \\  & Conv2\_3, BN3 \\  & Conv2\_4, BN1, ReLU, Conv2\_5, BN2 \\ Layer 4 & Conv3\_1, BN1, ReLU, Conv3\_2, BN2 \\  & Conv3\_3, BN3 \\  & Conv3\_4, BN1, ReLU, Conv3\_5, BN2 \\ Layer 5 & Conv4\_1, BN1, ReLU, Conv4\_2, BN2 \\  & Conv4\_3, BN3 \\  & Conv4\_4, BN1, ReLU, Conv4\_5, BN2 \\ Layer 6 & CBAM, Dropout \\ Layer 7 & ConvLSTM, AvgPool, FC \\ \hline \end{tabular}
\end{table}
Table 1: Overall network structure.  
\begin{table}
\begin{tabular}{l c c} \hline
**Characteristics** & **SMIC** & **CASME II** \\ \hline Samples & 164 & 255 \\ Subjects & 16 & 26 \\ Ethnicities & 3 & 1 \\ FPS & 100 & 200 \\ Resolution & \(640\times 480\) & \(640\times 480\) \\ Facial Area & \(160\times 130\) & \(340\times 280\) \\ Emotion Classes & 3 & 7 \\ Objective Classes & - & 7 \\ FACS Coded & NO & YES \\ \hline \end{tabular}
\end{table}
Table 2: Characteristics of datasets.  
The number of sample frames in the micro-expression datasets is not uniform. Therefore, we extract the continuous frames from the apex to the peak and fix the length as 11 frames. We preprocess each sample sequence by using face detection and alignment, facial motion feature amplification and optical flow feature extraction, as follows:  
1. Face detection and alignment  
Firstly, we select the first frame image in each sample, and use ASM [33] to extract 68 facial key points; then, we use the Local Weighted Average [34] to transform these facial key points into matrices, and normalize and align all frames of the sample with these matrices. Finally, we locate the left eye coordinates, determine the distance between the two eyes and cut all frames of the samples.
2. Facial motion feature amplification  
We use the video action amplification network based on deep learning to amplify the features of micro-expression. According to our experiment, when the magnification is too small, the micro-expression features are not obvious enough; when the magnification is too large, it will cause too much noise interference and distort the micro-expression image. After many experiments, we determine that the amplification factor is 3.
3. Optical flow extraction  
We use LiteFlowNet to extract optical flow features from two adjacent micro-expression images in each sample. After a pair of images are processed through this network, an optical flow file (.flo) is output and visualized. Therefore, one sequence of 11 frames can be turned into an optical flow sequence of 10 frames.