# Continuous Emotion Recognition with Spatiotemporal Convolutional Neural Networks
## 4 Experimental Results
### 2D CNN-LSTM Architecture  
After specializing the three pre-trained CNN architectures (VGG-11, VGG-16, and ResNet50) for emotion recognition by fine-tuning them on the RAF-DB dataset, we developed cascaded networks based on such architectures for spatiotemporal continuous emotion recognition. Two 2D CNN-LSTM models were developed: (i) one based on the VGG-16 architecture pre-trained on the VGG-Face dataset and fine-tuned on the RAF-DB dataset because such an architecture achieved the best results on the RAF-DB test set; and  
\begin{table}
\begin{tabular}{p{56.9 pt} p{56.9 pt} p{56.9 pt} p{56.9 pt}} \hline \hline
**Reference** & **Model** & **Dataset for Pre-Training** & **Convolution Block Block** & **Accuracy (\%)** \\ \hline \multirow{6}{*}{Proposed} & & & Full & 75.6 \\  & VGG-11 & ImageNet & Conv\_3\_1 & 75.9 \\  & & & Conv\_4\_1 & 75.9 \\  & & & Conv\_5\_1 & 70.3 \\ \cline{2-4}  & VGG-11-BN & ImageNet & Full & 77.8 \\ \cline{2-4}  & & & Full & 78.5 \\  & VGG-16 & VGG-Face & Conv\_3\_1 & 79.1 \\  & & & Conv\_4\_1 & **79.9** \\  & & & Conv\_5\_1 & 74.4 \\ \cline{2-4}  & VGG-16-BN & VGG-Face & Full & 78.4 \\ \cline{2-4}  & ResNet50 & VGG-Face & Full & 79.7 \\  & & & Conv\_4\_1 & 78.0 \\  & & & Conv\_5\_1 & 65.1 \\ \hline \multirow{2}{*}{[PERSON] et al. [62]} & RCNN & NA & NA & 76.5 \\  & CNN & NA & NA & 78.2 \\ \cline{2-4}  & ACNN & ImageNet & NA & 85.1 \\ \cline{2-4}  & CNN+RAN & MS-Celeb-1M & NA & 86.9 \\ \hline \hline \end{tabular}  
* The level of schooling refers to the mothers of all schoolchildren, who are used as a proxy for socio-economic status.  
\end{table}
Table 1: Results of the fine-tuning of VGG and ResNet50 models on RAF-DB. Convolution block denotes the initial level of fine-tuning of each model. Full denotes that all layers were fine-tuned.  
(ii) a second one without fine-tuning on the RAF-DB dataset. The 2D CNN sequentially provides spatial features for each input frame, and the LSTM unit models the temporal information from a single clip. Different configurations were evaluated by varying the length of sequences, the overlapping ratio, and the strategy to fuse the labels within a clip. The architectures were fine-tuned on the development set of SEWA-DB, and the mean squared error (MSE) was used as a cost function. Some other works also considered CCC as a cost function [72] since it provides information about correspondence and correlation between predictions and annotations. However, a better convergence is observed while using the MSE.  
Table 2 shows the results in terms of PCC and CCC, considering different frames rates (fps), sequence lengths (SL), overlapping ratios (OR), and fusion modes (FM). In general, both extremum and mean fusion performed well, and the best results for both valence and arousal were achieved for sequences of 64 frames at 10 fps. The VGG-16 architecture benefited from fine-tuning on the RAF-DB, and it achieved CCC values of 0.625 for valence and 0.557 for arousal on the validation set of SEWA-DB. In addition to the correlation metrics, the proposed 2D CNN-LSTM achieved an overall MAE of 0.06 (among 2D CNN-LSTM models), indicating a good correspondence between predictions and annotations. Since the best performance was obtained with post-processing steps, thus remodeling our set of predictions and annotations, MAPE was also computed to evaluate the error ratio between predictions and annotations.