# Continuous Emotion Recognition with Spatiotemporal Convolutional Neural Networks
## 6 Conclusions  
This paper presented two CNN architectures for continuous emotion prediction in the wild. The first architecture is a combination of a fine-tuned VGG-16 CNN and an LSTM unit. This architecture achieved state-of-the-art results on the SEWA-DB dataset, producing CCC values of 0.625 and 0.557 for valence and arousal, respectively. The second architecture is based on the concept of inflation, which transfers knowledge from pre-trained 2D CNN models into a 3D one to model temporal features. The best proposed i3D CNN architecture achieved CCC values of 0.304 and 0.326 for valence and arousal prediction, respectively. These values are far below those achieved by the 2D CNN-LSTM. Due to the high number of parameters of i3D CNNs (barely three times greater than 2D CNNs), fine-tuning and hyperparameter tuning of such architectures require a substantial computational effort \begin{table}
\begin{tabular}{l l c c c c} \hline \hline  & & \multicolumn{2}{c}{**Valence**} & \multicolumn{2}{c}{**Arousal**} \\ \cline{3-6}
**Reference** & **Model** & **PCC\(\uparrow\)** & **CCC\(\uparrow\)** & **PCC\(\uparrow\)** & **CCC\(\uparrow\)** \\ \hline \multirow{2}{*}{Proposed} & 2D-CNN-LSTM 1 ** & **0.631** & **0.625** & **0.558** & **0.557** \\  & i3D-CNN 2  & 0.346 & 0.304 & 0.382 & 0.326 \\ \cline{2-6}  & SVR & 0.321 & 0.312 & 0.182 & 0.202 \\  & RF & 0.268 & 0.207 & 0.181 & 0.123 \\  & LSTM & 0.322 & 0.281 & 0.173 & 0.115 \\  & ResNet18 (RMSE) & 0.290 & 0.270 & 0.130 & 0.110 \\  & ResNet18 (CCC) & 0.350 & 0.350 & 0.350 & 0.290 \\ \hline \multirow{2}{*}{
\begin{tabular}{l} [PERSON] et al. [75]\({}^{\star}\) \\ [PERSON] et al. [76]\({}^{\star}\) \\ \end{tabular} } & ST-GCN & NA & 0.540 & NA & 0.581 \\  & DenseNet-style CNN & NA & 0.580 & NA & 0.594 \\ \hline \hline \end{tabular}  
* VGG-16 fine-tuned with RAF-DR.2  
\end{table} 
Table 7: Comparison of the best results achieved by the proposed models and the state-of-the-art for the SEWA-DB dataset. The best results for valence and arousal are in boldface. and massive datasets. Unfortunately, facial expression datasets for continuous emotion recognition are relatively small to accomplish such a task. We also showed that a double transfer learning strategy over VGG and ResNet architectures with ImageNet and RAF-DB datasets can improve the accuracy of the baseline models. However, subjects mostly faced the camera with a relatively clear view of the whole face in this work. To some extent, this could imply some bias in the results when presenting diverse real-world scenarios. Moreover, the complexity of the i3D CNN architecture could, at this time, be a drag for live applications. Finally, to the best of our knowledge, it was the first time that 3D CNNs were used in regression applications for predicting valence and arousal values for emotion recognition. There are some promising directions to expand the approaches proposed in this paper. For example, one could take advantage of the development of vast and complex cross-cultural datasets, such as the Aff-Wild dataset, to exploit occlusion cases, pose variations, or even scene breaks. We believe that i3D CNN architectures can deal with these specific cases, as they can learn discriminant spatiotemporal features that may improve separability. Furthermore, the recent Aff-Wild2 dataset, the largest existing in-the-wild video dataset, could also be used for pre-training the 2D CNNs on the same task (valence-arousal estimation) instead of using the RAF-DB dataset, which contains annotations for discrete emotion classification. Finally, the Aff-Wild2 dataset could also be used to train and evaluate the proposed 2D CNN-LTSM and i3D CNN architectures. Finally, we showed a peculiar and flexible way of fine-tuning inflated CNNs; maybe this strategy could also be transferred to other applications, such as object and action recognition on video sequences. Conceptualization, T.T., E.G. and A.L.K. ; methodology, T.T. ; writing--original draft preparation, T.T. ; writing--review and editing, A.L.K. ; supervision, E.G. and A.L.K.