# Continuous Emotion Recognition with Spatiotemporal Convolutional Neural Networks
## 2 Related Works
[46] combined linear discriminant analysis and weighted principal component analysis with VGG-Face for feature extraction and dimension reduction in a face recognition task. [PERSON] et al. [47], as part of the EmotiW challenge, fine-tuned a VGG-Face on the FER2013 dataset [15] and aggregated frame features for classifying emotions on video sequences with a linear SVM. Finally, [PERSON] et al. [48] proposed peculiar fine-tuning techniques with VGG-Face. They constrained their network to act like a VGG-Face network by transferring outputs from late layers rather than transferring weights. Recent works used 3D convolutional kernels for a spatiotemporal description of visual information. The first 3D-CNN models were developed for the action recognition task [30; 34; 38]. Three-dimensional CNNs pre-trained on action recognition datasets were then made available and transferred to affect computing research [32; 33]. [PERSON] et al. [37] combined VGG-Face and LSTM, among other CNN-RNN and 3D CNN networks, to build an ensemble network for multi-modality fusion (video+audio), which predicts seven categories of emotions. We believe no 3D CNN model has been evaluated so far for predicting continuous emotions, and only a few approaches have been proposed for discrete emotion prediction. This is mainly due to the lack of video datasets for FER, which allow exploiting the temporal dimension. To circumvent this issue, [PERSON] and [PERSON] [49] developed the i3D network, which can learn 3D feature representations based on 2D datasets. They inflated a 2D inception CNN to extend learned weights in 2D to a third dimension. In this way, they developed several pre-trained networks based on the same architecture with a combination of ImageNet and Kinetics datasets on images or optical flow inputs. As demonstrated by [PERSON] et al. [29], 3D CNNs for emotion recognition and depression detection [32; 33; 37] have a lower capacity to produce discriminant spatiotemporal features than i3D [49]. This is mainly because the i3D-CNN is deeper and benefits from efficient neural connections through the inception module. In this way, with inflated 2D networks, efficient 3D CNNs can be built from competitive 2D architectures. [PERSON] et al. [29] proposed a deep maximization-differentiation network (MDN) and compared this architecture with i3D-CNN and T-3D CNN [50], showing that i3D requires fewer parameters than other models and is computationally faster. Finally, [PERSON] et al. [51; 52] applied i3D networks in pain intensity estimation with ordinal regression. Their approach achieved state-of-the-art results, notably by using deep weakly supervised domain adaptation based on adversarial learning. Most of the previous studies on FER are based on the categorical representation of emotion. Still, some studies have also dealt with continuous representations, which have proven effective on both image and video datasets. Discrete models are a straightforward representation of emotion, and for instance, they do not generalize well across cultures. For example, smiles can be attributed to happiness, fearfulness, or disgust, depending on the context. On the other hand, dimensional models can distinguish emotions upon a better basis, which are levels of arousal and valence [53]. These two values, widely used in the psychology field, can assign a broader range of emotional states. Researchers have shown that low- and high-level features complement each other. Their combination could shrink the affective gap, defined as the concordance between signal properties or features and the desired output values. For instance, [PERSON] and [PERSON] [54] and [PERSON] et al. [55] built feed-forward networks combining color features, texture features (LBP), and shape features (SIFT descriptors). Other works focused on emotion recognition at the group level by studying not only facial expressions, but also body posture or context [56], as well as by exploring various physiological signals, such as electrocardiogram and respiration volume [53; 57].