# Continuous Emotion Recognition with Spatiotemporal Convolutional Neural Networks
## 2 Related Works  
A conventional approach for dealing with video frames is to aggregate features extracted from each frame into a clip before final emotion prediction [PERSON] et al. [27]. In addition to feature aggregation, [PERSON] et al. [26] also aggregated the mean, variance, minimum and maximum over a sequence of features, thus adding some statistical information. However, since feature aggregation cannot exploit inter-correlations between frames and cannot depict temporal dependencies, this approach has substantial limitations. To circumvent this issue, recurrent neural networks (RNN), such as LSTM or 3D CNN architectures, can integrate data series as input, provided that data are sequentially ordered, and transitions have a substantial potential for information. While LSTMs can deal with sequential data of variable length in both directions, 3D CNNs exploit textured variations from a sequence of images by extending convolutional kernels to a third dimension. Hence, 3D CNNs are well suited to encode spatiotemporal information in video-based FER applications. For example, [PERSON] et al. [30] proposed a 3D CNN model for action recognition on the UCF101 dataset, which encompasses videos classified over 101 action categories. They showed that 3D CNNs can outperform 2D CNNs on different video analysis benchmarks and bring efficient and compact features. Several recent studies [31; 32; 33; 34; 35; 36; 37] proposed approaches based on 3D CNNs for FER. Nevertheless, all of them deal with discrete emotion prediction. A popular approach for dealing with temporal sequences of frames is a cascaded network in which architectures for representation learning and discrimination are stacked on top of each other. Thus, various levels of features are learned by each block and processed by the following until the final prediction. Notably, the combination of CNNs and LSTM units has been shown to be effective in learning spatiotemporal representations [30; 38]. For instance, [PERSON] et al. [37] used a VGG-16 CNN to extract a 16-frame sequence of features and fed an LSTM unit to predict six emotion categories. They preprocessed video frames with a multi-task cascade CNN (MTCNN) to detect faces and described each video by a single 16-frame window. Similarly, [PERSON] et al. [39] used a VGG-16 CNN and an LSTM unit as part of an ensemble with a 3D-CNN and an audio network. Mainly, they used the multi-instance learning (MIL) method to create bag-of-windows for each video with a specific overlapping ratio. Each sequence was described by a single label and contributed to the overall prediction of the matching video clip. Since deep neural networks (DNNs) are highly data-dependent, there are strong limitations for designing FER systems based on DNNs, even more since FER datasets are often small and task-oriented [1]. Considering this fact, training deep models on FER datasets usually leads to overfitting. In other words, end-to-end training is not feasible if one may learn representation and a discriminant with deep architectures on images with little pre-processing. In this way, some previous works showed that additional task-oriented data for pre-training networks or fine-tuning on well-known pre-trained models could greatly help in building better FER models [40; 41]. Pre-training deep neural networks is then essential for not leading DL models to overfit. In this way, several state-of-the-art models were developed and shared for research purposes. For example, VGG-Face [42] is a CNN based on the VGG-16 architecture [PERSON] and Zisserman [43] to circumvent the lack of data by building an architecture for face identification and verification and make it available for the research community. This CNN trained on about three million images of 2600 subjects makes this architecture especially adapted for face and emotion recognition. Recent works that have performed well in FER challenges, such as EmotiW [44] or AVEC [45], are based on VGG-Face architecture. [PERSON] et al.