# Continuous Emotion Recognition with Spatiotemporal Convolutional Neural Networks
## 1 Introduction  
Facial expressions are the results of peculiar positions and movements of facial muscles over time. According to previous studies, face images and videos provide an essential source of information for representing the emotional state of an individual [1]. Therefore, facial expression recognition (FER) has attracted a growing interest in recent years. However, the detection of spontaneous facial expressions in the wild is a very challenging task. The performance depends on several factors, such as variations among individuals, the identity bias of subjects (e.g., gender, age, culture, and ethnicity), and the quality of recordings (e.g., illumination, resolution, head pose, and capture conditions). The fundamentals of human affect theory [2, 3] inspired early research on FER systems. Even if facial expressions can also be performed voluntarily, without any relationship to the emotions that a person actually feels [4, 5], FER systems assume that facial expressions are associated with either discrete or multidimensional emotion models. Discrete models are employed to classify facial images into discrete categories that can be recognizable across cultures, such as anger, disgust, fear, happiness, sadness, and surprise. The limited generalization capacity of these models has paved the way for multidimensional spaces that can improve the representativeness and accuracy for describing emotions [6]. There are three levels for describing emotion: pleasantness, attention, and activation. Specifically, emotion recognition datasets annotate emotional states with two values namedvalence and arousal. The former represents the level of pleasantness, and the latter represents the level of activation, each of these values lying in the \([-1;1]\) range. These values can be used to project an individual emotional state into a 2D space called the circumplex model, which is shown in Figure 1. The level of arousal is represented on the vertical axis, whereas the valence level is represented on the horizontal axis. Early FER systems relied on shallow approaches, which combined handcrafted features, such as eigenfaces [8], Gabor wavelets [9], local binary patterns (LBP) [10; 11] and its variants (such as LBP-TOP [12] and LBP-SIP [13]), and Weber local descriptor [14] with support vector machines (SVM). However, recent advances in deep learning (DL) and computing technologies have led FER systems to learn discriminant representations directly from face images. Previous studies first exploited static FER datasets, where subjects are associated with discrete and mutually exclusive categories of emotions. State-of-the-art performance on benchmarking datasets, such as FER2013 [15], TFD [16], and SFEW [17], was achieved using CNN-based approaches [18; 19; 20; 21; 22; 23; 24]. Temporal information and dynamic facial components can play a crucial role in describing facial expressions [1]. By definition, people express emotions in a dynamic process, and learning spatiotemporal structures has become the current trend. Recent studies have proposed deep architectures for FER, which were trained on video data [25]. In the literature, video sequences are mostly processed using aggregation and concatenation of features for each facial image into clips. Yet, they cannot leverage the temporal dependencies over a video [26; 27]. To circumvent this issue, convolutional recurrent neural networks (CRNN) and 3D-CNN architectures have been proposed to encode spatiotemporal relations among video frames [28; 29]. This paper investigates and compares state-of-the-art DL models based on CNNs for video-based emotion recognition, where affective states are represented as continuous values in the bidimensional space of valence and arousal. The novelty of this paper is the extension of state-of-the-art 2D CNN architectures to model spatiotemporal relations of facial features in videos with the aim of improving the predictions for continuous values of emotion.