# Continuous Emotion Recognition with Spatiotemporal Convolutional Neural Networks
## 3 Spatiotemporal Models for Continuous Emotion Recognition
### Spatiotemporal Models
#### 3.3.2 Inflated 3D CNN (i3D CNN)  
The need to analyze a sequence of frames led us to the use of 3D CNNs. Three-dimensional CNNs produce activation maps that allow analyzing data where temporal information is relevant. The main advantage of 3D CNNs is to learn representation from clips that can strengthen the spatiotemporal relationship between frames. Different from 2D CNNs, 3D CNNs are directly trained on batches of frame sequences rather than batches of frames. On the other hand, adding a third dimension to the CNN architecture increases the number of parameters of the model, and that requires much larger training datasets than those needed by 2D models. The main downside of using such an architecture for FER tasks is the lack of pre-trained models. Besides that, it is difficult to train 3D CNN architectures end to end for continuous emotion recognition, due to the limited amount of training data. Therefore, a feasible solution is to resort to weight inflation of 2D CNN pre-trained models [49]. Inflating a 2D CNN minimizes the need for large amounts of data for training a 3D CNN properly, as the inflation process reuses the weights of the 2D CNNs. Figure 7 shows that the weight inflation consists of enlarging kernels of each convolution filter by one dimension. Figure 6: The architecture of the VGG-16-LSTM. A \(3\times 3\) kernel is used in each convolutional layer. The numbers of filters and units in LSTM and FC layers are also indicated. Figure 7: Representation of the inflation method for a single convolutional filter. The 2D convolutional kernels are replicated along a new dimension (temporal dimension) to obtain 3D convolutional kernels. Basically, \(n\times n\) kernels are made cubic to obtain \(n\times n\times n\) kernels. This process is applied to every convolutional filter to transform 2D convolutional layers into 3D convolutional layers. Our target task implies extending each neuron's receptive field to the time dimension (i.e., a sequence of frames). The 2D convolutional kernels are replicated to fit the third dimension and form a 3D convolutional kernel. At first glance, pre-trained weights are just copied through the time dimension and provide a better approximation for initialization than randomness. However, they do not constitute an adequate distribution for the time dimension yet. With this in mind, the next issue is finding a method that best fits the transfer learning to the time dimension with weight inflation by varying some parameters, such as initialization, masking, multiplier, and dilation. Initialization: When replicating kernels for weight inflation, it is possible to copy the weights \(n\) times (\(n\) being the dimension of the time axis) or to center the weights. Centering means copying once the weights of a 2D kernel and initializing the weights of the surrounding kernels that form the 3D filter either randomly (with a uniform distribution) or with zeros. Assuming that pre-trained 2D kernels have a good capacity for generalization for images, giving sufficiently distant distribution for all but one 2D kernel from the copied 2D kernel could positively impact model convergence. Masking: Assuming that copied 2D kernels are pre-trained properly considering a very similar task and that they perform well on images, the idea of masking is to train adequately inflated weights on the time dimension. Therefore, to disseminate the spatial representation learned from pre-trained weights to inflated weights, centered weights are not modified during training. Multiplier: The distribution of CNN weights and the range of targeted values for regression are closely related. Since values of valence and arousal range from \(-1\) to \(1\) and standard values of the weights often take values between \(10^{-3}\) and \(10^{-1}\), then rising targeted values by a factor could allow to scale up the distribution space and improve convergence. Dilation: Dilated convolutions are used in our models, as suggested by [PERSON] and [PERSON] (2017).