# Continuous Emotion Recognition with Spatiotemporal Convolutional Neural Networks
## 4 Experimental Results
### i3D CNN Architecture  
Another alternative for spatiotemporal modeling is to use the i3D CNN. In this way, strong spatiotemporal correlations between frames are directly learned from video clips by a single network. Thanks to weight inflation, the pre-trained 2D CNNs can be used to build i3D CNN architectures. The inflation method allows us to transpose learned information from various static tasks to dynamic ones, performing the essential transfer learning for learning spatiotemporal features. With this in mind, the 2D CNN architectures shown in Table 1 are reused, and their convolutional layers are expanded to build i3D CNNs considering two configurations, denoted as C1 and C2 in Table 3.  
Due to the high number of trainable parameters, i3D CNNs are particularly time consuming to train. Therefore, some basic hyperparameters had a fixed value instead of performing exploratory experiments to set them. Consequently, we evaluated only the best configuration found for the 2D CNN-LSTM, as shown in Table 1, which uses a batch of size 8, a sequence length of 64 frames, overlapping ratio of 0.8, and frame rate of 10 fps. This is the main downside of our approach based on i3D CNNs. The number of trainable parameters of i3D CNNs is three times greater than the counterpart 2D CNNs. \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline \multirow{2}{*}{**Dataset for**} & \multicolumn{5}{c}{**Configuration**} \\ \cline{2-7}  & **fps** & **Label** & **(SL, OR, FM)** & **PCC\(\uparrow\)** & **CCC\(\uparrow\)** & **MAPE(\%)\(\downarrow\)** \\ \hline \multirow{4}{*}{VGG-Face} & 10 & Valence & 16, 0.2, extremum & 0.590 & 0.560 & 3.8 \\  & & Arousal & 16, 0.2, mean & 0.549 & 0.542 & 8.7 \\ \cline{2-7}  & \multirow{2}{*}{50} & Valence & 64, 0.2, mean & 0.541 & 0.511 & 6.8 \\  & & Arousal & 64, 0.2, extremum & 0.495 & 0.492 & 3.4 \\ \hline \multirow{4}{*}{RAF-DB} & \multirow{2}{*}{10} & Valence & 64, 0.8, mean & **0.631** & **0.625** & **3.7** \\  & & Arousal & 64, 0.8, extremum & **0.558** & **0.557** & **9.4** \\ \cline{1-1} \cline{2-7}  & \multirow{2}{*}{50} & Valence & 64, 0.2, mean & 0.582 & 0.568 & 8.2 \\ \cline{1-1}  & & Arousal & 64, 0.2, extremum & 0.517 & 0.517 & 4.4 \\ \hline \hline \end{tabular}  
* SL: sequence length (in frames), OR: overlapping ratio, FM: fusion mode.  
\end{table} 
Table 2: Performance of the 2D CNN–LSTM based on the VGG-16 architecture on the SEWA-DB dataset. The best results for valence and arousal are in boldface. Tables 4 and 5 show only the best configurations of each architecture for valence and arousal prediction, respectively. These configuration were obtained from an exhaustive combination of the parameters' values shown in Table 3 for each base model and considering different datasets for initializing such base models. Globally, different inflation types, masking, and dilation values do not impact the results achieved by i3D models. Table 6 shows the best performance obtained for each architecture for valence and arousal in terms of PCC and CCC values. Inflated 3D CNNs for regression seem sensitive to some training configurations regarding the range of results achieved by different base models and datasets used in their initialization. In these conditions, it is difficult to state the effect of a single parameter for inflation. VGG-16 with batch normalization and ResNet50 achieved the best results for valence and arousal and showed a good ability to predict these values, compared to other base models. Surprisingly, the VGG-16 pre-trained on ImageNet achieved higher PCC and CCC for both valence and arousal than those base models pre-trained on VGG-Face and RAF-DB, which are source datasets closer to the target one. On the other hand, ResNet50 benefited from initialization with VGG-Face. In summary, the best results ranged from 0.313 to 0.406 for PCC and from 0.253 to 0.326 for CCC. The performance still shows a poor correlation between predictions and annotations.