## 4 Results and Discussion  
Table 1 and Figure 4 present the accuracy of the models on test data varying the considered sequence duration. A general rising trend in accuracy can be noticed, indicating that providing more temporal context to the model increases the classification accuracy. The trend plateaus between 35 and  
Figure 3: Schematic of the CNN lip model for 60 frames. The input dimensions represent _(butch size, channels, number of frames, feature space dimension)_. The feature space dimension is the sum of 512 dimensions of facial features and 96 dimensions of LipNet features. Figure 2: Schematic of the RNN lip model for 60 frames. The input dimensions represent _(batch size, number of frames, feature space dimension)_. The feature space dimension is the sum of 512 dimensions of facial features and 96 dimensions of LipNet features. 45 frames for all but the two lip models, which boasts the trend all the way to the maximum number of frames. The worst performing model is confidently the frame model, while the two best performing models are the CNN lip and RNN lip models, with the RNN lip model slightly outperforming the CNN lip model at higher frames. Most importantly, however, we see that there is, at frames greater than 25, a clear and persistent accuracy gain of the lip models with respect the corresponding face models. Looking at only the difference in accuracy between the lip models and the corresponding face models, as depicted in Figure 5, we can confirm both the increase in accuracy starting at 25 frames, and that it follows a seemingly linear rising trend, as we increase the number of frames in input to the model. Running linear regression on the data in Figure 5 reveals a statistically significant slope of 0.0014, representing the increase in accuracy per additional frame in input. However, the real trend is likely not linear due to there being an upper limit to the accuracy metric, and increasing further the number of frames would raise questions as to the persistence of expressions of base emotions in spontaneous conversations. Nonetheless, the accuracy gains for the data at hand seem to rise linearly with positive values starting from 1 s in input sequence duration. While attributing interpretation to the workings of a black box system, such as a neural network, is a slippery endeavour, we retain the reason behind the lip models not plateauing and the increases rising linearly: the information-related to articulation boosts the utility of the information-related to the face. Under the assumption that articulation-related features in speaking subjects carry no information on affect, they induce the network to become invariant to articulation. Thus, when providing more information in input with a greater number of frames, the lip models are able to classify better. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Frames** & **CNN** & **CNN Lip** & **RNN** & **RNN Lip** & **Frame** \\ \hline
5 & 0.429 & 0.446 & 0.423 & 0.444 & 0.413 \\
10 & 0.413 & 0.448 & 0.443 & 0.452 & 0.421 \\
15 & 0.458 & 0.458 & 0.458 & 0.442 & 0.431 \\
20 & 0.459 & 0.442 & 0.462 & 0.475 & 0.437 \\
25 & 0.465 & 0.509 & 0.452 & 0.470 & 0.439 \\
30 & 0.475 & 0.516 & 0.465 & 0.513 & 0.441 \\
35 & **0.479** & 0.485 & 0.476 & 0.530 & 0.449 \\
40 & 0.459 & 0.509 & **0.485** & 0.505 & 0.450 \\
45 & 0.452 & 0.479 & 0.473 & 0.533 & **0.457** \\
50 & 0.465 & 0.492 & 0.479 & 0.519 & 0.449 \\
55 & 0.466 & 0.508 & 0.482 & 0.538 & 0.444 \\
60 & 0.462 & 0.544 & 0.470 & **0.594** & 0.447 \\
65 & 0.448 & **0.569** & 0.482 & 0.563 & 0.454 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Test data accuracy of the models at different frame values. Models with highest accuracy are indicated in bold. Figure 4: A plot of accuracy of the models on test data. On Figure 6 a representative example of the training behavior of the models can be seen. It depicts the training set and validation set accuracy values for different epochs for two models: the best performing RNN lip and the corresponding RNN face model.