## 4 Experiments
### Modality Experimentation  
Before attempting classification of the fused multi-modal embeddings, an initial testing of each individual modality was performed to assess the extrapolation quality and obtain a term of comparison for later on. Accordingly, multilayer perceptrons (MLP) were attached to the end of the feature extraction layers of VGGVox and the developed RN-LSTM, for classification. Additionally, level of model generalization was also assessed by comparing dependence and independence scenarios in terms of speaker for the Columbia dataset and in terms of video for AVA-ActiveSpeaker--a solution to the unknown speaker amount issue. Essentially, for dependent scenario experiments (closed set) the test data includes instances from speakers who are also represented in the training data, albeit by completely distinct instances. On the contrary, for independent scenario experiments (open set) it is guaranteed that the speakers from the test data are completely different and never before seen by the model during training. These two types of testing allow for a better evaluation of the model's generalization ability, by examining how much the model's performance decreases from dependency to independency. The obtained results are presented in Tables 2 and 3, respectively. The novel SCF layer, as well as the entirety of the implemented experiments, were realized using the Keras v2.4.3 framework [35] with Tensorflow backend. Test runs were performed on a Nvidia RTX 2080 GPU with 8 GB memory. In order to assess its superiority against state-of-the-art standards and suitability to the task at hand, we compared its performance to a naive baseline. This was composed of an audiovisual concatenated embedding array fed directly to a multi-layer perceptron, as seen in Figure 3. The results of this experiment and those obtained with the SCF layer alternative are shown in Tables 4 and 5 for the Columbia and AVA datasets, respectively. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Dataset** & **Sequences** & **Frames** & **Unique Facial** & **Image** \\  & & & **Trackings** & **Conditions** \\ \hline Columbia & 4914 & 30 & 34 (6 speakers) & Identical \\ AVA- & 5510 (3 labels) & [30; 30] & 4321 (20 videos) & Varied \\ ActiveSpeaker & 5470 (2 labels) & [30; 30] & 4321 (20 videos) & Varied \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Overview of the data used in the experiments. The Unique Facial Trackings factor is provided in place of number of speakers considering that AVA-ActiveSpeaker does not inform on speaker amount. Same person has been manually verified to not exist in two different videos. \begin{table}
\begin{tabular}{c c c} \hline \hline  & **Speaker Dependent** & **Speaker Independent** \\ \hline \(VGGVox+MLP_{2}\) & & & \\ (Audio Only) & 47.37 (0.52) [\%] & 17.34 (9.14) [\%] \\ \(RN\)-\(LSTM+MLP_{1}\) & & & \\ (Video Only) & 97.68 (0.39) [\%] & 64.89 (14.44) [\%] \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Audio/video results using the Columbia dataset with respect to Mean Accuracy (Standard Deviation). \begin{table}
\begin{tabular}{c c c} \hline \hline  & **Video Dependent** & **Video Independent** \\ \hline \(VGGVox+MLP_{2}\) & & & \\ (Audio Only) & 87.73 (0.87) [\%] & 84.96 (1.78) [\%] \\ \(RN\)-\(LSTM+MLP_{1}\) & & & \\ (Video Only) & 78.68 (1.79) [\%] & 72.48 (2.17) [\%] \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Audio/video results using the AVA-ActiveSpeaker videos with respect to Mean Accuracy (Standard Deviation). The SCF layer's neural areas were kept at a \(17\times 17\) dimension, for the sake of computational simplicity. Each of the presented values in all tables was obtained using 5-fold cross validation for increased robustness. Furthermore, the adaptive moment estimation (Adam) optimizer [36] was employed during the training phase of the models for weight adjustment. Batch normalization [37] was also integrated in the training, in addition to minor dropout [38].