## 2 Materials and Methods
### Early Fusion Methodology  
For the Early Fusion, extracted features from both the audio and the visual model were merged into a unified dataset. Considering that the two datasets bear a direct correlation with each other, audio and visual features were randomly assigned using a concatenation technique. After loading the datasets into memory, they were sorted and aligned by their common column \"label\" to ensure proper correspondence for the visual and audio data. For each entry of a visual feature in the dataset, a random audio feature entry was concatenated with the visual feature. In order to properly merge the feature data, a Python script that separated each dataset based on the label was developed, ensuring that both subsets have the same number of samples for each class, concatenated the label = 1 and label = 2 instances separately, and then recombined them into a single dataset. This technique ensures proper Early Fusion with matched samples while keeping features from both modalities without data loss [41].  
Finally, with the aim of forming a comprehensive multimodal dataset, the datasets were concatenated along the feature axis. This resulted in a dataset where each entry contained both audio and visual features. The final dataset was saved as **early_fusion_features_final_dataset.csv** for future use in the training of the meta-classifier. The final dataset consisted of two subsequent datasets that were already filtered for noise and artifacts, and they were already preprocessed. Furthermore, imputation and feature standardization techniques for possible missing values that occurred during the fusion were applied [42] (**S1.6**, **S1.7**).  
After the fusion of the visual data (body landmarks) and audio data (speech features), the final result was a high-dimensional multimodal feature space that contained the information for both modalities Table 1. More specifically, as follows:  
* **Body Landmark Features** (1629 features): These represent key points of the body, face, and hands in the three-dimensional space with corresponding (x,y,z values).
* **Audio Features (39 features):*
* Audio speech signals extracted from the audio file, including the following:
* **Mel Frequency Cepstral Coefficients (MFCCs)*
* (mean and standard deviation): 26 features;
* (mean and standard deviation): 2 features;
* **Spectral Features*
* (centroid, bandwidth, flatness, contrast): 4 features;
* **Zero-Crossing Rate (ZCR)*
* (mean): 1 feature;
* **Root Mean Square Energy (RMS)*
* (mean): 1 feature.
* **Class Label (True Value):** This column contains the class for each entry, aggressive or non-aggressive (1). The total number of Early Fusion Data Entries after the merging of the two multimodal datasets is 932.  
\begin{table}
\begin{tabular}{l c c c c c c c c c c c c c c c c c c} \hline \hline
**Length** & **Length** & **Length** & **Length** & **Length** & **Micro.** & **Micro.** & **Micro.** & **Micro.** & **Micro.** & **Micro.** & **Micro.** & **Micro.** & **Micro.** & **Micro.** & **Micro.** & **Micro.** & **Micro.** & **Micro.** & **Micro.** & **Micro.** & **Micro.** & **Micro.** & **Micro.** & **Micro.** \\
**AR, p** & **arx.1** & **arx.n** & **arx.1** & **arx.2** & **arx.3** & **Mem.1** & **Mem.2** & **std.1** & **Std.1** & **Mem.3** & **Std.2** & **Mem.4** & **Micro.2** & **Micro.3** & **Micro.4** & **Micro.5** & **Micro.6** \\ \hline
0.62511 & 0.63794 & -0.0233 & -0.4564 & -0.75 & -1.32 & -1.2303 & -0.4840 & -0.2329 & 0.47584 & 0.46798 & -0.554811 & -0.2396 & 0 \\
0.62897 & 857303 &... & -0.64860 & 59020 &... & 7296 &... & 99353 & 773684 & 88002 & 0.000766 & 0.7227662 & 57258 & 0 \\
0.5071 & 2277 & 88903 & 63195 & 0.000569 &... & 0.0005699 & 7512306 & 34824 & 7956 & 33126 & 0.000766 & 0.0195 & 0 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Example of the first row of the Early Fusion Merged Dataset.