## 2 Materials and Methods
### Audiovisual Emotion Recognition Model
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Layer** & **Arguments** & **Number of Parameters** \\ \hline Convolution2D & Filters = 64; kernel size = (7, 7); stride = (2, 2); & 3200 \\  & input shape = (128, 170, 1) & \\ MaxPooling2D & Filters = 128; kernel size = (7, 7); stride = (2, 2) & 401,536 \\ AveragingPooling2D & Filters = 256; kernel size = (3, 3); stride = (2, 2) & 295,168 \\ AveragePooling2D & Filters = 512; kernel size = (3, 3); stride = (2, 2) & 1,180,160 \\  & Flatten & \\ Dense\_1 & Nodes = 4096 & 4,198,400 \\  & Rate = 0.5 & \\ Dense\_2 & Nodes = 6 & 24,582 \\ \hline Total number of parameters & & 6,103,046 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: The proposed CNN model for speech emotion recognition. The obtained STFT coefficients are then converted to PNG image of size 128 \(\times\) 170 pixels. It was demonstrated in the literature that Hanning and Hamming windowing functions usually provide better performance than the other in the case of speech emotion recognition task [31]. A similar approach of using spectrograms of fixed size can be found in the literature as a common choice for processing speech in the case of speaker verification [32]. The final speech processing model consists of approximately 6.1 million trainable parameters, resulting in a size of around 23.3 MB. We also experimented with larger VGGish-like architectures [33; 34], but found that their performance on the small client subsets was either similar or inferior. In order to maximize classification accuracy at the utterance level, we apply a sequential voting procedure in the post-processing phase for both modalities, inspired by the approach described in [33]. Similar approaches for video processing are already available in the literature [35], simplifying the overall complexity, avoiding a need to utilize LSTM or 3D-CNN for a sequence of frames. In the case of facial expression analysis, after training a classifier to make decisions for individual input video frames, we buffer the outputs for all frames corresponding to a single utterance and make an utterance-level decision using average probability voting. Similarly, for audio signals, neural networks are trained to make decisions for individual 1.0 s long spectrograms. The outputs corresponding to a single utterance are buffered, and average probability voting is applied. Previously, we found that the average probability voting is a preferable choice comparing to the maximum probability voting and the majority voting for a scenario related to the CNN-based speech emotion recognition [33]. The outputs from both modalities are fused at the decision level, and the final decision is made by averaging scores of these two outputs. The decision fusion process allows us to leverage the complementary information from both facial expressions and speech for a more robust emotion recognition decision. The architecture of the proposed audiovisual emotion recognition model is presented in Figure 2, with an accompanied inference example. Without loss of generality, we present an example where three different classes are analyzed, whereas the outputs for other classes tend to be zero. In the next subsection, we explain the federated learning protocol applied to the audio modality of the proposed audiovisual emotion recognition model. Figure 1: Example of spectrograms extracted from a single speech stream. Figure 2: The architecture of the proposed audiovisual emotion recognition model with an inference example.