## 2 Materials and Methods  
### Audiovisual Emotion Recognition Model  
In our study, a multimodal model is proposed, designed to process signals from two different modalities: facial expressions, given as a sequence of images, and speech, given as a set of image spectrograms created in the pre-processing phase. To process facial expressions, a MobileNetV2 model is utilized [26], pre-trained on the ImageNet [27], a large-scale dataset widely used for image classification tasks. The idea of transfer learning is introduced due to the fact that available audiovisual emotion-capturing datasets are relatively small, and it is often impossible or at least highly impractical to record large datasets for speaker-dependent emotion recognition tasks. To adapt the MobileNetV2 model for our specific task (described further ahead in Section 2.3), we introduce a new prediction layer consisting of six classes corresponding to different facial expressions. While incorporating key frame detection could potentially enhance the overall accuracy of emotion recognition in video signals [28], this approach is computationally intensive, which contradicts our objective of creating a simple model suitable for deployment on edge devices. Consequently, we made the decision to proceed with a uniform extraction of frames from the video. In the pre-processing phase, we extracted frames from the original videos with a step size of 8 and resized each frame to the resolution of 224 \(\times\) 224 pixels, so that our inputs would be compatible with ImageNet resolution used in MobileNetV2. Considering the chosen input shape and the aforementioned output layer, we designed a neural network model for analyzing facial expressions, which comprised approximately 2.63 million parameters. The overall size of the model is approximately 10.3 MB, with 0.37 million trainable parameters. Since the number of trainable parameters is relatively small and the impact of federated learning on such a small parameter set yielded negligible changes in our experiments, we opted to further simplify the proposed system by applying federated learning to the audio (i.e., speech) modality only. By focusing on federated learning solely for the audio modality, we aimed to simplify the system and reduce the computational requirements, aligning with our objective of creating an efficient model suitable for edge devices. For the task of emotion detection from the audio modality, we propose a model inspired by the network from [29]. However, we simplify the method by excluding one fully connected layer from the 2D CNN network, reducing its complexity. The proposed model is described in Table 1.  
Unlike the method proposed in [29], we do not utilize MFCC (Mel frequency cepstral coefficients) as inputs. Instead, we adopt the approach of generating spectrograms using short-time Fourier transform (STFT), following the methodology described in [30]. We decided to choose spectrograms, as their computing is straightforward and there are efficient algorithms and libraries for further deployment, whereas scalogram computation might be more computationally intensive. Recent studies have shown that spectrograms and scalograms lead to the similar performance on the speech emotion recognition tasks [9], so we decided to continue with the STFT approach. In the first step of the spectrogram creation, a speech file is divided into 1.0 s long segments with an overlap of 0.5 s as shown in Figure 1. If the last segment is shorter than 0.7 s, it is expelled from the training set. If the last segment's length falls between 0.7 and 1.0 s, it is extended to a 1.0 s length by replicating samples from the end of that segment. Spectrograms are calculated using an 11.6 ms Hanning window with a time shift of 5.8 ms.