## 3 Experimental Results
\begin{table}
\begin{tabular}{c c c c c c c c c c} \hline \hline
**Emotion** & \multicolumn{3}{c}{**Client 1**} & \multicolumn{3}{c}{**Client 2**} & \multicolumn{3}{c}{**Client 3**} \\ \hline  & **Precision** & **Recall** & **F1-Score** & **Precision** & **Recall** & **F1-Score** & **Precision** & **Recall** & **F1-Score** \\ \hline Anger & 0.54 & 0.55 & 0.55 & 0.60 & 0.69 & 0.64 & 0.55 & 0.69 & 0.61 \\ \hline Disgust & 0.42 & 0.48 & 0.45 & 0.44 & 0.34 & 0.39 & 0.33 & 0.38 & 0.35 \\ \hline Fear & 0.45 & 0.37 & 0.40 & 0.43 & 0.38 & 0.40 & 0.27 & 0.27 & 0.27 \\ \hline Happiness & 0.38 & 0.42 & 0.40 & 0.52 & 0.47 & 0.49 & 0.45 & 0.41 & 0.43 \\ \hline Sadness & 0.56 & 0.59 & 0.57 & 0.47 & 0.53 & 0.50 & 0.67 & 0.53 & 0.59 \\ \hline Surprise & 0.34 & 0.30 & 0.32 & 0.41 & 0.45 & 0.43 & 0.35 & 0.28 & 0.31 \\ \hline
**Average** & 0.45 & 0.45 & 0.45 & 0.48 & 0.48 & 0.47 & 0.43 & 0.43 & 0.43 \\ \hline
**Weighted average** & 0.46 & 0.46 & 0.46 & 0.48 & 0.48 & 0.48 & 0.44 & 0.44 & 0.44 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Detailed performance–audio modality. \begin{table}
\begin{tabular}{c c c c c c c c c c} \hline \hline
**Emotion** & \multicolumn{3}{c}{**Client 1**} & \multicolumn{3}{c}{**Client 2**} & \multicolumn{3}{c}{**Client 3**} \\ \hline  & **Precision** & **Recall** & **F1-Score** & **Precision** & **Recall** & **F1-Score** & **Precision** & **Recall** & **F1-Score** \\ \hline Anger & 0.91 & 0.76 & 0.83 & 0.84 & 0.70 & 0.76 & 0.97 & 0.84 & 0.90 \\ \hline Disgust & 0.75 & 0.92 & 0.83 & 0.79 & 0.78 & 0.79 & 0.69 & 1.00 & 0.82 \\ \hline Fear & 0.85 & 0.81 & 0.83 & 0.89 & 0.75 & 0.81 & 0.97 & 0.73 & 0.83 \\ \hline Happiness & 0.68 & 0.95 & 0.79 & 0.72 & 0.87 & 0.79 & 0.87 & 0.81 & 0.84 \\ \hline Sadness & 0.95 & 0.65 & 0.77 & 0.92 & 0.75 & 0.82 & 0.97 & 0.93 & 0.95 \\ \hline Surprise & 0.82 & 0.89 & 0.85 & 0.65 & 0.90 & 0.75 & 0.90 & 0.96 & 0.93 \\ \hline
**Average** & 0.83 & 0.83 & 0.82 & 0.80 & 0.79 & 0.79 & 0.89 & 0.88 & 0.88 \\ \hline
**Weighted average** & 0.84 & 0.82 & 0.82 & 0.80 & 0.79 & 0.79 & 0.90 & 0.88 & 0.88 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Detailed performance–video modality. As already described in Section 2, we do not apply federated learning to the video modality as we exploit transfer learning and there is only a small number of trainable parameters. By observing Figure 3, we can see that the implementation of federated learning significantly improves classification accuracy in the case of the audio modality, both for making decisions using the single spectrogram (the blue line) and after the sequential voting process (the red line), demonstrating the importance of implementing such a technique. It can also be noticed that there is a gain in the range of 1.19-3.57 (%) in the case of multimodal inference, which is significant, considering that the non-federated learning audiovisual model provides a relatively high classification accuracy in the (already high) range of \begin{table}
\begin{tabular}{c c c c c c} \hline \hline  & \multicolumn{4}{c}{**Classification Accuracy (\%)**} \\ \cline{2-6}  & \multicolumn{2}{c}{**Audio**} & \multicolumn{2}{c}{**Video**} & \multicolumn{1}{c}{**Multimodal**} \\ \cline{2-6}  & \multicolumn{1}{c}{**Single Spectrogram**} & \multicolumn{1}{c}{**Voting**} & \multicolumn{1}{c}{**Single Frame**} & \multicolumn{1}{c}{**Voting**} & \multicolumn{1}{c}{**Multimodal**} \\ \hline Client 1 & 49.29 & 67.86 & / & / & 92.86 \\ \hline Client 2 & 51.27 & 64.29 & / & / & 92.86 \\ \hline Client 3 & 51.89 & 71.43 & / & / & 94.05 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Classification accuracy of the proposed AVER model with federated learning. Figure 3: Classification accuracy gain \(Gain_{f}[\%]=CA_{f}-CA_{nf}\) of the proposed emotion recognition approach accompanied with federated learning, compared to the non-federated learning approach. 89.29-92.86 (%). This is mainly due to the good performance of the classifier in the case of the video modality, whereas the audio modality behaves like a supporting modality within this experiment.