## 3 Analysis of Emotions and Classification Results
### Classification Results
In weighted \(F1\) (\(F1_{W}\)) (Equation (2)) instead, the \(F1\)-scores were calculated for each label, and then, their average is weighted with the number of true instances for each label. \[F1_{W}=\frac{\sum\limits_{i=1}^{N_{c}}n_{C_{i}}F1^{i}}{n} \tag{2}\]  
where \(n_{C_{i}}\) is the number of samples in \(C_{i}\) class and \(n\) is the total number of samples in the test set. Note that, hereafter, macro averages are denoted as P, R, and \(F1\), whereas weighted averages are denoted as \(P_{W}\), \(R_{W}\), and \(F1_{W}\). In the results associated with the TV Debates experiments, a macro \(F1\)-score of 0.56 was achieved in the categorical model. Interestingly, all of the evaluation metrics (P, R, and \(F1\)) provided results in the same range and were quite compensated for. Weighted \(F1\) (\(F1_{W}\)) provided better results (about 0.65) than macro \(F1\) due to the imbalance that could be appreciated in the dataset (the minority class comprises only 8% of the corpus, as seen in Table 6). If we focus on the specific categories, the best results were achieved for the most frequent one (CALM), but the \(F1\) scores for the rest were still acceptable. Focusing on the VAD model, we notice that arousal provided a much better \(F1\)-score, reaching 0.7; the \(F1\)-score dropped again for valence (0.47) and, then, increased a bit for dominance (0.58). Let us note that three different labels were provided for valence, which made the classification task more difficult, while only two were provided for arousal and dominance. Dominance was the most difficult dimension to perceive for the annotators and the most ambiguous one. Nevertheless, in this dataset, dominance had a significant presence and was efficiently perceived and classified. The experiments in the Empathic VA task resulted in lower performances. The categorical model provided a much lower \(F1\)-score when compared with those obtained in the TV Debates dataset, which may be due to the imbalance being even higher in this dataset. The minority class comprised 2.8% of the whole corpus, which was lower than that found in the TV Debates dataset (8%), as shown in Table 6. In fact, looking at the independent categories, the evaluation metrics were very low for less-frequent classes, such as PUZZ or ANN. Moreover, in this set, the number of labels was higher (four instead of three), which also leads, in general, to more confusion and lower performance. The VAD model followed the same tendency observed in the TV Debates dataset, with the highest performance for arousal and lower values for valence and dominance. However, in this case, the results achieved in the previous corpus were not reached, either, because it was a more neutral dataset and little emotional information could be learned. Preliminary Classification Results Using wav2 vec Model  
Some preliminary experiments were also carried out using the wav2 vec model, as shown in Table 9. The performance achieved was significantly lower. Minority categories, such as HAPPY and PUZZ, were almost never recognized. However, the same tendency observed with the spectrogram was also perceived, here: the VAD model performed better than the categorical one, where arousal was the best recognized dimension and weighted averages provided better results due to the imbalanced nature of these scenarios. Thus, the results achieved were promising, considering the pretrained nature of the model and the specific datasets employed in the training process. These datasets were based on speech that is quite far from the conversational nature of the scenarios we deal with in this work. Their contents were mostly neutral, and Spanish was scarcely included. A fine-tuning process would be needed, in this case, to adapt the model to specific features of the task and language. However, the aforementioned corpora might not be large enough to robustly perform such an adaptation, which, currently, makes the use of pretrained representations of the speech signal to model emotions in Spanish really difficult.