## 2 Materials and Methods
### Annotation Procedure
To this end, the annotation work was divided in micro-tasks that were performed by a large number of untrained annotators who did not speak to each other. This division in tasks made the annotations diverse, which is a plus for our dataset [53] and was made possible by the wide variety of different annotators. In this work, each audio chunk was annotated by five different annotators who were asked to fill in the following questionnaire for each audio clip. Some categories have two or three names as a result of the preliminary task adaptation carried out over the categories selected from [51]. Figure 1: Illustration of [PERSON]’s circumplex [52], which shows categories represented by the dimensions arousal and valence. * How do you perceive the speaker? * Slightly Excited
* His/her mood is
* Rather Positive
* Neither Positive nor Negative
* Rather Negative
* How do you perceive the speaker in relation to the situation which he/she is in? * Rather dominant/controlling the situation
* Rather intimidated/defensive
* Neither dominant nor intimidated
* Select the emotion that you think best describes the speaker's mood: (TV Debates) (Empathic VA)  
Given that each audio chunk was labeled by five different annotators, an analysis of the agreement among the annotators was carried out. Table 3 gathers the statistics of agreement per audio chunk for the categorical model. This table shows that, for about 70% of the data in the TV Debates dataset, the agreement was 2/5 or lower. For the Empathic VA dataset, a higher agreement was achieved due to the lower number of categories that were selected. However, almost 50% of the samples still showed an agreement of lower than 4/5. This confirms the ambiguity and subjectivity of the task. Moreover, [PERSON]'s _alpha_ coefficient [54] was also low for both tasks, resulting in values of 0.11 and 0.2, respectively. This coefficient reflects the degree of agreement but is very dependent on the number of labels. In the rest of the document, we do not consider samples with an agreement below 3/5 for the categorical model, which means that we used 30.74% of the annotated audio files of \begin{table}
\begin{tabular}{c c c c c} \hline \hline  & \multicolumn{2}{c}{**TV Debates**} & \multicolumn{2}{c}{**Empathic VA**} \\ \hline
**Agr Level** & **No. Audios** & **\% Audios** & **No. Audios** & **\% Audios** \\ \hline \(\geqslant\)1/5 & 4118 & 100.00\% & 4525 & 100.00\% \\ \(\geqslant\)2/5 & 3035 & 73.70\% & 4522 & 99.93\% \\ \(\geqslant\)3/5 & 1266 & 30.74\% & 4023 & 88.91\% \\ \(\geqslant\)4/5 & 392 & 9.52\% & 2519 & 55.67\% \\ \(\geqslant\)5/5 & 82 & 1.99\% & 1086 & 24.00\% \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Statistics of the agreement per audio chunk for each corpus. Column Agr. Level indicates the condition, i.e., the minimum inter-annotator agreement, and the next two columns (No. Audios and % audios), indicate how many samples and what percentage of them fulfilled the agreement condition. the TV Debates dataset and 88.81% of the Empathic VA for the experiments with emotional categories. Then, the majority voting method was used to establish the ground truth for these sets. The answers to the questionnaires related to the VAD model were transformed into real values, ranging from 0 to 1, by applying the rules of Table 4 to each response. Then, these values were averaged per sample over all five annotators to obtain a real value in the 3D space. In this case, carrying out majority voting and thus obtaining a minimum agreement level were not required. The average was computed due to the vast diversity derived from the subjectivity of this task, which was reflected in the different answers provided by the diverse labels generated by the annotators. The size of the resulting labeled corpus (100% of the audio clips shown in Table 3) was bigger than the corpus labeled in terms of the categorical model.