# Analysis of Deep Learning-Based Decision-Making in an Emotional Spontaneous Speech Task
## 3 Emotion Detection
### Classification Results  
The performance of the proposed classification system for both emotion categories and VAD dimensions are compared to the VGG16 model in Tables 2 and 3. All results were obtained after a 10-fold cross-validation procedure. For the categorical model, two different results are given, one corresponding to the direct categorical classification associated with output B in Figure 1 (CLS C) and another one making use of the predicted VAD floating point values associated with output D (CLS D). The average and standard deviation of five metrics commonly employed to evaluate emotion classification systems [81; 82] are reported: F1 score, Unweighted Accuracy (UA, also known as balanced accuracy or unweighted average recall), average precision, Matthews Correlation Coefficient and Area Under the ROC Curve (AUC). Additionally, paired \(t\)-tests were computed to assess the statistical significance of the performance differences between our model and the VGG16 network. First, and most importantly, our network performs slightly better than the fine-tuned VGG16 network for most classification tasks and metrics. This is already remarkable because our architecture uses around 3000 times fewer parameters than the VGG16 CNN. Furthermore, the differences in performance are statistically significant in four comparisons: when measuring the AUC for CLS C, CLS D and arousal, and also for the Unweighted Accuracy in the arousal prediction task. Importantly, these results support the use of our light network to apply XAI techniques. \begin{table}
\begin{tabular}{l c c} \hline \hline
**Our Net/VGG16** & **CLS C** & **CLS D** \\ \hline F1 & \(\mathbf{0.58\pm 0.04}\)/0.57 \(\pm\) 0.06 & \(\mathbf{0.59\pm 0.05}\)/0.57 \(\pm\) 0.05 \\ UA & \(\mathbf{0.57\pm 0.04}\)/0.54 \(\pm\) 0.06 & \(\mathbf{0.58\pm 0.04}\)/0.54 \(\pm\) 0.05 \\ Average precision & \(0.60\pm 0.05\)/\(\mathbf{0.63\pm 0.07}\) & \(0.60\pm 0.06\)/\(\mathbf{0.63\pm 0.07}\) \\ Matthews corr. coef. & \(\mathbf{0.39\pm 0.06}\)/0.38 \(\pm\) 0.08 & \(\mathbf{0.41\pm 0.05}\)/0.38 \(\pm\) 0.06 \\ AUC & \(\mathbf{0.80^{*}\pm 0.03}\)/0.75 \(\pm\) 0.03 & \(\mathbf{0.81\pm 0.02}\)/0.74 \(\pm\) 0.04 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Classification performance of our proposal and VGG16 for the categorical model prediction task. Two ways of predicting the emotion categories were tested. Comparisons where a network significantly outperforms the other (i.e., \(p\)-value < 0.05) are marked with the symbol \({}^{*}\). The best result for each comparison is highlighted in bold. \begin{table}
\begin{tabular}{l c c c} \hline \hline
**Our Net/VGG16** & **Acausal** & **Valence** & **Dominance** \\ \hline F1 & \(\mathbf{0.67\pm 0.11}\)/0.67 \(\pm\) 0.02 & \(\mathbf{0.42\pm 0.03}\)/0.41 \(\pm\) 0.05 & \(\mathbf{0.57\pm 0.05}\)/0.56 \(\pm\) 0.03 \\ UA & \(\mathbf{0.67\pm 0.09}\)/0.66 \(\pm\) 0.03 & \(\mathbf{0.45\pm 0.04}\)/0.41 \(\pm\) 0.03 & \(\mathbf{0.57\pm 0.03}\)/0.56 \(\pm\) 0.03 \\ Average precision & \(0.69\pm 0.13\)/\(\mathbf{0.69\pm 0.02}\) & \(0.44\pm 0.05\)/\(\mathbf{0.45\pm 0.09}\) & \(0.58\pm 0.07\)/\(\mathbf{0.58\pm 0.04}\) \\ Matthews corr. coef. & \(0.35\pm 0.17\)/\(\mathbf{0.35\pm 0.03}\) & \(0.14\pm 0.04\)/0.12 \(\pm\) 0.04 & \(\mathbf{0.15\pm 0.06}\)/0.14 \(\pm\) 0.06 \\ AUC & \(\mathbf{0.74\pm 0.11}\)/0.72 \(\pm\) 0.02 & \(\mathbf{0.65\pm 0.03}\)/0.60 \(\pm\) 0.02 & \(\mathbf{0.63\pm 0.07}\)/0.60 \(\pm\) 0.03 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Classification performance of our proposal and VGG16 for the VAD model prediction task. Comparisons where a network significantly outperforms the other (i.e., \(p\)-value < 0.05) are marked with the symbol \({}^{*}\). The best result for each comparison is highlighted in bold. If we further analyse the results, Table 2 shows a slight tendency towards a better performance of CLS D, i.e., the classifier that uses the predicted VAD values before classifying the categories, particularly in the case of our proposed network.