# Analysis of Deep Learning-Based Decision-Making in an Emotional Spontaneous Speech Task
## 3 Emotion Detection
### Convolutional Neural Model  
Convolutional neural architectures have become a standard in image processing over the last few years. However, other types of tasks have also taken advantage of these architectures by adapting the problem and addressing it with computer vision techniques. For example, audio analysis can be performed with computer vision techniques if the \begin{table}
\begin{tabular}{c c c c} \hline \hline \multirow{2}{*}{**Categorical Model (\%)**} & \multicolumn{3}{c}{**Dimensional Model**} \\ \cline{2-4}  & **Arousal (\%)** & **Valence (\%)** & **Dominance (\%)** \\ \hline Angry: 30.2 & Excited: 25.5 & Positive: 29.0 & Dominant: 26.2 \\ Happy: 15.3 & Neutral: 74.5 & Neutral: 54.4 & Neutral: 73.8 \\ Calm: 54.5 & & Negative: 16.6 & \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Class distribution of the annotated data for categorical and VAD model. audio is represented by a spectrogram. In addition, other areas such as speech and natural language processing have also taken advantage of the potential of convolutions for the analysis of temporal sequences [80]. In this work, we propose a simple and light convolutional network architecture (a network with 43K parameters) and compare it with the VGG16 convolutional network [9], a model widely used in the literature but that consists of 134M parameters, which makes it difficult to understand its behaviour. Both network architectures are designed to obtain, from the speech spectrogram, the joint classification of emotional state in terms of both representations, the categorical model and VAD dimensions, as shown in Figure 1. On the one hand, after a number of convolutional and pooling layers are applied, three scalar values corresponding to each dimension of the VAD model are computed through three linear layers with a sigmoid activation function at point A. These values are then converted to discrete VAD predictions linearly, at point B. On the other hand, the categorical model is inferred in two ways. First, at point C, the categories are predicted based on the output of the CNNs. Second, at point D, the scalar predictions of the VAD model are used instead. Intuitively, this second prediction might perform better as it could explicitly take advantage of the multi-tasking capabilities of the networks. As for the training procedure, a resampling strategy is used when training both networks to deal with the imbalance of the data, as observed in Table 1. The employed method consisted of selecting the samples inside a batch using a random function that can take into account the weight given to each sample. The weight \(W_{x}\) for each sample \(x\) was computed, as Equation (1) shows  
\[W_{x}=min\left\{\frac{|X|}{|X_{c}|},\beta\cdot\underset{\forall c\in C}{\text{ min}}\left\{\frac{|X|}{|X_{c}|}\right\}\right\} \tag{1}\]  
Figure 1: Representation of the structure of the VGG16 [9] network and our proposal. In both cases, the convolutions (blue boxes) and max-poolings (red boxes) help to extract a set of appropriate features, while the linear ones are in charge of performing the class discrimination. Point A represents a dimensional prediction of the VAD model, while point B provides the discretisation of each of the dimensions. On the other hand, the categorical model is inferred directly in point C, and from the scalar VAD model predictions, in point D.  
where \(|X|\) is the number of samples in the corpus, \(|X_{c}|\) is the number of samples in the class which sample \(x\) belongs to, and \(\beta\) is the oversampling coefficient (in this work it was chosen a value \(\beta=2\)). In this way, the samples of the minority classes appear proportionally more times, but never more than twice as much as a sample from the majority class. Regarding the optimisation hyperparameters, the Adam optimiser was chosen with a learning rate of \(10^{-4}\) and with a batch size of 16. The models were trained throughout 7K iterations. Note that our network is trained from scratch, whereas VGG16 is fine-tuned from the publicly available pretrained checkpoint.