# An Ensemble Model for Multi-Level Speech Emotion Recognition
## 4 Design of Experiments and Analysis of Results
### Independent Experiment for Each Expert  
Many experimental verifications were carried out for each expert. The TensorFlow framework was used to build the network model structure. Experiments were performed on the spectrogram features and low-level descriptor features. To avoid the impact of different emotional imbalances, weighted accuracy (WA) and unweighted accuracy (UA) were used as indicators. The different sentiment classification models were tested. The first experiments focused on the features of the spectrogram, comparing the effectiveness of each expert. Experiment 1: Mainly used for testing expert 1, with 21,517 samples + spectrogram + CNN model as the baseline; the feature information does not use any processing method. Model 1: 5531 samples + spectrogram + CNN model; model 2: 2 \(\times\) 21,517 samples + spectrogram + CNN model (two-layer convolution and two-layer pooling); the original picture adds time and frequency domain transformation. Model 3: 3 \(\times\) 21,517 samples + spectrogram + CNN model; feature information adds original picture and PCA channel. Model 4: Expert 1, 8 \(\times\) 21,517 samples + spectrogram + CNN model, combined with time and frequency domain transformation. The accuracy of each model's emotion recognition was calculated separately. Table 2 shows the accuracy of the speech emotion recognition models after experimental verification. It can be seen from Table 2 that the fused set of time and frequency domain transforms had the best performance and the best WA and UA, which is superior to the single feature combination and the original picture form, and also superior to the CNN model with PCA structure. It can be determined that the current organization form of expert 1 can maximize the recognition accuracy of the spectrogram. Experiment 2: Mainly used for testing expert 2, with 21,517 samples + 513-dimensional + single-layer single-directional GRU model as the baseline; feature information is obtained by using the mean processing method. Model 1: 2 \(\times\) 513 dimensions (combined mean and mean square error) + single-layer single-directional GRU model. Model 2: 513-dimensional + single-layer single-directional GRU model + attention mechanism. Model 3: 513-dimensional + double-layer single-directional GRU model + attention mechanism. Model 4: Expert 2, 513-dimensional + three-layer bi-directional GRU model + attention mechanism. The accuracy of each model for emotion recognition was calculated separately. Table 3 shows the accuracy of speech emotion recognition models after experimental verification. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Model/Sample Set** & **WA** & **UA** \\ \hline Baseline: 21,517 samples & 46\% & 43\% \\ Model 1: 5531 samples & 33\% & 34\% \\ Model 2: 2 \(\times\) 21,517 samples & 54\% & 53\% \\ Model 3: 3 \(\times\) 21,517 samples & 50\% & 51\% \\ Current System: Expert 1, 8 \(\times\) & 69\% & 68\% \\
21,517 samples & & \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Test results of expert 1. It can be seen from Table 3 that because expert 2 combines the three-layer bi-directional GRU model and attention mechanism, it has the best WA and UA, so it performs best, surpassing the single-layer single-directional GRU model and the model that does not use the attention mechanism. However, it should be noted that after increasing the mean square error characteristic, the overall performance of the model is degraded. It can be determined that the current organization form of expert 2 can maximize the recognition accuracy of the comprehensive information in the local data of the spectrogram, but it is necessary to carefully select a part of the effective integration method. Experiment 3: Mainly used for testing expert 3. Feature sets that have LLD and HSF characteristics are added to expert 3, then the validity of the feature sets is confirmed. The baseline for this selection is a combination of the INTERSPEECH 2010 Paralinguistic Challenge feature set [25] and the LSTM model.