# An Ensemble Model for Multi-Level Speech Emotion Recognition
## 4 Design of Experiments and Analysis of Results  
### Experimental Preparation  
The IEMOCAP dataset, which was designed to study the interaction of binary expressions in multiple modes, was used for the experiments. It was collected in 5 binary sessions of 10 themes using motion capture and audio/video recording. Each session consists of a different conversation in which a male and a female actor execute a script and participate in a spontaneous impromptu conversation triggered by an emotional scene cue. At least three evaluators used the classified emotional tags selected from the collection to annotate each utterance in the dataset. The optional categories were happy, sad, neutral, angry, astonished, excited, frustrated, disgusted, fearful, and others. When at least two-thirds of the evaluators gave the same emotional tag, the existing types of scripts and impromptu conversations in the dataset were no longer subdivided.  
For the 10,039 items of standard voice data in this dataset, related similar emotions were combined to remove less correlated samples. Finally, four types of emotional sample data are summarized. The excited and happy categories were combined to form a new happy category. In addition, there are sad categories, angry categories, and neutral categories. The remaining 6 categories of sample data were discarded. Based on the classification method, a total of 5531 samples were retained, with the following number belonging to each type: angry: 1103; happy: 1636; neutral: 1708; and sad: 1084.  
Then, in units of 1 s, the batch of samples was subjected to basic segmentation, and a total of 21,517 pieces of sample data was obtained. After the division, the numbers of samples were: angry: 3448; happy: 5181; neutral: 4313; and sad: 4270. Aside from the small size of the angry category, the data volume of the other categories was more balanced. In the preprocessing of this corpus, the speech segmentation length used was less than the traditional average segmentation length of 2.5 s, based on the sample data volume balance of this corpus. Experiments were performed using five-fold cross-validation; 80% of the data was used to train deep neural networks, and the remaining 20% was used for testing and verification of accuracy.  
When designing the overall model, each spectrogram has a size of \(32\times 513\) dimensions. In the CNN and CRNN models, the batch size is 100 and the maximum number of training items is 100,000. The learning rate is set to 0.001 and the dropout is set to 0.7. In addition, Adam is used as the optimizer. The mean square error is used as a loss function.  
The CNN infrastructure consists of three convolutional layers. The first layer has 64 filters of size \(3\times 3\), and layers 2 and 3 contain 96 filters of size \(5\times 5\). The ReLu function is used for all three convolutional layers. The size of the two largest pooling layers is \(2\times 2\), and the size of the stride is 1. There are 2 nodes in the Softmax layer, and cross entropy is used as the objective function for training.  
In the LSTM model, a bi-directional three-layer structure is adopted. The batch size is 300. The maximum number of training items, learning rate, loss function, and optimizer are the same as CNN, and the dropout is set to 0.5. In the HSF level, 16 LLD features and 4 HSFs are selected to form a 20-dimensional feature set.