# An Ensemble Model for Multi-Level Speech Emotion Recognition
## 3 Proposed Method  
### Design Route for the Overall Model  
The overall model framework adopts the model structure of ensemble learning, which involves 3 emotional judgment experts with different focuses, which make emotional decisions in their respective fields. Through ensemble learning, the emotional decision with the highest recognition rate is finally obtained.  
As an important branch of information fusion, ensemble learning is generally used for classifier integration. Its goal is to integrate multiple kinds of information to obtain better classification accuracy. According to the characteristics of the differences in emotional speech, the fusion method of weighted voting at the decision level can be adopted; that is, the output of multiple experts is negotiated according to the customary rules, and finally a unified opinion is reached. The emotional marker with the highest number of votes is used as the final result of the sample to be tested.  
Assuming that the sample to be tested is \(x\) and the number of votes identified as \(W_{j}\) is \(v_{j}\), then the formula for weighted voting is as shown in Equation (1):  
\[v_{j}(x)=\sum_{j=1}^{m}W_{j}d_{j}(x) \tag{1}\]  
Here, the weight is designed in the voting process and is determined according to the training data to estimate the confidence of the classifier. \(d_{j}(x)\) indicates that sample \(x\) is determined by the NO. \(j\) expert as a weighted label of the emotion category \(W_{j}\).  
If the accuracy of the voting results of each expert is improved, the accuracy of the recognition can be improved according to this design idea. Once some experts have scored mistakes, their confidence would be reduced, and other experts lead the recognition of speech emotions. This design idea has certain generalization capability. In summary, in the above scheme, the accuracy and confidence distribution method of the expert is the key element, and it is also the content and experimental point to be described later.  
Expert 1: A spectrogram is a kind of 3-dimensional spectrum that contains rich time and frequency information. The expert enhances the data of the original speech signal. The data source of the model is the map of the original audio generated under various amplitudes, the time domain, and the frequency domain transformed map. Then they are fed into a 2-channel CNN model. Expert 1 can make more effective use of the time-domain and frequency-domain features of speech to highlight the expression of local features.  
Expert 2: Based on the characteristics of short-term invariance of speech, expert 2 focuses on the subdivided audio files. After processing the PCA, the mean value features of the spectrogram can be fused. Speech is sequential, so it is suitable for GRU, which is a deep model of time series. Combined with the attention mechanism, the model can further enhance the expression of global features of short-term speech.  
Expert 3: LLD features and their HSFs can describe features from a global point of view, and can be used as a powerful supplement to existing time series features. Therefore, for segmented speech, the local original speech signal is fed into the CRNN model based on the features of the speech spectrum, generating an abstract representation of advanced features obtained from the deep learning model, which is then combined with HSFs, and the joint representation is fed into the BLSTM model, which is suitable for temporal data. This expert extracts global features of speech from multiple perspectives.  
Each expert's focus and input are different. When setting the confidence level, comprehensive consideration is needed. The design method of confidence is described in detail later.  
Considering that the recognition performance of the deep neural network requires high data volume, a large amount of new data is generated in the preprocessing segment, which can minimize the impact on the model due to insufficient data volume, and can also speed up the network training and reduce the impact of overfitting during training on small datasets.