# An Ensemble Model for Multi-Level Speech Emotion Recognition
## 3 Proposed Method
### Expert 1: Double-Channel Model Based on CNN
#### 3.2.2 Design of Double-Channel Model Based on CNN  
A convolutional neural network is a deep neural network in which alternating convolution and pooled layers are stacked. The role of the convolution layer is to extract features. The neural unit of the current layer is connected to several feature maps of the previous layer through a set of convolution kernels, the convolution operation is performed, and the feature map of the current layer is obtained by adding the bias.  
The convolution kernel is shared by all neural units of the same feature map, which is called weight sharing, greatly reducing the size of the parameters. Each neural unit is only connected to a local area of the previous feature map. Each neural unit extracts features of the local area, and all of the neural units are combined to obtain global features.  
To obtain more comprehensive information from the feature parameters, multiple convolution kernels are used in the same layer network to obtain multiple feature maps.  
After the convolution operation, the feature map performs a down sampling operation at the pooling layer. The pooling unit calculates the main information of the local area in the feature map, removes the redundant information, and reduces the scale of the operation.  
Based on the above CNN model structure, a double-channel model structure was designed, as shown in Figure 1. Each channel of this model is composed of a CNN model, in which the input data source is different, which is the spectrogram generated under various amplitudes, the time and frequency domain transformed image. Since the CNN structure requires large-scale training data,it is necessary to dynamically generate different forms of pictures for the time and frequency domain features of the spectrogram.  
Each channel has a similar CNN structure except for the difference in the processing scheme of the spectrum. The CNN structure consists of 3 convolutional layers, 3 pooled layers, and 2 fully connected layers, for a total of 8 layers. The input information size of the first convolutional layer is \(310\times 310\times 3\), where 310 is the length and width of the spectrum and 3 is the 3 channels red (R), green (G), and blue (B). The spectrogram passes through 64 convolution kernels with a size of \(3\times 3\). After the convolution operation with a step size of 1, 64 feature maps are generated, and then the rectified linear unit (ReLu) activate function is used. After the maximum pooling operation, 64 feature maps are obtained. The input source of the second convolutional layer is the output characteristic values of the first layer. The calculation process is the same as the first layer. The third layer is the same, followed by a fully connected layer, which has a total of 1024 neurons. Finally, a dropout operation is performed on this layer to prevent overfitting of the model.  
After the first 2 layers are fully connected in the above 2 channels, all the outputs are combined as the input of the next fully connected layer, and the emotional class value is obtained by the softmax function.  
Through the combination of multiple levels of information, the parameters learned by the network cannot change drastically in the local area, and the generalization ability of the model for data feature learning is improved.