# A Novel Heterogeneous Parallel Convolution Bi-LSTM for Speech Emotion Recognition
## 3 Methods
### Heterogeneous Parallel Conv-BiLSTM (HPCB)  
HPCB contains two heterogeneous branches, as shown in Figure 1. The purpose of designing the two heterogeneous branches is to project the original data into different transformation spaces for calculation, so as to better represent the original emotional speech.  
The left one contains two dense layers and a Bi-LSTM layer, and it processes the temporal information of input data, the number of neurons in the two dense layers is 512, and the number of memory cells in the Bi-LSTM layer is 256.  
The right one contains a dense layer, a convolution layer, and a Bi-LSTM layer, and it handles the spatiotemporal information of input data. The number of neurons in the dense layer and one-dimensional convolution layer is 512, and the number of memory cells in the Bi-LSTM layer is 256. 1D convolution is used to extract the spatial information of speech emotion signals in the time dimension, and Bi-LSTM is used to extract context information from the front and back ends of speech.  
To represent emotional speech more completely, the features extracted from the left- and right-branches are fused through  
\(Concatenate(\cdot)\) operation, where \(Concatenate(\cdot)\) is the joint feature matrix. This operation increases the dimension of the features describing the original data, but the information corresponding to each dimension feature does not increase.  
A \(Softmax(\cdot)\) function is used to classify emotions according to emotional signals from the concatenation layer that concatenates and fuses the information from the two heterogeneous branches. The number of neurons in the \(Softmax(\cdot)\) layer is equal to the number of emotion categories in the corresponding database.  
The proposed parallel learning architecture accelerates the convergence in deep learning, it also contributes to capture and retrieve spatiotemporal coherence information, which plays an essential role in improving learning performance of the model.  
The proposed HPCB employs a valid convolution operation, and it performs convolution operation only for time dimensional tensor. This means that the convolution kernel moves inside the one-dimensional tensor. The output \(h\) of convolution is calculated as:  
\(h\ =f(h)\)  
\[h=f(\frac{h^{1}*F}{S}\times N). \tag{1}\]  
where \(h^{1}\) denotes the output of dense layer, \(F\ =\ [k_{1},\ k_{2},\ \ldots,\ k_{512}]\) denotes the convolution kernel, \(N\) denotes the number of filters and is set to 512. \(S\) denotes the stride and is set to 1 by default.  
Bi-LSTM is adept at context modeling on time series data. Different from the traditional neural network, there is a connection between any two neurons in the same hidden  
Figure 1: The topology of the proposed deep learning model HPCB.  
layer. Bi-LSTM receives the input from the convolution layer, and it helps the HPCB model to extract spatial and temporal coherence emotion features more effectively.  
The outputs \(y_{L}^{B}\) and \(y_{R}^{B}\) of the left and right branches are concatenated in the concatenate layer to merge information:  
\[F_{c}=concatenate(y_{L}^{B},y_{R}^{B}). \tag{2}\]  
On the top of model HPCB, there is an output layer using \(Softmax(\cdot)\) to classify emotion. It is noted that HPCB employs the Adam optimization in its learning procedure. Compared to the original Bi-LSTM or CNN, HPCB automatically extracts information both in the spatial and time domains in a parallel learning architecture by exploiting the pros of the two models.