## 4 The Construction of the Hybrid Deep Neural Networks
### The Construction of LSTM Recurrent Neural Networks  
In the DEAP experiment, the stimulus intensity changes over 60 s. The emotion scores by the Subjects are often based on the most exciting part of the entire video. Therefore, we needed to model the context information for long-term sequences. As mentioned before, RNN is good at sequential modeling. However, a simple RNN must face the challenge of 'gradient vanish or explode' in back propagation when its dependencies are too long [47]. LSTM units have been adopted to replace the simple units of a traditional RNN. LSTM units combine gate mechanisms in their structures so that the key features of the timing data are effectively maintained and transmitted during the long-period calculation. The gate is able to forget the used information and the self-loop structure allows the gradient to flow for long durations [48]. A typical structure of a LSTM unit is illustrated in Figure 7. For comparison, Figure 7 shows the structure of two neural network units. The upper left corner of the figure is a simple recurrent neural network unit, and the LSTM unit is below the graph. As seen in Figure 7, the simple RNN unit only contains the feedback from the output to the input. However, the LSTM unit contains three gate structures, i.e., input gate, forget gate, and output gate, which determine what information from the prior step should be forgotten and what information in the current time step should be added into the main data flow. \(f_{i}\), \(f_{o}\) and \(f_{g}\) are the activation function of the input data, output data, and gate, respectively. In this study, they are all sigmoid functions. \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**Input Data** & **Convolutional** & **Max Pooling** & **Convolutional** & **Max Pooling** & **Dense Layer** & **LSTM RNN** & **RNN Output** \\  & **Layer 1** & **Layer 1** & **Layer 2** & **Layer 2** & & \\ \hline \multirow{2}{*}{\(<\)200 \(\times\) 200\(\times\) 3} & \(<\)2 \(\times\) 30 & \(<\)2 \(\times\) 2\(>\) & \(<\)2 \(\times\) 10 & \(<\)2 \(\times\) 2\(>\) & 6250\(\delta\)25 & 625\(\delta\)25 & \\  & \(<\)5 \(\times\) 5 \(\times\) 30 & \(<\)2 \(\times\) 2\(>\) & \(<\)2 \(\times\) 2\(>\) 10 & \(<\)2 \(\times\) 2\(>\) & 4000\(\cdot\)400 & 400\(\cdot\)400 & 4 \\  & \(<\)10 \(\times\) 10\(>\) 30 & \(<\)2 \(\times\) 2\(>\) & \(<\)2 \(>\) 2\(>\) 10 & \(<\)2 \(>\) & 1000\(\cdot\)100 & 100\(\cdot\)100 & \\ \hline \hline \end{tabular}
\end{table} 
Table 4: The configurations of CNN. The parameters are denoted as \(<\)input size/receptive field size/pooling size\(>\)\(\times\) \(<\)number of kernels/channels/out size\(>\). Figure 7: The Long Short-Term-Memory (LSTM) unit and simple recurrent network unit. Different gates generate decision vectors to decide what candidate information will be selected. Using '_Input Gate_' as an example, this generates vector \(\mathbf{i_{t}}\) with the hidden state \(\mathbf{h_{t-1}}\) from the prior LSTM cell and the current step's input \(\mathbf{x_{t}}\). The process of generating \(\mathbf{i_{t}}\) can be formalized as in Equation (4):  
\[\mathbf{i_{t}}=f_{g}(\mathbf{w_{i}x_{t}}+\mathbf{w_{i}h_{t-1}}+\mathbf{b_{i}}), \tag{4}\]  
where \(\mathbf{w_{i}}\) is the weighted matrix of the input function; and \(\mathbf{b_{i}}\) is the bias. The input candidate information \(\widetilde{\mathbf{C}_{t}}\) is also generated with \(\mathbf{h_{t-1}}\) and \(\mathbf{x_{t}}\). \(\widetilde{\mathbf{C}_{t}}\) can be formalized as Equation (5):  
\[\widetilde{\mathbf{C}_{t}}=f_{i}(\mathbf{w_{c}x_{t}}+\mathbf{w_{c}h_{t-1}}+ \mathbf{b_{c}}). \tag{5}\]  
The final updating information is the multiplication of the candidate information by the decision vector \(\widetilde{\mathbf{C}_{t}}\times\mathbf{i_{t}}\). Another gate is the forget gate, which generates vector \(f_{t}\) to determine if the prior unit's state \(\mathbf{C_{t-1}}\) should be reserved by multiplication \(\mathbf{C_{t-1}}\times f_{t}\).