## 3 Methods
### Long Short-Term Memory RNNs  
Due to the long-term correlation of the speech signal, the recurrent neural network is more suitable for completing the speech recognition modeling task than the conventional DNN. The RNN adds a feedback connection to the hidden layer. That is to say, part of the input of the RNN hidden layer at the current moment is the hidden layer output of the previous moment, which allows the RNN to obtain information of all previous moments through the recurrent feedback connection, and it gives RNN memory. However, the information of RNN can only be passed to the neighboring successors. When the output is close to the relevant input information, an ordinary RNN can be competent. When this time interval is very long, although theoretically RNN can handle this long-term dependence problem, it has not been successful in practice. [PERSON] et al. [22] and others conducted an in-depth study of the problem. They found that the fundamental reason for making RNN training difficult is the vanishing gradient and gradient explosion problem. Therefore, [PERSON] & [PERSON] [PERSON] & [PERSON] (2013) proposed a long short-term memory recurrent neural network, which was recently improved and promoted by [PERSON] (2014); [PERSON] & [PERSON] (2015); [PERSON] & [PERSON] (2015). The LSTM maintains the error at a more constant level through a specific gating unit, allowing the RNN to learn multiple time steps, thereby avoiding the vanishing gradient. The LSTM stores information in gated units outside the normal flow of the recurrent neural network. These units can store, write, or read information, just like data in the computer's memory. The unit determines which information is stored by the switch of the door and when it is allowed to read, write or clear the information. However, unlike digital memories in computers, these gates are analog and contain element-by-element multiplication of sigmoid functions whose output ranges are all between 0 and 1. When compared to digital storage, analog values have the advantage of being differentiable. So it is suitable for back propagation. These gates switch on the basis of the received signal. Similar to the nodes of the neural network, they use their own set of weights to filter the information and decide whether to allow the information to pass based on their strength and the content of the import. These weights, like the weights of modulation inputs and hidden states, are adjusted through the learning process of a recurrent neural network. The memory unit learns when to allow data to enter, leave, or be deleted by an iterative process in which the error propagates back and the weights are adjusted by gradient descent. A graphical comparison between RNN blocks and LSTM blocks is shown in Figure 1. The LSTM block consists of a set of memory cells c and three gates: input gate i, forget gate f, and output gate o, which are used to control the flow of information. The following is a brief description of several gating units:  
* Memory unit c: they store the network's time status;
* Input gates i: they decide whether to pass input information into cell c;
* Output gate o: they decide whether to output the cell information; and,
* Forget gate f: These gates adaptively reset the cell state. Figure 1: A comparison between Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) blocks. (**a**) The internal structure of the RNN block. The bold blue line indicates that RNN blocks store memories in the form of hidden state activation. (**b**) The internal structure of the LSTM block. Dashed lines indicate peephole connections. The bold red line denotes that the LSTM block retains memories in the form of memory cell states, and memory cell states across two consecutive time steps are linearly related by the gating unit. At the time step t, the formula of the LSTM block can be described, as follows:  
\[