## 3 Methodology
### Deep Neural Networks
#### 3.2.2 Recurrent Neural Network and LSTM  
A recurrent neural network (RNN) is a natural feed-forward neural network that has been applied successfully owing to its capacity to model highly non-linear data with a sequential nature [49]. A standard RNN is composed of a series of recurrent units to cope with time series problems. The unfold overview of RNN cell architecture is shown in Figure 4. It's different from the general fully connected neural network, as the output at the current time is also the input of the next time. In Figure 4, the same layer weight matrices \(U\), \(V\), and \(W\) are re-utilized for each step of computation throughout the sensor data, i.e., weight sharing. \(L(t)\) is the loss function; \(y(t)\) is the actual output. The symbol in the parentheses denotes the time-step for the recurrent unit. The RNN cell forward propagation is formulated as follows:  
\[h(t)=\tanh(Ux(t)+Wh(t-1)+b_{h}), \tag{12}\]  
\[o(t)=\sigma(Vh(t)+b_{o}), \tag{13}\]  
where \(o(t)\) is the prognostics output of the RNN cell. \(h(t)\) is the hidden layer. \(\sigma(\cdot)\) is the element-wise sigmoid activation function. \(tanh(\cdot)\) is the element-wise hyperbolic tangent activation function. \(b_{h}\) and \(b_{o}\) are the bias vectors. During the training process, a back-propagation through time (BPTT) algorithm is executed [50]. It is formulated as follows:  
\[\frac{\partial L(t)}{\partial V}=\sum_{t=1}^{n}\frac{\partial L(t)}{\partial o (t)}\cdot\frac{\partial o(t)}{\partial V}, \tag{14}\]  
\[\frac{\partial L(t)}{\partial W}=\sum_{k=1}^{t}\frac{\partial L(t)}{\partial o (t)}\cdot\frac{\partial o(t)}{\partial h(t)}\cdot(\prod_{j=k+1}^{t}\frac{ \partial h(j)}{\partial h(j-1)})\cdot\frac{\partial h(k)}{\partial W}, \tag{15}\]  
\[\frac{\partial L(t)}{\partial U}=\sum_{k=1}^{t}\frac{\partial L(t)}{\partial o (t)}\cdot\frac{\partial o(t)}{\partial h(t)}\cdot(\prod_{j=k+1}^{t}\frac{ \partial h(j)}{\partial h(j-1)})\cdot\frac{\partial h(k)}{\partial U}. \tag{16}\]  
Figure 4: Recurrent neural network (RNN) cell unfold architecture. However, there will be an accumulation of derivatives of the activation function during backpropagation process. Gradient vanishing or gradient explosion may occur. Hence, improving the RNN cell internal structure is positively efficient. The LSTM neural network was proposed by [PERSON] and [PERSON] in 1997 [PERSON] and [PERSON] in 1997 [PERSON] and [PERSON] (1997). LSTM is intended to address two special issues that standard RNNs can't achieve: one issue associated with the standard RNN is the \"fading memory\". Once the number of time-steps becomes large, the \"future\" time-steps will contain virtually no memory of the first inputs as there is no structure in the standard recurrent layer that individually controls the flow of the memory itself. The other issue is that an RNN model is required to confirm the window length in advance, which is formidable to automatically acquire in practical applications [PERSON] and [PERSON] (1997). LSTM is a special architecture of RNN. As is shown in Figure 5, the series recurrent unit is the LSTM cell rather than a simple RNN cell. An LSTM cell possesses long-term memory, which is attributable to three gates modulating the flow of information in the LSTM cell: the input gate, forget gate, and output gate. **Input gate \(i(t)\):** It controls what information will be passed to the memory cell based on previous output and current sensor measurement data. **Forget gate \(f(t)\):** It controls how the memory cell will be updated. **Output gate \(o(t)\):** It controls which information will be carried to next time-step. The LSTM forward propagation algorithm is given by Algorithm 1. ```
1:Inputs: Feature matrix \(x(t)\) and target matrix \(h(t)\). 2:Initialize: Randomly initialize the weight matrix \(W\) and bias vectors \(b\). 3:Forget gate:\(f(t)=\sigma(W_{xf}x(t)+W_{h_{0}}h(t-1)+W_{cf}c(t-1)+b_{f})\). 4:Input gate:\(i(t)=\sigma(W_{xi}x(t)+W_{hi}h(t-1)+W_{ci}c(t-1)+b_{i})\).