## 2 The Proposed DCLNN Method for Failure Time Prediction
### Convolutional LSTM Neural Network
#### 2.4.2 Long Short-Term Memory Neural Network  
After data sequences have been preprocessed, a long short-term memory neural network (LSTM NN) is applied in this paper for data modeling. To overcome the gradient vanish of traditional RNNs, a LSTM NN was proposed. LSTM is one of the special structures of RNNs that can solve the memory problem of neural networks. It can be applied to process and predict important events with long intervals and delays in time series.  
With a special gate and cell in the hidden layer, LSTM can effectively update and deliver critical information in time series. Compared to traditional RNNs, LSTM has stronger capacities of information selection and time series learning, which can solve the problem of long-term dependencies by using remote context information for current prediction tasks. An LSTM is composed of one input layer, one hidden layer, and one output layer. Unlike the traditional NN, the basic unit of the hidden layer is a memory block, and LSTM adds a 'processor' in the algorithm to judge whether the information is useful or not, which is called a cell [27].  
The typical structure of an LSTM cell is shown in Figure 6. An LSTM cell is configured mainly by three gates: Input gate, forget gate, and output gate. The basic idea of LSTM is that when information enters the network, the information that meets the requirements is left by the input gate, the non-compliant information is discarded by the forget gate, and finally new information is generated by the output gate.  
Given \(x(t)\) is the input time series, the output is denoted as \(h(t)\) and the memory of the cell is denoted as \(s(t)\). Three gates of the LSTM NN can be given as follows:  
(1) Forget gate  
This unit is decided by considering the current input, the previous output, and the previous memory. It will produce a new output and change its memory. \(\sigma\) determines how much information can be left.  
\[f(t)=\sigma\Big{(}W_{f}\cdot[h(t-1),x(t)]+b_{f}\Big{)} \tag{10}\]  
(2) Input gate  
Figure 5: Sparse connection and shared weights in CNN.  
Figure 6: A cell of a long short-term memory neural network (LSTM NN).  
The input of this unit is the same with the forget gate, but the input gate determines the extent to which new memories should affect old memories. Meanwhile, this unit determines how much new information should be delivered to the next cell. Finally, the cell state is updated through discarding the information that needs to be discarded and adding new information.  
\[i(t)=\sigma(W_{i}\cdot[h(t-1),x(t)]+b_{i}) \tag{11}\]  
\[g(t)=\tanh(W_{C}\cdot[h(t-1),x(t)]+b_{C}) \tag{12}\]  
\[s(t)=f(t)*s(t-1)+i(t)*g(t) \tag{13}\]  
(3) Output gate  
Based on the new cell state, some information of the new state should be added to the output and the output information of the cell will finally be determined by deterministic information that needs to be output.  
\[o(t)=\sigma(W_{o}[h(t-1),x(t)]+b_{o}) \tag{14}\]  
\[h(t)=o(t)*\tanh(s(t)) \tag{15}\]  
where \(\sigma\) denotes the sigmoid function defined in the equation and the tanh function is defined in the equation:  
\[\sigma(x)=\frac{1}{1+e^{-x}} \tag{16}\]  
\[\tanh(x)=\frac{\sinh(x)}{\cosh(x)}=\frac{e^{x}-e^{-x}}{e^{x}+e^{-x}} \tag{17}\]  
The learning process of LSTM mainly includes the error backpropagation process and optimization algorithm. The backpropagation through time (BPTT) algorithm is applied in the error backpropagation process of LSTM [28].