# Prediction of Machine Failure in Industry 4.0: A Hybrid CNN-LSTM Framework
## 5 Hybrid CNN-LSTM Model for PdM
### Hybrid CNN-LSTM  
This section explains the proposed hybrid CNN-LSTM model as shown in Figure 4. The model contains a sequence of convolutional layers and a sequence of LSTM blocks that are combined using a fully connected layer at the final stage to produce an output. The input data are first passed through a convolutional layer and a pooling layer, and the corresponding features are extracted. The obtained features are given as input to the LSTM block, which is designed in such a way that it memorizes the historical data and the relative long-term dependencies. The traditional LSTM fails to capture long-term correlations; therefore we introduce a novel temporal skip connection that takes advantage of the regular pattern in input variables. For instance, the dataset used in this work shows a clear pattern on a daily basis. Based on the records at time \(t\) in historical data, our model can make predictions regarding the machine failure. Regular LSTMs can barely capture this type of information because of the long length of the past time-frame (24 h) and the subsequent optimization process. Thus we introduce an LSTM with a temporal skip connection capable of prolonging the temporal span of the information flow, thereby making optimization efficient and easy. The skip connection is added between the current hidden cells and the hidden cells in the same phase in an adjacent period. The updating process is formulated as follows:  
\[\begin{split} v_{t}&=\sigma(z_{t}W_{zv}+q_{t-x}W_{qv }+b_{v})\\ u_{t}&=\sigma(z_{t}W_{zu}+q_{t-x}W_{qu}+b_{u})\\ y_{t}&=tanh\big{(}z_{t}W_{zy}+v_{t}\odot(q_{t-x}W_{ qy})+b_{y}\big{)}\\ a_{t}&=(1-u_{t})\odot q_{t-x}+u_{t}\odot y_{t} \end{split} \tag{9}\]  
where the output of the CNN layer is given as input to this layer, and \(x\) is the number of hidden cells. The final layer of the hybrid model is the fully connected layer. The input to the fully connected layer is the hidden states of the skip connection at time \(t\), which is represented as \(p_{t}^{V}\), and \(x\) is the hidden states of the skip connection from the timestamp \(t-x+1\) to \(t\), which is in the form of \(p_{t-x+1}^{S}\), \(p_{t-x+2}^{S}\cdots p_{t}^{S}\). The final output of the fully connected layer is computed as follows:  
\[p_{t}^{D}=W^{v}p_{t}^{v}+Sum\sum_{i=0}^{x-1}W_{i}^{S}p_{t-i}^{S}+b \tag{10}\]  
where \(p_{t}^{D}\) is the final prediction result of the hybrid CNN-LSTM model.  
The parameter setting of the proposed CNN-LSTM for this experiment are given in Table 4.  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Parameters** & **Value** \\ \hline Convolutional layer filters & 64 \\ Convolutional kernel size & 1 \\ Convolutional layer activation function & ReLU \\ Convolutional layer padding & Same \\ Pooling layer pool size & 1 \\ Pooling layer padding & Same \\ Pooling layer activation function & ReLU \\ Number of LSTM hidden cells & 128 \\ Number of skip connections & 2 \\ LSTM activation function & tanh \\ Batch size & 32 \\ Loss function & RMSE, MAE \\ Learning rate & 0.0001 \\ Epochs & 100 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Different parameter settings.  
The data are first fed into a one-dimensional convolution layer, which extracts features and produces a three-dimensional output vector (None, 10, 64), with 64 being the size of the convolution layer filters. The vector then enters the pooling layer, where it is converted into a three-dimensional output vector (None, 10, 64). The output vector is then fed into the LSTM layer for training, and the output data (None, 128) from the previous is fed into another complete connection layer after training to acquire the output value; 128 is the number of hidden units in the LSTM layer. Skip connections are added between LSTM cells that receive input from the previous layers. Our hybrid CNN-LSTM has the settings as shown in Table 4. The hyperparameter settings for different experiments are summarized in the Table 5.