## 3 Methodology
### Deep Neural Networks
#### 3.2.2 Recurrent Neural Network and LSTM
5:



### Hybrid Scheme

Inspired by prior investigations, CNNs can expand spatially and LSTM can expand in long-term temporary memory. If there are high-frequency sensor measurements or various sensors involved, it is superior to add a convolutional neural network layer before the LSTM layer. Convolution layers are interspersed with pooling layers to reduce computation time and to gradually build up further spatial and configural invariance. LSTM can discover long-term temporary dependency features. The fully connected neural network is good at mapping, and the hybrid DNN will eventually learn an excellent regression model to predict equipment RUL. The proposed scheme is shown in Figure 6. The objective of the proposed scheme is to feed the raw sensor data after feature extracting and nonlinear regression by multilayer networks, then the equipment RUL is obtained. We utilize a one-layer convolutional neural network that is composed of one convolution layer and one pooling layer. Meanwhile, we utilize two LSTM layers. For the fully connected neural network, we utilize a multilayer perceptron, which consists of two hidden layers and 50 neurons included in every hidden layer. The mean square error (MSE) is used as the cost function. \[MSE=\frac{1}{N}\sum_{i=1}^{N}\left\|RUL_{g_{i}}-RUL_{y_{i}}\right\|^{2}, \tag{17}\]

where \(RUL_{g_{i}}\) and \(RUL_{y_{i}}\) are the prediction value and true RUL, respectively. \(N\) is the total number of samples in the testing set. The Adam optimizer (shown in Algorithm 2), an adaptive learning rate optimization algorithm [52], is employed to train the model. Figure 6: Hybrid neural networks for RUL prognostics. ### Data Preparation

#### 3.4.1 Variance Threshold

The measure data is complex and multidimensional. Features with low variance are convergent, i.e., features are not obviously distinguishable during a sequence, they are ineffective against prognostics performance [53]. Features with a dataset variance lower than _threshold_ will be removed. It is formulated as

\[h(t)=\tanh(Ux(t)+Wh(t-1)+b_{h}), \tag{12}\]0

where \(m\) is the sample size and \(\bar{x}\) represents the mean of the feature. #### 3.4.2 Normalization

Sensor data value scales may be diversiform. To accelerate the convergence rate, we need to normalize the process with respect to each sensor before utilizing data. Two main data normalization methods are exploited widely [54]: z-score normalization (Equation (19)) and min-max normalization (Equation (20)). Specifically, the z-score normalization method makes the sensor data follow the normal distribution, and the min-max normalization method makes the sensor data scale within the range of \([0,1]\). \[h(t)=\tanh(Ux(t)+Wh(t-1)+b_{h}), \tag{12}\]1

\[h(t)=\tanh(Ux(t)+Wh(t-1)+b_{h}), \tag{12}\]2

where \(x_{std}\), \(x_{max}\), and \(x_{min}\) are the standard deviation, maximum, and minimum with respect to each sensor, respectively. ### Metrics

For the sake of evaluating the performance of the proposed scheme, some evaluation metrics of prediction performance are adopted. These are root mean square error (RMSE), mean absolute percentageerror (MAPE), mean absolute error (MAE), and penalty score function. The first four metrics adopted are popularly applied in prognostics tasks. Dissimilarly, the penalty score function is given by PHM2008 competition [44] specifically for RUL prognostics evaluation. The penalty score function is asymmetric, as early prediction is preferred over late prediction. It can be seen from Figure 7 that the tolerance of advanced prognostics is greater than delayed prognostics for the same penalty score value. Evaluation metrics are formulated as follows:

\[h(t)=\tanh(Ux(t)+Wh(t-1)+b_{h}), \tag{12}\]3

\[h(t)=\tanh(Ux(t)+Wh(t-1)+b_{h}), \tag{12}\]4

\[h(t)=\tanh(Ux(t)+Wh(t-1)+b_{h}), \tag{12}\]5

\[score=\left\{