## 3 Experimental Study
### Experimental Setup
#### 3.1.8 Network Configuration and Hyperparameter Setting  
The proposed RUL prediction approach contains deep neural networks. The hyperparameters for network design affect the prediction performance of deep neural networks. Figure 11: Relationship between the RUL versus the difference between the last and first data points for all engines in training set. Grid search [31; 33; 61]; random search [24; 62]; and heuristic algorithms, such as genetic algorithm [49], are used for selecting the best hyperparameter combination. In this paper, the proposed neural network structure 1D-CNN-LSTM is given in Section 2.6. The proposed hybrid neural network structure was compared with the base versions of a 1D-CNN and the LSTM network. The structures of the base 1D-CNN and base LSTM network are presented in Figures 12 and 13, respectively. The hyperparameters evaluated were the number of layers, batch size, number of epochs, filter size, and number of units in the first layer of the network. In this study, a general procedure was created to design the two-layer neural networks. For clarity, in the two-layer network, the filter size or the number of units in the first layer started with a value, and it was halved in the second layer. This was the general procedure used to design the two-stage neural networks in this study. It also gave helpful insight into the effects of the low and high values of the filter size and the number of units used in these neural networks. The values of hyperparameters are shown in Table 5. For the 1D-CNN-LSTM hybrid neural network, the value of 1 in the number of layers meant one 1D-CNN layer followed by one LSTM layer. Similarly, the value of 2 means two consecutive 1D-CNN layers and two consecutive LSTM layers. Figure 12: Base 1D-CNN structure. Figure 13: Base LSTM structure. \begin{table}
\begin{tabular}{c c} \hline \hline
**Hyperparameter** & **Range Values** \\ \hline Number of layers & \{1, 2\} \\ Number of epochs & \{20, 25, 30, 40, 50, 60, 80, 100\} \\ Batch size & \{64, 128, 256\} \\ Filter size of the first layer & \{32, 64, 128, 256\} \\ Number of units of the first layer & \{32, 64, 128, 256\} \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Hyperparameter values evaluated in the proposed approach. Cross-validation (CV) is an appropriate method for measuring a model's performance [63]. In k-fold cross-validation, the dataset is split into k non-overlapping folds, where k-1 folds are used for training and one-fold is used for testing. RMSE is the performance measure used in the k-fold CV process. The Adam algorithm was employed for the optimization of network weights. For the 1D-CNN, the kernel size was set to three, and max pooling with a pool size of three was used for the pooling operations. The activation functions used in all network layers except the output layer were ReLU. The number of fully connected layers was two, and the number of nodes used for those two layers was set to 50 for each neural network. Experiments were run on a personal computer with Intel Core i5-8250U (1.60 GHz) CPU, 8 GB memory, and Microsoft Windows 11 operating system. The programming language was Python, and the libraries used for building the neural networks were Tensorflow [64] and Keras [65]. The best hyperparameter combination was determined according to the main effects plots. First, we created a hyperparameter space by considering the full range of values for each hyperparameter. In this space, there existed 192 unique combinations resulting from various hyperparameter value permutations (two discrete values for the first hyperparameter, eight for the second, three for the third, and four for the fourth). We then ran each combination through ten statistically independent experiments to thoroughly evaluate the overall effects of each one. After calculating the mean performance of each hyperparameter value, main effects plots helped to examine the impact of a hyperparameter on the performance. Figure 14 shows the main effects plots for the hyperparameters of the base and hybrid networks.