# Optimizing the Parameters of Long Short-Term Memory Networks Using the Bees Algorithm
## 4 Results and Discussion
### Turbofan Engine Degradation Simulation Dataset  
The dataset consists of time series data of 100 engines that start normally at the beginning, and then some faults appear during the series. The numerical data contain 26 columns starting with a unit number, time in cycles, 3 operational settings, and 21 sensor measurements [48]. The datasets contain 100 sequences, as each engine represents a sequence that varies in length. There are 100 observations for each of the training, validation, and testing sets. Because the dataset is numerical, a CNN was not needed to extract the features, so the hybrid BA-LSTM algorithm was applied using the MATLAB platform to predict the remaining operational cycles before engine failure. Since the forward LSTM was used in developing the LSTM architecture [45], only four parameters related to this  
Figure 16: Validation accuracy for the algorithms using 10-fold cross-validation (ECG dataset). \begin{table}
\begin{tabular}{c c c c c} \hline \hline  & **CNN** & **CNN-LSTM** & **BO-CNN-LSTM** & **BA-CNN-LSTM** \\ \hline Training set & 100\% & 100\% & 100\% & 100\% \\ Validation set & 85.37\% & 87.80\% & 87.80\% & 87.80\% \\ Testing set & 92.50\% & 92.50\% & 95\% & 95\% \\ Computation time & 3 min 47 s & 3 min 52 s & 4 min 2 s & 3 min 31 s \\ \hline \hline \end{tabular}
\end{table} 
Table 9: The classification accuracy and time (ECG dataset). layer were optimized using the BA, which were the learning rate factors for the input gate, forget gate, cell candidate, and fully connected layer. The following Table 10 shows the values of the LSTM parameters for the four evaluations of the BA. As can be seen in the previous table, the first evaluation yielded the minimum prediction error for the validation set with a value of 4.4272, so the global learning rate of 0.01 [45] was adjusted for the input gate by multiplying it by 1.0712, which resulted in a more optimum learning rate value of 0.0107. Similarly, the learning rate for the forget gate was improved to 0.0101 by using an adjustment factor of 1.0167. The new learning rate value for the cell candidate was 0.0107 after multiplying the global learning rate by 1.0747. The adjustment factor for the output gate was 1.0526, which resulted in a learning rate value of 0.0105. Finally, the performance of the fully connected layer was improved as well by specifying a customized learning rate of 0.0099 after multiplying the global learning rate by 0.9915. The following Table 11 summarizes the new learning rate values for the LSTM parameters. The following Figure 17 shows the training progress for the BA-LSTM algorithm using the new learning rate values stated above. The blue line represents the training progress, and the black line represents the validation set. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**LSTM Parameter** & **1** & **2** & **3** & **4** \\ \hline Learning rate factor for input gate & 1.0712 & 0.9557 & 1.0413 & 0.9048 \\ Learning rate factor for forget gate & 1.0167 & 0.9147 & 1.0244 & 0.9898 \\ Learning rate factor for cell candidate & 1.0747 & 1.0939 & 1.0716 & 1.0878 \\ Learning rate factor for output gate & 1.0526 & 0.9989 & 0.9378 & 1.0041 \\ Learning rate factor for fully connected layer & 0.9915 & 1.0902 & 0.9613 & 1.0519 \\ Prediction error for the validation set & 4.4272 & 5.2740 & 6.7158 & 11.6459 \\ \hline \hline \end{tabular}
\end{table} 
Table 10: The values of LSTM parameters in the four evaluations of BA (engine dataset). Figure 17: Training progress for the hybrid BA-LSTM algorithm (engine dataset). As can be seen from the figure, the training started with a root mean square error (RMSE) of almost 80 that decreased significantly during the first 20 iterations, and then the chart experienced a steady state around an RMSE value of 20. In the validation set, the chart followed the same pattern and reached an RMSE value of 17.062 at the end of the chart.