# Stacked Ensemble of Recurrent Neural Networks for Predicting Turbocharger Remaining Useful Life
## 7 Experimental Results  
In this section, first, the results of applying covariate shift detection method are presented. Then, since we decided to label the data according to TTE labeling, the results of a corresponding regression problem will be reported. The results of stacked ensemble will be compared to simple ensemble aggregation such as average and median of all base models predictions. Also, ensemble result will be compared to the best individual base model result as well. In order to explain the results, we will also analyse model correlations and their relations to prediction errors. In Algorithm 1, LightGBM model with the parameters reported in Table 4 is used as the classifier for feature selection. Now, we need to find a proper threshold (\(tr\)) to compare against LightGBM's AUC for removing features. Different values of (\(tr\)) would lead to selection of different subsets of features. For each subset of features obtained for a specific (\(tr\)), an LSTM with the Mean Absolute Error(MAE) cost function and the architecture shown in Table 1 is trained. Finally, the obtained MAEs for different values of (\(tr\)) are compared to get the best (\(tr\)) and features subset. \begin{table}
\begin{tabular}{c c c} \hline \multicolumn{2}{c}{**First Architecture**} & \multicolumn{2}{c}{**Second Architecture**} \\ \hline No. of layers & 4 & 3 \\ \hline Layers type & (1-D CONV,LSTM,Dense,Dense) & (1-D CONV,LSTM,Dense) \\ \hline No. of CONV filters & 128 & 64 \\ \hline No. of units & (50,20,1) & (100,1) \\ \hline Units & (relu,relu,linear) & (relu,linear) \\ \hline Dropout & (0.5,0.2) & (0.5) \\ \hline Optimizer & ADAM & ADAM \\ \hline learning rate & 0.001 & 0.001 \\ \hline Cost function & MAE & MAE \\ \hline \end{tabular}
\end{table} 
Table 2: CONV-LSTM architectures and parameters. \begin{table}
\begin{tabular}{c c} \hline No. of layers & 2 \\ \hline Layers type & (Dense,Dense) \\ \hline No. of units & (40) \\ \hline Dropout & (0.2) \\ \hline Optimizer & ADAM \\ \hline learning rate & 0.001 \\ \hline Cost function & MAE \\ \hline \end{tabular}
\end{table} 
Table 3: Meta model architecture and parameters. For training the LSTMs, \(n_{timesteps}\) in the tensor structure \((n_{samples},n_{timesteps},n_{features})\) is 20, and \(n_{features}\) comes from the selected threshold for Algorithm 1. The result of this experiment is shown in Figure 7, which indicates threshold of 0.8 as the best choice--which leads to 355 features. The problem of predicting time to failure is formulated as a regression task using TTE labeling. Table 5 shows the MAE of different models including ensembles and best individual model. The first row in Table 5 is a very naive estimation of TTE, namely the average TTE across all vehicles. This is included as a baseline for comparison. From Table 5, it can be seen that all ensemble approaches beat the best individual model on test data. The best individual model reported is the first CONV-LSTM architecture defined in Table 2. However, the result of Stacked ensemble outperforms both simpler approaches (mean and median aggregation of the results of base models). Figures 8 and 9 show the prediction of stacked ensemble predictions and best model prediction against target values, respectively. As can be seen in Figure 9, predictions of the best model generally drop too early in comparison to the target TTE. Also, it can be seen that there are more unnecessary fluctuations in its predictions. On the other hand, the stacked ensemble predictions drop much closer to the target TTE, with not nearly as much fluctuation. \begin{table}
\begin{tabular}{c c} \hline Objective & binary \\ \hline Metric & AUC \\ \hline Boosting & GBDT \\ \hline Learning rate & 0.001 \\ \hline Max depth & 7 \\ \hline feature\_fraction & 0.7 \\ \hline bagging\_fraction & 0.7 \\ \hline \end{tabular}
\end{table} 
Table 4: LightGBM setting and parameters. Figure 7: LSTM error results for choosing threshold for feature selection in Algorithm 1.