# Profit-Efficient Elastic Allocation of Cloud Resources Using Two-Stage Adaptive Workload Prediction
## 6 Performance Evaluation
### Prediction Performance Evaluation  
To evaluate the performance of the proposed prediction method, we compared it to other prediction models, including ATSMNN (which uses an NN instead of the LSTM used in ATSMNN-LSTM), LR, ARIMA, SVR, LSTM, Transformer, LSTM-Transformer, and NN. The prediction results for 30 randomly selected time points are shown in Figure 17.  
From Figure 17, all prediction methods fitted the workload trend, which means that the ATSMNN-LSTM and other methods could effectively predict the workload trend. To demonstrate that ATSMNN-LSTM can obtain a higher prediction accuracy, we compared the metrics used for assessment of the workload prediction using the different methods, as shown in Table 1.  
According to Table 1, the RMSE and MAPE values obtained with ATSMNN-LSTM (3.7805 and 0.0276, respectively) were lower than those of other models. Moreover, the \(R^{2}\) of the ATSMNN-LSTM model was 0.9522, which was closer to 1 and higher than those of the other models, demonstrating that the ATSMNN-LSTM had improved prediction performance. ATSMNN-LSTM achieved better prediction accuracy than a single LSTM or an NN. Furthermore, due to the high prediction accuracy of the LSTM, the prediction results of ATSMNN-LSTM were better than those obtained with the ATSMNN model. Moreover, the performance of the Transformer model was better than that of the ARIMA,  
Figure 17: Comparison of the prediction results for 30 time points. SVR, LR, and NN models, demonstrating that the Transformer model is more suitable for time-series forecasting compared to traditional approaches. However, the experimental results indicated that the Transformer model did not outperform the LSTM in terms of prediction accuracy. Although the Transformer exhibited excellent performance in sequence processing, its complex model architecture is prone to overfitting when using limited data and presents training difficulties. For the Wikimedia workload time-series used in this study, the Transformer may not outperform the LSTM in terms of single-task, short-term prediction performance. This finding aligns with the conclusions reported in [21]. Additionally, the experimental results indicated that the LSTM-Transformer model did not achieve high performance. Although the LSTM-Transformer model enhances the Transformer architecture by incorporating LSTM modules, it further increases the model complexity, leading to greater training difficulties and a higher risk of overfitting. To present the details of the prediction graphically, we compare the relative error (\(|\frac{\hat{y}_{i}-y_{i}}{y_{i}}|\)) and absolute error (\(|\hat{y}_{i}-y_{i}|\)) at 70 time points in Figures 18 and 19, respectively. As shown in Figures 18 and 19, compared to the other models, ATSMNN-LSTM guaranteed low and stable relative and absolute errors. Additionally, we show the cumulative distribution percentage error in Figure 20, where the \(X\)-axis indicates the percentage of the error threshold (PET), while the \(Y\)-axis indicates the percentage of prediction values below the error threshold percentage (PBET). According to Figure 20, the cumulative distribution percentage error of ATSMNN-LSTM was higher than that of other models at PBET values less than 60%. Thus, the prediction error of ATSMNN-LSTM was mostly between 0% and 60%, and its number of high prediction errors (i.e., 60-100%) was lower than that of the other models. Furthermore, when the cumulative distribution percentage error of the ATSMNN-LSTM decreased, the interval of the cumulative distribution percentage error between ATSMNN-LSTM and the other models increased. Thus, the number of small error prediction results of ATSMNN-LSTM is larger than that of the other models. These results prove that ATSMNN-LSTM achieved higher prediction accuracy in the comparison.