# Deep Learning-Based Autoscaling Using Bidirectional Long Short-Term Memory for Kubernetes
## 6 Experiment and Evaluation
### Experiment Results
#### 6.4.3 Evaluation of Autoscaler  
Because the ARIMA model exhibits a poor prediction speed, it is not suitable for real-time autoscaling systems and thus is not considered in this experiment. For this experiment, we took a testing subset of the NASA dataset from 7 July 1998 09:51:00 to 1 July 1998 16:31:00. Figures 16 and 17 show the number of pods produced by the LSTM and Bi-LSTM autoscalers for the one-step prediction. A comparison of both autoscalers is shown in Table 6, where Bi-LSTM achieves better performance in \(\Theta_{O}\) and \(T_{O}\) in one-step and five-step predictions. These results show that the proposed Bi-LSTM autoscaler tends not to have over-provision, and when it does, it occurs for a shorter time as compared to LSTM. In contrast, the LSTM autoscaler achieves better performance in \(\Theta_{U}\) (both one-step and five-step predictions), and the \(T_{U}\) of both autoscalers are almost equal in the one-step prediction. It is shown that LSTM tends not to have under-provision as compared to Bi-LSTM. Finally, the Bi-LSTM autoscaler outperforms the LSTM autoscaler in elasticity speedup \(\epsilon_{n}\) in both one-step and five-step predictions, which indicates that Bi-LSTM has a higher autoscaling gain over LSTM.  
Figure 14: Forward LSTM with one-step prediction on the testing set (from 1 July 1998 20:31:00 to 2 July 1998 05:00:00).  
Figure 15: Bi-LSTM with one-step prediction on the testing set (from 1 July 1998 20:31:00 to 2 July 1998 05:00:00).  
Figure 18 shows the changes in the number of pods of the proposed proactive custom autoscaler (PCA) and the HPA for the given workload in a real environment. By default, every 15 seconds, also known as the sync period, the HPA will compare the collected metric values against their predefined thresholds. In this experiment, the metric for the HPA is CPU usage, and its threshold is 70%. Once the average CPU usage exceeds the predefined threshold, the HPA will calculate and increase the number of pods in the cluster. There will be a delay time for creating new pods and assigning them to the nodes, which is the common drawback of the reactive approach. It only reacts to workload changes after it happens and spends a certain time reconfiguring the system to adapt to the new workload. In addition, the default value for down scaling is 5 minutes. This means that the HPA will keep monitoring the average CPU usage, and if its value is below the threshold for 5 min, HPA will start scaling down. This mechanism of HPA is called cooldown/delay [12]. It is shown in Figure 18 that the scaling down actions happen quite late, around the 20 th and 83 rd min, which leads to a waste of resources due to a slow response to traffic demand  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Type** & **No-Autoscaling** & \begin{tabular}{c} **LSTM** \\ **1 Step** \\ \end{tabular} & \begin{tabular}{c} **BI-LSTM** \\ **1 Step** \\ \end{tabular} & \
\begin{tabular}{c} **LSTM** \\ **5 Steps** \\ \end{tabular} &
\begin{tabular}{c} **BI-LSTM** \\ **5 Steps** \\ \end{tabular} \\ \hline \(\Theta_{O}[\%]\) & 593.775 & 1.610 & **1.287** & 3.405 & **1.157** \\ \(T_{O}\) [\%] & 97.60 & 49.400 & **42.80** & 55.6 & **18.2** \\ \(\Theta_{U}[\%]\) & 0.189 & **0.806** & 0.964 & **1.784** & 3.312 \\ \(T_{U}\) [\%] & 2.2 & 30.0 & **29.8** & **35.80** & 63.2 \\ \(\epsilon_{n}\) & 1 & 1.881 & **1.974** & 1.188 & **1.529** \\ \hline \hline \end{tabular} Bold values indicate the best results.  
\end{table}
Table 6: Evaluation of auto-scaler.  
Figure 16: Number of pods produced by the LSTM auto-scaler.  
Figure 17: Number of pods produced by the Bi-LSTM auto-scaler.  
change. In contrast, the proactive custom autoscaler (PCA) has better performance in accuracy and speed than HPA because PCA can predict the trend of workload and perform the scaling up/down actions more accurately and ahead of time. Moreover, when the workload decreases, the PCA was found to remove a part of pod replicas, thus making it efficient when handling a burst of workload in the near future.