## 6 Experiment
### LSTM-Based Scheduling  
Our experiment is based on LSTM-based scheduling, utilizing a model trained on Google Cluster Trace data [20]. In the work presented in [20], it is stated that \"Most resource utilization measurements and requests have been normalized, including: (1) CPU (core count or core seconds/second); (2) memory (bytes); (3) disk space (bytes); (4) disk time fraction (I/O seconds/second). For each of the foregoing, we compute separate normalizations. The normalization is a scaling relative to the largest capacity of the resource on any machine in the trace (which is 1.0)\". In this study, we also assume that all CPU resources are homogeneous and normalized regarding maximum consumption as in Google Cluster Trace data [20]. The data includes cluster CPU usage rates at 5 min intervals, enabling the training of a model to predict the cluster CPU usage rate 5 min into the future. The model, built on insights from five experiments, analyzes CPU usage rates over the past 15 min to predict the rate for the next 5 min. The choice of predicting the CPU usage rate for the next 5 min is due to the high volatility in the Google Cluster Trace data. Through LSTM-based scheduling, the algorithm selects clusters with higher stability compared to algorithms considering only the current CPU usage. Figure 9: Flowchart of the automated cluster-recovery experiment. To validate this, we conducted two types of experiments. In the first experiment, we utilized test data comprising 20% of the total data from Figure 6. For each data point, we compared the current CPU utilization value with the value 5 min ahead. This process started from the 6681 st data point and iterated through the last data point, calculating the RMSE (Root Mean Square Error) metric. In the second experiment, we constructed an xLSTM model, following the predictive model architecture presented in Section 5.2. After hyperparameter tuning, we obtained the best parameters as follows: lookback window size = 3; prediction window size = 1; batch size = 64; learning rate = 0.0001; trainable LSTM parameters = 789,761; and training epoch = 100. The model was trained on data from the past 15 min corresponding to the lookback window value of 3 to predict CPU utilization 5 min ahead (prediction window value of 1). We compared the predicted values with the actual values in the test data and calculated the RMSE metric. As a result of the experiments, the RMSE value in the second experiment utilizing LSTM improved by 2.21% compared to the first experiment. This outcome validates the excellence of CPU utilization prediction using LSTM. Based on these results, we applied the experiment to real-world environments. In the experiments, Google Cluster Trace data is applied to predict CPU usage rates in the current environment. The cluster's CPU usage rates are adjusted at 5 min intervals, aligned with the patterns in the Google Cluster Trace data. This process aims to validate the model's prediction accuracy by adapting it to the real environment. In the event of a disaster, the model mimics data patterns, predicts CPU usage based on applied data in the actual cluster environment, and selects the cluster with the lowest predicted CPU usage rate for recovery. Figure 10 shows the Rancher configuration for the LSTM-based scheduling experiment. In the figure, green represents the systems we added, orange represents Velero, and blue represents components of Rancher. Each of the six clusters linked to the Rancher cluster comprises one master node and two worker nodes. The master node is a t2.medium EC2 instance with 2 vCPU, 4 GiB memory, and 30 GiB storage, whereas the worker nodes are t2.small EC2 instances with 1v CPU, 2 GiB memory, and 30 GiB storage. The CPU-utilization rates of the clusters were varied, which was intentional for the scheduling test. Figure 11 illustrates a YAML file for a pod operating to adjust the CPU-utilization rates of the clusters. This pod uses a very resource-light busybox image.