## 6 Experiment
### Automated Cluster Recovery  
Figure 8 shows the Rancher configuration for the automatic cluster-recovery experiment. In the figure, green represents the systems we added, orange represents Velero, and blue represents components of Rancher. The Kubernetes cluster running Rancher is referred to as the Rancher cluster. Although Rancher can operate as a container on a single server, in our experiment, it runs within a Kubernetes cluster because of the use of Velero. Clusters 1 and 2, linked to the Rancher cluster, each comprise one master node and two worker nodes. The master node is a t2.medium EC2 instance type with 2 vCPU, 4 GB memory, and 30 GB storage, and the worker nodes use t2.small EC2 instances with 1 vCPU, 2 GB memory, and 30 GB storage.  
Figure 9 shows the flowchart of the cluster auto-recovery experiment. In the figure, green represents the systems we added, orange represents Velero, and blue represents components of Rancher. Solid arrows represent the flow of backup file data, while dashed arrows indicate communication, commands, and similar actions. Velero, connected to the AWS S3 bucket, is installed in all clusters, including Rancher Cluster, Cluster 1, and Cluster 2. In step 2, if Cluster 1 loses functionality due to a disaster, Rancher detects this. In step 3, the detected status of Cluster 1 changes to disconnected, triggering the auto-recovery system. In step 4, the auto-recovery system uses Velero to look up the latest backup file of Cluster 1. In step 5, the restore command, including the name of this backup file, is transmitted to Cluster 2. In step 6, Cluster 2 executes the received command, using the backup file in the S3 bucket to restore the application that was running in Cluster 1 to Cluster 2.  
Figure 8: Architecture for the automated cluster-recovery experiment.  
The automatic recovery experiment was conducted as follows. First, Nginx was run on Cluster 1, and a backup file was created in the AWS S3 bucket by using the Velero backup command. Then, the master node of Cluster 1 was stopped to disconnect it from the Rancher cluster. When the disconnection is detected by the Rancher cluster, the proposed automatic recovery system is initiated for the cluster. Because Clusters 1 and 2 both have a total of 4 vCPU allocated, Cluster 2 is a viable cluster for restoration. In this experiment, because Cluster 2 was the only target cluster available for restoration, the restoration proceeded to Cluster 2 without scheduling. First, Velero in the Rancher cluster queried the latest Nginx backup file in the S3 bucket and sent the restoration command, including the name of the backup file, to Cluster 2. Cluster 2 executed the received command, restoring the Nginx running on Cluster 1 by using the backup file in the S3 bucket.  
In this experiment, excluding the process of creating the backup file, the time taken from when the master node of Cluster 1 was stopped, from the creation of an artificial disaster scenario to when the Nginx on Cluster 2 was completely restored was measured. This experiment was repeated 10 times to measure the time. The time taken from executing the [PERSON] restoration command in Cluster 2 to the completion of the restoration is referred to as the restoration time. The time utilized by the process, excluding the restoration time, is attributed to the proposed system. The measurement and analysis of these times are explained in Section 7.