# Q8S: Emulation of Heterogeneous Kubernetes Clusters Using QEMU
## 3 Results
### Implementation
Even during the benchmark, the CPU usage of the OpenStack node barely goes up except for the UDP communication, where it is significantly increased for the client node but not the server node. The reason for the increased CPU usage likely lies in the emulation, as, for each CPU instruction, the hypervisor has to perform additional operations to check and translate the instruction in addition to actually performing the instruction. This overhead factor even varies between client and server, such that we can assume that sending and receiving requests require different quantities of overhead operations by the hypervisor. Regarding the increased client CPU usage for UDP, when comparing the measurements to other benchmarks created with k8s-bench-suite [35], these also show an increase in CPU usage for UDP clients. Our test setup features only 2 CPU cores; therefore, we can speculate that the performance differences would be less pronounced when using more cores, as in the external example. The memory usage of the nodes is given in Table 4 and shows no significant changes during the benchmark. Notably, the memory usage of the emulated server is even slightly lower than that of the OpenStack server. Overall, the memory usage does not vary significantly, such that it appears to be handled well by the emulation layer. \begin{table}
\begin{tabular}{l c c c c} \hline \hline \multirow{2}{*}{**Node Type**} & \multicolumn{2}{c}{**Pod-to-Pod**} & \multicolumn{2}{c}{**Pod-to-Service**} \\  & **TCP** & **UDP** & **TCP** & **UDP** \\ \hline OpenStack Instance Nodes & 93 & 94.5 & 93 & 94.5 \\ Emulated Nodes & 93 & 89.7 & 93 & 89.4 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Internal cluster network bandwidth in Mbit/s. Measured by k8s-bench-suite, 5 min per measurement. \begin{table}
\begin{tabular}{l c c c c c} \hline \hline \multirow{2}{*}{**Node Type**} & \multirow{2}{*}{**Idle**} & \multicolumn{2}{c}{**Pod-to-Pod**} & \multicolumn{2}{c}{**Pod-to-Service**} \\  & & **TCP** & **UDP** & **TCP** & **UDP** \\ \hline OpenStack, client node & 1.52 & 1.98 & 51.43 & 2.21 & 56.93 \\ Emulated, client node & 14.33 & 31.66 & 63.99 & 32.74 & 63.22 \\ OpenStack, server node & 1.13 & 3.94 & 3.58 & 5.03 & 3.55 \\ Emulated, server node & 15.14 & 43.81 & 56.59 & 44.20 & 55.61 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Comparison of CPU usage in %, with nodes idling and during pod-to-pod and pod-to-service communication. Measured by k8s-bench-suite, 5 min per measurement Moreover, to verify that QSS is capable of deploying larger clusters, we have deployed a cluster with 50 emulated worker nodes including 30 emulated x86_64 nodes and 20 emulated ARM64 nodes as well as two additional master nodes. The time to complete the deployment was 77 min, with the times taken for the individual stages shown in Table 5.

The first stage in Table 5 represent the time taken for the OS to provision the nodes and is stopped when the OS reports all nodes as ready. However, for the OS, a node appears ready when it has successfully booted, but it might still be starting up and launching services such that access to it via SSH is not yet possible. In the next stage, QSS distributes setup scripts and configuration files to the nodes, which takes only a few seconds per node but is performed sequentially in our implementation and nodes might not be ready to establish an SSH connection yet, such that the script has to wait. The third major stage in Table 5 is the setup of the hosts, which involves installing QEMU as well as downloading and preparing the guest image for the emulated workers. This stage is processed in parallel. Afterwards, the cloud-init script has to run in all emulated workers to install Kubernetes and finally join the cluster. After joining the cluster, the nodes must all report ready status for the next status before the network setup is completed in the last stage.