# Q8S: Emulation of Heterogeneous Kubernetes Clusters Using QEMU
## 2 Materials and Methods
### Q8S Design  
Here, we present and discuss the design considerations when creating Q8S. The problem of providing a tool for creating heterogeneous Kubernetes clusters on top of OpenStack can be split into three layers:  
* User interface;
* OpenStack instances;
* QEMU VMs. These layers are also partially reflected in Figure 2, a sequence diagram depicting the workflow of Q8S. As the user interface, Q8S provides a CLI tool, which reads from a YAML file. The user can specify the desired cluster by creating said YAML file as a human- and machine-readable record of the cluster to be created. Moreover, the user is expected to provide credentials for OpenStack such that Q8S can request resources in the name of the user. This operation is reflected in the first step shown in Figure 2, where Q8S represents the CLI tool as the user interface. The next layer involves Q8S using the provided credentials to request instances via the OpenStack API. Each OpenStack VM serves as the host for a single Kubernetes node, which may be deployed directly on the host or as a nested VM via QEMU. Kubernetes nodes hosting control plane components are deployed without emulation, while all worker nodes use QEMU VMs as emulated environments. The provisioning of the OpenStack VMs is depicted in the next three steps in Figure 2.  
We considered operating multiple emulated Kubernetes workers from a single OpenStack VM by having QEMU run multiple VMs but decided on a one-node-per-host solution to avoid adding additional complexity to the network configuration. This decision comes at the price of having to run a host operating system for each emulated node instead of providing the combined resources to a single VM host to emulate multiple nodes. When scaling out a Q8S cluster, 2 CPU cores and 2 GB RAM should be dedicated to the Ubuntu hosts for the emulated nodes such that the emulations can run without interruption due to background activities from the host. By grouping, for example, four emulated nodes on the  
Figure 2: Sequence diagram of the Q8S workflow. same host, this overhead would be reduced by a factor of four. This is not yet implemented in the presented version of Q8S. The last layer consists of setting up the Kubernetes cluster and the QEMU VMs and joining the cluster together. When initializing the cluster, Q8S uses the node on which it is running as the initial Kubernetes control plane member that generates the Kubernetes certificates and administrator configurations. On all other control plane nodes, Q8S installs Kubernetes and joins it to the already initialized cluster. On the worker nodes, Q8S installs QEMU and libvirt and is then ready to fetch the OS images used for the respective emulated environments. We considered multiple approaches for providing the OS images:  
* VM Snapshots This approach involves preparing a VM with all the required dependencies and saving it as a snapshot, which is then provided to the QEMU hosts. However, we wanted Q8S to be flexible, and this approach would require recreating said snapshots for every update to the images for every specified node type. * Autoinstall for Ubuntu Server Ubuntu servers have a feature such that they can automatically set up a system when given an autoinstall configuration. We explored this option, and it worked for our x86 images, but we encountered problems for our ARM images. * Ubuntu Cloud-Images Besides Ubuntu server images, Canonical also provides Ubuntu Cloud-Images, which are lightweight for Ubuntu OS images with around 600 MB and can be configured via cloud-init settings. Cloud-Images are available for different system architectures including x86_64 and ARM64 while utilizing identical setup instructions. While this decision locked Q8S into only supporting Ubuntu based hosts, we consider this a reasonable trade-off compared to specifying setup instructions for various Linux distributions. Q8S provides the cloud-init settings to the QEMU hosts such that the Cloud-Images can be set up in the emulated environments. Another major aspect of the design of Q8S is networking.