# Q8S: Emulation of Heterogeneous Kubernetes Clusters Using QEMU
## 3 Results
### Implementation  
In the following, we discuss the implementation details of our Q8S prototype with examples taken from our OpenStack environment. Networking  
Depending on the configuration of a given OpenStack cloud, the default network masks may vary. In our OpenStack environment, all hosts are part of the 10.254.1.0/24 network range, and the QEMU VMs use 192.11.3.0/24. When assigning the IPs of the QEMU VM, we configured Q8S to automatically match the host IP, e.g., the host with 10.254.1.3 would internally run 192.11.3.3. This also ensures that the QEMU VM IPs are unique across both networks. Figure 3 provides an overview of a cluster created by Q8S with the 10.254.1.0/24 IP range used for the hosts. The name of the default network interface of the hosts in the OpenStack network is ens3 here, but it may be different in other OpenStack clouds. Figure 3: Overview of a cluster created by Q8S. The starting instance in the top left of Figure 3 is the node on which the user has started Q8S and that initialized the cluster as its first control plane node. Below that are the other control plane nodes, which do not install an additional virtualization layer. On the right are workers depicted with numbers from 1 to n. It should be noted that in Figure 3, there are two worker types: x86 and arm64. The worker nodes deploy QEMU and connect its internal network via a bridge to the host network. From the perspective of the other nodes, the additional virtualization layer provided by QEMU is not visible on the network layer, as any requests sent to a worker node are forwarded by the iptables rules to the internal VM via NAT. The internal networking of a worker host are depicted in detail in Figure 4, with the white box on the left listing the iptables rules. These include prerouting DNAT rules to forward any traffic send to the host to the QEMU VM on the same port. The exception is port 22 of the host, which still provides regular SSH access to the host node, and port 222, which is redirected to port 22 of the QEMU VM for SSH access. For outgoing traffic, SNAT is used to masquerade requests as the host node. The right half of Figure 4 shows the network bridge provided by libvirt labeled as virbr0, which connects the host and QEMU VM networks. Inside the QEMU VM is the enp1s0 interface, which holds the IP of the overall QEMU VM. Inside the VM is the local network created by Flannel, which is used to assign IPs to the individual containers. Flannel is shown in Figure 4 to use the node's public-ip for inter-node communication, which is necessary for it to accept the requests forwarded via the iptables rules. Configuration  
Q8S requires two configuration files at the start: the YAML cluster configuration file and a file with credentials for the OpenStack cloud. The official Python OpenStack SDK used by Q8S is able to process clouds.yaml files, which can be downloaded through the OpenStack Horizon web interface under application credentials. With these application credentials, Q8S is able to query the OpenStack API for the OpenStack project for which the credentials were created. The cluster configuration file includes the user-provided settings for the cluster to be created. An example of such a file is given in Listing 1. The configuration includes the following fields, which the users are expected to adjust to their needs:  
Figure 4: Internal view of a worker host and its networking. git_url:A URL pointing to a Q8S git repository, which is either public or accessible via embedded access tokens. This is required for the later installation stages to download the respective setup scripts on the new nodes. * private_network_id: The id of the internal network in which all hosts reside; it can be found in the OpenStack Horizon web interface under networks as the id of the private subnet. This is needed to request new VMs via the OpenStack API. * remote_ip_prefix: The IP mask of the OpenStack network. In our environment, this is 10.254.1.0/24. * default_image_name: The name of the OS image that should be used for the OpenStack images.