# Q8S: Emulation of Heterogeneous Kubernetes Clusters Using QEMU
## 3 Results
### Implementation
Creation of the security group q8s-cluster if it does not exist and the configuration of the rules as specified in the settings file;
2. Creation of an SSH key pair, which is uploaded to OpenStack such that the new VMs are initialized with it and can later on be accessed;
3. Creation of the OpenStack VMs via its API for the control plane and worker nodes;
4. Waiting for all new OpenStack VMs to be reachable via SSH;
5. Installation of Kubernetes dependencies and system configurations required for Kubernetes on the instance running Q8S;
6. Initializing the Kubernetes cluster and installing Flannel;
7. Extracting the Kubernetes token for joining of worker nodes and uploading cluster certificates, required for joining control plane nodes;
8. Installation of Kubernetes dependencies and system configurations on the additional control plane nodes;
9. Joining of the additional control plane nodes to the cluster;
10. Installation of QEMU and libvirt on the worker hosts;
11. Configuring of the QEMU network to ensure the desired IP address that matches the host IP will be assigned to the QEMU VM;
12. Downloading of the Ubuntu Cloud-Images;
13. Preparation of the user-data and meta-data files for the Cloud-Image VM;
14. Preparation of the Cloud-Image for QEMU;
15. Creation of the QEMU VM from the Cloud-Image;
16. Installation of Kubernetes and dependencies and system configurations on the QEMU VMs as triggered by cloud-init;
17. Waiting for all worker nodes to join the Kubernetes cluster. Once all these steps are completed, the cluster is ready to be used, and Q8S is no longer required. In the prototype implementation of Q8S, the above steps are executed sequentially and are still open to optimization through parallelization, for example, by letting the deployment process of the additional control plane nodes and the emulated worker nodes run in parallel. ### Evaluation

Benchmark

In this section, we discuss the measurements that were acquired according to the methods described in Section 2.2. Table 2 shows the bandwidth measured through the benchmark for pod-to-pod and pod-to-service communication for emulated and non-emulated nodes via TCP and UDP. Except for UDP being slightly slower on the emulated nodes, the performance is almost identical despite the additional layer of networking that applies for the emulated nodes. The difference in UDP speed could be related to the UDP packet size [34] being non-optimal for wrapping in the Flannel environment. As the difference is marginal, we consider the networking performance to be effectively identical. We measured the following network latencies:

* Worker-Node-to-Worker-Node round-trip latency: \(\approx\)3.1 ms;
* Pod-to-Pod (different nodes) round-trip latency: \(\approx\)4.8 ms;
* Pod-to-Pod (same node) round-trip latency: \(\approx\)1.1 ms;* Worker-Node-to-Control-Plane-Node round-trip latency: \(\approx\)1.5 ms;
* Control-Plane-Node-to-Control-Plane-Node round-trip latency: \(\approx\)0.5 ms. Between control plane nodes, no additional iptables rules are applied; between a worker node and a control plane node, one set of iptables rules are applied; and between worker nodes, the iptables are applied both on being sent and on being received. Going from no iptables rules to one pass adds about 1 ms of latency, and adding another pass adds another 1.6 ms. Compared to pod-to-pod on two worker nodes, this becomes even slower, as it now also has to pass through the Kubernetes internal networking. Considering the low latency for direct pings between two hosts, these latencies created by the additional iptables and network virtualization layers seem high but are overall still relatively low and acceptable for the majority of use cases. The CPU usage of the nodes during the throughput benchmark as well as in the idle state is given in Table 3. The benchmark consists of a client-and-server pairing, where the client sends requests to the server. The CPU usage is significantly higher for the emulated nodes than for the OpenStack nodes.