# Q8S: Emulation of Heterogeneous Kubernetes Clusters Using QEMU
## 1 Introduction
### Background  
QSS depends on a complex stack of technologies including nested virtualization, hardware emulation, and manipulation of Linux iptables. The following paragraphs explain these technologies, as they are relevant for understanding the design of QSS. For heterogeneous systems, we derive our definition from [23], such that a system is homogeneous if the hardware, configuration, resources, and operating systems are the same. Therefore, a node is heterogeneous if it deviates in any of these areas. Differences in hardware may consist of different CPU architectures such as x86 and Advanced RISC Machines (ARM) but can also consist of different CPUs of the same architecture, varying amounts of installed system memory, or different network cabling. Moreover, some nodes might include additional hardware components such as GPUs. All these differences lead to variations in the resources that a given node can contribute to a heterogeneous cluster. Based on deployed operating systems and configurations as well as hardware variations, a given workload may perform better or worse on a specific node or even be incompatible. Having a scheduler determine the optimal placement may be an NP-complete problem [9]. In order to find near-optimal placements in a heterogeneous cluster, all these factors should be considered. Nested Virtualization  
Virtualization includes operating a Hypervisor or Virtual Machine Manager (VMM) on a node that, in turn, controls one or more virtual machines (VMs) [24]. The hypervisor must handle the virtualization of CPU and memory as well as any Input/Output (IO) devices including disks and networking [25], which results in additional overhead for operations carried out in a VM because the VMM must perform translations between the VM and the hardware. There exist optimization techniques such as para-virtualization [26], which often require a VM to recognize that it is running on top of a hypervisor such that it can utilize hypercalls, requests going directly to the hypervisor without translation. In nested virtualization, a VM is run within another VM by layering hypervisors [27]. This creates a hierarchy of hypervisors, where the outermost hypervisor operating on the actual hardware and hardware interactions from a VM running on an inner hypervisor are passed from inner hypervisor to outer hypervisor. For the inner hypervisor to work properly, certain operations that it would expect to be carried out directly on the underlying hardware need to be trapped and emulated by the outer hypervisor. Due to the multiple layers of hypervisors, nested virtualization further adds to the performance overhead of virtualization. Fannel is a lightweight networking plugin for Kubernetes with decent performance [28; 29]. It operates via a Layer 3 IPv4 network and routes packages based on target IP addresses to the respective ports of the Kubernetes containers. Its recommended backend is Virtual Extensible LAN (VXLAN), a networking virtualization, which employs encapsulation of Ethernet traffic to construct Layer 2 (data link layer) connections via Layer 3 networks. For inter-node communication, Flannel sets up a VXLAN Tunnel Endpoint (VTEP) that enables any two nodes to communicate on Layer 2 by sending UDP packages, which encapsulate a VXLAN header. The inter-node traffic uses Layer 3 IP addresses, while node internal addressing takes place on Layer 2 using MAC addresses. Linux Firewall  
Networking packages are processed by the Linux kernel based on routing tables and iptables in order to determine where a given package should be sent. If a package is not addressed to a local IP, its next hop is determined by the routing tables, while iptables serve to apply further operations on incoming packages.