# Q8S: Emulation of Heterogeneous Kubernetes Clusters Using QEMU
## 1 Introduction  
Cloud computing has brought a major paradigm shift to the IT landscape [1] and enabled the creation of large compute continuums. These continuums are often utilized for a variety of tasks [2] including compute-intensive workloads as well as providing web services and are commonly operated through Kubernetes [3]. While Kubernetes is capable of effectively scheduling container workloads, taking into account various constraints and policies [4], the cloud landscape is growing increasingly complex through the addition of heterogeneous components such as Internet of Things (IoT) devices [5]. Kubernetes scheduling is an ongoing area of research [4, 6, 7] with different approaches focusing on metrics such as reliability, energy consumption, and resource utilization. However, scheduling becomes increasingly difficult with the inclusion of heterogeneous components, which can be edge devices, data centers using heterogeneous hardware, or outdated hardware being replaced by newer components [8]. Researchers working on scheduling algorithms for heterogeneous compute clusters may struggle to evaluate their work properly due to limited access to heterogeneous components. Heterogeneous components include specialized and often expensive hardware or IoT devices, which may be reserved by other users or not available at all. Moreover, available heterogeneous components may already be installed in production systems such that a newly developed scheduling algorithm cannot be deployed in production without being evaluated first; however, for the evaluation, the heterogeneous components would need to be moved into a test system and be missing from production. One such research effort is the DECICE project [9], which aims to develop an AI-based scheduler for Kubernetes that is suitable for heterogeneous Kubernetes clusters. To train machine learning models and to test frameworks, one or more heterogeneous Kubernetes clusters are required. As a possible solution, we considered simulations such as CloudSim [10], its extensions [11; 12; 13; 14; 15], EMUSIM [16], and K88 Sim [17]; however, we found that the grade of detail available, especially for heterogeneous hardware, was insufficient for our work. Instead, we investigated techniques to emulate a heterogeneous Kubernetes cluster consisting of nodes with different emulated hardware architectures and operating systems on existing homogeneous OpenStack cloud infrastructure. Emulations provide higher accuracy than simulations [18], as simulations apply abstractions that omit details not explicitly included by the developers of a given simulation. Moreover, an emulation enables the execution of real-world workloads such that technical problems with scheduling algorithms and end-user applications can be discovered. As the result of this, we present QSS, which allows users to automatically set up a Kubernetes cluster with heterogeneous nodes running on a homogeneous OpenStack cloud using Quick Emulator (QEMU) [19]. Other emulation efforts such as [18; 20] focus on the emulation of networks and large numbers of nodes, respectively. QSS, on the other hand, fills a gap in the range of options by emulating heterogeneous hardware through actual hardware emulation techniques instead of requiring physical heterogeneous hardware or simulating heterogeneous capabilities of nodes. QSS offers a greater level of detail than simulations, as, instead of a simulated cluster with artificial nodes and tasks, QSS provides a real Kubernetes cluster, which can execute actual workloads. Therefore, QSS enables researchers to observe details in the implementation of a given algorithm that might not otherwise surface until the algorithm was deployed in a production environment. Furthermore, we have benchmarked QSS using k8s-bench-suite [21], showing that the emulated clusters work without significant changes in communication bandwidth, although CPU performance suffers from emulation overhead.