# Enhanced Scheduling of AI Applications in Multi-Tenant Cloud Using Genetic Optimizations
## 1 Introduction  
With the rapid advancement in artificial intelligence (AI) industries and the explosion of large datasets, heterogeneous GPUs within MLaaS (Machine Learning as a Service) frameworks are being swiftly deployed to execute various AI applications [1]. These extensive GPU clusters are crucial for handling a huge number of machine learning (ML) tasks [2; 3], such as training and inference, significantly speeding up processes across numerous AI applications [4; 5].  
However, the diversity of ML tasks coupled with varied GPU configurations, poses significant challenges in resource management and scheduling. Task scheduling has been widely studied in a variety of systems, ranging from low-end mobile devices to high-end industrial systems [6; 7; 8]. However, ML workloads have significantly different resource usage patterns from traditional applications, making it difficult to efficiently manage system resources [9; 10; 11]. In particular, GPUs in large-scale production clusters often exhibit low utilizations of less than 50% when executing ML workloads [12; 13; 14]. This underutilization of advanced GPU features is primarily due to the increasing number of small tasks, such as inference, and this trend is expected to continue in the near future [15].  
In response to these challenges, the Multi Instance-GPU (MIG) feature within NVIDIA's Ampere architecture has been designed to enable GPU sharing across tasks [16]. Nevertheless, this solution does not ensure high GPU utilization, as fragmentation issues often arise when GPUs are only partially allocated. This fragmentation is a recurring problem in shared-GPU environments.  
Traditional solutions such as packing strategies have been utilized to mitigate fragmentation issues [17]. Conceptually, task scheduling in GPU environments can be likened to a multi-dimensional bin-packing problem, where tasks and nodes are comparable tospheres and boxes, aiming to minimize the number of boxes used. However, traditional bin-packing methods often fall short in environments with multiple GPUs, due to intrinsic differences in the scheduling needs of GPU sharing.  
To better tackle these issues, this paper explores two bin-packing models tailored for GPU scheduling. The first model seeks to consolidate multiple GPUs into a single machine by aggregating their capacities. Although theoretically appealing, this model falls short in practical scenarios due to its failure to consider individual GPU fragmentation. The second model treats each GPU as a separate resource dimension, acknowledging that GPUs, unlike CPUs and memory, are interdependent, with tasks on one GPU often relying on others, rendering traditional bin-packing strategies insufficient for real-world GPU scheduling challenges.  
To address these complexities, we propose a novel scheduling method that incorporates a genetic optimization technique within the Simpy 4.1.1 discrete-event simulation framework [18]. This approach specifically targets the issue of GPU underutilization due to fragmentation and aims to enhance overall GPU efficiency. By conducting large-scale simulations on Alibaba's PAI (Platform for AI) cluster, which includes over 1800 nodes and more than 6000 GPUs [19; 20], we demonstrate that our scheduling policy improves GPU utilization by 12.8% compared to Round-Robin scheduling and also performs better than the established bin-packing-based scheduling method, Tetris [21]. This substantial improvement highlights the effectiveness of our approach in optimizing resource allocation within cloud-based ML environments.  
The remainder of this paper is organized as follows. In Section 2, we will briefly discuss the problem definition and the basic model for our scheduling. Section 3 describes the genetic optimization of our task scheduling in detail. Section 4 validates the effectiveness of the proposed scheduling based on simulation experiments. Finally, Section 5 concludes this paper.