# Algorithmic Techniques for GPU Scheduling: A Comprehensive Survey
## 4 Looking Forward: LLM Focus
### Scaling Scheduling for Future GPU Clusters  
This directly re-engages the heavy-tail mismatch documented in Section 1.5, where we showed that naive exponential assumptions can inflate 99 th-percentile queueing delays by up to 4 times. It implies that LLM-serving schedulers must embed explicit tail-risk predictors. Designing queueing policies that both learn heavy-tail statistics and retain closed-form performance bounds therefore remains a high-leverage research frontier. Next-generationdeep learning platforms comprise tens of thousands of heterogeneous accelerators shared by diverse workloads, pushing legacy schedulers--designed for smaller, homogeneous fleets--well beyond their comfort zone. Modern controllers must therefore (i) scale to cluster sizes that rival supercomputers, (ii) embrace heterogeneity as a first-class dimension, and (iii) juggle multi-tenant fairness, network topology, and security in a single optimization loop. Early systems such as Sia make headway by explicitly modeling device diversity and scaling to 2000-GPU clusters, trimming average JCT by 30\(\sim\)93% relative to prior heuristics [55]. Their key insight--jointly optimizing accelerator _type_, _quantity_, and _elasticity_--highlights the cost of ignoring any one axis: penalties of 40\(\sim\)70% JCT when hardware heterogeneity or malleability is left out.  
As jobs fan out across nodes, interconnect topology becomes equally decisive. Synchronization and sharding traffic can dominate run time unless placements are fabric-aware--co-locating tasks on NVSwitch or InfiniBand islands whenever possible. The problem generalizes classical \"unrelated-parallel-machine\" scheduling: the controller must select not merely how many resources to grant but which class (GPU, TPU, FPGA) under throughput and QoS targets. Extending dominant-resource or finish-time fairness to these mixed pools remains open; prototypes such as Pollux and Gavel offer initial road-maps [57; 68]. At fleet scale, single-point schedulers risk bottlenecks and failures, motivating hierarchical or decentralized control planes that approximate global optima with partial information.  
Network contention adds a further layer. Distributed training generates _coflows_--sets of synchronized flows whose slowest member gates progress. Coflow-aware algorithms allocate bandwidth holistically and cut JCTs by up to 61% on NVSwitch and InfiniBand-NDR fabrics [199; 200]. By reasoning over coflows, these schedulers confront the 20\(\sim\)40% cross-rack traffic inflation we quantified in Section 2.4. However, the empirical pathway is still blocked: no public trace today exposes per-flow affinity tags, leaving a conspicuous data void for reproducible evaluation.  
Finally, multi-tenant isolation unifies security, performance, and billing accuracy. Hardware-level virtualization (e.g., NVIDIA MIG) partitions GPUs into spatial slices but may strand capacity when demand fluctuates, surfacing the very burst-induced fragmentation cost that Section 1.5 flagged as a barrier to fleet-wide efficiency. Runtime-level sandboxing enables finer time-sharing at the cost of 10\(\sim\)100 ms context-switch delays--well above the sub-millisecond target of interactive DL jobs [192]. Hypervisor or enclave isolation delivers the strongest confidentiality guarantees yet restricts dynamic scaling and invites vendor lock-in [201; 202]. The paucity of public traces with microsecond-level contention data leaves designers flying blind; closing this gap will require open telemetry alongside hardware-assisted, low-latency isolation primitives.  
Looking ahead, schedulers for exascale GPU fleets must weave together topology-aware placement, coflow-level bandwidth management, and secure, fair multi-tenancy across heterogeneous accelerators. Whether their control planes are centralized, hierarchical, or fully decentralized, these next-generation systems will have to reconcile all three objectives at supercomputer scale--a challenge that promises rich opportunities for both system builders and scheduling theorists.