# Algorithmic Techniques for GPU Scheduling: A Comprehensive Survey
## 4 Looking Forward: LLM Focus
### Serving LLMs and Complex Inference Workloads: New Scheduling Frontiers  
Building on the heterogeneity- and scale-oriented challenges discussed in earlier sections, we now shift focus from _training_ to the equally demanding domain of production-scale _inference_. While training jobs consume more total GPU hours, LLM serving imposes far stricter real-time constraints and exposes a very different--and highly volatile--scheduling surface. Inference workloads exhibit extreme heterogeneity: queries vary in prompt length, generate an unpredictable number of output tokens, and span latency requirements from interactive chat to offline batch scoring. Each output token triggers a full forward pass, and the GPU-resident key-value (KV) cache grows monotonically during generation [208]. As a result, request durations unfold online and are unknown _a priori_, making inference scheduling a multi-tenant analog of operating system job management under unknown runtimes.  
Empirically, naive FIFO batching yields erratic latency and poor hardware utilization. Even state-of-the-art systems such as Alpa on Ray and MegaScale achieve only around 50% FLOP utilization on LLM workloads [209].  
Recent work addresses this volatility along three main axes. First, _prediction-driven size-based scheduling_ leverages intermediate activations to forecast the remaining runtime of each query. TRAIL, for example, reduces prediction error by 2.6\(\times\) compared to prompt-only estimates and applies a lightly preemptive SRPT-style policy that halves mean latency while substantially improving 99 th-percentile tail performance [210]. Second, _runtime adaptivity via migration_ enables in-flight transfers of full inference sessions--including multi-GB KV caches--between GPUs. Llumnix uses this to rebalance load, defragment memory, and accelerate high-priority queries by 1.5\(\times\), reducing tail latency by up to 10\(\times\)[208]. Third, at fleet scale, inference platforms must multiplex dozens of models and tenants. Static reservations waste capacity, while opportunistic sharing risks SLA violations. Emerging token-based fair schedulers manage this tradeoff by bounding per-request service time variance within 2\(\times\) while sustaining high utilization [97].  
When models exceed single-GPU memory, scheduling must extend to graph-level placement across devices. Helix tackles this by casting layer placement and routing as a MILP, achieving a 2.7\(\times\) throughput gain over baseline heuristics [211]. These results suggest that periodic global optimization, paired with lightweight local adjustments, can deliver practical performance despite the underlying NP-hardness.  
Looking ahead, four challenges dominate the inference scheduling agenda. First, _low-overhead preemption and migration_ require the ability to suspend or relocate multi-GB KV caches with minimal disruption. Second, _accurate yet robust prediction_ is crucial, as even modest errors can cause severe tail-latency spikes under tight SLAs. Third, _GPU memory fragmentation_, driven by variable request lengths and batch sizes, demands online repacking strategies that avoid long compaction stalls. Fourth, maintaining _steady-state stability under bursty load_ will require robust admission control and autoscaling mechanisms that prevent oscillatory behavior.  
Addressing these challenges will require both systems innovations--such as efficient GPU context switching, live migration, and fine-grained telemetry--and theoretical advances in queueing theory, real-time scheduling, and worst-case performance analysis. As demand for LLM inference grows, schedulers that integrate prediction, adaptive control, and rigorous guarantees will be key to building scalable, responsive, and cost-effective inference services.