Taming the Chaos: Coordinated Autoscaling for Heterogeneous and
  Disaggregated LLM Inference
Authors: name: Rongzhi Li; name: Ruogu Du; name: Zefang Chu; name: Sida Zhao; name: Chunlei Han; name: Zuocheng Shi; name: Yiwen Shao; name: Huanle Han; name: Long Huang; name: Zherui Liu; name: Shufan Liu
Published: 2025-08-27 04:22:02

Abstract: Serving Large Language Models (LLMs) is a GPU-intensive task where
traditional autoscalers fall short, particularly for modern Prefill-Decode
(P/D) disaggregated architectures. This architectural shift, while powerful,
introduces significant operational challenges, including inefficient use of
heterogeneous hardware, network bottlenecks, and critical imbalances between
prefill and decode stages. We introduce HeteroScale, a coordinated autoscaling
framework that addresses the core challenges of P/D disaggregated serving.
HeteroScale combines a topology-aware scheduler that adapts to heterogeneous
hardware and network constraints with a novel metric-driven policy derived from
the first large-scale empirical study of autoscaling signals in production. By
leveraging a single, robust metric to jointly scale prefill and decode pools,
HeteroScale maintains architectural balance while ensuring efficient, adaptive
resource management. Deployed in a massive production environment on tens of
thousands of GPUs, HeteroScale has proven its effectiveness, increasing average
GPU utilization by a significant 26.6 percentage points and saving hundreds of
thousands of GPU-hours daily, all while upholding stringent service level
objectives.