Title: ATHENA-Serve: An Intelligent Scheduling LLM Serving System via Horizon-Cost Prediction and Hierarchical RL
Primary area: foundation or frontier models, including LLMs
Keywords: LLM serving systems, Tail-latency SLOs, Bursty traffic scheduling, Resource budgeting

Abstract: Online inference serving for large language models (LLMs) is foundational infrastructure for conversational agents, retrieval-augmented generation, and multi-tenant intelligent applications. Its core objective is to meet strict latency SLOs under heterogeneous and bursty workloads. However, existing systems suffer from bursty arrivals and long-tailed output lengths that drive peak cache pressure and bandwidth contention, as well as the brittleness of FCFS or shortest-job heuristics under noisy length regression and distribution shift—ultimately compounding tail-latency violations and head-of-line (HoL) blocking. We present ATHENA-Serve, a deployable, horizon–cost–aware LLM serving scheduler. ATHENA-Serve converts predicted generation horizons into calibrated memory and compute budgets. Rather than forecasting exact trajectories, it senses each request’s KV-cache usage patterns and peak-footprint signals. Guided by these budgeted signals, ATHENA-Serve proactively constrains batching and concurrency to smooth memory peaks, while conditioning scheduling decisions on global system signals. On the ShareGPT dataset, ATHENA-Serve achieves up to 1.64× lower latency compared to state-of-the-art serving systems.

TL;DR: ATHENA-Serve fuses calibrated horizon-to-budget mapping (ORACLE) with hierarchical scheduling (HERA) to deliver robust, low-latency LLM serving under bursty workloads.