Title: Scorpio: Serving the Right Requests at the Right Time for Heterogeneous SLOs in LLM Inference
Venue: NeurIPS 2025
Accepted: no
Keywords: Machine Learning System; Large Language Model; Service Level Objectives
Abstract: Existing Large Language Model (LLM) serving systems prioritize maximum throughput. They often neglect Service Level Objectives (SLOs) such as Time to First Token (TTFT) and Time Per Output Token (TPOT), which leads to suboptimal SLO attainment. This paper introduces SCORPIO, an SLO-oriented LLM serving system designed to maximize system goodput and SLO attainment for workloads with heterogeneous SLOs. Our core insight is to exploit SLO heterogeneity for adaptive scheduling across admission control, queue management, and batch selection. SCORPIO features a TTFT Guard, which employs least-deadline-first reordering and rejects unattainable requests, and a TPOT Guard, which utilizes a VBS-based admission control and a novel credit-based batching mechanism. Both guards are supported by a predictive module. Evaluations demonstrate that SCORPIO improves system goodput by up to 14.4X and SLO adherence by up to 46.5% compared to state-of-the-art baselines.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a heterogeneous SLO-oriented serving system, SCORPIO, that actively exploits per-request latency targets (TTFT and TPOT) rather than treating every prompt identically. The system couples lightweight sequence-length prediction with two cooperating control loops—TTFT Guard (least-deadline-first re-ordering plus early rejection) and TPOT Guard (virtual-batch-size admission plus credit-based batching). Across workloads, SCORPIO boosts goodput versus existing baselines, while keeping the scheduling overhead low.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. This paper focuses on a timely problem, moves beyond throughput to heterogeneous latency guarantees, a gap in mainstream systems like vLLM.\n2. The paper is well written and easy to follow\n3. This paper proposes a system with modular design, two orthogonal guards plus a tiny predictor; each module’s contribution is dissected via ablation.\n4. Evaluation is thorough with multiple models (8 B/27 B), two public traces, and QPS scaling experiments substantiate the claims. It also evaluates against three representative schedulers covering greediness, length-based ranking, and admission-control paradigms. \n\nWeaknesses:\n1. This paper follows a similar design decision, which is to introduce an output sequence length predictor. However, it’s not clear that the fine-tuned OPT-125 M would be able to generalize across different datasets. Do you need to train a separate predictor for each dataset?\n2. As shown in this paper, previous works are focusing on similar problems, and the core of this line of work is the length predictor. Including this paper, they are all sharing the same methodology (length prediction) with papers only considering homogeneous SLOs, which makes it hard to judge relative novelty. The remained components are quite trivial, which is just the earliest deadline first. Can you give more justification for why such a length predictor would even work? What’s new and special about your length predictor? Especially, when you compare with S3, is the performance gap mainly due to the different performance of the length predictor? What’s S3’s performance if it uses your length predictor?\n3. In the main evaluation, there are cases where Scorpio yields lower goodput at relatively lower request rates. Can you elaborate on that?\n\nQUESTIONS:\nPlease refer to the previous comments.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces SCORPIO, an LLM serving system designed to handle heterogeneous Service Level Objectives (SLOs), specifically addressing TTFT and TPOT. The authors identify that current LLM serving systems prioritize throughput at the expense of meeting diverse latency requirements across different requests. To tackle this, SCORPIO leverages adaptive scheduling using two specialized mechanisms:\n\n- **TTFT Guard:** Implements least-deadline-first reordering and rejects requests unlikely to meet TTFT.\n- **TPOT Guard:** Uses Virtual Batch Size (VBS)-based admission control and a credit-based batching mechanism tailored to TPOT constraints.\n\nAdditionally, SCORPIO employs a sequence length predictive module to support decision-making. Empirical evaluations show that SCORPIO significantly improves system goodput (up to 14.4×) and SLO adherence (up to 46.5%) compared to state-of-the-art systems.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths:\n- **Quality:** Rigorous evaluation demonstrating significant improvements over existing methods, with well-chosen baseline comparisons.\n- **Clarity:** Clearly structured and detailed explanations of methods and system components, supported by informative diagrams and algorithms.\n- **Significance:** Directly addresses a highly relevant problem in LLM inference serving—heterogeneous SLO management—which is crucial in practical deployment scenarios.\n\n# Weaknesses\n- **SLO setting concerns:** SCORPIO is a system specifically designed to enhance SLO attainment and goodput. Different SLOs should have a significant impact on experimental results, but there is no appropriate basis for setting SLOs during the experiment (although the authors have set many SLOs and conducted ablation experiments). I suggest that the authors search for authoritative SLO setting guidelines to set the SLOs for the experiment.\n- **Starvation concerns:** This paper does not explicitly discuss or handle the starvation problem. The scheduling method described in the paper might cause a potential starvation problem: The TTFT Guard will place requests with closer deadlines at the front. Under continuous high-load scenarios, requests with looser TTFT requirements may be repeatedly delayed, failing to receive timely service, thus causing starvation.TPOP Guard is designed to use credit accumulation to determine the frequency of request batching, in extreme cases, when strict TPOT SLO requests keep pouring in, the rate at which loose TPOT requests accumulate credit will significantly slow down, which may lead to requests failing to obtain service opportunities for a long time, thus causing a situation of request starvation.\nTherefore, I think that the SCORPIO scheduling system, although it effectively improves the satisfaction rate of heterogeneous SLOs, indeed has potential starvation risks under high-load and long-term operation scenarios.The author does not explicitly discuss this issue in the paper and does not propose any related mitigation strategies. Subsequent research may need to further explore how to better balance the relationship between throughput, SLO achievement rate, and request fairness to reduce the risk of starvation.\n\nQUESTIONS:\nPlease see Weaknesses.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-28"}

comments: SUMMARY:
The paper introduces SCORPIO, an SLO-oriented serving system for Large Language Models (LLMs) designed to maximize system throughput and adherence to Service Level Objectives (SLOs) for requests with heterogeneous SLOs. It incorporates two specialized mechanisms, TPOT Guard and TTFT Guard, supported by a predictive module to ensure high system goodput and optimized SLO adherence. SCORPIO improves system performance by up to 14.4× and enhances SLO adherence by 46.5%, demonstrating its superior performance over traditional throughput-oriented approaches.

STRENGTHS AND WEAKNESSES:
### Strengths

- The paper addresses a practical and meaningful problem: SLO heterogeneity for adaptive scheduling in LLM serving.
- The paper is clearly written, with effective figure illustrations that help convey key ideas.
- Extensive experimental results and a thorough overhead analysis are provided, demonstrating the effectiveness of the proposed method.

### Weaknesses
- Adaptability to "long output" or "long input" scenarios: While the paper effectively addresses SLO heterogeneity, it would be valuable to discuss the applicability of the proposed framework to two iconic LLM serving scenarios: "reasoning" (which involves long output tokens) and "long input" (which involves long input tokens). Assessing how well the framework handles these scenarios is crucial for its broader practical usage.

Despite the paper provides valid contribution, the supplementary materials seem to violate the double-blind rule. The author's code contained at least 21 files with the author's name, “tingfeng,” appearing 143 times. Based on reference [20], One could infer that one of the authors is Tingfeng Lan.

QUESTIONS:
Please refer to the weakness section.

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-01