LLM Query Scheduling with Prefix Reuse and Latency Constraints
Authors: Gregory Dexter
Shao Tang
Ata Fatahibaarzi
Qingquan Song
Tejas Dharamsi
Aman Gupta
Venue: NeurIPS 2025 poster ()
Keywords: queueing theory
scheduling theory
Large Language Models (LLMs)
prefix caching
Primary area: theory
TL;DR: Theoretical analysis of scheduling algorithms for LLM queries with latency constraints when using RadixAttention along with a novel scheduling algorithm.

Abstract: The efficient deployment of large language models (LLMs) in online settings requires optimizing inference performance under stringent latency constraints, particularly the time-to-first-token (TTFT) and time-per-output-token (TPOT). This paper focuses on the query scheduling problem for LLM inference with prefix reuse, a technique that leverages shared prefixes across queries to reduce computational overhead. Our work reveals previously unknown limitations of the existing first-come-first-serve (FCFS) and longest-prefix-match (LPM) scheduling strategies with respect to satisfying latency constraints. We present a formal theoretical framework for LLM query scheduling under RadixAttention, a prefix reuse mechanism that stores and reuses intermediate representations in a radix tree structure. Our analysis establishes the NP-hardness of the scheduling problem with prefix reuse under TTFT constraints and proposes a novel scheduling algorithm, $k$-LPM, which generalizes existing methods by balancing prefix reuse and fairness in query processing. Theoretical guarantees demonstrate that $k$-LPM achieves improved TTFT performance under realistic traffic patterns captured by a data generative model. Empirical evaluations in a realistic serving setting validates our findings, showing significant reductions in P99 TTFT compared to baseline methods.

LLM Query Scheduling with Prefix Reuse and Latency Constraints
Gregory Dexter
Shao Tang
LinkedIn gdexter@linkedin. com
LinkedIn shatangol inwedin..onndinooo
Ata Fatahibaarzi
Qingquan Song
Tejas Dharamsi LinkedIn
LinkedIn
LinkedIn
afatahibbaarziilinneeede com
ustcsqg@gaii.ccom
@charammi@linkeedee com
Aman Gupta
Nubank
aman www.gguuttuubnn....mmmm.m com. br
Abstract
The efficient deployment of large language models (LLMs) in online settings requires optimizing inference performance under stringent latency constraints, particularly the mime-to-fist-tokee (TTFT) and time-per-outppu-tooen (TPOT). This paper focuses on the query scheduling problem for LLM inference with prefix reuse, a technique that leverages shared prefixes across queries to reduce computational overhead. Our work reveals previously unknown limitations of the existing first-come-rt first-serve (FCFS) and taggest-prefix prettt---fiieeeehe match (LPM) scheduling strategies with respect satisfying latency constraints. We present a formal theoretical framework for LLM query scheduling under RadixAttentin,, prefix reuse mechanism that stores and reuses intermediate representations structure. Our analysis establishes the NP-hardness of the scheduling problem with prefix reuse under TTFT constraints and proposes a novel scheduling algorithm, K-LPM, which generalizes existing methods by balancing prefix reuse and fairness in query processing. Theoretical guarantees demonstrate that k-LPM achieves improved TTFT performance under realistic traffic patterns captured by data generative model. Empirical evaluations realistic serving setting validates our findings, showing significant reductions in P99 TTFT compared to baseline methods.
radix tree
Introduction
The rapid integration of large language models (LLMs) into online systems has spurred significant research aimed at improving their inference efficiency. Unlike traditional batch- aatch-processing environ- ments, online usage demands nuanced understanding of performance, prioritizing what often termed "goodput"" "Zhong et al. 2024 the maximum number of requests that can be served while meeting stringent constraints on time-to- to-rs-t-ttttt (TTFT) and time-ped time-per-output-tokk
(TPOT). This shift in focus underscores the necessity of developing advanced inference and serving algorithms that optimize goodput, ensuring efficient and effeccive deployment of LLMs in latency- applications.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

An important approach to improving LLM inference efficiency is prefix reuse. In autoregressive LLMs, prompts with shared prefixes can leverage the intermediate representations (i.e. key and value tensors, often stored in the key-value (KV) cache) of the common prefix, reducing both memory usage and computational overhead during inference. A notable implementation of this idea is RadixAttentionn, which stores the KV cache of processed queries and automatically reuses shared prefixes by maintaining a radix tree Zheng et al.) (2024). This tree tracks cached values and matches them with new incoming prompts, enabling efficient reuse without requiring manual intervention. To manage memory constraints imposed by the fixed size of the KV cache, the system employs a least recently used (LRU) eviction policy for cached values. This method is particularly impactful in scenarios where many prompts share prefixes, offering substantial savings in compute and memory. Moreover, its "fut-oo-the--ueee the-box"" functionality simplifies adoption by eliminating the need for users to manually analyze or encode prefix structures, making it a practical solution for real-world applications.
Automatic prefix reuse enabled by RadixAttention can substantially improve goodput in many realistic settings, particularly when there is large overlap among query prompts. However, under a stream of queries, the performance of RadixAttention becomes dependent on the order in which queries are processed (i.e., the "schedule"), and its implications for latency remain poorly understood.
The main scheduling algorithms considered in prior work are First Come First Serve (FCFS). which processes queries in arrival order, and Longest Prefix Match (LPM [
which greedily maximizes KV cache reuse each scheduling step. In the offline setting (where all queries are known in advance). Theorem 1 in Zheng et al. (2024) shows that LPM indeed maximizes cache reuse. However, the online setting, where queries arrive over time under tight fime-to-fiss-toker constraints, was not characterized.
In this work, we provide a theoretical exploration of RadixAttention scheduling in the online regime, focusing on non-preemptive scheduling. Our analysis and experiments show that existing scheduling approaches can lead to large TTFT spikes under heavy traffic, motivating a new scheduling algorithm that more robustly balances prefix reuse and waiting time. Specifically, we design mechanism that exploits the benefits of LPM but mitigates its performance risks in high-load scenarios, resulting in superior performance for prompp, short-output (i.e., prefill- prefill-ominant) queries where Radix- Attention delivers significant efficiency gains. Examples of such applications include document summarization, coding assistants, and prompts with detailed instruction sets among others Anthropic (024). Recently, a well-studied generation use case that can benefit from faster prefill is the scaling of test-time compute (TTC) via best-of-N sampling, where N outputs are generated in parallel and verifier model is used to score them Snell et al. (2024), Cobbe et al. 2021 TTC has enabled recent LLMs to dramatically improve performance on reasoning benchmarks for domains like coding and
math Guo et al. 2025).
1.1 Contributions
In Section 2 we introduce a formal model for analyzing the L.M query scheduling problem. Our model draws inspiration from prior work on the roofline model Imai et al. (2024) and incorporates experimental observations to ensure practical relevance. Despite its grounding in real-world consid- crations, the model remains sufficiently simple to facilitate analytical insights, including a formal specification of the "query stream" (Definition L and a computational model for LLMs (Definition 2). enabling further exploration of the problem.
We show that the decision problem of determining whether TTFT constraint can be satisfied for a given query stream is NP-Hard when using Radix Attention (Theorem) I.. This is in contrast to the case without Radix Attention, where latency is trivially minimized by FCFS, or the case with Radix Attention and uniform arrival times, where latency is minimized by LPM Zheng et al. 22024].
Although the decision problem is NP-hard, we introduce data generative model (see Definition 4 that effectively captures the behavior of realistic query streams in key applications. Additionally, we present generalized algorithm, k-LPM (Algorithm [). which outperforms both FCFS and LPM under this data generative model (sce Theorem 2 and Corollary 3
'LPM is the default query scheduling algorithm in SGLang v0.4.1. Meanwhile, FCFS is an option in SGLang and the default scheduling algorithm in vLLM v0.6.6.

---

We validate our theoretical results with experiments demonstrating that our predictions hold in practice by running a Llama-3.1 3.1--B..BB-----.. -B--Intrrct model on the SGLang serving framework using real prompts. In particular, the k-LPM algorithm is able to attain reduced P99 TTFT across a range of request rates and settings of the hyperparameter k.
Finally, in Appendix A we prove that approximation algorithm exists that, for length query stream Q and TTFT constraint T. either 1) certifies no schedule exists satisfying the TTFT constraint, or, 2) returns schedule such that the (1 p)-th percentile TTFT is at most T for p (0, 1). Furthermore, this algorithm runs in On exp(1/p log 1/p)) time.
1.2 Related work
There has been significant interest in prior work on scheduling LLM queries in order to maximize throughput while satisfying TTFT constraints. Some examples are FastSwitch, which employs priority-based scheduler with preemption to dynamically allocate resources to effectively reduce TTFT and GPU idleness Shen al. [2024), and Orca, which employs iteration-level scheduling. processing each model iteration separately allowing for flexible batching and immediate response to newly arrived requests Yu al. 2022]. Many other papers increase throughput subject to latency constraints by innovations to the processing schedule of incoming prompts Zhong et al.) (2024), Patel al.] (2024), , Jain et al.] [2024), Agrawal al.] [2024]. Collectively, this prior work underscores the importance of dynamic scheduling in achieving high throughput without compromising latency guarantees.
The most relevant work to ours addresses scheduling of LLM queries with consideration of prefix reuse. Zheng et al. [2024) introduced RadixAttention and the LPM scheduler, which we build upon. but their work provides only limited exploration of scheduling strategies. (Srivatsa et al. (2024) proposes priority- based local scheduler in Section 3.3 aimed at balancing prefix reuse and waiting times. The scheduler works by assigning requests to priority groups based on prefix cache hit rate and then selects a number of prompts each group proportionally its priority, but no accompanying analysis is provided. [Qin et al.] [2024) integrates RadixAttentn
in local instances but primarily emphasizes maintaining a disaggregated KV cache to balance the decode and prefill phases, enhancing throughput while adhering to latency constraints in highly overloaded scenarios.
Analytic exploration of LLM inference efficiency is relatively limited, with some notable examples. Kim et al. [2024) propose INFERMAX, an analytical framework for evaluating schedulers and deriving theoretical performance bounds, while identifying prefix-sharing awareness a future direction. Yang et al. [2024] analyze the behavior of LLM queries using an M/G/1 queue while accounting for unknown decoding length, and Guldogan al. [0224] examine multi-bin batching to boost throughput. These studies underscore the value of theoretical approaches, which our work advances through focus on prefi--awar scheduling.
1.3
Notation
Let bold lower case letters denote strings, c.g., x or y, over some fixed alphabet. Let |x] denote the length of string Let Overlap(x, y) denote the length of the maximal prefix overlap between x and y.
Computational model of RadixAttention
In this section, we provide model of LLM computation that allows for theoretical study of RadixAt- tention under different scheduling algorithms. Consider single L.LM instance (that may be on a single GPU or parallelized across multiple GPUs in various ways). This L.M instance can process queries using continuous batching. The time for a single pass can be understood from the "roofline model* (see Appendix A.2 of Imai et al. (024 for details on applying the roofline model to LLM inference).
First, we define the query stream as a collection of prompts with associated arrival times.
Definition 1. (Query stream) Let query stream of length n be denoted by Q (x4, (e) where x; is an arbitrary length string in some fixed alphabe and t; 2 0 is the arrival time of the i-th query.
This alphabet will be the of tokens in practice.
3

---

Note that this definition specifies fixed finite collection of queries rather than a distribution as is common in queuing theory. We make this choice simplify the analysis under the added complexity of prefix reuse, and to enable focus on the "burst traffic" behavior of TTFT rather than stable state behavior of the queue. That is, we are most interested in the behavior of scheduling algorithms in the periods of time where queries arrive faster than they can be processed and so the queue becomes temporarily long.
We present a formal model for the computation time of an LLM operating with batch size of one under a specified queue ordering. This model captures both the time to process each prompt- accounting for prefix reuse from the preceding query and the constraint that processing cannot start before a prompt's 's arrival time. We focus on scenarios where the prefill stage constitutes the primary contributor to total inference time, and where RadixAttention provides the largest improvement within that stage.
Definition 2 (LLM Instance Computation), Consider a streum of queries Q = {(xi, 4i) ) heEn)] Suppose these queries are processed in the order X), Xf22)..,,K,, Let R(ja) denote the completion time of the jæ-th query. Then R(h1) = |x51| Cattn Xx, 1. and,
R(ju) max (Bixx (.x 3 (1 Cattn
Overlap(xje 1)).
In other words, the jæ-th query cannot start processing until its arrival time ji Its processing cost is proportional to the prompt length minus the prefix it shares with the previous query. We assume the cache is empty initially, so the first query costs [x,1 l-
During inference, the prefill stage involves a forward pass over all input tokens. The above formula for R(jx) captures the dominant computational cost of a forward pass through an autoregressive transformer architecture. Layers such as the query, key, and value projections or MI.P modules scale linearly with the number of tokens that must be processed in a query. Meanwhile in the attention mechanism, each token must attend to all previous tokens. Taking into account the cached prefix, this step scales with Xx, (|x;, Overlap(x,, x))-))). For short to medium sequence lengths, the point-wise feed-forward network (FFN) typically dominates computation time, For long sequence lengths, self-attention dominates, as scales quadratically. We capture the relative cost of the MLP versus self-attention operations for fixed architecture by the Cattn constant. By focusing on a profitldominated regime, we simplify the analysis while retaining real-world relevance.
2.1 Batch size and prefix cache
The original specification of RadixAttention does not leverage reuse of prefixes of prompts in a single batch (see Algorithm in Zheng et al. 20244). However, more recent development in SGLang has enabled within batch prefix sharing. In this case, behavior of scheduling algorithms in the batched setting can be closely approximated by the batch size one setting in Definition 2 Computing a query processing order and then dividing the queries into B sized bins sequentially provides a schedule for the B batch size setting that attains the same amount of prefix reuse. The only difference is that the time that the i-th query is finished being processed will be the same as when the [ B] -th query finished being processed. This difference will be negligible when the query arrival rate high relative to the batch size.
A discrepancy between Definition 2 and the behavior of Radix is that model assumes only the last processed prompt is cached, whereas Radix/Atentin allocates a fixed amount of memory for caching. We choose to make this simplification avoid introducing an additional hyperparameter and because the behavior remains similar under standard settings. Specifically, we expect the behavior to be similar when prompt length is fairly uniform and the query arrival rate is high relative the TTFT constraint.
In our experiments (Section 5) we use dynamic batch size and a fixed memory pool for the prefix cache in the SGLang framework and observe that our theoretical predictions based on Definition are still accurate.
3 Complexity of scheduling under TTFT constraints
We first explore the complexity of the decision problem for determining whether there exists a schedule for a fixed queue that satisfies given TTFT constraint T for each query. By reduction to

---

('t + 13/ (tu
Di 138
8
x
13
Time
Schodule Time
Figure 1: This figure graphically represents the imposed structure for any feasible schedule in the query stream construction of Theorem [ Note that the only flexibility in the schedule is how the set of strings {x, [x)mm[[3m) fits into the m time windows size H. The solid lines represent arrival times and the dashed lines represent processing start times.
the 3-PARTITION problem, we show that it is NP-Hard to decide feasibility of a queue under TTFT constraint, despite this problem being trivial in the case where there is zero prefix reuse (where FCFS optimal) in the offline setting where all arrival times are uniform (where LPM is optimal).
3.1 NP-hardness of feasibility determination
We provide the definition of the 3 S-PARTITION problem, which is NP-Hard in the strong sense. See SP15 in the appendix of Garey and Johnson 1979] for reference.
Definition 3. (3-PARTITION) Let m E N and H > O. Given set A of 3m integers such that H/4 < a < H/2 for all a E A and E mH, decide if can be partitioned into m disjoint sets ,,, Am such that E.EA. a H for all E [m]]
The intuition behind the proof of Theorem iis as follows. The introduction of prefix reuse along with non-uniform arrival times allows us construct query pairs that must be processed time. As an example, let query x arrive at time H and query arrive at T + H where x and have the same prompt. In order to achieve prefix reuse between these two prompts without any idle time while satisfying the TTFT constraint T, must finish being processed at T + H and must start being processed at this time. By using this idea introduce constraints on processing times, we may construct "windows" of size H that set of prompts may be feasibly scheduled within. By then constructing a set of queries with processing time equal to the integers in Definition 3 deciding the existence of feasible schedule solves the 3-PARTITION problem.
particular
Theorem 1. Deciding if there a processing order in query stream Q (Definition 7 such that a TTFT constraint T is satisfied under the computational model of Definition 2 an NP-Hard problem.
We construct query stream Q such that determining whether the queries can be scheduled to meet the TTFT constraint T (m + m²) under the computation model of Definition equivalent to deciding the 3-PARTITION instance.
Proof. Let A an instance of 3-PARTITION problem defined Definition 3
Query construction.
Define:
X: A set of 3m queries (x:). where each prompt consists of a unique character, leading to zero prefix overlap between queries. Furthermore, |x: a; for some indexing that matches A. Assign
all x; the same arrival time / = (m + m²)H.
Y: An ordered set of m queries (y:). each of length miH, each composed of a unique character
not appearing in X. Assign arrival time t; i(H -mH) y..
Z: A set of queries (z,) where each z, is identical to y; in content but arrives time
T + i(H + mH).
Wi and Wa: Two additional queries, each length T, composed characters distinct from those
in X' Y.Y Let w1 arrive at é = 0 and wa arrive at 27.
5

---

Set the overall query stream to be Q = X' y U 2 U [W1, w2). Observe:
1. W1 must begin processing time / 0. Since |w| 7 and there no prior cache, it finishes
exactly at t=t== 7
2. Since W2 composed of characters distinct from those in UU W2 has zero cache overlap when it arrives at t = 27 Then, W2 must begin processing exactly at t 27 to finish by 37' thereby forcing all queries in X UUUUZ Z to be completed within the time window [T, 2T)).
Within [T, 27), the only possible nontrivial cache overlaps come from pairs and (y;, z;) because all other queries have prompts with distinct characters. For each i [m]] the maximal prefix overlap between and reduces the processing time by ly./- However, to fit all queries from X' U UYUZ into [T, 27), every possible overlap must fully utilized. This rigid constraint implies:
The total time to process y and 2 (accounting for maximum y,-2, overlap) is: Σyey ly: = m T -mH.
Processing must then be continuous (no idle time) in [T. 27), and must attain maximum overlap between every Y; and z, turn, this implies that y; must finish being processed at exactly T + i(H + mH) so that z; can be processed immediately as it arrives without violating the TTFT constraint on y::
Partition of X and relation to 3-PARTITION.
We next show that X' must be partitioned into disjoint batches, each of total length H. Observe that for each i E [m]. since z, must start right when y, finishes, we have
Yi end (and thus) 2; begin at T + i(H + m.H)-
Then Y++1 must be processed starting from T + i(H mH) H in order to finish by T (i () + +m) and maintain the "no idle time" schedule. Since 2; is identical to yı, its processing time is zero in this case. Therefore, between z, finishing T' i(/1 + mH) and Yi+1 starting at T++i( + -mH) ) H, there is exactly length-// sub-interval available. Since Ex.xx |x|| mfl, the only way to fill these m sub-intervals continuously is to divide } into m disjoint groups, each summing exactly H. But deciding such a partition is precisely the 3-PARTITION problem. Consequently, scheduling Q to meet the TTFT constraint is possible if and only if the instance A of 3-PARTITION admits a feasible partition.
In Appendix A we further explore the general problem of scheduling queries to satisfy a TTFT constraint. In Theorem [ we prove that there exists an algorithm that accepts a query stream and TTFT constraint T and either certifies that there is no schedule satisfying constraint T or returns schedule such that the (1 - p)-th percentile TTFT at most T in O(n eexpeexppp)))) exp(1/plog plog 1/p)) time.
k--LPM scheduling algorithm
In this section, we introduce our proposed scheduling algorithm, k-LPM, that generalizes the FCFS and LPM scheduling algorithms. Note that it reduces L.PM when oo and reduces FCFS when h = 1.
Below, we show that k-LPM can achieve superior performance terms of TTFT on random queues under a data generative model that captures the relevant properties of realistic use cases for RadixAttention (Theorem 2). This shows that, despite the intractability of the general scheduling problem, it is still possible to obtain theoretically grounded improvement over existing methods in realistic settings. Section[5] we further support this improvement with experiments showing that the A-L algorithm achieves better TTFT performance than FCFS or I.MM on queues constructed using real prompt distributions.
The intuition behind Algorithm [ is that it first performs greedy prefix-match steps in the spirit of LPM to maximize prefix reuse. After these & steps, processes the oldest query in the queue, mirroring FCFS. This strategy circumvents the LPM failure case, where a query could be unprocessed if its prompt never have sufficiently high prefix overlap. At the same time, retains the significant prefix-reus reuse advantage that LPM provides.
'It is trivial to show that some queries must be processed between Wi and w2 in order to meet the TTFT constraint.

---

Algorithm 1 k-LPM
1: Input: Input queue of prompts and arrival times Q = (x:, (4)
2: while true do
3: Process the oldest query, i.e., x; such that i = argmin Lj
4: for = l...... h do
5: Process x that maximizes the prefix cache hit rate
6: end for
7: end while
4.1 Data generative model
Although deciding whether a TTFT constraint can be satisfied for given query stream is NP-Hard (see Theorem D. we are able to show that under data generative model capturing properties of practical use cases, k-LPM achieves an improvement on the maximum TTFT. Our data generative model has the following additional structure.
Tree structured queries:
Recall that the maximum prefix reuse is attained for fixed set of prompts by DFS traversal of the radix tree constructed from all prompts. Hence, RadixAAtennion can only provide significant efficiency gains if the sum of edges of the radix tree constructed from prompts in a query steam is significantly less than the sum of prompt lengths. Fortunately, many applications of LLMs fulfill this assumption.
In this section, we restrict our attention to instances where queries in queue approximately follow a tree structure of low height. An example of such prompt structure used in Team [2025] is x = (base _prompt) buser_conteet
(doc). Here, all queries share the same (base ppompp)) and multiple queries may share the same (user .osteex.......
Examples of such structures include generative usecases: personalized content generation Zhang et al. [024], conversational context-aware question answering Zaib et al. [2022), and the predictive usecase with LLMs as engagement predictors in recommendation systems Wu et al. (2024). These scenarios exemplify applications where the prompt structure remains consistent while the user context varies, allowing for efficient processing and relevant responses. By focusing such structured instances, we can better analyze and optimize the computational models for LLMSs under constrained scheduling conditions.
Our data generative model considers the simplest case of prompts constructed from a height two prefix tree, where the edges at each depth are constant. The arrivals of the queries are regular, but the order the arrivals is sampled uniformly from all permutations model that captures the interplay between the tree structure of the prompt prefixes and the arrival rate of the queries under randomness in the queue arrival order. For ease of exposition, we keep with the (user) and (doc) terminology of the previous example use case. However, these ideas apply generally to query streams with approximately tree-structured prompts.
the queue. This is the simplest
Definition 4 (Regular Arrival Shuffled Queue). Let n,, u, k, d, N be parameters such that divides We form collection of n queries, each denoted by (user) (doc), where:
(user) is substring of length u. repeated in exactly distinct queries.
(doc) is substring of length unique each query.
Each (user) and (doc) starts with distinct character from large enough alphabet, ensuring zero overlap among different (user) or (doc) substrings.
Thus, there are n/k distinct user substrings, each used times, and n distinct doc substrings, one per query, yielding total prompts. We construct the regular arrival shuffled queue Q. {(xa(): s. o(i) i E [n]} by sampling permutation a uniformly randomly from the symmetric group Sn, and assigning arrival time s a(i) to the i-th prompt.
The structural assumptions in Definition 4 can certainly relaxed. In real settings, there would likely be negligible but non-zero overlap between unique user and documents, and the repitions of each user may not be uniform. Here, we avoid these details focus on clarity regarding the most pertinent structure.
7

---

4.2 TTFT improvement from k-LPM
In scenarios where each query can be processed swiftl-- when the processing time of given query is less than the inter-arrival interval 8-the FCFS scheduling algorithm is optimal. However, in more practically relevant burst-traffic regimes where queries arrive in rapid succession and cannot be processed fast enough, backlog of unprocessed queries inevitably forms. To illustrate this, consider a toy example with Cattn = 0, n = queries, a replication factor & = 2, and parameters = and d 5. The queries are denoted as X1 (user); (doc) 1. X2 (user) (doc) X3 (user), 1 (doc) and X₁ = (user), (doc) In the case where the inter-arrival time 8 10, FCFS scheduling is clearly optimal, resulting in uniform TTFT of 10 units for each query. Conversely, in the case with = 0, representing a burst-traffic scenario, the LPM scheduling strategy becomes optimal. Under FCFS, the processing order is XI, X2, X3. X4 and TTFT; = 10 i with max(TTFT,) = TTFT, 40. Under LPM, the processing order is rearranged to x1, X3. X2, X4 with TTFTss of 10, 15, 25, and 30, and max(TTFT,) = 30. This example underscores the improvement of TTFT from cache reuse in practical scenarios.
In the following thcorem, show that k-LPM has lower maximum TTFT than FCFS LPM with high probability on instances of the regular arrival shuffled queue (Definition 4). We set the hyperparameter l in k-LPM match the number of user repetitions defined in Definition 3 for simplicity. In practice, the hyperparameter & in k-LPM can be determined through back-testing or by employing adaptive multi-armes bandit approach to achieve better performance than the provable improvement observed in the simple setting. In our experiments (Section 5 the performance for varying values of the hyperparameter k.
we empirically measure
Intuitively, the theorem shows that when 8 is small and u is large, I.PM much better than FCFS, and k-LPM retains this advantage. On the other hand, when is relatively large, FCFS is better and k-LPM retains a E factor the sn reduction in TTFT. For intermediate values of s and u, k-LPM prove in Corollary 3 Proofs for these results are provided in
better than both algorithms
Appendix B
Theorem 2 (LPM/FCFS vs. k-LPM), Let Qn be a regular arrival shuffled queue (Definition F of length n with i repetitions of each user prefix, user history length document length d and inter-arrival gap s. Suppose the queue starts being processed at time 2 s n, and let TTFT, denote the time-to-fine to-fistt-token of the i-th query under specified scheduling algorithm. Then, under the computational model of Desinition 2 with Cattn = 0:
(LPM) For every > 0 and 8 E 1). there is an no such that for all n 2 no. with probability at least 1 (with respect to random shuffle and randomness LPM):
max4e[1] TTFT, T + (1 (1-e) <)
+ +d).
(FCFS) For every e > and (0, 1), there is an no such that for all n 2 ng. with probability at least 1 5 (over the random shuffle):
max4e]j] TTFT, 2 T + (1 n (u= (u+00 d-8).
(k-LPM) Deterministically (i.e. for any arrival order), Algorithm maXig]n] TIFT, ST+ tnnne +d-
satisfies:
For any values of k, u, and d such that 0 < s < and k > L, and for any 8 E 1), there exists no N such that k-LPM achieves a lower maximum TTFT that LPM and FCFS simultaneously on the regular arrival shuffled queue (Definition F for any value of 2 no with probability at least 1 5. This result holds for any value of Catta 2 in the computational
Corollary 3.
model of Definition 2
5 Experiments
In this section, measure the performance of k-LPM versus FCFS and LPM in realistic setting. Our results validate the predictive power of the computational model from Section 2 and the data generative model from Definition 4 in real-world serving scenarios.
We assume that ties in prefix overlap broken by uniform sampling in the LPM algorithm.

---

In our experiments, we use the 1amma-31-8B -B-Instruct model Dubey et al. (2024) with tensor paral- lelism across eight A100 GPUs. We run the experiment using the SGLang v0.4.1 serving framework. In particular, we evaluate the timing metrics using SGLangg' Lang's serving benchmark utility [SGLang [2224] and only modify the benchmarking dataset. We implement the k-LPM algorithm as an extension to the current LPM implementation in SGLang. Finally, we construct the dataset used for benchmarking by sampling four prompts with shared user history from the 8k context length prompts described in Team 2025) for 2100 prompts in total. We follow the textual interface of Team [ Member Profile, Past Interactions, Question) where the first three constitute the shared prefix and only the Question varies; 360Brew, Table 1 for example of this exact layout. We then randomly shuffle the ordering of these prompts and use 100 for warm up of the benchmarking server.
5.1 Performance of k-LPM
We measure P99 TTFT vs. request rate for vary- ing values of k in the k-LPM algorithm. Our key observation (Figure 2 that setting the hyper- parameter to k 2 achieves reduced P99 TTFT across a wide range of request rates compared to FCFS and LPM. Note that SGLang's bench- marking utility uses a Poisson arrival process, "request rate" refers the average number of requests per second. Additional experiment re- sults provided in Appendix C showing other TTFT percentile metrics and performance when varying the amount of prefix reuse.
200
k-1000
100
The experimental results not only highlight the benefits of the k-LPM scheduling algorithm but also demonstrate that our theoretical framework- as encapsulated by Theorem -accuurtely pre- dicts scheduling behavior in real-world settings. This holds true even though the experiments do not strictly adhere to all of the assumptions re- quired by the theorem. In particular, we make the following observations:
Bequest
Figure We measure P99 TTFT versus request rate for five values of the hyperparameter k on 2000 randomly shuffled prompts from the use- case described in Team [2025]. Note that k = 1 corresponds to FCFS and oo corresponds to LPM.
k-LPM consistently outperforms both FCFS and LPM. Over wide range of request rates considered, k-LPM achieves lower TTFT. This underscores its robust advantage, especially under moderate to heavy loads. Additionally, we note that large values of k (e.g.. k = 1000) behave like k == 00.
FCFS shows better performance at low request rates, while LPM is advantageous at higher rates. This matches our theoretical insight in Theorem 2 where a larger inter-arrival time favors FCFS, but decreases (i.e., the request rate grows), LPM becomes more efficient than FCFS.
k-LPM performance remains robust under realistic models and prompt data, when theoretical assumptions are relaxed. Despite using a real LLM, an off-the-shelf serving framework, and an industrial prompt dataset, the observed scheduling behavior closely matches our theory. Here, prefix overlap structure is only approximately tree-like and random tie-breaking holds in practice; streaming processing (starting first arrival) shows the "7 2 sn" assumption is inconsequential; and even when & differs from the true replica count (we used four replicas), choices like k or 3 still yield substantial P99 TTFT improvements.
Future work
The primary objectives of this work were to formalize the LLM query scheduling problem in the context of Radix Attention and to develop practical scheduling algorithm for latency-sensitive appli- cations. This theoretical framework not only informs the design of new methods, but is particularly

---

valuable given the current lack of empirical benchmarks for comparing scheduling strategies. How- ever, further advancements in this area will necessitate empirical evaluation using real-world arrival patterns. In particular, the described behavior depends on the complex interaction between constant factors dictating the arrival rate, query processing rate, and average prefix reuse, along with batch size, radix tree memory limit, and other factors in real serving frameworks. Greater understanding of LLM query scheduling in real systems necessitates a standardized, measurement-drivee evaluation suite to characterize the practically relevant regime.
The proposed problem formulation and theoretical results leave many interesting extensions open. One open question is whether there is an algorithm which returns schedule satisfying a constraint on the (1 - p) p)-th percentile TTFT in (oly(1/p) time, i.e., a polynomial time approximation scheme, Additional directions of interest would be extending the computational model of Section to handle distributional query streams or non-constant decoding length. Finally, generalizing the data generative model of Definition may be interesting to capture other properties of real data.
References
Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S Gulavani, Alexey Tumanov, and Ramachandran Ramjee. Taming shouuhput-latencyy tradeoff in LLM inference with sarathi-serve. arXiv preprint 24Xivv2403.02310, 2024.
Anthropic. Prompt caching for faster, cheaper LLM inference, 2024. URL https: //www http://c.ccc//ees/s////ptttte
Accessed: 2025-01-29.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2 10. 14/68, 2021.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 2024.
Michael R Garey and David S Johnson. Computers and intractability, volume 174, freeman San Francisco, 1979.
Ozgur Guldogan, Jackson Kunde, Kangwook Lee, and Ramtin Pedarsani. Multi-bin batching for increasing LLM inference throughput. arXiv preprint arXiv:2412.04504, 2024.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv preprint arXiv:2501. /2948, 2025.
Saki Imai, Rina Nakazawa, Marcelo Amaral, Sunyanan Choochotkaew, and Tatsuhiro Chiba. Pre- dicting LLM inference latency: A roofline-driven ml method. In Annual Conference on Neural Information Processing Systems, 2024.
Kunal Jain, Anjaly Parayil, Ankur Mallick, Esha Choukse, Xiaoting Qin, Jue Zhang, fnigo Goiri, Rujia Wang, Chetan Bansal, Victor Rühle, et al. Intelligent router for L.LM workloads: Improving performance through workload-awan scheduling. arXiv preprint arXiv:2408. 135 2024.
Kyoungmin Kim, Kijae Hong, Caglar Gulcehre, and Anastasia Ailamaki. The effect of scheduling and preemption on the efficiency of LLM inference serving. arXiv preprint 27X.4.2411.07447, 2024.
Pratyush Patel. Esha Choukse, Chaojie Zhang, Aashaka Shah, fñigo Goiri, Saced Maleki, and Ricardo Bianchini. Splitwise: Efficient generative LLM inference using phase splitting. In 2024 ACM/IEEE 5/ Annual International Symposium Computer Architectree (ISCA), pages 118-132, IEEE, 2024.
Ruoyu Qin, Zheming Li, Weiran He, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. Mooncake: kveache-centric disaggregated architecture for LLM serving. arXiv preprint a7Xiv:2407.00079 2024.
10

---

SGLang. SGLang's serving benchmark utility, 2024. URL http:tt:://ghhhuub ctm hatp/ggl-pppro] projjettt// sglang/blob/vo. .4. http:////pppppooo/gggg/////aaaaa
Accessed: 2025-01-29.
Ao Shen, Zhiyao Li, and Mingyu Gao. Fastswitch: Optimizing context switching efficiency in fairness-aware large language model serving. arXiv preprini arXiv:2411.18444 2024.
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM test-time compute optimally can be more effective than scaling model parameters. arXiv preprint wxXv::2408.03314, 2024.
Vikranth Srivatsa, Zijian He, Reyna Abhyankar, Dongming Li, and Yiying Zhang. Preble: Efficient distributed prompt scheduling for LLM serving. arXiv preprint arXiv:2407.00023, 2024.
360Brew Team. 360brew decoder-only foundation model for personalized ranking and recom- mendation. arXiv preprint 2644:02000111400,, 2025.
Likang Wu, Zhi Zheng. Zhaopeng Qiu, Hao Wang, Hongchao Gu, Tingjia Shen, Chuan Qin, Chen Zhu, Hengshu Zhu, Qi Liu, al. A survey on large language models for recommendation. World Wide Web, 27(5):60, 2024.
Yuqing Yang, Yuedong Xu, and Lei Jiao. A queueing theoretic perspective low-latency Ilm inference with variable token length. arXiv preprint arXiv:2407 005477, 2024.
Gyeong-In Yu, Joo Scong Jeong, Geon-Woo Woo Kim, Soojeong Kim, and Byung- HoungGGon Chun. Orca: A distributed serving system for (Transformer-Basee) generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pages 521-538, 2022.
Munazza Zaib, Wei Emma Zhang, Quan 7. Sheng, Adnan Mahmood, and Yang Zhang. Conversational question answering: survey. Knowledge and Information Systems, 64(12):315--319 2022.
Zhehao Zhang, Ryan Rossi, Branislav Kveton, Yijia Shao, Diyi Yang, Hamed Zamani, Franck Demoncourt, Joe Barrow, Tong Yu, Sungchul Kim, et al. Personalization of large language models: A survey, arXiv preprint arXiv:2411.0000 2024.
Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph Gonzalez, et al. Sglang: Efficient execution of structured language model programs. arXiv preprint urXiv:2312.07/11 004 2024.
Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. Distserve: Disaggregating prefill and decoding for goodput-optimized large language model serving. arXiv preprint arXiv:2401 2401.0970, 2024.

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: See our contributions section in the introduction for detailed connections between our technical results and the overal goals of this paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See the Future work section for discussion of limitations in the current work and ideas for future extensions.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sellsspeciiiationn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
12

---

Answer: [Yes]
Justification:
Yes, every theorem/lemma has an accompanying formal proof, with proofs for results in Section 4 provided in the appendix due to space restrictions.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provided the accurate pscudocode of the k-LPM algorithm in Section 4 and necessary details of the experiment setup in Section 5. The experiment can be reproduced by making a minor change to the SGLang version we specified to add the k-LPM algorithm and then running with the setup we described. However, we are not permitted to share the exact data of our experiments, though this does not affect the core behavior.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the partieular way they provide for reproducibilitty In the case of closed-source models, it may be that access to the model is limited in
13

---

some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: The algorithm can easily be reproduced as described above. We are not permitted to share the exact data we used. However, we' ve've described the relevant properties of the data and so the results should be reproducible by using other similar synthetic or real data sets.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
pbblic/guiiees/ddeettnniii
onPolicy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips.
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Details for the core setup provided in Section 5 that sufficient to reproduce the behavior. As mentioned previously, we are unable to share the specific real data set.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Due to the computational burden of running repeated experiments for multiple al algorithms across many parameter values, we not use report error bars. However, one can from the plots that clear patterns present nonetheless.
14

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the eπror bar is the standard deviation or the standard error of the mean.
It is OK to report I-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (c.g. negative error rates).
show in tables
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification:
We discuss this in Section 5.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
We follow all NeurIPS ethical guidelines.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [NA]
15

---

This paper is primarily focused on the theoretical behavior of query scheduling algorithms for LLMs deployed in the online context.
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has socictal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: These concerns are unrelated to this paper.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We cite the SGLang paper for the LLM serving framework in the experiments and the Meta LLAMA paper for the model we benchmark on. We cite Team (2025) for the real data.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
16

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's ' creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release any new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the datasct/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: No human subjects were involved.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No human subjects related concerns applicable.
Guidelines:
17

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question:
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: This paper is focused on scheduling queries LLMs for online usage, which we make clear throughout the paper.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to LLM policy http::///aarrpppocccc///feeerree for what should or should not be described.
18

---

A Percentile TTFT constraint approximation
The hardness result of Theorem I motivates us to consider approximation guarantees for computing a schedule satisfying constraints on the TTFT. One important relaxation of the problem is to instead consider scheduling algorithms that satisfy constraint on a fixed percentile of the per-query TTFTs. This problem is of interest since latency constrained applications typically seek to bound the P95 or P99 lateney of a response practice. In this section, we show that such a relaxation is tractable when the interaction distance of queries in a queue (controlled by the latency constraint T and maximum prompt length m) is bounded.
Theorem 4 proves there is an algorithm that runs in polynomial time with respect to the query stream length that either returns schedule satisfying constraint T on the (1 p)-th percentile TTFT or certifies that no feasible schedules exists such that all queries satisfy the TTFT constraint T', Note that there is necessary gap where the the positive case pertain the the percentile TTFT problem and the negative case certifies no schedule satisfies a constraint on the maximum TTFT. This gap is fundamental, since otherwise we could solve the problem for the maximum TTFT in by adding enough unsatisfiable queries and solving the percentile TTFT decision problem.
Theorem There exists an algorithm that takes lenghh n query stream Q (see Definition] 1 and TTFT constraint T > 0 as input and either:
1. Returns a certificate that no schedule exists for Q that satisfies TTFT constraint T, or
2. Returns schedule for Q such that the (1 - -)tth percentile TTFT is most T.
under the computational model of Definition with Cattn = O. Furthermore, if every query is at most length mn and no query an exact prefix of another, then this algorithm runs (xexxp(1pplog 1/D)) time when m and T are bounded by a constant.
Proof. At high level will prove that, if there exists schedule for Q satisfying the TTFT constraint T for all queries, then there exists a subset Q' C Q of size (1 p)- that also satisfies the constraint T for all queries. By the contrapositive of the statement, if Q' does not exist, then no such schedule exists for Q. We show that Q' can be constructed by decomposing Q into ((n) subproblems, each of which can be solved in time independent from n, thereby providing an algorithm that is tractable with respect to n.
Without loss of generality, assume that p n is an integer and that is divisible by no reasons we will explain later. First, we partition the query stream Q into d no disjoint blocks Q1,-.., 2n that are contiguous with respect arrival time.
2T for p
Let this partition satisfy the constraint that for any < j. every query in Q, has an equal or earlier arrival time than every query in Qj. For each block Q2, we remove 2T queries with the latest arrival times to form the reduced block Gj- Define -' U L/.
First, note that if there exists a feasible schedule for L' under a uniform TTFT constraint T, then there exists schedule for Q where the (1 p)-th percentile TTFT is at most T. This schedule can be constructed by following the schedule for all queries in Q' and then processing the remaining queries afterwards. By the contrapositive statement, if there does not exist such a schedule for Q'. then there does not exist such a schedule for @
We will next show that it is possible to efficiently compute such schedule for Q' or certify that none exists due to the decomposable nature of the problem. Since 2/2 was constructed by removing the last 2T queries of QR, the latest query arrival time in Ø/ must be at least T units of time before the arrival time of any query in 2/++1 if a schedule for @ satisfying the uniform TTFT constraint exists. This is because query is exact prefix of another, and so it must take at least one unit of time to process a query. Therefore, if all queries in 2s \ Q/ satisfy the TTFT constraint, then their arrival times must span at least T length interval of time.
The partition { [Qa was constructed to partition the arrival times of Q into d contiguous intervals. Then, the previous argument implies that the queries in Q/ must finished being processed before the earliest arrival time in Q'+1 and so the processing time of queries in Q' (the earliest arrival time to the latest completion time) under a processing order of Q satisfying constraint can be partitioned into d contiguous time windows, each corresponding to C/ block. However, this does
19

---

not completely decompose the problem, as the processing order of Ø may still affect the the next block 2x+1 through prefix reuse.
We may handle this dependency by keeping track of the feasible last queries for each block that possible under schedules that satisfy the TTFT constraint, as these dictate the potential cache states when computing the next block. More concretely, if there is feasible schedule for Q then the following procedure must return feasible schedule for O':
1. Let the possible cache initialization of 2g be the feasible end queries of 2 or the empty string if k = 1. Then, in order of l = L....,d, d, compute all possible pairs of cache initialization and last query processed in 2x where a feasible schedule satisfying the constraint T exists.
2. Consider the queries vertices in graph along with vertex representing the empty string initialization and the set of pairs computed in the last step as directed edges in this graph. Compute path from the empty string vertex to vertex representing query in 2
3. There exists feasible a schedule for Q' where the latest processed queries in each Q is provided by the path computed in the last step. Hence, we may compute schedule for each O with the fixed last query with the constraint that the eache initialization and last query processed is dictated by the returned path.
Note that in step one above, if the query in 2/_1 that is processed latest under the feasible schedule for Q is recorded as possible initialization of 2 then the procedure will correctly identify the last query in 2/ processed under the feasible schedule for as a possible initialization for 2 Since the only feasible initialization of the cache at 2 is the empty string, by induction we conclude that step one correctly identifies tuples of feasible cache initialization and end queries for block. From this, there must exist path returned by step two, since the sequence of tuples that occur correspond to the feasible schedule for Q must be a directed path from the empty string vertex to the last query processed under the schedule.
Next, we show that the above procedure runs in O(n exwpe/xppppppp///1 log 1/p)) time for each of the above steps:
1. To compute the set of tuples for 2/- must consider at most no possible cache initialization and no! orderings of queries in Ø. We may verify if a fixed combination satisfies the constraint Ono) time. Therefore, computing set of tuples for fixed k has the following time complexity:
Ong no!) = O(1 (2)) = exp(no log = exp(1/p log 1/p).
Performing this procedure for each h then takes O(n exp(1/p- 1 1/p)) time.
2. For calculating the path, each directed edge is either between a query in 2x to query in Q/+11 or from the empty string vertex to a query in Q,- Hence, the directed graph is acyclic, and the degree of every vertex is most no- Therefore, finding a path from the empty set vertex to a vertex corresponding to query in Q'2 can be accomplished by BFS in O(n (noy( time.
3. Finally, given a feasible sequence of last processed queries for each C/. we may compute the schedule for the other queries in Ø/ while considering fixed cache initialization and last processed query. This can be done by evaluating all possible schedules in O(no!) = exp(1/p- log 1/p) time. Doing this for all d blocks then takes O(n exx((1p log 1/p)) time.
Note that computing the partition (Ox heedd depends solely on the arrival times and so it can be done in (n) time. Finally, to complete the proof, note that do not need to remove the 27 last queries
in Ca to construct O'2 as no other blocks will become dependent on it. Hence, we may adjust the
argument by constant factor, and the assumption that is exactly divisible by 2T is not needed. D p
20

---

B Proofs for Section 4
B.1 Proof of Theorem 2
Proof. LPM: Label the queries so that 1.2,..., are ascending order arrival times, i.c. & S .... tn- Let o--) be the permutation specifying the processing order under LPM. We show that with high probability, some query among the earliest arrivals (say indices [g]) is processed in very late position (> j)-
Concretely, for integers q j. define the event
{3i [q] o(i) j} (some earliest-g arrival is not processed among the first positions
Since the processing order of LPM with uniformly random tie breaking is independent from the arrival times, a is uniform random permutation of [n]] A standard combinatorial bound then gives:
q-1
P Vi (a), ( S)) -
II
- S
(
p=
We set 3// and = n 1/2 Then,
11 - n2/22 3/4
P( o(1) > 2 2)
71
= 1 (1 1/n 1/2
= ((1 1/n 1/2 32133/111 3
By the known limit lim 1)
1
we can conclude that there exists no E such that, for
any 2 ng. ((3 aii) > 2 8. e
For fixed ordering x x the time which the j-th query is finished being processed can be written as:
ITFT, + t + [j/k] u u d
(1)
since under LPM, [j/k] unique user prefixes and unique document suffixes will be processed.
the event occurs, then there must be a query of index i E [g] that has not been processed at time T [5/k] u d Since t S 9 s for all [g]. this implies there exists [g] such that:
TTFT,
> T + [ + d. s
2 T juu/jj/uu d) qs
By substituting in the values = 3/4 and j = n n /2, we conclude the theorem bound for LPM
in the theorem statement.
FCFS: Label the queries so that 1, 2,.,.... . n are in ascending order of arrival times, i.e. f S lo S & In. FCFS processes these queries in the order 1.2....,
Under the Cattn 0, query i's computational time equals:
(u + d), if the user prefix differs from that of query i 1,
d. if query i has the same user prefix as query - 1.
(For = 1, there is no previous query, so the time cost is always a + d.)
Define an indicator variable
1,
if queries and i 1 share the same user prefix,
1
0, otherwise.
Note that [T[]]]
for all 2, ..., n, since the prohahility that the and (i 1)-th queries share the Then, the time needed to process the entire queue is given by:
(user) prefix
k-1 n-1
TTFT. T' X1
([x=]
uls) tn
T T d) sn L.
2=2
21

---

Markov's inequality states that P(X a) I
Hxi where X is non-negative value and > 0. Applying this X Ei= Is with respect randomness in the the queue order implies:
(k
1)vn
k
1 vn
Vn
j=2
This implies that 00, 2iz l converges
in probability. Then the formula can be written
TTFT, 2 ni + =.,)
where e, converges to zero in probability as n - cc. By the relation ma (TTFT,) 2 TTFT,, we conclude the lower bound for the FCFS algorithm in the theorem statement.
k-LPM: First, note that all queries in the queue must finish being processed by T (+= + d). Additionally, the i-th query must be processed time 7' i(u + kd), since this is the time needed to complete i groups of user queries, and hence complete FCFS steps after the start time
TTFT, & min + i(u kd) 1. T
nl
+ d) tx
k
min (T + i(u kd) - i. + ml
u
+ d)
T
min T i(u + kd - s), T + nl + d) is}.
We want uniform bound for all i E [n]] Hence, we may bound over the maximum of all indices and then relax the domain to the entire real line.
TTFT, S max min T + i(u + kd - s). T + n k
+ d) - is}
:E(n]
max min T + i(u - kd s), T + EER
n²
+ d) - -ss}
k
Observe that (i( - i(u + kd - a) is increasing in i, while (2(1) = T + n +d) is is decreasing in i. The maximum of min (1), f2(1) occurs where these two lines intersect. We solve
i(u + kd - 8) nl k
is,
which implies,
i(u + kil) = nl( k
+ d).
1" n + + d)
+
Plugging i" into fi and fo yields
f1)) = To [u + kd - s). fe(t")
= k
n
(u + kd s),
max (f: fa(1)} k
(u kd - s).
Hence for all i E [n].
TIFT, < max min i(u kd s), n 2/4 +
is
E[al
b (u kd s).
B.2 Proof of Corollary 3
Proof. From the conditions that > and s > 0, it follows that:
T nl + - < T nl 2
d).
Furthermore, from the condition that u > s. follows that:
T + n u + d -
$ ))
< T + n(u + d d-
b
22

---

Hence, by Theorem 2 the corollary statement holds when Cattn = 0. We extend to all Cattn > 0 by observing that |x;| = u is constant under Definition query of the regular arrival shuffled queue under Definition 2
Therefore, the time needed process with order indexed by > 1 is:
(1 + Catin (attm/x Overlap(x,, Xy-1))
(1 +) Overlap(x): xz-1));
for some constant that depends only on and d. Rescaling by this 1 + c term extends the result to all Cattn 0 and concludes the proof.
C Additional experiment results
C.1 Real prompt distribution latency metrics
In this section, we provide additional plots for latency metrics corresponding to the experiment of Figure 2
Note that the relative performance of the scheduling algorithms is accurately predicted by Theorem 2 and surrounding discussion. That LPM (equivalent k-LPM with oc) attains the best median TTFT as it maximizes prefix reuse. However, k-LPM with intermediate values of k attains nearly the same median TTFT and better p99 TTFT by ensuring that no queued query waiting for too long.
197
1.0
100000
200
TIF
120
20
25
Request Tate
Eaquum Rattt
(a) Median TTFT
(b) P90 TTFT
1m1
1=2000
TITT 34
130
150
Roque5t
Reques: Ratt
P95 TTFT
(d) P99 TTFT
Figure 3: We measure P50, P90. P95, and P99 TTFT versus request rate for five values of the hyperparameter on 2000 randomly shuffled prompts from the usecase described in Team (2025). Note that à = corresponds to FCFS and k = co corresponds to LPM.
23

---

C.2 Varying prefix length ratio
In this section, we provide experiments which vary u and d in the Regular Arrival Shuffled Queue of Definition 4 We set 2000, &o = (the number of replications per prefixes), and measure at 80 requests per second. For this experiment, we use single H100 GPU and the Meta-Llama-33.2 3B- Instruct model Dubey et al. 22224..
Table showws latency results for k-LPM with & equal 4, and Do respectively, while varying the ratio the replicated prefix to the constant prompt length (see Definition 4 These results match the relative performance of the settings of & predicted by Theorem That LPM consistently results in lower median TTFT due to greater prefix reuse. However, when the ratio of the prefix length to the prompt length (u + d) is small, FCFS achieves lower TTFT high percentiles by avoiding any query from waiting for too long. As the the ratio of the prefix length to the prompt length increases, the impact of better prefix reuse becomes greater. and so LPM performs better even regarding higher percentile TTFT. Finally, k-LPM with intermediate k never performs simultaneously worse than FCFS and LPM, while performs better than both the intermediate regime.
Table I: TTFT timings seconds while varying the ratio of the repeated prefix to a constant total string length.
(a) FCFS
d
Median
P90
P95
1000
5000
3466.2
5604.33
5843.80
3000
3000
509.93
1040.45
1081.62
5000
1000
20.15
524.91
564.44
(b) k-LPM
d
Median
P90
P95
1000
5000
1940.35
8610.27
9128.23
3000
3000
177.40
821.41
1647.49
5000
1000
19.61
273.55
447.37
(c) LPM
d
Median
P90
P95
1000
5000
677.07
11 297.53
14 733.45
3000
3000
176.32
1245.15
2821.72
5000
1000
20.70
216.88
565.92
'Note that generate the prefix and suffix as random strings of the specified length, which results in non-zero negligible between unique prefix suffix substrings.


Reviews:
[{"review_id": "7FVr2WeK9K", "reviewer": "Reviewer_gHUX", "summary": "The paper formulates the LLM query scheduling problem with a computational model and proves its hardness as an NP-hard problem. The paper further introduces k-LPM —a hybrid scheduler that interpolates the FCFS scheduler (k = 1) and LPM scheduler (k = ∞). Theoretical analysis also shows that k-LPM achieves better TTFT performance than FCFS and LPM on realistic, tree-structured query streams. Experiments on an Llama-3 8B model show that with k = 2 the method significantly reduces P99-TTFT compared to FCFS and LPM under different serving loads.", "questions": "Although as shown in the paper, the algorithm is generally robust to the choice of k, I still wonder if there could be some method to choose k principally. For instance, how should k change adaptively as workload changes?\n\nHow would your algorithm interact with pre-emptive batching or other priority/fairness scheduling that is widely used?", "limitations": "My main concerns are in the experiment sections. Also, see my questions for some further places for improvements.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strength**\n- This paper targets on an important problem that is highly relevant to the wide application of multi-agent systems and complicated tree-based reasonings.\n- This paper gives a theoretical model to describe the scheduling problem under TTFT constraints. The paper further reduces this problem to 3-partition problem to show that this problem is NP-hard. This shed light on future research to find better approximations instead of exact solutions for this problem.\n- The proposed method(L-kpm) is intuitive and the author gives theoretical guarntees on why this method is better than FCFS and LPM\n- Experiments on Llama-3.1 8b shows that this method can significantly reduce the TTFT over different loads\n\n**Weakness**\n- Lack of evaluation on different traces:  The experiment is conducted on a relatively small model and limited workloads with only 2k prompts. More different workloads with different level of reusable prompts are helpful to comprehensively evaluate the performance of k-LPM.\n- Lack of evaluation with different metrics: Only the P99 TTFT is reported. What about the P50 TTFT, P95 TTFT?\n- The generative model is based on simple assumption “Our data generative model considers the simplest case of prompts constructed from a height two prefix tree, where the edges at each depth are constant”. This may hurt the robustness of k-LPM of the further proofs.\n- Although it has been shown in the paper that the overhead is minimal theoretically, it would be better to provide some breakdown experiments on the overhead", "was_revised": true, "final_justification": "The reviewer addressed my concerns in the lack of experiments, assumptions, and theories part, and promised to add additional results in the camera ready version. I will maintain my score after the rebuttal.", "rebuttal": "Thank you for providing this review. Please see our responses below.\n\n> Lack of evaluation on different traces: The experiment is conducted on a relatively small model and limited workloads with only 2k prompts. More different workloads with different level of reusable prompts are helpful to comprehensively evaluate the performance of k-LPM.\n\nWe agree that evaluating the behavior at different levels of prefix overlap would be useful. Hence, we have conducted additional experiments using the SGLang serving framework by generating synthetic data matching Definition 4 in our paper so that we can systematically control this prompt structure. We will add the full results to our camera ready version, but as an example we observe on a Llama-3.2 1B model that with ($u=4000$, $d=2000$, $k=4$, request\\_rate=48), the P99 TTFT of $k$-LPM, FCFS, and LPM is [326, 456, 340] ms respectively, showing that the advantage of $k$-LPM holds across different prompt distributions and models.\n\n\n> Lack of evaluation with different metrics: Only the P99 TTFT is reported. What about the P50 TTFT, P95 TTFT?\n\nWe provide P99 TTFT since our focus is on tail-end latency, which is a critical consideration of online systems. However, we will add graphs for P50 and P95 in the appendix to complement Figure 2. Interestingly, we again see the behavior predicted by our theory. That is, P50 is better for large values of $k$ across all request rates (since prefix reuse is maximized), but this doesn't translate to better P99 TTFT at high request rates as Figure 2 shows (as predicted by Theorem 2).\n\n> The generative model is based on simple assumption “Our data generative model considers the simplest case of prompts constructed from a height two prefix tree, where the edges at each depth are constant”. This may hurt the robustness of k-LPM of the further proofs.\n\nWe make these assumptions since it does not materially affect the conclusion of our results while greatly simplifying the notation and exposition, allowing the core ideas to be presented more clearly. We see from our experiments on real world data that these assumptions are not necessary for the predicted behavior to hold.\n\nOne can prove an analogous result for any query distribution where queries have variable arrival times and variable prefix overlaps, since the key idea that over-optimizing for prefix reuse results in long tail latency and ignoring prefix reuse result in low throughput still holds. The precise theorem and proof would depend on the parameterization of the query distribution.\n\n> Although it has been shown in the paper that the overhead is minimal theoretically, it would be better to provide some breakdown experiments on the overhead\n\nThe additional scheduling overhead of $k$-LPM over LPM is to additionally maintain an ordering with respect to arrival time. While this could be done very efficiently by maintaining pointers between the arriving queries (in order) and the prefix-overlap sorted list, this efficiency is not needed. One can sort a randomly ordered list of 10k floats in Python in 0.006 seconds, which is negligible compared to query processing speed, and so the algorithmic overhead is negligible.\n\n> Although as shown in the paper, the algorithm is generally robust to the choice of k, I still wonder if there could be some method to choose k principally. For instance, how should k change adaptively as workload changes?e\n\nWe agree this is an interesting direction to explore, and briefly touch on some potential methods at line 313. Two initial ideas are to use backtesting which works well on stationary data or multi-armed bandit algorithms which could attain optimal asymptotic regret. However, there are many possible approaches (e.g., controller methods) and it is worth exploring further.\n\n> How would your algorithm interact with pre-emptive batching or other priority/fairness scheduling that is widely used?\n\nWhile it is difficult to answer precisely without knowing the exact rules of the scheduling method, we generally expect that they should be rather orthogonal to $k$-LPM. Pre-emption has the downside that it wastes computation, and hence achieves lower average throughput compared to $k$-LPM, but may enforce a harder constraint on the fairness. \n\nSuch ideas may be complementary, since one can use the $k$-LPM algorithm by default to achieve a smooth tradeoff between prefix reuse and prioritizing older queries. Pre-emption can then be used as a layer on top of this to protect against resource starvation of long-waiting queries, though it would be preferable to handle this by setting $k$ optimally instead."}, {"review_id": "ctio97femH", "reviewer": "Reviewer_9K3f", "summary": "The authors study the query scheduling (with prefix reuse) for LLM's problem. They establish that deciding if there is a processing order in query stream such that a time to first token (TTFT) constraint is satisfied is a NP-hard problem. A new query scheduling algorithm k-LPM is introduced that generalizes and outperforms prior methods. Empirical validation of this is carried out on the Llama-3.1-8B-Instruct model.", "questions": "* Can the authors discuss the challenges/difficulties of studying the problem with a time per output token constraint? Can techniques in this paper be used to extend results to this case?\n* In the experimental section, can the authors add a more thorough description of the data used? In particular, it is not easy to discern the structure of the prompts used in the current format.", "limitations": "The authors could consider adding a limitations section", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "**Strengths**\n\n* A novel theoretical definition of completion time $R(j_k)$ that accounts for query size, size of re-used prefix, and the computational cost of a forward pass through the model architecture.\n* A clear and intuitive proof of Theorem 1 that reduces the query scheduling problem to the 3-partition problem.\n* The work identifies clear limitations of the LPM/FCFS methods for scheduling and also demonstrates that their method $k-LPM$ can alleviate these issues.\n\n**Weaknesses**\n\n* Experiments are limited to one model and one set of prompts.", "was_revised": true, "final_justification": "The authors addressed all points during rebuttal", "rebuttal": "Thank you for providing this review. Please see our responses below.\n\n> Experiments are limited to one model and one set of prompts.\n\nWe agree that evaluating the behavior on additional prompt distributions would be useful. Hence, we have conducted additional experiments using the SGLang serving framework by generating synthetic data matching Definition 4 in our paper so that we can systematically control this prompt structure. We will add the full results to our camera ready version, but as an example we observe on a Llama-3.2 1B model that with ($u=4000$, $d=2000$, $k=4$, request_rate=48), the P99 TTFT of $k$-LPM, FCFS, and LPM is [326, 456, 340] ms respectively, showing that the advantage of $k$-LPM holds across different prompt distributions and models.\n\n\n> Can the authors discuss the challenges/difficulties of studying the problem with a time per output token constraint? Can techniques in this paper be used to extend results to this case?\n\nFrom a theoretical viewpoint, the difficulty in studying TPOT is that the number of output tokens is random and dependent on input text and conditional probability distribution represented by the LLM. Handling this requires simplifying assumption, for example, as done in [Yang, 2024]\n\nWhile it may be possible to rigorously study TPOT under our model, it is not high priority for the following reason. Prefix reuse only affects TPOT when an entire prompt overlaps with another and the decoding step leverages this to reduce compute time. Exact prompt matches in a query stream are likely uncommon in practice, and hence existing systems do not optimize for this case.\n\n> In the experimental section, can the authors add a more thorough description of the data used? In particular, it is not easy to discern the structure of the prompts used in the current format.\n\nThe prompt structure of our experiments is described at a high-level in lines 264-267, and the referenced source [Team, 2024] provides more in-depth examples of how these prompts look. We will use the extra space in the camera ready version to expand on this explanation to clarify the detailed prompt structure.\n\n> The authors could consider adding a limitations section\n\nWe will expand our future work section to both point out limitations in the current analysis and propose potential next steps to address them."}, {"review_id": "BPgqaGPZjZ", "reviewer": "Reviewer_a5sr", "summary": "This paper presents a formal theoretical framework for scheduling large language model (LLM) queries, particularly in online settings where inference performance needs to be optimized under strict latency constraints like time-to-first-token (TTFT). The research focuses on \"prefix reuse\", a technique that leverages shared prefixes across queries to reduce computational overhead. The authors reveal limitations of existing scheduling strategies like first-come-first-serve (FCFS) and longest-prefix-match (LPM) and establish that the scheduling problem with prefix reuse under TTFT constraints is NP-Hard. To address this, they propose a novel scheduling algorithm called k-LPM, which generalizes existing methods by balancing prefix reuse and fairness in query processing. The paper provides theoretical guarantees that k-LPM improves TTFT performance under realistic traffic patterns and validates these findings with empirical evaluations showing significant reductions in P99 TTFT compared to baseline methods.", "questions": "1. For emerging inference traffic such as long CoT reasoning and deep research, TTFT is not as important as the real-time serving scenarios. Are there any plans to extend the framework to handle such scenarios? How would k-LPM perform in these cases?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\n1. Solid Theoretical Foundation: The paper introduces a formal theoretical framework for LLM query scheduling, which provides a robust and well-defined foundation for their analysis.\n\n2. The paper proposes k-LPM, a new scheduling algorithm that generalizes existing approaches like FCFS and LPM, offering a balanced solution for prefix reuse and fairness. Empirical validation shows that k-LPM can reduce the P99 TTFT compared to existing methods.\n\n### Weaknesses\n\n1. My major concern is that the paper presents very little empirical evidence to support its claims. While the theoretical analysis part is strong, the experiments are only limited to one model and one metric (P99 TTFT) in Figure 2. What about other models such as long CoT reasoning models? What about the latency per output token, average TTFT etc? Just to name a few.\n\n2. Online serving systems usually deploy abundant capacity than estimated traffic, while the proposed k-LPM algorithm seems to only benefit when the traffic is high. This would make the proposed algorithm less practical.", "was_revised": true, "final_justification": "I would like to thank the author's rebuttal response and keep my original scores. The experimental part of this paper can be further strengthened, but I still support the acceptance of this paper due to its theoretical contribution.", "rebuttal": "Thank you for providing this review. Please see our responses below.\n\n> My major concern is that the paper presents very little empirical evidence to support its claims. While the theoretical analysis part is strong, the experiments are only limited to one model and one metric (P99 TTFT) in Figure 2. What about other models such as long CoT reasoning models? What about the latency per output token, average TTFT etc? Just to name a few.\n\nThe purpose of our experiments is to show that the theoretical prediction of our formal model holds even under the nuances present in real-world serving frameworks (e.g., dynamic batch size, fixed memory budget, etc.).\n\nThe choice of model should not materially affect our described results as long as it fits under the computational model described in Definition 2. Choosing a different model would change the memory needed to store the KV-cache and the amount of time needed for a forward pass which would change the constant scaling with respect to arrival time and prompt length, but not the core behavior described in our theory.\n\nWe have conducted additional experiments using the SGLang serving framework by generating synthetic data matching Definition 4 in our paper so that we can systematically control this prompt structure. We will add the full results to our camera ready version, but as an example we observe on a Llama-3.2 1B model that with ($u=4000$, $d=2000$, $k=4$, request\\_rate=48), the P99 TTFT of $k$-LPM, FCFS, and LPM is [326, 456, 340] ms respectively, showing that the advantage of $k$-LPM holds across different prompt distributions and models.\n\nRegarding other metrics: TPOT is not generally impacted by prefix reuse since prefix reuse could only have effect if the entire prompt was duplicated, hence we omit this metric. We will add the P50 behavior, and we note that it behaves as theoretically expected. Namely, it is in minimized by $k$-LPM with large $k$ since this maximizes prefix reuse, but this doesn't lead to lower P99 at low request rates as Figure 2 shows (and Theorem 2 predicts).\n\n\n> Online serving systems usually deploy abundant capacity than estimated traffic, while the proposed k-LPM algorithm seems to only benefit when the traffic is high. This would make the proposed algorithm less practical.\n\nWhile it is true that a server must not be in the high traffic regime on average (otherwise the TTFT would diverge towards infinity), robustness to high traffic is an important property of real serving systems. Tail-end latency is a key quality metric for online systems, and this is roughly equivalent to studying it's behavior in high-traffic periods studied here. In practice, tail-end latency from traffic spikes can be much larger than average latency, and the $k$-LPM algorithm allows for improved robustness, mitigating the impact of these spikes.\n\n> For emerging inference traffic such as long CoT reasoning and deep research, TTFT is not as important as the real-time serving scenarios. Are there any plans to extend the framework to handle such scenarios? How would k-LPM perform in these cases?\n\nGenerally, the answer to this question heavily depends on the serving framework and what specifically are the \"scheduled\" queries. If the \"query\" being scheduled is the initial prompt that is immediately followed by decoding steps for all reasoning and answer tokens, then it is equivalent to a non-reasoning LLMs from a scheduling perspective. Alternatively, if reasoning is split into separate \"queries\" to an LLM, then $k$-LPM would automatically account for reuse between the reasoning steps, and each step would behave as a non-reasoning LLM. We are happy to dive into detail for any particular setup you have in mind."}, {"review_id": "VM3cwiZxx0", "reviewer": "Reviewer_erQW", "summary": "This paper addresses the problem of online query scheduling for LLM inference in systems using prefix reuse, such as RadixAttention. The authors analyze limitations of First Come First Serve (FCFS) and Longest Prefix Match (LPM) schedulers under latency constraints (particularly TTFT) and formally establish the NP-hardness of the scheduling problem. They propose a new scheduling algorithm, k-LPM, that interpolates between FCFS and LPM, and demonstrate both theoretical improvements under a realistic generative model and empirical gains (e.g., P99 TTFT reductions) on Llama-3.1-8B-Instruct using the SGLang serving framework.", "questions": "- How sensitive is the performance of `k-LPM` to different prompt distributions, particularly those with low or noisy prefix overlap?\n- What is the runtime overhead of computing the k-LPM schedule compared to FCFS or LPM?", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n- **Problem Formalization.** The paper presents a well-defined and structured model for query scheduling with prefix reuse under latency constraints, and establishes the NP-hardness of the problem via a reduction from 3-PARTITION. This offers a theoretical foundation for understanding the complexity of prefix-aware scheduling in LLM inference.\n- **Theoretical Analysis of k-LPM.** The analysis of the proposed `k-LPM` scheduling strategy under a structured generative model (the regular arrival shuffled queue) appears mathematically sound. Theoretical results suggest that `k-LPM` can achieve improved worst-case latency compared to both FCFS and LPM in this setting.\n- **Empirical Validation.** The authors support their theoretical claims with experiments on real LLMs and infrastructure. Evaluation on Llama-3.1-8B-Instruct using the SGLang framework demonstrates that `k-LPM` offers tangible improvements in P99 TTFT across a range of request rates, indicating its practical value.\n\nWeaknesses\n\n- **Simplifying Assumptions.** The theoretical model introduces several simplifications—for example, assuming a batch size of one and caching only the most recently processed prompt—which may limit how well the results generalize to production systems with more complex memory and batching behaviors.", "was_revised": true, "final_justification": "The paper proposes a hybrid method that combines FCFS and LPM, achieving a notable reduction in TTFT. The authors provide a rigorous theoretical analysis in a simplified setting and prove that their approach is never worse than either FCFS or LPM (I don't find any problems related to the theorem). I consider this a meaningful and well-justified theoretical contribution.\n\nThat said, my recommendation remains a weak accept due to the following limitations:\n\nThe theoretical guarantees are established under a simplified problem formulation, which may limit applicability to more complex or realistic scenarios.\n\nIn practice, systems often adopt more sophisticated strategies beyond pure FCFS or LPM, which reduces the practical relevance of the baseline comparison.", "rebuttal": "Thank you for providing this review. Please see our responses below.\n\n> Simplifying Assumptions. The theoretical model introduces several simplifications—for example, assuming a batch size of one and caching only the most recently processed prompt—which may limit how well the results generalize to production systems with more complex memory and batching behaviors.\n\nWe make such theoretical assumptions to formalize the problem in a way that allows for clean analysis of the core behavior. While it is possible to assume more complex memory/batching behavior, this would introduce dependency on many hyperparameters (e.g., how long are the prompts, how large is the GPU memory, etc.) which would obscure key ideas. Furthermore, LLM serving frameworks do not have unified behavior that we can model, e.g., treatment of prefix reuse within a batch.\n\nInstead, we structure our paper to theoretically explore the behavior of prefix reuse vs. arrival ordering on tail-end latency under the simplified model. Then, we experimentally validate that these behaviors do indeed hold in a popular production-grade serving framework.\n\n> How sensitive is the performance of k-LPM to different prompt distributions, particularly those with low or noisy prefix overlap?\n\nTheorem 2 of our paper describes rigorously how varying prefix overlap, query length, and arrival time affect expected behavior of $k$-LPM, FCFS, and LPM. For very low prefix reuse, FCFS becomes advantageous and for very high arrival rate, LPM becomes advantageous. The general $k$-LPM method generalizes and interpolates these two extremes. However, $k$-LPM will never be simultaneously worse than FCFS ($k = 1$) or LPM ($k = \\infty$), and it will be better in the intermediate regime. In summary, $k$-LPM with intermediate values of $k$ is actually more \"robust\" than pure FCFS or LPM across variations in these parameters.\n\nMinor noise in the prefix overlap which appears in real world settings does not materially affect the behavior in Theorem 2, since the bulk behavior will be the same. This noise is modeled by the random choice selection use of LPM (see Footnote 4).\n\n> What is the runtime overhead of computing the k-LPM schedule compared to FCFS or LPM?\n\nThe additional scheduling overhead of $k$-LPM over LPM is to additionally maintain an ordering with respect to arrival time. While this could be done very efficiently by maintaining pointers between the arriving queries (in order) and the prefix-overlap sorted list, this efficiency is not needed. Sorting a random list of 10000 floats in Python on a CPU takes just 0.006 seconds—essentially nothing compared to typical query‐processing times. Moreover, we can tap into highly optimized C++ sorting libraries to eliminate even that minimal overhead."}]

Author rebuttal:

Meta-review: The paper studies LLM query scheduling, which is an important problem. All the reviewers agree on the paper's solid theoretical contribution. While some reviewers raised concerns about experimental parts, the authors effectively addressed these concerns during the rebuttal.
Decision: Accept (poster)