Cascadia: An Efficient Cascade Serving System for Large Language Models
Authors: Youhe Jiang
Fangcheng Fu
Wanru Zhao
Stephan Rabanser
Jintao Zhang
Nicholas D. Lane
Binhang Yuan
Venue: ICLR 2026 Poster (2026)
Keywords: Distributed
Parallel
and Cluster Computing
Primary area: infrastructure, software libraries, hardware, systems, etc.

Abstract: Recent advances in large language models (LLMs) have intensified the need to deliver both rapid responses and high-quality outputs. More powerful models yield better results but incur higher inference latency, whereas smaller models are faster yet less capable. Recent work proposes balancing this latency–quality trade-off using model cascades, which route simpler queries to smaller models and more complex ones to larger models. However, enabling efficient cascade serving remains challenging. Current frameworks lack effective mechanisms for handling (i) the huge and varying resource demands of different LLMs, (ii) the inherent heterogeneity of LLM workloads, and (iii) the co-optimization of system deployment and routing strategy.

Motivated by these observations, we introduce Cascadia, a novel cascade serving framework designed explicitly to schedule request routing and deploy model cascades for fast, quality-preserving LLM serving. Cascadia employs a bi-level optimization method: at the deployment level, it uses a mixed-integer linear program to select resource allocations and parallelism strategies based on LLM information and workload characteristics; at the routing level, it applies a Chebyshev-guided method to iteratively co-optimize the routing strategy and the system deployment produced by the deployment level. Our extensive evaluation on diverse workload traces and different model cascades (DeepSeek and the Llama series) demonstrates that Cascadia significantly outperforms both single-model deployments and the state-of-the-art cascade serving baseline, achieving up to 4$\times$ (2.3$\times$ on average) tighter latency SLOs and up to 5$\times$ (2.4$\times$ on average) higher throughput while maintaining target answer quality.

Published as a conference paper at ICLR 2026
CASCADIA: AN EFFICIENT CASCADE SERVING SYS- TEM FOR LARGE LANGUAGE MODELS
Youhe Jiang- Fangcheng Fu Wanru Zhao Stephan R Jintao Zhang", Nicholas D. Lane Binhang Yuan
1. The Hong Kong University of Science and Technology Shanghai Jiaotong University
3. University of Cambridge Princeton University
5. Tsinghua University
jlag@gmia... com, ccchergffesjtu.eeo
rabanser@princeton.cc 2hang-jt24@maills wetaaaisssssssniiiie tsinghua. edu. cn
(wz341, nd132) @cam.ac. biyuanlust.
ABSTRACT
Recent advances in large language models (LLMs) have intensified the need to deliver both rapid responses and high-quality outputs. More powerful models yield better results but incur higher inference latency, whereas smaller models are faster yet less capable. Recent work proposes balancing this latency-qualee trade-off using model cascades, which route simpler queries to smaller models and more complex ones to larger models. However, enabling efficient cascade serving remains challenging. Current frameworks lack effective mechanisms for handling (i) the huge and varying resource demands of different LLMs, (ii) the inherent het- erogeneity of LLM workloads, and (iii) the co-optimization of system deployment and routing strategy. Motivated by these observations, we introduce CASCADIA, a novel cascade serving framework designed explicitly to schedule request routing LLM serving. CASCADIA
and deploy model cascades for fast, employs bi-level optimization method: the deployment level, it uses a mixed- integer linear program to select resource allocations and parallelism strategies based on LLM information and workload characteristics; at the routing level, it applics a Chebyshev- shev-guded method to iteratively co-optimize the routing strategy and the system deployment produced by the deployment level. Our extensive evaluation on diverse workload traces and different model cascades (DeepSeek and the Llama series) demonstrates that CASCADIA significantly outperforms both single-model deployments and the state-of-the-ar cascade serving baseline, achieving up 4x (2.3x on average) tighter latency SLOS and up to 5x (2.4x on average) higher throughput while maintaining target answer quality.
INTRODUCTION
Large language models (LLMs) such DeepSeek-RI (Guo al. 2025). OpenAl o3 (OpenAl 2025), Claude (Anthropic) 2024), Gemini (Reid et al. 2024) and Llama-3 (Dubey et al. 2024) have demonstrated outstanding performance across wide range of real-world applications (e.g.. chatbots,
healthcare and education) (Jeon& & Lee 2023 Peng et al. 2023 GitHub 2024). largely influence
human lives. However, serving LLMs can be costly (Jiang al. 2024 2025b Miao et al,, 2024b), since significant computational resources (e.g., GPUs) are required to meet certain service demands,
such meeting certain latency deadlines (i.e., SLO attainment- the proportion of requests served within a specified response- time target) and generation throughput. In this paper, we explore an alternative solution that strategically utilizes model cascades to better balance the response latency and quality trade-offs inherent in LLM serving.
represents equal contribution, Correspondence Binhang Yuan.

---

Published as a conference paper at ICLR 2026
Cascade model serving refers to a serving architecture where multiple models of varying sizes and capabilities arranged in sequential pipeline, creating hierarchy of models that process requests with increasing levels of
DeepSerk-67BB
DrepSenk-dit-78
DospSica--iss-788
Qualtty
7.00
80
92
78
IIJ
sophistication (Aggarwal et al.] [024 Chen et al. Koss-
2.3
40
0.4
mann et al. 2024, Kolawole et al. Lebovitz et al., 2023]
Different Models
Different Models
Streeter 2018). As shown in Figure I larger models typically provide higher response quality but also incur greater latency, which in turn leads increased energy consumption and compute usage (Samsi et al. 2023). In this approach, incoming requests are initially handled by smaller, computationally efficient models that can rapidly process simpler requests. Only when these lightweight models determine that a request exceeds their capabilities or requires higher-quality responses does the system escalate the request to larger, more powerful models in the cascade.
Figure I: Average response quality and laten- cies of different DeepSeek models. Quality is judged by GPT-40 using the LLM-assa-Judge framework (Zheng et al. 2023).
This progressive delegation mechanism enables service providers to optimize system performance by matching request complexity with appropriate model capacity, thereby significantly reducing computational costs while maintaining high-quality responses for complex request. Several recent studies have focused on optimizing LLM serving using model cascades (Chen et al. Aggarwal et al.
2024 Kossmann et al. 2024 Gupta et al. Narasimhan et al. 2024).
The cascade model serving architecture, which adaptively routes simpler and more complex requests to smaller and larger models, respectively, presents significant opportunities for optimizing the cost-efficiency of LLM serving. In this work, we focus specifically on the setting where service providers host and manage every model in the cascade themselves. However, effectively adapting this paradigm LLM scenarios is much harder to implement than to propose, as we enumerate below:
Model heterogeneity. LLMs require large amounts of compute and memory, and different models have varying resource demands for efficient serving (Duan et al. 2024). With fixed resource pool, suboptimal allocation across models in the cascade can degrade overall serving efficiency.
Workload heterogeneity. LLM workloads exhibit considerable heterogeneity (Sun et al., 2024
Zheng al.) Zhao et al.). Models within the cascade often face incoming requests with varying characteristics (c.g., input/output lengths, arrival rates) and favor different deployment strategies (e.g., replication. parallel configuration), further adding complexity to optimal system deployment.
Cascade-aware load balancing. The request routing strategy directly impacts the system load of each model in the cascade. For instance, if more requests are routed to a particular model, its load increases; the resource allocation and deployment strategy for that model should then be adjusted to balance loads across all models. Consequently, the deployment of multiple models must be co-optimized with the routing strategy to manage load across the cascade.
In order to overcome these challenges, we propose CASCADIA, a novel cascade serving system that optimized for LLM characteristics and that co-optimizes the deployment of multiple models in the cascade together with the request routing strategy. Our contributions are as follows:
Contribution 1. We formulate cascade serving-covering system deployment and request rout- ing- as constrained optimization problem. To solve it, we propose a bi-level approach that jointly optimizes deployment and routing. The deployment level uses mixed- mixed-integer linear programming (MILP) to determine the optimal deployment plan given routing strategy, while the routing level applies a Chebyshev-guides method to optimize routing, balancing latency and quality.
Contribution 2. We implement CASCADIA, an efficient cascade serving system tailored to LLMs. CASCADIA enables an adaptive model cascade paradigm that allocates resources and routes requests across a hierarchy of model sizes (e.g., small, medium, and large), thereby balancing response latency and output quality. Within each cascade stage, CASCADIA supports various parallelism strategies (c.g., tensor and pipeline parallelism), which allows it to automatically select the optimal strategy based on model size, incoming workload, and routing decisions.
Contribution 3. We empirically evaluate CASCADIA by comparing it to both single-model and existing cascade serving systems across a variety of scenarios, including diverse workload traces (e.g.. coding and mathematies), different model cascades (DeepSeek and the Llama series), and multiple evaluation metrics (SLO attainment and throughput). The results show that, compared with state-of-the-ar non-cascade and cascade solutions, CASCADIA achieves up to 4x lower latency deadlines (2.3 x on average) and boosts system throughput by up to 5x (2.4x on average).
2

---

Published as a conference paper at ICLR 2026
2 PRELIMINARY AND RELATED WORK
LLM inference phases and workload heterogeneity. There are two phases within LLM inference: prefill and decoding. During the prefill phase, the model processes the imput prompt to compute the key-value (KV) cache and generates the first token in a single step. In contrast, the decoding phase the last generated token and the KV cache inputs to generate subsequent tokens in token-by-token manner. Generally, the prefill phase is compute-bound, while the decoding phase memory-bound (Patel et al. 2024 Zhong al., 2024 Agrawal et 2024). LLM inference workloads exhibit heterogeneity in input, output token lengths and request arrival rate, which is called workload heterogeneity. For instance, conversation workloads (short input and long output lengths) typically require more memory resources to handle the memory-bound decoding phase, while coding workloads (long input and short output lengths) demand more compute resources to manage the compute- bound prefill phase. Therefore, appropriately allocating resources based on workload demands is critical for optimal performance (Zhao et al. 2024, Jiang et al., 2025a).
Cascade model inference. Current LLMs come in various sizes and configurations, offering broad spectrum of choices. Effectively leveraging this diversity can balance trade-offs between response latency and quality during inference. Recent efforts propose cascade model inference to utilize models of differing complexities (Dekoninck et al.] 2025 Narasimhan et al.) (225).. In such architectures, an input prompt is processed through increasingly complex models, using threshold-basee routing that stops computation once a cheaper model produces a confident enough answer. For instance, FrugaIGPT (Chen al.) employs dynamic LLM cascade strategy that routes queries through progressively stronger models (c.g.. GPT-3.5 - GPT-4) based on real-time difficulty estimation, optimizing cost-efficiency without sacrificing accuracy. Similarly, AutoMix (Aggarwal al. 2024) uses intelligent layer-wise token routing to dynamically allocate computation based on input dificulty. CascadeServe (Kossmann al. 2024) automates and optimizes end-to-end inference with cascades, adjusting model deployment and request routing based on real-time system loads. However, existing systems overlook key L.L--spccffic workload characteristies and neglect the importance of co- optimizing system deployment with request routing (i.e., system-algorithh co-design).
Throughput (1oap))
(4.1))
7B (2.2.1)
(2,1.2
70B (4.2.)
70B (4.1.21
70B (2,4.1) DeepSeck
SeepSeck-dist-22
dist-
DeepSeek-dist dist-7D
DeepSeek-dist dis
DeepSeck Seek--ist--O000 d0s
SeckSeekdiss-700
14.0]
19:26
261
9.82
9,54
13
L.74
2,99
6.58
1.86
4.67-
6.42
0.87
1.50
3.29
0.95
0.00
Iale:44 long ourputs
rate-88
req Fale=44
TM long outpats
req rate:4 shart outputs
Fall=4
long outputs
long ourputs
short outputs
Figure 2: Benchmarked performance of different parallelism strategies across different workloads and model sizes. Long and short outputs represent different workloads with average output sequence length to be 512 and 1024; the threc-clement array represents the DP, TP, and PP degrees.
Limitations of existing cascade serving systems. We summarize the limitations of cxisting cascade serving systems: (i Ineffective resource allocation for different model types within a cascade. Different model types have distinct memory and computation resource needs. For example, DeepSeek- 671B typically requires more allocated resources than DeepSeek-dist-7000 due its larger memory and computational demands. Current systems ignore the importance of adjusting resource allocation according to the needs of different model types, leading to unbalanced system loads. (ii) Inadequate adaptation of parallelism strategies to varying workloads and model sizes. The optimal parallelism strategies vary across different workloads (e.g., different input and output request sequence lengths and request arrival rates) and model sizes. As shown in Figure 2, choosing the optimal parallelism strategy can achieve up to 3x higher system throughput. Current systems do not optimize parallelism strategies according to specific workload and model size, resulting in degraded overall system performance. (iii) Insufficient co-optimization between system deployment and routing strategy. The routing strategy decides the request portion processed by each model type within cascade, which in turn determines the system loads for different model types. Existing systems neglect to adapt system deployment configurations based on routing outcomes, resulting suboptimal resouree usage. To address these challenges, a cascade serving system tailored for LLMs is necessary. Such a system must optimize end-to-end performance and ensure stringent SL.O adherence.
3

---

Published as a conference paper at ICLR 2026
3 SCHEDULING ALGORITHM IN CASCADIA
3.1 PROBLEM FORMULATION
To optimize the cascade serving system under different LLM workloads and user-specific requirements (e.g., system response quality requirements), the scheduling algorithm should determine two essential components: (i) The model deployment plan, which specifies the resource allocations and parallelism strategies for multiple model types (e.g.. small, medium, large) within the cascade to minimize the system response latency (c.g., p95 latency-thh response time threshold below which 95% of all requests complete); and (ii) the routing strategy, which balances the trade-off' between system response latency and quality to decide the appropriate model path for each incoming request. We term a solution addressing these two components as cascade plan.
Note that the routing strategy determines the request distribution over different model types, which in turn dictates the optimal model deploy- ment plan, while the model deployment plan de- fines the system response lateney that feeds back into the routing decision. Given the interdepen- dent and exponentially large search space, de- termining the optimal cascade plan is an NP- hard problem. To solve this problem, adopt a bi-level optimization method that enables sys- tem- algoritti co-design, which is shown in Al-
Algorithm 1: Bi-level Scheduling Workflow
Require: Og: initial routing strategy; O: routing strat-
egy; mmin quality requirement; I: subsampled
input workload; W: workload distribution; Q: system response quality; N: resource limit; D: deployment plan; L: system response latency; J: latency-quality score; K: consecutive stable iterations to break
Ensure: final routing strategy 0 and deployment D 1: 0 < Gu /" Op detailed in B33 %
while true do
3:
(W; Q) + derived frrm (0, Ż)
gorithm 1, and can be summarized as:
4:
/* Optimize deployment ($3.2 */
MlLeP- deployment solver: Given the
5:
(D, L. <- DeploymentSolverr W. N)
6:
1e Optimize routing strategy
routing strategy, the deployment solver ($ (3.2) employs an mixed-integer linear programming
(0, JJ <- RoutingSolver(L. Q. (min)
8:
/tt Terminate upon convergence */
(MILP) formulation to capture system resource
9:
if J is stable for K iters then
constraints and compute the optimal deployment
10:
break
plan that minimizes system response latency.
11: return (9, D)
Chebydhev-guided routing solver: Based on the system response latency generated from the deployment solver and the user-specific quality requirement, the routing solver ($3.3) applies Chebyshev-guider method to find the optimal routing strategy that optimizes system response latency with respect to the quality requirement.
3.2 MILP-BASED DEPLOYMENT SOLVER
As shown in Algorithm the routing strategy (obtained from routing solver) determines how many requests should be routed to each model in the cascade, thus determining the workload distribution among models. Given the workload distribution and resource limit, the deployment solver aims to determine the optimal deployment plan, which includes the resource allocation and parallelism strategies for models within cascades. An example deployment plan is shown in Figure
Assume a total of N GPUs serve model cascade with C model types, {ci, C2, .,....,.C. cc where c; de- notes the i-th model type. The incoming workload information is denoted as W (wi, W22 ........... wc } where each w/ includes the distributions of input/output sequence lengths and the request arrival rate for the i-th model type. We use F [/,,,,,... f2...., ScJ to denote the number of GPUs allocated per model, the total allocation must not exceed the resource limit, i.c., E f. < N. Given this setup, our deployment solver (i determines the parallelism strategy for each specific resource allocation f., and (ii) uses an MILP to optimize the overall resource allocation F..
Parallelism strategy search. Given the workload information w; and a specific resource allocation fi. this optimization determines the optimal parallelism strategy and computes the corresponding system response latency 1, for the model type CASCADIA provides three forms of parallelism: data parallelism (i.e., model replication, DP) (L.i et al., 2023), tensor model parallelism (TP) (Shoeybi) al. 2019) and pipeline parallelism (PP) (Huang et al. (019). Denoting the degrees of data, tensor,
'Given @ and Σ. W is derived by aggregating per-madel routed requests (including arrival rates and sequence statistics), while Q is derived by aggregating quality scores of accepted outputs across all models (Chen et al.).
4

---

Published as a conference paper at ICLR 2026
and pipeline parallelism for the model type by dp, tp, and pp, any feasible parallelism strategy must satisfy the following resource constraint: (2j=) tpij PPIJ) S fú, i.e., one model type can be replicate into multiple replicas, each replica can have varied tensor and pipeline parallelism degrees, as shown in Figure 3] the summation of different parallelism degrees should be less or equal than the total number of GPUs assigned. Based on the workload information ur; and the resource allocation fi. we iterate over all feasible parallelism combinations to select the strategy that minimizes the response latency 4 for the model type i. The latency 4 is computed using the simulator Sim(-) as 4; Sim w,, fi) Note that the parallelism strategy optimization can be precomputed for all possible resource allocations f to provide latency lookup tables for the MILP formulation.
MILP formulation for resource allocation op- timization. Our MILP problem formulation aims to minimize the maximum system response latency among all model types in the cascade, Let L denote the maximum latency across all model types. We discretize the GPU allocations into candidate values E 21, N). For each model type i and candidate allocation f. we use the precomputed latency table from the parallelism strategy optimization to obtain 4((f). We then introduce binary assignment variables fi where f 1 if model rype is assigned f GPUs and xi.f 0 otherwise, for all i E {1,.... C} and feasible f. The constraints of our MILP include: (i) For each model type i, exactly one GPU alloca- tion f must be selected, i.e.. Σx= Zi,f v, = 1, ..... C: (ii) the total number of GPUs assigned across all model types should be equal to the available GPUs N, i.e., Σ Ey=1 = N; and (iii) the maximum latency L must be at least as large as the latency li(s) corresponding to each selected allocation, i.e., L Σÿ= 4(f) æi vi I....,C C. We explicitly enforce variable domains and integrality constraints as follows: Fis E [0,1], and L 2 0. If certain GPU allocations/ f are infeasible for specific model such as when the total memory of the allocated f GPUs is less than the minimum memory required by the model type-we explicitly set æ,, = 0 for these allocation pairs. Our objective is to minimize the maximum system response latency L. which serves as the input for the routing layer optimization.
Workload 5. Nodel Rephea 4P Teneor Parallelian "> Pipulina Parallaiam
BP 2:31 2.10
Figure 3: Illustration of a model deployment plan.
3.3 CHEBYSHEY-GUIDER ROUTING SOLVER
As shown in Algorithm I, the deployment plan (obtained from the deployment solver) determines the system response latency. Given the system response latency and quality requirement, the routing solver aims to optimize the routing strategy (i.e., co-optimize system latency and quality).
Thresholds tuning and request routing. We adopt the threshold-basee cascade routing workflow consistent with prior works (Aggarwal et al., 2024, Chen et al.) (Figure 4). Initially, every incoming request is sent to the first (smallest) model type e₁ in the cascade. A judger then evaluates the quality of the output responses from model types e₁ to CC-1, and a set of thresholds H (h,, ,, fug, ... li-11} is defined to decide whether the requests at each model type should be accepted or forwarded to the next model type. In this framework, the rout- ing strategy is directly determined by the thresholds H. i.e., 0 = (H)) Each routing strategy 0 is associated with a system response latency L(O) (determined by the deployment solver optimization) and quality (0) (determined by the judger
Router
Routing Management
Regs
[U.MM
LLM Cascade Inference
Responses Responses
Responses
Responses Accepted
Threshold-based cascade routing workflow. The router deter- mines whether a request accepted forwarded to the next model type based on predefined thresholds.
Figure 4:
"W use the ETH EASL Scratchpad simulator (ETH-EASL, 2025) to estimate system p95 latency from workload and resource allocation. We show detailed simulator design (e.g.. simulator inputs, batching strategy, queuing mechanism, parallelism strategy modeling) and evaluation in[Appendix B
3 AAnalogous to (Chen et al.). we estimate Q(0) by profiling a subsample of the input workload across all cascade models to obtain per model quality score distributions. During scheduling, given any threshold vector H. and the quality score distributions, we can determine which model's 's response would be accepted for each request under routing policy (HH), then aggregate these final model scores to compute the overall system quality Q()).
5

---

Published as a conference paper at ICLR 2026
Our routing solver uses Chebyshev-guides method to optimize the routing strategy. We initialize the routing strategy 8g as proportional routing, where the i-th model receives i/i requests.
Chehysher-guided optimization for routing strategy. Given the routing strategy and user- specified quality requirement ymin: we employ the Chebyshev-guided method (Steuer & Choo 1983) to minimize the system response latency L(0) with respect gmin- First, we define a utopia point 2 (all requests processed by the largest model ec) and nadir point 2 (all requests processed by the smallest model 61) representing the best and worst achievable system response quality. Then, for
given quality requirement quin. we minimize the system response latency subject to meeting the quality requirement by solving the single-objective penalty problem:
arg min J(O) = arg mijn [L(0) + μ pmax (0, (ymin Q(0))((zi zi)
where J0) represents the latency-quality score, 11 is penalty weight that enforces the quality constraint (for sufficiently large μt, any minimizer of ((O) satisfies ((0) 2 Amin). and zं and z
are used to normalize the quality shortfall so the penalty dimensionless and well-conditioned across workloads. Note that our routing solver can also optimize system response quality under
user-specified latency requirement using a similar procedure, as detailed in/pppendix C
Illustrative example for Chebyshev-guide optimization. Assume the utopia and nadir points zj
and z2 equal 0.95 and 0.75. The user-specific quality requirement qmin 0.90 and the penalty weight is 100. Consider a strategy On with p95 latency L(01) = 11.0 s and overall quality Q(0:) = 0.88. The normalized shortfall from the requirement is (0.90 0.88)/(0 (88)/(009) (0.95 90 0.75) = 0.02/0.20 = 0.10,
yielding (0:) 11.0 100 0.10 21.0. Consider another strategy B2 with latency L(62) 11.4 s and quality ((22) 0.91, which results in J (02) ) 11.4. Strategy 0g is preferable under this setting due to its significantly lower objective value. Additionally. higher-quality strategy 0g with latency T.(0g)= = 12.2 s and quality ((Os) 0.93 yields ((s) = 12.2. Although both 0g and fg satisfy the quality requirement gmin, strategy 02 is preferable since it achieves lower latency while meeting the constraint. This example demonstrates how the Chebyshev-guies method effectively penalizes infeasible solutions while optimizing system response latency.
Putting them together. In our bi-level optimization framework, the routing solver (i.e., Chebyshev- guided optimization) iteratively searches for the next 0, invokes deployment solver (i.e., MILP optimization) to obtain the minimized system response latency L(0). and then minimizes the objective function (i.c., arg ming- (()). Finally, an optimal routing strategy selected that guarantees a minimal system response latency while fulfilling the quality requirement.
Impact of LLM workloads on optimal cascade plan selection. The characteristics of incoming LLM workloads strongly influence the selection of cascade plans. This influence stems from two key factors: (i) Request input/output length and arrival rate affect system response latency-longer sequences higher loads increase compute demand, necessitating plan adjustments to balance latency and quality; (ii) Request complexity impacts system response quality-comples requests or difficult queries require larger models, necessitating plan adjustments to maintain quality while managing latency. Therefore, our bi-level optimization framework considers both system performance (e.g., deployment solver) and algorithmic behavior (e.g., routing solver), enabling efficient, adaptive optimization across different incoming LLM workloads. Additionally, our framework incorporates a re-scheduling mechanism to handle online fluctuating workloads, as detailed and tested in $4.4
The complete mathematical formulation for our bi-level optimization is provided in Appendix D)
4 EVALUATION
4.1 EXPERIMENTAL SETUP
Environments. Our experiments are conducted on 4 GPU servers, where each server equipped with 8 NVIDIA H100-80GB GPUs. Within each server, the GPUs are connected via. NVLink with bandwidth of 400GB/s, and the servers are connected via Inifiband with a bandwidth of 200GB/s.
Model cascade construction. We construct model cascade using the DeepSeek series models for CASCADIA, which are representative and popular open-source transformer models. Specifically, we (distilled version), and DeepSeck-671B AWQ with INT4 quantized weights (Lin et al. (2024) as three model types within our system. We employ a GPT-4o (L.LM-as-a-Judge (Zheng et al. 2023) the judger mentioned in $33 which assesses the output
DeepSeck-dist-755 DeepSeek-dist-

---

Published as a conference paper at ICLR 2026
Cascudia ava, quality
CascudeServe
DeepSeek-dist-
5L.O Amiimmen (N)
Trace ang. quality
90
85
loe qualit
Trace evg quality
2 jag quality
2| Ba quality
100
100
100
100
86
85
08
65
53
60
55
10
Trace 2 ang. quality
80
Trace aVg quality-70
Trece quiliy-90
Trace PPg quality-55
55
Trace 3 log qual 50
Trace 31 ong quality 70
Antiimment
100
100
100
86
71
72
16
2273
2.00
46
1120
44
2.0 3.4 4.8
6.1
75
0.5
1.6
288
5.0
3.0 6.2 9.5
128
160
3.0
5.2
988
12.0
0.5
1.4
2.2 51 4.0
Scalo
SLO Scale
Scale
Scale
Scale
Figure 5: End-to-end SLO attainment results evaluating CASCADIA against two baseline systems. Each row corresponds to particular LLM workload trace, and each column corresponds to a specific quality requirement. The stars indicate the 95% SLO attainment for each system.
responses of each model type within the cascade and assigns scores between 0 and 100. The judging overhead is included in our experiments.
Baselines. We compare CASCADIA with two baselines:
Compare with stand-alone LLMs served by SGLang. We compare CASCADIA against stand- alone LLMs that are directly served on SGLang (Zheng et al. (024) under various response quality constraints (e.g.. 90, 85, 80, 70) to demonstrate the effectiveness of LLM serving with model cascades. For quality requirement of 90 and 85, choose stand-alone DeepSeck-671B for comparison, and for quality reqirement of 80 and 70, we choose stand-alone DeepSeck-dist- for comparison. For fair comparison, we tune the parallelism strategy using our MILP algorithm mentioned in 43.2 for each of the stand-alone model and report the best values in all experiments.
Compare with cascade model serving system CascadeServe (Kossmann et al., 2024]. We compare CASCADIA against an existing cascade model serving system CascadeServe. chooses model cascade deployment plan based on system load (e.g., request arrival rate), enables model replication on hardware and adaptively dispatches incoming requests. We tune the parallelism and request routing strategies for CascadeServe based on the real-time system load and report the best values in all experiments.
Traces. We follow prior work generate workload traces based on real-world data (Jiang al.) 2024, Zhong et al. (024). Our testing traces are subsampled from MT-Menchl (Zheng et al. (023)) multi-turn conversation benchmark that contains multiple types LLM workloads (e.g.. coding, mathematics and reasoning). Each of our subsampled traces have different workload characteristics and different complexities as mentioned in $3.3
Evaluation metrics. Following previous evaluation setups (Li et al. 2023 Duan et al. 2024 Agrawal et al. 2024), we evaluate system performance based on SLO attainment and system throughput. The SLO is determined empirically based on the system's 's average single-request processing latency, and we scale it to various multiples (SLO Scale in Figuue 5) to assess performance under different levels of operational stringency. We focus identifying the minimum SLO Scale at which the system achieves 95% SLO attainment.
4.2 END-TO-END EXPERIMENTAL RESULTS
End-to-end system performance. We evaluate the SLO attainment and throughput of CASCADIA across multiple traces and quality requirements, comparing it with two baselines. Results Figure and Figure 6 show that CASCADIA outperforms all baselines:
CASCADIA achieves up to 4x and on average 2.8x x lower latency deadlines, and up to 5x and average 3x higher system throughput compared with stand-alone LLMs. For instance, when testing on trace with an average quality requirement of 85, stand-alone DeepSeek-66 IB requires
"The judger takes a Q&A pair as input and outputs quality grades (1-2 tokens), resulting in significantly lower latency and cost than full request inference (on average 0.27s for single judge). We benchmark the judge overhead in Appendix H We also demonstrate sensitivity experiments when replacing GPT-4o with weaker judgers (c.g., GPT-4o-mini and Llama3. 1--70B) in Appenddxx K
7

---

Published as a conference paper at ICLR 2026
Cascadia
CascadeServe
DoppSok-diit-7066677
Treec || qually
50
Traee log. uuguqulllyy 50
Laxe 3 Log cudiy 90
Trace lira, quality
65
Trace auguqualltyy 55
Tiaee 3 quall i5
0.56
1.13
Thouulou (rogkee) (po
0.15
0.28
0.42
0.38
1.2K
0.28
0.41
1.7-
0.56
0.84
145
0.79
0S5
L
0.08-
0.14
0.19
2,5M
0.14
0.28
0.26
A27kL
0.00
0.00
0.00
0.00
0.00
Tieec [| quglqullly
Logg cog_ccadityy
Taaco [ara, qualltyy
Tiace sxg, quality
3 quag_qualyy
1.23
1:59
3.14
4.76
5.28
1.20
3.57
0.00
0.80
1.4
2.64
0.30
0.40
0.78
0.7
1.19
1.35x
1.92
0.00
0.00
0.00
0.0
0.00
0.00
Figure 6: End-to-end throughput results evaluating CASCADIA against two baseline systems across different LLM workload traces and quality requirements.
Cascadia
CascadeServe
Llama-70B
At (9)
Trace jovg quallty-50
Trace 2 lagg qualit
50
Trace 3 layy qualtt
Trace aug quality 70
Trace aavg quality
avg. quality
100-
100
100
100
100-
1024
78-
74-
04-
70-
70
67-
48
18-
12.3
40
15
40H
10.0
1.0
2.5
6.4
1.0
28
8.0
62
SLO Scale
SLO Scale
SLO Scale
SLO Scale
SLO Scale
SLO Scale
Figure 7: End-to-end SLO attainment results evaluating CASCADIA against two baselines using a Llama cascade (Llama3-8B; Llama3-70B) across LLM workload traces and quality requirements.
11.88 SLO scales to achieve 95% attainment, while CASCADIA with different model types that uses smaller models to process simpler requests only requires 3.75 SLO scales.
CASCADIA achieves up to 2.5x and on average 1.7x lower latency deadlines, and up to 3.3x and on average 1.7x higher throughput than CascadeServe. While CascadeServe optimizes model deployment and routing based on real-time load, it overlooks LLM-specific workload characteristics (e.g., input/output lengths) and request complexity, leading to sub-optimal parallelism and routing. For example, trace with an average quality requirement of 90, CascadeServe needs 17.3 SLO scales to reach 95% SLO attainment, whereas CASCADIA requires only 11.73.
System performance with different model cascades and serving optimizations. We further evaluate CASCADIA using a different model cascade by replacing the DeepSeek series with the Llama series (Llama3-8B and Llama3-70B). As shown in FFigure 7, CASCADIA outperforms baselines by up to 3.8x and on average 2.6x, demonstrating strong performance across LLM cascades. We also compare CASCADIA with Sarathi-Serve (Agrawal et al. 2024), serving system with chunked prefill optimizations. CASCADIA achieves 1.95 higher performance (1 (1.64x average), validating our approach against advanced systems with scheduling optimizations. Detailed results are in in/ppendix
Compare with RouteLLM. We added additional experiments comparing CASCADIA with RouteLLM, LLM routing framework. CASCADIA achieves on average 21.3% lower SLO scale in achieving 95% SL.O attainment and 18.8% higher throughput compared to RouteLL.M. CASCADIA'S performance advantage stems from its system-algorithm co-design, as detailed Appendix I
Cost efficiency results. In addition to performance metrics, we conducted an analysis of cost efficiency comparing CASCADIA against baselines. Our results, detailed in [Appendix L demonstrate that CASCADIA significantly reduces operational expenditure. Specifically, CASCADIA achieves an average cost reduction of 20-39% compared to CascadeServe and a 33-61% reduction compared to stand-alona model serving, confirming its economic viability.
4.3 CASE STUDIES ON MODEL DEPLOYMENT PLANS AND ROUTING STRATEGIES
Case study on resource allocation and routing strategies. We benchmarked the thresholds, processing ratios and allocated resources for different model types across different testing cases. For instance, when testing on trace with an average quality requirement of 90, model types ej to c3 process 100%, 94% and 50% of the total requests, and the assigned GPU numbers are 4, 8 and 20. When the quality requirement changes to 85, less requests are required to be processed by the largest model cs (from 50% 21 1)), and less resources are allocated to c3 accordingly (from 20 to 16). This algorithm and system co-optimization enables CASCADIA to adjust system resource allocation and request routing based on user requirements, ensuring balanced load across different
8

---

Published as a conference paper at ICLR 2026
Cascadia
Uniferm Parallelism Strategy
Uniform Resounce Allocation
AsoouneeAllocttion
Trase I lova quality
90
T pug.ooiit 93
Treer 5 quralaulll 50
Traee I| qully 85
Taxe ooaaqulll 85
Traee 3 oog_culityy 85
Thooodipu (toqle))
0.30
0.55
0.77
0.55
1.13
1.06
0.15
0.26
0.38
177e
0.56
0.53
L4%
0.05
0.14
0.19
0.14
0.28
0.26
0.00
0.00
0.00
0.00
0.00
0.00
Figure 9: Ablation study resource allocation and parallelism strategy.
model types to boost system performance. Additionally, when testing on trace 3 with an average quality requirement of 70, CASCADIA deploys a subset of model types (DeepSeek-dist-
and -70B) to minimize the latencies required for requests processing. As shown in Figuuee 8] across different testing cases, CASCADIA always balances the loads among different model types to ensure optimized system performance. Table 2 in (Appeddix E demonstrates the thresholds, processing ratios and allocated resources for different model types across different testing cases.
Case study on parallelism strategies. We benehmarked the parallelism strategies for different model types across different testing cases. For example, when testing trace 1 with an average quality requirement of 90, the optimal parallelism strategy 82 for 02 is (DP=2, TP=4). In this case,
Niemaltaed Lanecy
1.2
1.0
0.8
04
02
0.0
we change the parallelism strategy to (DP=4, TP=2). the performance of this model type would drop by 33.7% 7.% Additionally, when the quality requirement drops to 85, the optimal parallelism strategy $2 for e2 shifts to (DP=6,
(90,
1)
(85.1) (50, 1)
(30.
()
(70.3)
Figure 8: Benehmarked p95 latency of cach model type within the cascade across differ- ent testing cases.
TP=2). This adjustment occurs because the change in quality requirements alters the LLM workloads, the request complexity routed and the resource allocated to C2- Consequently, 82 is updated to optimize the single model type's 's performance while balancing loads across all model types within the cascade. [Table in Appendix El presents the parallelism strategies for each model type within the cascade across different test cases.
Ablation study. We disable individual optimizations in CASCADIA to evaluate their impact, as shown in Figure 9] (i Replacing our parallelism strategy optimization with uniform parallelism strat- performance egy tensor parallelism within each server and data parallelism across servers- by up to 1.6x (1.4x average). For example, DeepSeek-7B and DeepSeek-671 1B requires higher degrees of data and tensor parallelism to maximize throughput and parameter sharding; a uniform approach fails accommodate these needs. (ii) Replacing our resource allocation optimization with uniform resource allocation reduces performance by up to 2.1x (1.7x on average). For instance, in trace with average quality requirement of 90, DeepSeek-671B B was originally allocated 20 GPUs, but uniform allocation assigns only 12, causing load imbalance.
4.4 EFFECTIVENESS OF THE SCHEDULING ALGORITIMM
Overall scheduling process. During scheduling, our Chebyshev-guide optimization ($3.3 explores different routing strategies to reduce response latency given a required quality. Simultaneously, our MILP-based optimization ($3.2) searches for resource allocations and parallelism strategies to balance load across model types and minimize latency. CASCADIA then selects the optimal plan-includiin thresholds, resource allocations, and parallelism strategies- based on quality requirements.
Scheduling algorithm runtime and scalability. Fig
Trace
Trace
Truce
ure 10 shows the runtime performance of CASCADIA'S scheduling algorithm. evaluated on a 12-core CPU in- stance. In our setup (32 GPUs), scheduling completes within 20s. For larger clusters (e.g., 80 GPUs), it fin- ishes within one minute, These results demonstrate the algorithm's 's efficiency and scalability across test cases and cluster sizes. Moreover, the algorithm is highly paral- lelizable, as resource allocations, parallelism, and routing strategies are independent- aaapeowwdnnlllooinnnggg execution time to scale down with more CPU cores. We added additional scheduling optimality analysis in Appendixx J
Nho Coet 69
39
16GPUs 52 GPUs
GPU
64GPUs
30GPUs
Figure 10: Algorithm running time when scaling from smaller clusters (e.g., 16 GPUs) to larger clusters (e.g., 80 GPUs).
Re-scheduling to adapt to online workload changes. As discussed in $3.3 LLM workload char- acteristics (e.g., distributions of input and output lengths, request rate and complexity) significantly

---

Published as a conference paper at ICLR 2026
affect the optimal model deployment plan and routing strategy. Thus, analogous to DistServe (Zhngg eet al. (024)) CASCADIA implement a re-scheduling mechanism to accommodate dynamic LLM workloads. Concretely, the system (i) subsampld" and record the real-time characteristies of the incoming LLM workloads (e.g.. subsample 50 requests every 5 minutes and record the workload characteristics). (ii) upon detecting significant shift in workload characteristics (e.g., increase in request arrival rate or request complexity), the scheduling algorithm is executed again, incorporating recent historical data produce an updated deployment plan and routing strategy.
We evaluated our system against baselines under online fluctuating workloads, where the work- load transitions trace 1 + trace 2 - trace 3 with segment lengths of 8, 16, and 10 minutes, eval- uated at different quality constraints. As shown Figure 11] CASCADIA consistently outper- forms baseline systems, achieving up to 4.4x workloads.
Cuocadia
CascadeServa
DecpSeck-dii- 7is:-10
0:42
0.065
1901
4.124
6400
1436
9,422
6030
0.166
0.000
Figure 11: Throughput evaluation under fluctuating
improvement with an average of 2.2x better performance. We further demonstrate the system lateney results of CASCADIA in comparison with CascadeServe and stand-alone model serving online fluctuating workloads (see (Appendixx G). Despite incurring additional scheduling overhead, CASCADIA maintains superior throughput and end-to-end efficiency under fluctuating workloads by dynamically optimizing cascade plans based on real-time L.M workload characteristics.
5
CONCLUSION
This paper proposes CASCADIA, a cascade serving system tailored for LLMs. Its core component is a scheduling algorithm that jointly optimizes resource allocation, parallelism, and routing within the cascade system. Extensive experiments on diverse workload traces and multiple model cascades show that this co-design substantially reduces request latency and boosts system throughput compared with both single-model and existing cascade baselines, while maintaining the target answer quality.
ACKNOWLEDGMENT
This work is supported by the HKUST startup grant R9895 from CSE; RGC-ECS project 26218024; RGC-NSFC project CRS HKUST601/24, We thank the support from National Supercomputer Center in Guangzhou Nansha Sub-Center.
REFERENCES
Pranjal Aggarwal, Aman Madaan, Ankit Anand, Srividya Pranavi Potharaju, Swaroop Mishra, Pei Zhou, Aditya Gupta, Dheeraj Rajagopal, Karthik Kappaganthu, Yiming Yang, et al. Automix: Automatically mixing language models. Advances Neural Information Processing Systems, 37: 131000-131034, 2024.
Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gulavani, Alexey Tumanov, and Ramachandran Ramjee. Taming 'Throughput-Latenu" tradeoff in (LLM} inference with {Sarathi-Serve Implementation (OSDI 24), pp. 117-134, 2024.
18th USENIX Symposium on Operating Systems Design and
Anthropic. The claude 3 model f'amily: Opus, sonnet, haiku, 2024. URL https: www cdn anthropic. Baa Card Claude 3.
8ba Baaebbabbbbllcc 9b01c htt.://oo....rraaaocceeeeeeeeee.
Lingjiao Chen, Matei Zaharia, and James Zou. Frugalgpt: How to use large language models while reducing cost and improving performance. Transactions on Machine Learning Research.
Jasper Dekoninck, Maximilian Baader, and Martin Vechev. A unified approach routing and cascading for llms. In Forty-second International Conference on Machine Learning, 2025.
'The query complexity is measured by subsampling 5% of incoming requests, routing them through all model types, and monitoring the quality score distributions from these sampled requests.
10

---

Published as a conference paper at ICLR 2026
Jiangfei Duan, Runyu Lu, Haojie Duanmu, Xiuhong Li, Xingcheng Zhang, Dahua Lin, lon Stoica, and Hao Zhang. Muxserve: flexible spatial- multiplexing for multiple Ilm serving. In Proceedings of the 41st International Conference Machine Learning, pp. 11905-11917, 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Ilama 3 herd of models. arXiv preprint arXiv:2407.21780 2024.
ETH-EASL. Scratchpad, 2025. httt:tttt//gtiitttt
GitHub. The world's most widely adopted developer tool, 2024. URL httpttttts:ggtbbbbb//.
Eeatures/copee copilot
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12945 2025.
Neha Gupta, Harikrishna Narasimhan, Wittawat Jitkrittum, Ankit Singh Rawat, Aditya Krishna Menon, and Sanjiv Kumar. Language model cascades: Token-level uncertainty and beyond. In The Twelfth International Conference on Learning Representations.
Yanping Huang. Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoe V Le, Yonghui Wu, et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. Advances in neural information processing systems, 32, 2019.
Jaeho Jeon and Seongyong Lee. Large language models in education: A focus on the complementary relationship between human teachers and chatgpt. Education and Information Technologies, 28 (12): 2023.
Youhe Jiang, Ran Yan, Xiaozhe Yao, Yang Zhou, Beidi Chen, and Binhang Yuan. Hexgen: generative inference of large language model over heterogeneous environment. In Proceedings of the 41st International Conference on Machine Learning, pp. 21946-21961, 2024.
Youhe Jiang, Fangcheng Fu, Xiaozhe Yao, Taiyi Wang, Bin Cui, Ana Klimovic, and Eiko Yoneki. Thunderserve: seag--eeffrmance and cost-efficient IIm serving in cloud environments. arXiv preprint arXiv:2502.094 2025a.
Youhe Jiang, Ran Yan, and Binhang Yuan. Hcxgen-2: Disaggregated generative inference of Ilms in heterogeneous environment. arXiv preprint arXiv:2502. 07903, 2025b.
Steven Kolawole, Don Dennis, Ameet Talwalkar, and Virginia Smith. Revisiting cascaded ensembles for efficient inference. In Workshop on Efficient Systems for Foundation Models II@ ICML2024.
Ferdi Kossmann, Ziniu Wu, Alex Turk, Nesime Tatbul, Lei Cao, and Samuel Madden. Cascadeserve: Unlocking model cascades for inference serving. arXiv preprint arXiv:2406.1442. 2024.
Luzian Lebovitz, Lukas Cavigelli, Michele Magno, and Lorenz K Muller. Efficient inference with model cascades. Transactions on Machine Learning Research, 2023.
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pp. 19274 19286. PMLR, 2023.
Zhuohan Li, Lianmin Zheng, Yinmin Zhong, Vincent Liu, Ying Sheng, Xin Jin, Yanping Huang, Zhifeng Chen, Hao Zhang, Joseph E Gonzalez, et al. (AlpaServe): Statistical multiplexing with model parallelism for deep learning serving. In 17th USENIX Symposium on Operating Systems Design and Implementation (OSDI 23), pp. 663-679, 2023.
Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang. Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for on-device Ilm compression and acceleration. Proceedings of Machine Learning and Systems, 6: 87-100, 2024.
Xiaoxuan Liu, Lanxiang Hu, Peter Bailis, Alvin Cheung, Zhijie Deng, lon Stoica, and Hao Zhang. Online speculative decoding. In Forty-first International Conference Machine Learning.

---

Published as a conference paper at ICLR 2026
Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang. Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang. Xiaoxiang Shi, et al. Specinfer: Accelerating large language model serving with tree-based speculative inference and verification. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume pp. 932-949, 2024a.
Xupeng Miao, Chunan Shi, Jiangfei Duan, Xiaoli Xi, Dahua Lin, Bin Cui, and Zhihao Jia. Spot- serve: Serving generative large language models on preemptible instances. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, pp. 1112-1127, 2024b
Harikrishna Narasimhan, Wittawat Jitkrittum, Ankit Singh Rawat, Seungyeon Kim, Neha Gupta, Aditya Krishna Menon, and Sanjiv Kumar. Faster cascades via speculative decoding. arXiv preprint arXiv:2405. 19261, 2024.
Harikrishna Narasimhan, Wittawat Jitkrittum, Ankit Singh Rawat, Seungyeon Kim, Neha Gupta, Aditya Krishna Menon, and Sanjiv Kumar. Faster cascades via speculative decoding. In The Thirteenth International Conference on Learning Representations, 2025.
OpenAL. Openai o3, 2025.
1s/o3
Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, fñigo Goiri, Saeed Maleki, and Ricardo Bianchini. Splitwise: Efficient generative Ilm inference using phase splitting. In 2024 ACM/IEEE 51st Annual International Symposium Computer Architecture (ISCA), pp. 118-132. IEEE, 2024.
Cheng Peng, Xi Yang, Aokun Chen, Kaleb E Smith, Nima PourNejatian, Anthony B Costa, Cheryl Martin, Mona G Flores, Ying Zhang, Tanja Magoc, et al. A study of generative large language model for medical research and healthcare. NPJ digital medicine, 6(1):210, 2023.
Haseena Rahmath P. Vishal Srivastava, Kuldeep Chaurasia, Roberto G Pacheco, and Rodrigo S Couto. Early-exit deep neural network-a comprehensive survey. ACM Computing Surveys, 57(3):1-37, 2024.
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac. Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.055300 2024.
Siddharth Samsi, Dan Zhao, Joseph McDonald, Baolin Li, Adam Michaleas, Michael Jones, William Bergeron, Jeremy Kepner, Devesh Tiwari, and Vijay Gadepally. From words to watts: Benchmark- ing the energy costs of large language model inference. In 2023 IEEE High Performance Extreme Computing Conference (IPEE), pp. 1-9, IEEE, 2023.
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catan- zaro. Megatron-Im: Training multi- multi-billion parameter language models using model parallelism. arXiv preprint 1XXv:1909008053 2019.
Ralph E Steuer and Eng-Ung Choo. An interactive weighted tchebycheff procedure for multiple objective programming. Mathematical programming, 26:326-344 2444, 1983.
Matthew Streeter. Approximation algorithms for cascading prediction models. In International conference on machine learning, pp. 4752-477666 PMLR, 2018
Biao Sun, Ziming Huang, Hanyu Zhao, Wencong Xiao, Xinyi Zhang, Yong Li, and Wei Lin. Llumnix: Dynamic scheduling for large language model serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pp. 173-191, 2024.
Surat Tecrapittayanon and Bradley McDanel. Branchynet: Fast inference via early exiting from deep neural networks. In 2016 23rd international conference on pattern recognition (ICPR), pp- IEEE, 2016.
Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. Wildchat: Im chatgpt interaction logs in the wild. In The Twelfth International Conference on Learning Representations.
12

---

Published as a conference paper at ICLR 2026
Yilong Zhao, Shuo Yang. Kan Zhu, Lianmin Zheng, Baris Kasikci, Yang Zhou, Jiarong Xing, and lon Stoica. Blendserve: Optimizing offline inference for ruto-regressive large models with resource-aware batching. arXiv preprint arXiv:2411.16102, 2024.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric Xing, et al. Lmsys-chat- Im: A large-scale real-world llm conversation dataset. In The Twelfth International Conference on Learning Representations.
Lianmin Zheng, Wei-Lin Chiang. Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Prucessing Systems, 36:46595-46622 2023.
Lianmin Zheng. Liangsheng Yin, Zhiqiang Xie, Chuyue Livia Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, al. Sglang: Efficient execution of structured language model programs. Advances in Neural Information Processing Systems, 37: 62557-62583, 2024.
Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. (DistServe): Disaggregating prefill and decoding for ciripuut-optmized large language model serving. 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pp. 193-210, 2024.
13

---

Published as a conference paper at ICLR 2026
Table I: Simulator accuracy across parallelism configurations Llama3-70B model under workload with average imput and output lengths of 1600 and 16. Errors are absolute percentage errors.
Config (DP,TP.PP)
Real (rea/s)
Estimated (rea/s)
Abs. % Error
(1.4,1)
0.21
0.219
4.29%
(I, 2)
0.26
0.280
7.69%
(I, 1, 4)
0.27
0.287
6.30%
(2, 2)
0.33
0.347
5.15%
2, 2, 1)
0.40
0.408
2.00%
(2, 4,
0.41
0.437
6.59%
(2, 2,
0,55
0.559
1.64%
A EXTENDED RELATED WORK
Parallelism strategies. LLMs with huge memory and computational resource requirements typically rely on parallelization across multiple GPUs @Li et al.) (023). There are three prevalent forms of parallelism: data parallelism (DP, i.e., model replication), tensor parallelism (TP) (Shoeybi et al., 2019), and pipeline parallelism (PP) (Huang et al. (019). DP replicates the model into multiple replicas, enabling parallel processing of requests. TP divides model weights and computationally intensive operations such as matrix multiplication across various GPUs, thereby splitting data scanning and computation to minimize LLM inference latency. PP divides the layers of a model into multiple stages. These stages assigned to distinct GPUs for execution and they establish pipeline. Only inter-layer activations are needed to be communicated between stages.
Speculative decoding and early-exit in LLM inference. Speculative decoding uses lightweight draft model to generate token blocks, which a larger target model porifies-leveran erogeneity to reduce computation and latency (Leviathan et al., 2023) Miao al.] [024a] Liu et al.). Similarly, early-exit networks add decision branches at intermediate layers, enabling inference to stop early when confidence is high cascading computation within a single model (Teerapittayanon
model het-
& McDanel 2016; Rahmath P et al. 2024). In contrast, we focus firmly on cascade model inference.
B SIMULATOR DESIGN AND VALIDATION
Our simulator employs round-robin strategy for request dispatching among multiple parallel models, and a first-come first-served strategy for per-model request processing. The single-GPU processing time is based on profiled characteristics like compute TFLOPS and memory bandwidth. The simulator also considers the phase-specific characteristics of LLMs. The prefill phase is compute-bound, so its batched processing capacity is determined by the sum of the individual latencies. In contrast, the decoding phase is memory-bound, and its batched processing capability is defined by a single latency value. This distinction has been validated in several studies (e.g.. DistServe (Zhong al. 2024). Splitwise (Patel et al. ))24)).
Inputs of the simulator. The simulator requires three fundamental inputs: (i) the distributions of input and output sequence lengths for each model type within the cascade; (ii) the request arrival rate corresponding to each model type within the cascade; and (iii) the resource allocation designated for each model type within the cascade.
Example. Consider workload distribution W that routes 100, 70, and 30 requests to model types 1, 2, and 3 respectively within the cascade, with corresponding GPU allocations of 2, 4, and 2 units. In this configuration, we record the distributions of input and output sequence lengths for each subset of requests (100, 70, and 30 respectively) input files to the simulator, configure the request arrival rates and resource allocations according to the specified parameters, and execute the simulation. Subsequently, the simulator undergoes iterative execution to identify the optimal parallelism strategy based on the provided input files, request arrival rates, and resource allocation constraints.
Batching strategy in simulator. The simulator's internal batching strategy is continuous batching, which iteratively batches request tokens to fully utilize the current resources. The GPU's memory limit constrains the maximum batch size for continuous batching.
14

---

Published as a conference paper at ICLR 2026
Queuing mechanism. Our simulator maintains individual queue for each model. Once there free memory on the GPU (one request has finished), the model will fetch the next request in the queue for processing.
Different parallelism. Tensor and pipeline parallelism both split the computation workload of a single model across multiple devices. For pipeline parallelism, the simulator models communication overhead by profiling the relationship between estimated communication volume and observed latency. For tensor parallelism, the simulator assumes that each operator's computation cost ideally scales down by a factor of I/N when split across N GPUs, and then adjusts this ideal cost using speed-up coefficient K (NN) obtained from berero-benehmarks to account for communication and synchronization overhead. All profiling is performed offline before scheduling begins.
Simulator evaluation. We present the accuracy of our simulator with real-time experiments in Table The table presents examples of our throughput estimation for the Llama3-70B model under a workload with average input and output lengths of 1600 and 16, respectively. The notation (1,2,2) indicates DP degree of I, TP degree of 2, and PP degree of 2. Although the estimations are not perfectly accurate, they are sufficiently reliable (with estimation errors within 2%-7%) for selecting optimal configurations.
C ROUTING SOLVER IN LATENCY-CONSTRAING CASE
The routing solver can also optimize system response quality under a user-specified latency budget by solving
arg min - Q(0) + max L(0) - Lmax
Plat, max LLat, min
where zjat, min and Sjat, max are the best (minimum) and worst (maximum) achievable latencies, Lmax is the allowable latency budget, and scales the penalty. The same nouting-deplyyment alternation, deployment solver, and convergence procedure are reused unchanged.
D COMPLETE BI-LEVEL OPTIMIZATION FORMULATION
Problem setup and notation. We consider a cascade with model types/stages indexed by {1...., C} and labeled C (c)....,cc). where c, denotes the i-th model type. The routing strategy is denoted by 0, parameterized by thresholds H = set of routing strategies. The GPU resource allocation is F = [fi..... fc]] where Si Z4 is the number of GPUs assigned to model type i, subject to a total budget N E Z.. The parallelism plan is S = (DP; TP:s ij., PP P...ii. where DP; denotes the number data-parallel replicas and, for each replica j. TPs; s) and PP:j denote its tensor- and pipeline-paralle degrees. Given routing and deployment (F, S), the estimated p95 latency is L(O, (., and the system quality is Q I) estimated by a judger using subsampled workload Ż. For Chebyshev-style normalization of quality, we use quality anchors 2 (utopia/best achievable quality, e.g.. all requests at cc) and 2 (nadir/worst credible quality, e.g., all requests cj). A user-specified quality requirement is Jmins and μ is penalty weight.
hc--), with θ the feasible
The routing is optimized by single scalar objective that penalizes quality shortfall, normalized by the utopia-nadir range, while the deployment is optimized under the GPU budget and parallelism feasibility:
Bi-level formulation.
1/00, F", St) + p max 0, Amin - QQ(0'; 7 }}
0 E arg min gee
2 -
DP:
(F", arg min min F,S) s.t. EASN,
Σ TP,, TP.y (i=l,...,CC) C),
F.5
j=1
f.. DP,, TP PP. Z+
Tractability and solution strategy. Because the problem couples routing, resource allocation, parallelism, heterogeneous L.M workloads, and user-specific quality requirements, monolithic
15

---

Published as a conference paper at ICLR 2026
solve is intractable. We therefore adopt bi-level strategy: The deployment problem is solved as MILP with latency values obtained from resource allocation and parallelism strategy optimization; the routing solver solves the Chebyshev-guided penalty problem. The two phases are executed iteratively, with the routing solver updating / and the deployment solver resolving (F*,S*) SS*) accordingly, and termination declared once the routing objective stabilizes under prescribed horizon.
Interpretation. The bi-level problem decomposes into routing and deployment subproblems that solved iteratively.
Deployment solver (deployment under resourceffeasibiliiy constraints). For a fixed routing 0' the deployment solver selects the latency-optimal deployment by choosing GPU allocations and parallelism plans subject to the budget and structural constraints:
C s.t. EASN,
DP. TPP = (,),...,
(F*) E arg arg F.S
1=1
fo DP,, TPG, PP,, E
This solver captures both hardware limits (GPU budget .) and parallelism feasibility.
Routing solver (routing, Stebyshev-guidee optimization). Given the current deployment (F*,S*) the routing solver updates the routing strategy (i.e., 0) by minimizing a single scalar objective that balances latency and normalized quality shortfall:
1(0', F*, St) + P max O, Qmin Q(''; 2) }1
0 E arg min P'E0
2 22
Here, (=j -=f-- provides Chebyshev (utopia-nadir) normalization for scale stability, and sets the severity of penalizing Q(0") < 9min- For sufficiently large μ (when the target is feasible), any minimizer is quality-compliy and the routing objective effectively reduces minimizing latency among feasible routings.
Coupling and procedure. The routing solver's determines the workload distribution seen by each model type within the cascade (and hence the optimal deployment plan for the deployment solver), while the deployment solver's (F*,S*) determines the latency used by the routing objective (and hence the optimal routing strategy for the routing solver). Alternating updates continue until the routing objective stabilizes under a prescribed termination horizon (e.g., best-so-far objective unchanged for K consecutive iterations).
E CASE STUDIES ON MODEL DEPLOYMENT PLANS AND ROUTING STRATEGIES
Case study on resource allocation and routing strategies. Table demonstrates the case study of thresholds, processing ratios and allocated resources for different model types across different testing cases.
Table Case study the thresholds (h1, h2), processing ratios (P1. P2, Ps), and allocated resources (f1, fa. fa) for each model type within the cascade across different testing cases. (90, 1) denotes testing on Trace with an average quality requirement of 90.
A
P1
Pa
P1
(90. 1)
99
91
100%
94%
50%
20
(85. 1)
74
64
100%
62%
21%
12
16
(80. 1)
69
25
100%
54%
11%
(80,
61
100%
31%
(80, 3)
0
100%
23%
0%
(70, 3)
10
0
100%
5%
0%
24
Case study on parallelism strategies, Table presents a case study on parallelism strategies for each model type within the cascade across different test cases.
16

---

Published as a conference paper at ICLR 2026
Table 3: Case study of the parallelism strategies for each model type within the cascade (81, 82, 83) across different testing cases.
Parallelism Strategies
(90, 1)
(DP=4), $2 (DP=2, TP=4), sy! (TP=4, PP=3), (TP=8)
(85. 1)
#1: (DP=2, TP=2). #2: (DP=6, TP=2), #g: (DP=2, TP=8)
(80. 1)
(DP=6), #yt (DP=5, TP=2), (TP=4), sy: (TP=4, PP=3)
(80,
$1: (DP=6), (TP=2), sg: (DP=8. TP=2), 83: (TP=8)
(80.
syz 10)=10), (DP=4. 1P=2). sz (DP 4). (DP=3. TP= ey!-
(70,
81: (DP=16), (DP=4, TP=2), eg (DP=4, TP=2), eg:
Table
End-to-end throughput results evaluating CASCADIA against Sarathi-errve
Trace Trace 0,2529 req/s
Ours
Sarathi-Serve
Speedup
% Improvement
0.1913 req/s
1.322
+32.20%
Trace 0,4659 req/s
0.2385 reg/s
1.953
+95.35%
Trace
0.6406 req/s
0.3977 reg/s
1.611
+61.08%
F COMPARISON WITH SARATHI-SERVE
We evaluated Sarathi-Serve under the same experimental setup as SGLang, as described $4.1 using traces 1-3 with an average quality requirement of 90. We used Sarathi-Serve's vL.LM implementation (its most efficient variant) and tuned the chunk size to be optimal for each case. As shown in Table 4 system achieves up to 1.95x higher throughput and averages 1.64x speedup across traces.
G LATENCY RESULTS ON FLUCTUATING WORKLOADS
Quantification of re-scheduling overheads. The re-scheduling overhead consists two components: (i) Algorithm runtime (~10-20s, shown in Figure 10), and (ii) model reconfiguration overhead 2~2)..
Re-scheduling impact on online serving. During rescheduling, requests continue to be processed using the current deployment configuration, so there is no service interruption.
Reconfiguration impact on online serving. Deployment plans typically have overlapping configurations between transitions (i.e., some model replicas retain the same deployment configuration), so these unchanged replicas continue processing requests during re- configuration. To further reduce the service interruption time, for replicas that do require reconfiguration, we perform rolling ricantts-rcconiguuiin them one at a time while others continue serving requests.
Re-scheduling impact on baseline methods. Note that CascadeServe also incurs similar reconfigu- ration overhead, while single-model baselines exhibit consistently poor performance due to lack of cascade optimization.
We further demonstrate the latency results of CASCADIA compared to CascadeServe and single-model deployment in our fluctuating workload experiments Figure I1} with average quality requirement of 90. CASCADIA achieves 34% and 45% reduction in SLO scale for achieving 95% SLO attainment compared to CascadeServe and single-model deployment.
Table 5: Benchmarked SLO Scale for 95% SLO Attainment (Avg. Quality 2 90).
Deployment Strategy
SLO Scale
Reduction vs. CASCADIA
CASCADIA
8.99
CascadeServe
13.55
34%
Single-Model Deployment
16.37
45%
17

---

Published as a conference paper at ICLR 2026
H BENCHMARK GPT-40 OVERHEAD
We conducted additional experiments on H100 GPUs benchmark single-request GPT-4o judging
vs. processing latency MT-Bench using Llama cascades (Llama3-8B Llama3-70B). The results demonstrate that the average single-request processing latency for the small (approximately 3.05s) and large model (approximately 7.35s) is approximately 5.21s. In contrast, the single-request GPT-4o judging latency is only approximately 0.27s, as judging is prefill-bound with minimal output (1 token). This overhead is negligible compared to overall inference cost, and is already included in all experimental results ($4) reported in our paper.
1 COMPARE WITH ROUTELLM
We conducted additional experiments comparing CASCADIA against RouteLLM with BERT-based router on Llama cascades (Llama3-8B Llama3-70B) following the setup in Section 4.1 with average quality requirement of 80 on Traces and 2. For fair comparison, we tune the deployment for each model for RouteLLM. Results show that CASCADIA achieves on average 21.3% lower SLO scale in achieving 95% SLO attainment (4.6, 2.8 vs. 5.8, 3.6) and 18.8% higher throughput (2.2, 3.5 vs. 1.9, 2.9) compared to RouteLLM.
CASCADIA's performance advantage stems from its system-algorithm co-design ($3): While RouteLLM focuses solely on routing optimization and fails to consider how system-side optimiza- tion (e.g., resource allocation, parallelism) impacts routing decisions and latency-quality trade-offs, CASCADIA jointly optimizes both aspects for better end-to-end performance.
J SCHEDULING OPTIMALITY
Due to the NP-hardness of the problem and the mutual dependencies between deployment and routing, providing theoretical optimality guarantees is intractable. However, empirically validate approach against exhaustive search, which enumerates all feasible resource allocations, parallelism strategies, and routing thresholds, serving additional experiments comparing our bi-level optimization against exhaustive search on Llama cascades (Llama3-8B - Llama3-70B) following the setup Section 4.1. To make exhaustive search computationally feasible, we applied the same deployment constraints from Section 3.2. Results show that our approach achieves near-optimal performance with only 2-6% gap compared exhaustive search, while reducing search time from 25 minutes to 20 seconds 2 15x speedup. Notably, exhaustive search time grows exponentially with cluster size, making our bi-level approach (grows linearly) essential for practical deployment at scale.
empirical optimum. Specifically, we conducted
While theoretical optimality is intractable, our method provides strong empirical performance with practical efficiency, making it suitable for real-world deployment scenarios where search overhead matters.
K SENSITIVITY EXPERIMENTS WITH WEAKER JUDGES
We conducted additional experiments to evaluate robustness by replacing GPT-4o with weaker judges 4GPT-40-mini and (lama3.1-70B) follwiinn the experimental setup Figure 7 (Llama cascade, Trace I, quality requirement qmin 80).
18

---

Published as a conference paper at ICLR 2026
K.1 EXPERIMENTAL RESULTS ANALYSIS
GPT-4o-mini and Llama3. assign scores that are on average 9.4% and 8.6% lower than GPT-4o for the same responses, exhibiting higher variance in quality assessment. This scoring bias causes the system to route 11.1% % and 9.4% more requests to the larger model compared to using GPT-4o. Nevertheless, Cascadia adaptively adjusts the deployment, allocating more resources to the larger model. As result, meeting the same Jmin requires only 6.8% and 5.5% increase in system latency, respectively. Importantly, the system continues to satisfy the quality requirement and avoids collapse into over-routing, demonstrating that Cascadia is robust to weaker or noisier judges.
Table 6: Sensitivity to judge quality:
Performance change compared to GPT-4o baseline.
Judge
Avg Score Deviation from GPT-4o
Routing to Larger Model
System Lateney Increase to meet Guain
Increase
GPT-4o (haseline)
0%
0%
0%
GPT-4o-mini
9.4%
+11.1%
+ +6.8%
Llama3. 1-70B
8.6%
+9..%%
+ +5.5%
K.2 JUDGE-AGNOSTU FRAMEWORK
Our framework is judge-agnostic: any model capable of pairwise comparison or quality scoring can be used as the judge, including open-source models (c.g., Llama-based judges).
L COST EFFICIENCY RESULTS
We provide a cost-efficiency analysis comparing CASCADIA against baselines. Following the experimental setup in Figure 6 with an average quality requirement of 90 (favg 90), we compute the cost per request based on GPU pricing (NVIDIA H100: $2.67/hour). Results demonstrate that CASCADIA achieves 20-39% cost reduction compared to CascadeServe and 33-61 % reduction compared to stand-alone serving.
Table Cost per request (USD/req) comparison (Avg.
Quality qaez
90, H100 GPU pricing).
Deployment Strategy
Trace
Trace 2
Trace 3
Stand-Alon Serving
0.15 S/req
0.26 S/req
0.31 S/req
CascadeServe
0.14 S/req
0.18 S/req
0.15 S/req
CASCADIA (Ours)
0.10 $/req
0.11 S/req
0.12 S/req
M DISCUSSION OF INTEGRATING PREFIX CACHING
Additional experiments with enabling prefix caching. In our experiments on MTT-Bench (Trace avg__ualiie quallttty== with the Llama cascade (Llama3-8B - Llama3-70B), enabling prefix caching changed the SLO scale required to achieve 95% SLO attainment from 4.6 to 4.5 (~2% system latency decrease, within measurement noise) and did not affect the relative gaps between CASCADIA and the baselines or the resulting scheduling decisions. Similarly, CascadeServe's 's SLO scale stayed the same, and stand-alone model serving changed from 8.2 to 8.1.
This minor impact is reasonable due to MT-Bench's 's workload characteristies. MT-Bench is decoding- heavy, so even perfect prefix reuse would have limited impact on overall latency dominated by the decoding phase.
Prefix caching impact on scheduling decision. In serving scenarios where many different requests share long, identical prefix, enabling prefix caching reduces absolute p95 latencies for all systems.
19

---

Published as a conference paper at ICLR 2026
However, the relative performance gains of CASCADIA over the baselines and the resulting optimal scheduling decisions remain largely unchanged, since prefix caching benefits all model replicas uniformly across the cluster.
How to incorporate prefix caching in our scheduling algorithm. To incorporate prefix caching into cost estimation, can model it as prefill reduction:
effective_prefilL_tteee _tokens ~ (1 hit riter x Bictinal_peefillttoeeee
where hit__aatt Late is obtained from profiling representative subsample of the input workloads. We note that prefix caching an orthogonal optimization technique-the bi-level scheduling methodology remains applicable and would operate on cache-adjusted latency profiles.
N FINE-TUNED BERT FOR CASCADING
We conducted two experiments evaluate BERT-based judging:
1. Realistic fine-tuning scenario. We fine-tuned BERT model using 70% of our experimental traces as training data and evaluated on the remaining 30% (unseen test set). The BERT judger trained request inputs, outputs, and quality grades across diffferntt models. We compared CASCADIA (GPT-40) against CASCADIA (BERT) on Llama cascades (Llama3-8B + -llama3-70B following the setup in Section 4.1 with quality requirement of 80. Results show that CASCADIA with the BERT judger exhibits large variance in quality assessment, leading ~8% degradation in system quality (80 74) compared to CASCADIA (GPT-4o). This demonstrates that less accurate judger fails to satisfy the quality requirement.
2. Oracle fine-tuning scenario. To isolate judging overhead from judging accuracy, we trained BERT 100% of our experimental traces (including the test set), creating oracle judger that perfectly replicates GPT-4o's judgments with minimal overhead. Even this idcalized scenario, CASCADIA (BERT-oracle) achieves only <5% better system latency than CASCADIA (GPT-4o) due to faster judging time, demonstrating that judging overhead is already negligible.
Why LLM-as-a-Judge over BERT-based routers. We choose LLM-as-a-Judge for two key reasons: (1) Overhead: As shown above, judging adds only ~0.27s overhead, which is negligible compared to inference savings from routing simple requests smaller models. (2) Generalization: BERT-based routers suffer from generalization problems when encountering diverse or out-of mut-of-distribution queries, whereas LLM-ass-a-Jdge (Zheng al. 2023) can evaluate response quality more robustly across varied workloads and domains.
O THE USE OF LLMS IN WRITING
We used LLM, namely OPENAI-GPTS, to polish the writing of this manuscript. No other generative AI functionality is used in the writing of this submission.
20


Reviews:
[{"review_id": "KuzJ26MULf", "reviewer": "Reviewer_HDzY", "summary": "The paper presents CASCADIA, an efficient cascade serving system for large language models that jointly optimizes resource allocation and request routing across hierarchies of model sizes, enabling fast, high-quality, and cost-effective LLM inference. By formulating cascade serving as a bi-level optimization problem and dynamically adapting deployment strategies, CASCADIA achieves significantly lower latency and higher throughput compared to single-model and existing cascade baselines, while maintaining answer quality across diverse workloads.", "questions": "1. Why do you try to minimize the maximum latency across models, L, in the MILP, when the latency of a query will be the sum of the latencies of the models that it passes through?\n\n2. Why do you consider separate thresholds for each model when determining the routing strategy even though the same input is passed through the models until a satisfactory output is obtained (if the input is the same then the threshold on the output score should also be the same for all models)?\n\n3. $L(\\theta)$ in line 263 appears to be non-differentiable. If that is indeed the case, please clarify how $\\theta$ is updated to converge to the minima of the optimization problem.\n\n4. Please provide a citation for the baseline CascadeServe in Section 4.\n\n5. How is query complexity measured when looking for a shift in workload characteristics in Section 4.4? \n\n6. How does CascadeServe handle distribution shifts? Does it not make any changes under distribution shift?\n\n7. Can you quantify the scheduling overhead (line 476) in terms of additional latency (and not just throughput) when re-scheduling under online workloads? For e.g. if there is a spike in the P95 latency during the rescheduling window then that would not be a good thing.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The joint formulation of resource allocation and adaptive inference via model cascades is a novel approach to cost-efficient LLM inference and is a very important problem that needs to be solved before cascades can be deployed in real world systems. The paper also considers heterogeneity in model and workload characteristics which are important considerations in real settings.\n\n2. The paper proposes a viable solution to the problem via bi-level optimization that helps to find an appropriate deployment and routing strategy and shows clear improvements over baselines in experiments", "weaknesses": "1. Some of the details of the approach are not clearly explained. I have added several questions below around points that were not clear to me.\n\n2. While the bi-level optimization itself doesn't seem to be taking too long to solve in online settings (Section 4.4), the latency of re-allocating the models/changing the parallelization may be high.\n\n3. The approach does not consider prefix caching even though the traces used in the experiments do contain multi-turn conversations and prefix caching in such settings and may affect the latencies of both the baselines and Cascadia. Even if it is difficult to incorporate prefix caching in the optimization formulation, I believe when running the traces prefix caching should be enabled to see if it causes the results to deviate significantly from what is expected after solving the optimization problem.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "32VpwIWmgt", "reviewer": "Reviewer_11UR", "summary": "This paper introduces CASCADIA, a novel cascade serving framework designed explicitly to schedule request routing and deploy model cascades for fast, quality-preserving LLM serving. CASCADIA employs a bi-level optimization method: at the deployment level, it uses a mixedinteger linear program to select resource allocations and parallelism strategies based on LLM information and workload characteristics; at the routing level, it applies a Chebyshev-guided method to iteratively co-optimize the routing strategy and the system deployment produced by the deployment level. \n\nIt uses mixed-integer linear programming (MILP) to determine the optimal deployment plan given a routing strategy. It balances response latency and output quality. Within each cascade stage, CASCADIA supports various parallelism strategies (e.g., tensor and pipeline parallelism), which allows it to automatically select the optimal strategy based on model size, incoming workload, and routing decisions.", "questions": "1. Maybe instead of using GPT-4, fine tune a BERT for cascading?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Stengths:\n\n1. Efficient serving multiple LLM to balance accuracy and latency is an important topic.\n\n2. The proposed cascading method intuitively can help the multi-model serving system.\n\n3. Extensive experiments show the performance.", "weaknesses": "1. The main concern is on the real-time efficiency and cost. LLM serving is an online process. If using GPT-4 to judge the small model response, runs GPT-4 takes a few seconds and the cost is expensive. \n\n2. Time to first token is also very long. For simple prompt, it also needs to wait until GPT-4 finishes the judge.\n\n3. The baselines are insufficient. BERT-based router [1, 2, 3] that directly routes prompt to multiple LLMs can be compared.\n\n\n[1] https://github.com/vllm-project/semantic-router\n\n[2] Tensoropera router: A multi-model router for efficient llm inference\n\n[3] RouteLLM: Learning to Route LLMs with Preference Data", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pr2PCralpM", "reviewer": "Reviewer_jBdC", "summary": "This paper proposes Cascadia, a novel cascade serving framework for efficient & effective LLM serving. Cascadia employs a bi-level optimization strategy to compute the optimal serving strategy which includes the deployment strategy (i.e., the resource allocation and parallelism strategies) as well as the routing strategy (i.e., which user requests should be consumed by which models). Extensive experiments on real-world datasets demonstrate the effectiveness of the proposed approach.", "questions": "What is the typical/expected number of iterations required to achieve stable solutions? If the number of iteration tends to be huge, it can lead to non-negligible overheads and compromise the efficiency gains.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper studies efficient and effective LLM serving, which is critical problem for a wide range of real-world applications.\n2. This paper introduces Cascadia with rigorous and solid technical developments.\n3. Cascadia achieves up to 4x lower latency deadlines and 5x higher system throughputs, which are impressive.", "weaknesses": "1. In Algorithm 1, Cascadia relies on iteratively optimizing both deployment and routing strategies. It remains unclear if the bi-level optimization is theoretically optimal or not. Such guarantees could be critical in practical scenarios.\n2. LLM routing is another well-studied technique aiming for efficient & effective LLM serving, which is under-discussed in this paper. Authors may want to discuss and compare to this line of work to better position the contribution of this paper. Several example references are as follows,\n\n[1] Ong, Isaac, et al. \"RouteLLM: Learning to Route LLMs from Preference Data.\" The Thirteenth International Conference on Learning Representations.  \n[2] Ding, Dujian, et al. \"BEST-Route: Adaptive LLM Routing with Test-Time Optimal Compute.\" Forty-second International Conference on Machine Learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "t170YLBN02", "reviewer": "Reviewer_m6H1", "summary": "The paper introduces CASCADIA, a cascade-serving system for LLMs that jointly optimizes (i) GPU/resource deployment of multiple models (including DP/TP/PP choices) and (ii) threshold-based routing across the cascade to meet a target latency–quality trade-off. It uses a bi-level optimization loop: a MILP picks per-model allocations/parallelism given a routing pattern, and a Chebyshev-guided routing solver adjusts thresholds to satisfy a user quality floor. On DeepSeek and Llama cascades, CASCADIA outperforms single-model serving and CascadeServe, reporting up to 4× tighter SLOs and 5× higher throughput, with ablations showing that both resource allocation and parallelism search are necessary.", "questions": "1. The routing quality depends on GPT-4o as the judge. If we swap in a weaker/open-source judge (e.g. Llama-3-70B with an arena prompt), does CASCADIA still satisfy the same 𝑞 min, or does it start over-routing to bigger models? A small sensitivity table would help.\n2. Rescheduling policy. How exactly do you detect a “significant” workload shift before re-running the scheduler, and do you use any hysteresis to avoid oscillating between plans?\n3. Can the deployment MILP be warm-started from the previous solution to keep solve time low when only the routing thresholds change slightly?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Real problem, 2026-relevant: serving fleets of heterogeneous LLMs under SLO and quality constraints is exactly what people are doing.\n- The bi-level loop (MILP for deployment + Chebyshev for routing) is a clean way to expose the coupling between “how many requests go to 70B” and “how many GPUs the 70B should get.”\n- Strong empirical numbers: up to 4× lower latency SLOs and up to 5× higher throughput than single-model; 1.7–2.5× over CascadeServe; wins also hold for Llama cascades.\n- Removing parallelism search or using uniform GPU allocation degrades performance notably, so the system is not just “a nicer scheduler,” it’s actually using its degrees of freedom.\n- Online re-scheduling: handling trace1 → trace2 → trace3 shifts and still beating baselines makes the system more believable for production.", "weaknesses": "- Routing quality is based on GPT-4o-as-a-judge. That’s fine for the paper but less fine for on-prem/air-gapped setups; we don’t see how robust the method is to weaker judges.\n- Incremental vs existing cascade work: CascadeServe, AutoMix, and even 2025 routing+speculative papers are moving in the same direction.\n- No cost/energy accounting, they motivate with resource efficiency and even cite energy papers, but do not report $/req or some energy unit /req, that’s what operators would care about.\n- If the quality distributions used in the routing solver are mis-estimated (domain shift, very hard math/coding workloads), the Chebyshev objective could over-route to large models and wipe out the latency gains.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)