Cascadia: An Efficient Cascade Serving System for Large Language Models

## Abstract

Abstract Recent advances in large language models (LLMs) have intensified the need to deliver both rapid responses and high-quality outputs. More powerful models yield better results but incur higher inference latency, whereas smaller models are faster yet less capable. Recent work proposes balancing this latency–quality trade-off using model cascades, which route simpler queries to smaller models and more complex ones to larger models. However, enabling efficient cascade serving remains challenging. Current frameworks lack effective mechanisms for handling (i) the huge and varying resource demands of different LLMs, (ii) the inherent heterogeneity of LLM workloads, and (iii) the co-optimization of system deployment and routing strategy.
Motivated by these observations, we introduce Cascadia , a novel cascade serving framework designed explicitly to schedule request routing and deploy model cascades for fast, quality-preserving LLM serving. Cascadia employs a bi-level optimization method: at the deployment level, it uses a mixed-integer linear program to select resource allocations and parallelism strategies based on LLM information and workload characteristics; at the routing level, it applies a Chebyshev-guided method to iteratively co-optimize the routing strategy and the system deployment produced by the deployment level. Our extensive evaluation on diverse workload traces and different model cascades (DeepSeek and the Llama series) demonstrates that Cascadia significantly outperforms both single-model deployments and the state-of-the-art cascade serving baseline, achieving up to 4 × \times (2.3 × \times on average) tighter latency SLOs and up to 5 × \times (2.4 × \times on average) higher throughput while maintaining target answer quality.

## 1 Introduction

Figure: Figure 1: Average response quality and latencies of different DeepSeek models. Quality is judged by GPT-4o using the LLM-as-a-Judge framework [1].
Refer to caption: x1.png

Large language models (LLMs) such as DeepSeek-R1 [2], OpenAI o3 [3], Claude [4], Gemini [5] and Llama-3 [6] have demonstrated outstanding performance across a wide range of real-world applications (e.g., chatbots, healthcare and education) [7, 8, 9], largely influence human lives. However, serving LLMs can be costly [10, 11, 12], since significant computational resources (e.g., GPUs) are required to meet certain service demands, such as meeting certain latency deadlines (i.e., SLO attainment—the proportion of requests served within a specified response-time target) and generation throughput. In this paper, we explore an alternative solution that strategically utilizes model cascades to better balance the response latency and quality trade-offs inherent in LLM serving.

Cascade model serving refers to a serving architecture where multiple models of varying sizes and capabilities are arranged in a sequential pipeline, creating a hierarchy of models that process requests with increasing levels of sophistication [13, 14, 15, 16, 17, 18]. As shown in [Figure 1](https://arxiv.org/html/2506.04203v2#S1.F1), larger models typically provide higher response quality but also incur greater latency, which in turn leads to increased energy consumption and compute usage [19].
In this approach, incoming requests are initially handled by smaller, computationally efficient models that can rapidly process simpler requests. Only when these lightweight models determine that a request exceeds their capabilities or requires higher-quality responses does the system escalate the request to larger, more powerful models in the cascade. This progressive delegation mechanism enables service providers to optimize system performance by matching request complexity with appropriate model capacity, thereby significantly reducing computational costs while maintaining high-quality responses for complex request. Several recent studies have focused on optimizing LLM serving using model cascades [14, 13, 15, 20, 21].

The cascade model serving architecture, which adaptively routes simpler and more complex requests to smaller and larger models, respectively, presents significant opportunities for optimizing the cost-efficiency of LLM serving. In this work, we focus specifically on the setting where service providers host and manage every model in the cascade themselves. However, effectively adapting this paradigm to LLM scenarios is much harder to implement than to propose, as we enumerate below:

- •
Model heterogeneity. LLMs require large amounts of compute and memory, and different models have varying resource demands for efficient serving [22]. With a fixed resource pool, suboptimal allocation across models in the cascade can degrade overall serving efficiency.
- •
Workload heterogeneity. LLM workloads exhibit considerable heterogeneity [23, 24, 25]. Models within the cascade often face incoming requests with varying characteristics (e.g., input/output lengths, arrival rates) and favor different deployment strategies (e.g., replication, parallel configuration), further adding complexity to optimal system deployment.
- •
Cascade-aware load balancing. The request routing strategy directly impacts the system load of each model in the cascade. For instance, if more requests are routed to a particular model, its load increases; the resource allocation and deployment strategy for that model should then be adjusted to balance loads across all models. Consequently, the deployment of multiple models must be co‑optimized with the routing strategy to manage load across the cascade.

In order to overcome these challenges, we propose Cascadia, a novel cascade serving system that is optimized for LLM characteristics and that co‐optimizes the deployment of multiple models in the cascade together with the request routing strategy.
Our contributions are as follows:

- •
Contribution 1. We formulate cascade serving—covering system deployment and request routing—as a constrained optimization problem. To solve it, we propose a bi-level approach that jointly optimizes deployment and routing. The *deployment* level uses mixed-integer linear programming (MILP) to determine the optimal deployment plan given a routing strategy, while the *routing* level applies a Chebyshev-guided method to optimize routing, balancing latency and quality.
- •
Contribution 2. We implement Cascadia, an efficient cascade serving system tailored to LLMs. Cascadia enables an adaptive model cascade paradigm that allocates resources and routes requests across a hierarchy of model sizes (e.g., small, medium, and large), thereby balancing response latency and output quality. Within each cascade stage, Cascadia supports various parallelism strategies (e.g., tensor and pipeline parallelism), which allows it to automatically select the optimal strategy based on model size, incoming workload, and routing decisions.
- •
Contribution 3. We empirically evaluate Cascadia by comparing it to both single-model and existing cascade serving systems across a variety of scenarios, including diverse workload traces (e.g., coding and mathematics), different model cascades (DeepSeek and the Llama series), and multiple evaluation metrics (SLO attainment and throughput). The results show that, compared with state-of-the-art non-cascade and cascade solutions, Cascadia achieves up to 4$\times$ lower latency deadlines (2.3$\times$ on average) and boosts system throughput by up to 5$\times$ (2.4$\times$ on average).

## 2 Preliminary and Related Work

LLM inference phases and workload heterogeneity. There are two phases within LLM inference: prefill and decoding. During the prefill phase, the model processes the input prompt to compute the key-value (KV) cache and generates the first token in a single step. In contrast, the decoding phase uses the last generated token and the KV cache as inputs to generate subsequent tokens in a token-by-token manner. Generally, the prefill phase is compute-bound, while the decoding phase is memory-bound [26, 27, 28, 29, 30].
LLM inference workloads exhibit heterogeneity in input, output token lengths and request arrival rate, which is called workload heterogeneity. For instance, conversation workloads (short input and long output lengths) typically require more memory resources to handle the memory-bound decoding phase, while coding workloads (long input and short output lengths) demand more compute resources to manage the compute-bound prefill phase. Therefore, appropriately allocating resources based on workload demands is critical for optimal performance [31, 32].

Cascade model inference.
Current LLMs come in various sizes and configurations, offering a broad spectrum of choices. Effectively leveraging this diversity can balance trade-offs between response latency and quality during inference.
Recent efforts propose cascade model inference to utilize models of differing complexities [33, 34]. In such architectures, an input prompt is processed through increasingly complex models, using threshold-based routing that stops computation once a cheaper model produces a confident enough answer. For instance, FrugalGPT [14] employs a dynamic LLM cascade strategy that routes queries through progressively stronger models (e.g., GPT-3.5 → GPT-4) based on real-time difficulty estimation, optimizing cost-efficiency without sacrificing accuracy. Similarly, AutoMix [13] uses intelligent layer-wise token routing to dynamically allocate computation based on input difficulty. CascadeServe [15] automates and optimizes end-to-end inference with cascades, adjusting model deployment and request routing based on real-time system loads. However, existing systems overlook key LLM-specific workload characteristics and neglect the importance of co-optimizing system deployment with request routing (i.e., system-algorithm co-design).

Figure: Figure 2: Benchmarked performance of different parallelism strategies across different workloads and model sizes. Long and short outputs represent two different workloads with average output sequence length to be 512 and 1024; the three-element array represents the DP, TP, and PP degrees.
Refer to caption: x2.png

Limitations of existing cascade serving systems. We summarize the limitations of existing cascade serving systems: (i) Ineffective resource allocation for different model types within a cascade. Different model types have distinct memory and computation resource needs. For example, DeepSeek-671B typically requires more allocated resources than DeepSeek-dist-70B due to its larger memory and computational demands. Current systems ignore the importance of adjusting resource allocation according to the needs of different model types,
leading to unbalanced system loads.
(ii) Inadequate adaptation of parallelism strategies to varying workloads and model sizes. The optimal parallelism strategies vary across different workloads (e.g., different input and output request sequence lengths and request arrival rates) and model sizes. As shown in [Figure 2](https://arxiv.org/html/2506.04203v2#S2.F2), choosing the optimal parallelism strategy can achieve up to 3$\times$ higher system throughput. Current systems do not optimize parallelism strategies according to specific workload and model size, resulting in degraded overall system performance.
(iii) Insufficient co-optimization between system deployment and routing strategy. The routing strategy decides the request portion processed by each model type within a cascade, which in turn determines the system loads for different model types. Existing systems neglect to adapt system deployment configurations based on routing outcomes, resulting in suboptimal resource usage. To address these challenges, a cascade serving system tailored for LLMs is necessary. Such a system must optimize end-to-end performance and ensure stringent SLO adherence.

## 3 Scheduling Algorithm in Cascadia

### 3.1 Problem Formulation

To optimize the cascade serving system under different LLM workloads and user-specific requirements (e.g., system response quality requirements), the scheduling algorithm should determine two essential components: (i) The model deployment plan, which specifies the resource allocations and parallelism strategies for multiple model types (e.g., small, medium, large) within the cascade to minimize the system response latency (e.g., p95 latency—the response time threshold below which 95% of all requests complete); and (ii) the routing strategy, which balances the trade-off between system response latency and quality to decide the appropriate model path for each incoming request. We term a solution addressing these two components as a cascade plan.

Figure: Algorithm 1 Bi-level Scheduling Workflow

Note that the routing strategy determines the request distribution over different model types, which in turn dictates the optimal model deployment plan, while the model deployment plan defines the system response latency that feeds back into the routing decision. Given the interdependent and exponentially large search space, determining the optimal cascade plan is an NP-hard problem. To solve this problem, we adopt a bi-level optimization method that enables system–algorithm co-design, which is shown in Algorithm 1, and can be summarized as:

- •
MILP-based deployment solver: Given the routing strategy, the deployment solver (§[3.2](https://arxiv.org/html/2506.04203v2#S3.SS2)) employs an mixed-integer linear programming (MILP) formulation to capture system resource constraints and compute the optimal deployment plan that minimizes system response latency.
- •
Chebyshev-guided routing solver: Based on the system response latency generated from the deployment solver and the user-specific quality requirement, the routing solver (§[3.3](https://arxiv.org/html/2506.04203v2#S3.SS3)) applies a Chebyshev-guided method to find the optimal routing strategy that optimizes system response latency with respect to the quality requirement.

### 3.2 MILP-Based Deployment Solver

As shown in Algorithm 1, the routing strategy (obtained from routing solver) determines how many requests should be routed to each model in the cascade, thus determining the workload distribution among models. Given the workload distribution and resource limit, the deployment solver aims to determine the optimal deployment plan, which includes the resource allocation and parallelism strategies for models within cascades. An example deployment plan is shown in [Figure 3](https://arxiv.org/html/2506.04203v2#S3.F3).

Assume a total of $N$ GPUs serve a model cascade with $C$ model types, $\{c_{1},c_{2},\dots,c_{C}\}$, where $c_{i}$ denotes the $i$-th model type. The incoming workload information is denoted as $\mathcal{W}=\{w_{1},w_{2},\dots,w_{C}\}$, where each $w_{i}$ includes the distributions of input/output sequence lengths and the request arrival rate for the $i$-th model type. We use $\mathcal{F}=\{f_{1},f_{2},\dots,f_{C}\}$ to denote the number of GPUs allocated per model, the total allocation must not exceed the resource limit, i.e., $\sum_{i=1}^{C}f_{i}\leq N$. Given this setup, our deployment solver (i) determines the parallelism strategy for each specific resource allocation $f_{i}$, and (ii) uses an MILP to optimize the overall resource allocation $\mathcal{F}$.

Parallelism strategy search. Given the workload information $w_{i}$ and a specific resource allocation $f_{i}$, this optimization determines the optimal parallelism strategy and computes the corresponding system response latency $l_{i}$ for the model type $i$. Cascadia provides three forms of parallelism: data parallelism (i.e., model replication, DP) [35], tensor model parallelism (TP) [36], and pipeline parallelism (PP) [37]. Denoting the degrees of data, tensor, and pipeline parallelism for the model type by $\mathrm{dp}$, $\mathrm{tp}$, and $\mathrm{pp}$, any feasible parallelism strategy must satisfy the following resource constraint: $(\sum_{j=1}^{\mathrm{dp_{i}}}\mathrm{tp_{i,j}}\times\mathrm{pp_{i,j}})\leq f_{i}$, i.e., one model type can be replicate into multiple replicas, each replica can have varied tensor and pipeline parallelism degrees, as shown in [Figure 3](https://arxiv.org/html/2506.04203v2#S3.F3), the summation of different parallelism degrees should be less or equal than the total number of GPUs assigned. Based on the workload information $w_{i}$ and the resource allocation $f_{i}$, we iterate over all feasible parallelism combinations to select the strategy that minimizes the response latency $l_{i}$ for the model type $i$. The latency $l_{i}$ is computed using the simulator $\mathbf{Sim}(\cdot)$ as $l_{i}=\mathbf{Sim}(w_{i},f_{i})$ (^1^11We use the ETH EASL Scratchpad simulator [38] to estimate system p95 latency from workload and resource allocation. We show detailed simulator design (e.g., simulator inputs, batching strategy, queuing mechanism, parallelism strategy modeling) and evaluation in [Appendix B](https://arxiv.org/html/2506.04203v2#A2).). Note that the parallelism strategy optimization can be precomputed for all possible resource allocations $f$ to provide latency lookup tables for the MILP formulation.

Figure: Figure 3: Illustration of a model deployment plan.
Refer to caption: x3.png

MILP formulation for resource allocation optimization.
Our MILP problem formulation aims to minimize the maximum system response latency among all model types in the cascade. Let $L$ denote the maximum latency across all model types. We discretize the GPU allocations into candidate values $f\in\{1,2,\dots,N\}$. For each model type $i$ and candidate allocation $f$, we use the precomputed latency table from the parallelism strategy optimization to obtain $l_{i}(f)$. We then introduce binary assignment variables $x_{i,f}$, where $x_{i,f}=1$ if model type $i$ is assigned $f$ GPUs and $x_{i,f}=0$ otherwise, for all $i\in\{1,\dots,C\}$ and feasible $f$. The constraints of our MILP include:
(i) For each model type $i$, exactly one GPU allocation $f$ must be selected, i.e., $\sum_{f=1}^{N}x_{i,f}=1,\forall\,i=1,\dots,C$;
(ii) the total number of GPUs assigned across all model types should be equal to the available GPUs $N$, i.e., $\sum_{i=1}^{C}\sum_{f=1}^{N}f\,x_{i,f}=N$;
and (iii) the maximum latency $L$ must be at least as large as the latency $l_{i}(f)$ corresponding to each selected allocation, i.e., $L\geq\sum_{f=1}^{N}l_{i}(f)\,x_{i,f},\forall\,i=1,\dots,C$.
We explicitly enforce variable domains and integrality constraints as follows: $x_{i,f}\in\{0,1\},\forall\,i,f$ and $L\geq 0$. If certain GPU allocations $f$ are infeasible for specific model types—such as when the total memory of the allocated $f$ GPUs is less than the minimum memory required by the model type—we explicitly set $x_{i,f}=0$ for these allocation pairs. Our objective is to minimize the maximum system response latency $L$, which serves as the input for the routing layer optimization.

### 3.3 Chebyshev-guided Routing Solver

Figure: Figure 4: Threshold-based cascade routing workflow. The router determines whether a request is accepted or forwarded to the next model type based on predefined thresholds.
Refer to caption: x4.png

As shown in Algorithm 1, the deployment plan (obtained from the deployment solver) determines the system response latency. Given the system response latency and quality requirement, the routing solver aims to optimize the routing strategy (i.e., co-optimize system latency and quality).

Thresholds tuning and request routing. We adopt the threshold-based cascade routing workflow consistent with prior works [13, 14] ([Figure 4](https://arxiv.org/html/2506.04203v2#S3.F4)).
Initially, every incoming request is sent to the first (smallest) model type $c_{1}$ in the cascade. A judger then evaluates the quality of the output responses from model types $c_{1}$ to $c_{C-1}$, and a set of thresholds $\mathcal{H}=\{h_{1},h_{2},\dots,h_{C-1}\}$ is defined to decide whether the requests at each model type should be accepted or forwarded to the next model type. In this framework, the routing strategy $\theta$ is directly determined by the thresholds $\mathcal{H}$, i.e., $\theta=\theta(\mathcal{H})$. Each routing strategy $\theta$ is associated with a system response latency $L(\theta)$ (determined by the deployment solver optimization) and quality $Q(\theta)$ (determined by the judger (^2^22Analogous to [14], we estimate $Q(\theta)$ by profiling a subsample of the input workload across all cascade models to obtain per-model quality score distributions. During scheduling, given any threshold vector $\mathcal{H}$ and the quality score distributions, we can determine which model’s response would be accepted for each request under routing policy $\theta(\mathcal{H})$, then aggregate these final model scores to compute the overall system quality $Q(\theta)$.)). Our routing solver uses a Chebyshev-guided method to optimize the routing strategy. We initialize the routing strategy $\theta_{0}$ as proportional routing, where the $i$-th model receives $1/i$ of requests.

Chebyshev-guided optimization for routing strategy. Given the routing strategy $\theta$
and user-specified quality requirement $q_{\min}$, we employ the Chebyshev-guided method [39] to minimize the system response latency $L(\theta)$ with respect to $q_{\min}$.
First, we define a utopia point $z_{1}^{*}$ (all requests processed by the largest model $c_{C}$) and nadir point $z_{2}^{*}$ (all requests processed by the smallest model $c_{1}$) representing the best and worst achievable system response quality.
Then, for a given quality requirement $q_{\min}$, we minimize the system response latency subject to meeting the quality requirement by solving the single-objective penalty problem:

$$ $\arg\min_{\theta}J(\theta)=\arg\min_{\theta}\left[L(\theta)+\mu\max\{0,\;(q_{\min}-Q(\theta))/(z_{1}^{*}-z_{2}^{*})\}\right]$ $$

where $J(\theta)$ represents the latency-quality score, $\mu>0$ is a penalty weight that enforces the quality constraint (for sufficiently large $\mu$, any minimizer of $J(\theta)$ satisfies $Q(\theta)\geq q_{\min}$), and $z_{1}^{*}$ and $z_{2}^{*}$ are used to normalize the quality shortfall so the penalty is dimensionless and well-conditioned across workloads. Note that our routing solver can also optimize system response quality under a user-specified latency requirement using a similar procedure, as detailed in [Appendix C](https://arxiv.org/html/2506.04203v2#A3).

Putting them together. In our bi-level optimization framework, the routing solver (i.e., Chebyshev-guided optimization) iteratively searches for the next $\theta$, invokes deployment solver (i.e., MILP optimization) to obtain the minimized system response latency $L(\theta)$, and then minimizes the objective function (i.e., $\arg\min_{\theta}J(\theta)$). Finally, an optimal routing strategy $\theta$ is selected that guarantees a minimal system response latency while fulfilling the quality requirement.

Impact of LLM workloads on optimal cascade plan selection.
The characteristics of incoming LLM workloads strongly influence the selection of cascade plans. This influence stems from two key factors:
(i) Request input/output length and arrival rate affect system response latency—longer sequences or higher loads increase compute demand, necessitating plan adjustments to balance latency and quality;
(ii) Request complexity impacts system response quality—complex requests or difficult queries require larger models, necessitating plan adjustments to maintain quality while managing latency.
Therefore, our bi-level optimization framework considers both system performance (e.g., deployment solver) and algorithmic behavior (e.g., routing solver), enabling efficient, adaptive optimization across different incoming LLM workloads. Additionally, our framework incorporates a re-scheduling mechanism to handle online fluctuating workloads, as detailed and tested in §[4.4](https://arxiv.org/html/2506.04203v2#S4.SS4).

The complete mathematical formulation for our bi-level optimization is provided in [Appendix D](https://arxiv.org/html/2506.04203v2#A4).

## 4 Evaluation

### 4.1 Experimental Setup

Environments. Our experiments are conducted on 4 GPU servers, where each server is equipped with 8 NVIDIA H100-80GB GPUs. Within each server, the GPUs are connected via NVLink with a bandwidth of 400GB/s, and the servers are connected via Inifiband with a bandwidth of 200GB/s.

Model cascade construction. We construct a model cascade using the DeepSeek series models for Cascadia, which are representative and popular open-source transformer models. Specifically, we use DeepSeek-dist-7B, DeepSeek-dist-70B (distilled version), and DeepSeek-671B AWQ with INT4 quantized weights [40] as three model types within our system. We employ a GPT-4o (LLM-as-a-Judge) [1] as the judger mentioned in §[3.3](https://arxiv.org/html/2506.04203v2#S3.SS3), which assesses the output responses of each model type within the cascade and assigns scores between 0 and 100. The judging overhead (^3^33The judger takes a Q&A pair as input and outputs quality grades (1–2 tokens), resulting in significantly lower latency and cost than full request inference, typically requiring only 1–2s.) is included in our experiments.

Baselines. We compare Cascadia with two baselines:

- •
Compare with stand-alone LLMs served by SGLang. We compare Cascadia against stand-alone LLMs that are directly served on SGLang [41] under various response quality constraints (e.g., 90, 85, 80, 70) to demonstrate the effectiveness of LLM serving with model cascades. For quality requirement of 90 and 85, we choose stand-alone DeepSeek-671B for comparison, and for quality reqirement of 80 and 70, we choose stand-alone DeepSeek-dist-70B for comparison. For fair comparison, we tune the parallelism strategy using our MILP algorithm mentioned in §[3.2](https://arxiv.org/html/2506.04203v2#S3.SS2) for each of the stand-alone model and report the best values in all experiments.
- •
Compare with cascade model serving system CascadeServe. We compare Cascadia against an existing cascade model serving system CascadeServe. It chooses model cascade deployment plan based on system load (e.g., request arrival rate), enables model replication on hardware and adaptively dispatches incoming requests. We tune the parallelism and request routing strategies for CascadeServe based on the real-time system load and report the best values in all experiments.

Traces. We follow prior work to generate workload traces based on real-world data [10, 27]. Our testing traces are subsampled from MT-Bench [1], a multi-turn conversation benchmark that contains multiple types of LLM workloads (e.g., coding, mathematics and reasoning). Each of our subsampled traces have different workload characteristics and different complexities as mentioned in §[3.3](https://arxiv.org/html/2506.04203v2#S3.SS3).

Evaluation metrics. Following previous evaluation setups [35, 22, 28], we evaluate system performance
based on SLO attainment and system throughput. The SLO is determined empirically based on the system’s average single-request processing latency, and we scale it to various multiples (SLO Scale in [Figure 5](https://arxiv.org/html/2506.04203v2#S4.F5)) to assess
performance under different levels of operational stringency. We focus on identifying the minimum SLO Scale at which the system achieves 95% SLO attainment.

Figure: Figure 5: End-to-end SLO attainment results evaluating Cascadia against two baseline systems. Each row corresponds to a particular LLM workload trace, and each column corresponds to a specific quality requirement. The stars indicate the 95% SLO attainment for each system.
Refer to caption: x5.png

### 4.2 End-to-end Experimental Results

End-to-end system performance. We evaluate the SLO attainment and throughput of Cascadia across multiple traces and quality requirements, comparing it with two baselines. Results in [Figure 5](https://arxiv.org/html/2506.04203v2#S4.F5) and [Figure 6](https://arxiv.org/html/2506.04203v2#S4.F6) show that Cascadia outperforms all baselines:

- •
Cascadia achieves up to 4$\times$ and on average 2.8$\times$ lower latency deadlines, and up to 5$\times$ and on average 3$\times$ higher system throughput compared with stand-alone LLMs. For instance, when testing on trace 3 with an average quality requirement of 85, stand-alone DeepSeek-671B requires 11.88 SLO scales to achieve 95% attainment, while Cascadia with different model types that uses smaller models to process simpler requests only requires 3.75 SLO scales.
- •
Cascadia achieves up to 2.5$\times$ and on average 1.7$\times$ lower latency deadlines, and up to 3.3$\times$ and on average 1.7$\times$ higher throughput than CascadeServe. While CascadeServe optimizes model deployment and routing based on real-time load, it overlooks LLM-specific workload characteristics (e.g., input/output lengths) and request complexity, leading to sub-optimal parallelism and routing. For example, on trace 1 with an average quality requirement of 90, CascadeServe needs 17.3 SLO scales to reach 95% SLO attainment, whereas Cascadia requires only 11.73.

Figure: Figure 6: End-to-end throughput results evaluating Cascadia against two baseline systems across different LLM workload traces and quality requirements.
Refer to caption: x6.png

System performance with different model cascades and serving optimizations. We further evaluate Cascadia using a different model cascade by replacing the DeepSeek series with the Llama series (Llama3-8B and Llama3-70B). As shown in [Figure 7](https://arxiv.org/html/2506.04203v2#S4.F7), Cascadia outperforms baselines by up to 3.8$\times$ and on average 2.6$\times$, demonstrating strong performance across LLM cascades. We also compare Cascadia with Sarathi-Serve [28], a serving system with chunked prefill optimizations. Cascadia achieves 1.95$\times$ higher performance (1.64$\times$ average), validating our approach against advanced systems with scheduling optimizations. Detailed results are in [Appendix F](https://arxiv.org/html/2506.04203v2#A6).

Figure: Figure 7: End-to-end SLO attainment results evaluating Cascadia against two baselines using a Llama cascade (Llama3-8B; Llama3-70B) across LLM workload traces and quality requirements.
Refer to caption: x7.png

### 4.3 Case studies on Model Deployment Plans and Routing Strategies

Case study on resource allocation and routing strategies. We benchmarked the thresholds, processing ratios and allocated resources for different model types across different testing cases. For instance, when testing on trace 1 with an average quality requirement of 90, model types $c_{1}$ to $c_{3}$ process 100%, 94% and 50% of the total requests, and the assigned GPU numbers are 4, 8 and 20. When the quality requirement changes to 85, less requests are required to be processed by the largest model $c_{3}$ (from 50% to 21%), and less resources are allocated to $c_{3}$ accordingly (from 20 to 16). This algorithm and system co-optimization enables Cascadia to adjust system resource allocation and request routing based on user requirements, ensuring balanced load across different model types to boost system performance. Additionally, when testing on trace 3 with an average quality requirement of 70, Cascadia deploys a subset of model types (DeepSeek-dist-7B and -70B) to minimize the latencies required for requests processing. As shown in [Figure 8](https://arxiv.org/html/2506.04203v2#S4.F8), across different testing cases, Cascadia always balances the loads among different model types to ensure optimized system performance. [Table 2](https://arxiv.org/html/2506.04203v2#A5.T2) in [Appendix E](https://arxiv.org/html/2506.04203v2#A5) demonstrates the thresholds, processing ratios and allocated resources for different model types across different testing cases.

Figure: Figure 8: Benchmarked p95 latency of each model type within the cascade across different testing cases.
Refer to caption: x8.png

Case study on parallelism strategies. We benchmarked the parallelism strategies for different model types across different testing cases. For example, when testing on trace 1 with an average quality requirement of 90, the optimal parallelism strategy $s_{2}$ for $c_{2}$ is (DP=2, TP=4). In this case, if we change the parallelism strategy to (DP=4, TP=2), the performance of this model type would drop by 33.7%.
Additionally, when the quality requirement drops to 85, the optimal parallelism strategy $s_{2}$ for $c_{2}$ shifts to (DP=6, TP=2). This adjustment occurs because the change in quality requirements alters the LLM workloads, the request complexity routed and the resource allocated to $c_{2}$. Consequently, $s_{2}$ is updated to optimize the single model type’s performance while balancing loads across all model types within the cascade. [Table 3](https://arxiv.org/html/2506.04203v2#A5.T3) in [Appendix E](https://arxiv.org/html/2506.04203v2#A5) presents the parallelism strategies for each model type within the cascade across different test cases.

Figure: Figure 9: Ablation study on resource allocation and parallelism strategy.
Refer to caption: x9.png

Ablation study. We disable individual optimizations in Cascadia to evaluate their impact, as shown in [Figure 9](https://arxiv.org/html/2506.04203v2#S4.F9): (i) Replacing our parallelism strategy optimization with a uniform parallelism strategy—tensor parallelism within each server and data parallelism across servers—reduces performance by up to 1.6$\times$ (1.4$\times$ on average). For example, DeepSeek‐7B and DeepSeek‐671B requires higher degrees of data and tensor parallelism to maximize throughput and parameter sharding; a uniform approach fails to accommodate these needs. (ii) Replacing our resource allocation optimization with uniform resource allocation reduces performance by up to 2.1$\times$ (1.7$\times$ on average). For instance, in trace 1 with an average quality requirement of 90, DeepSeek‐671B was originally allocated 20 GPUs, but uniform allocation assigns only 12, causing load imbalance.

### 4.4 Effectiveness of the Scheduling Algorithm

Overall scheduling process.
During scheduling, our Chebyshev-guided optimization (§[3.3](https://arxiv.org/html/2506.04203v2#S3.SS3)) explores different routing strategies to reduce response latency given a required quality. Simultaneously, our MILP-based optimization (§[3.2](https://arxiv.org/html/2506.04203v2#S3.SS2)) searches for resource allocations and parallelism strategies to balance load across model types and minimize latency. Cascadia then selects the optimal plan—including thresholds, resource allocations, and parallelism strategies—based on quality requirements.

Figure: Figure 10: Algorithm running time when scaling from smaller clusters (e.g., 16 GPUs) to larger clusters (e.g., 80 GPUs).
Refer to caption: x10.png

Scheduling algorithm runtime and scalability. [Figure 10](https://arxiv.org/html/2506.04203v2#S4.F10) shows the runtime performance of Cascadia’s scheduling algorithm, evaluated on a 12-core CPU instance. In our setup (32 GPUs), scheduling completes within 20s. For larger clusters (e.g., 80 GPUs), it finishes within one minute. These results demonstrate the algorithm’s efficiency and scalability across test cases and cluster sizes. Moreover, the algorithm is highly parallelizable, as resource allocations, parallelism, and routing strategies are independent—allowing execution time to scale down with more CPU cores.

Figure: Figure 11: Throughput evaluation under fluctuating workloads.
Refer to caption: x11.png

Re-scheduling to adapt to online workload changes.
As discussed in §[3.3](https://arxiv.org/html/2506.04203v2#S3.SS3), LLM workload characteristics (e.g., distributions of input and output lengths, request rate and complexity) significantly affect the optimal model deployment plan and routing strategy. Thus, analogous to DistServe [27], Cascadia implement a re-scheduling mechanism to accommodate dynamic LLM workloads. Concretely, the system (i) subsample and record the real-time characteristics of the incoming LLM workloads (e.g., subsample 50 requests every 5 minutes and record the workload characteristics), (ii) upon detecting a significant shift in workload characteristics (e.g., an increase in request arrival rate or request complexity), the scheduling algorithm is executed again, incorporating recent historical data to produce an updated deployment plan and routing strategy.
We evaluated our system against baselines under online fluctuating workloads, where the workload transitions trace 1 → trace 2 → trace 3 with segment lengths of 8, 16, and 10 minutes, evaluated at different quality constraints. As shown in [Figure 11](https://arxiv.org/html/2506.04203v2#S4.F11), Cascadia consistently outperforms baseline systems, achieving up to 4.4$\times$ improvement with an average of 2.2$\times$ better performance. Despite incurring additional scheduling overhead, Cascadia maintains superior throughput and end-to-end efficiency under fluctuating workloads by dynamically optimizing cascade plans based on real-time LLM workload characteristics.

## 5 Conclusion

This paper proposes Cascadia, a cascade serving system tailored for LLMs. Its core component is a scheduling algorithm that jointly optimizes resource allocation, parallelism, and routing within the cascade system. Extensive experiments on diverse workload traces and multiple model cascades show that this co‑design substantially reduces request latency and boosts system throughput compared with both single‑model and existing cascade baselines, while maintaining the target answer quality.

## References

- [1]
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al.
Judging llm-as-a-judge with mt-bench and chatbot arena.
Advances in Neural Information Processing Systems, 36:46595–46623, 2023.
- [2]
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al.
Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.
arXiv preprint arXiv:2501.12948, 2025.
- [3]
OpenAI.
Openai o3, 2025.
- [4]
Anthropic.
The claude 3 model family: Opus, sonnet, haiku, 2024.
- [5]
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al.
Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.
arXiv preprint arXiv:2403.05530, 2024.
- [6]
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al.
The llama 3 herd of models.
arXiv preprint arXiv:2407.21783, 2024.
- [7]
Jaeho Jeon and Seongyong Lee.
Large language models in education: A focus on the complementary relationship between human teachers and chatgpt.
Education and Information Technologies, 28(12):15873–15892, 2023.
- [8]
Cheng Peng, Xi Yang, Aokun Chen, Kaleb E Smith, Nima PourNejatian, Anthony B Costa, Cheryl Martin, Mona G Flores, Ying Zhang, Tanja Magoc, et al.
A study of generative large language model for medical research and healthcare.
NPJ digital medicine, 6(1):210, 2023.
- [9]
GitHub.
The world’s most widely adopted ai developer tool, 2024.
- [10]
Youhe Jiang, Ran Yan, Xiaozhe Yao, Yang Zhou, Beidi Chen, and Binhang Yuan.
Hexgen: generative inference of large language model over heterogeneous environment.
In Proceedings of the 41st International Conference on Machine Learning, pages 21946–21961, 2024.
- [11]
Youhe Jiang, Ran Yan, and Binhang Yuan.
Hexgen-2: Disaggregated generative inference of llms in heterogeneous environment.
arXiv preprint arXiv:2502.07903, 2025.
- [12]
Xupeng Miao, Chunan Shi, Jiangfei Duan, Xiaoli Xi, Dahua Lin, Bin Cui, and Zhihao Jia.
Spotserve: Serving generative large language models on preemptible instances.
In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, pages 1112–1127, 2024.
- [13]
Pranjal Aggarwal, Aman Madaan, Ankit Anand, Srividya Pranavi Potharaju, Swaroop Mishra, Pei Zhou, Aditya Gupta, Dheeraj Rajagopal, Karthik Kappaganthu, Yiming Yang, et al.
Automix: Automatically mixing language models.
Advances in Neural Information Processing Systems, 37:131000–131034, 2024.
- [14]
Lingjiao Chen, Matei Zaharia, and James Zou.
Frugalgpt: How to use large language models while reducing cost and improving performance.
Transactions on Machine Learning Research.
- [15]
Ferdi Kossmann, Ziniu Wu, Alex Turk, Nesime Tatbul, Lei Cao, and Samuel Madden.
Cascadeserve: Unlocking model cascades for inference serving.
arXiv preprint arXiv:2406.14424, 2024.
- [16]
Steven Kolawole, Don Dennis, Ameet Talwalkar, and Virginia Smith.
Revisiting cascaded ensembles for efficient inference.
In Workshop on Efficient Systems for Foundation Models II@ ICML2024.
- [17]
Luzian Lebovitz, Lukas Cavigelli, Michele Magno, and Lorenz K Muller.
Efficient inference with model cascades.
Transactions on Machine Learning Research, 2023.
- [18]
Matthew Streeter.
Approximation algorithms for cascading prediction models.
In International conference on machine learning, pages 4752–4760. PMLR, 2018.
- [19]
Siddharth Samsi, Dan Zhao, Joseph McDonald, Baolin Li, Adam Michaleas, Michael Jones, William Bergeron, Jeremy Kepner, Devesh Tiwari, and Vijay Gadepally.
From words to watts: Benchmarking the energy costs of large language model inference.
In 2023 IEEE High Performance Extreme Computing Conference (HPEC), pages 1–9. IEEE, 2023.
- [20]
Neha Gupta, Harikrishna Narasimhan, Wittawat Jitkrittum, Ankit Singh Rawat, Aditya Krishna Menon, and Sanjiv Kumar.
Language model cascades: Token-level uncertainty and beyond.
In The Twelfth International Conference on Learning Representations.
- [21]
Harikrishna Narasimhan, Wittawat Jitkrittum, Ankit Singh Rawat, Seungyeon Kim, Neha Gupta, Aditya Krishna Menon, and Sanjiv Kumar.
Faster cascades via speculative decoding.
arXiv preprint arXiv:2405.19261, 2024.
- [22]
Jiangfei Duan, Runyu Lu, Haojie Duanmu, Xiuhong Li, Xingcheng Zhang, Dahua Lin, Ion Stoica, and Hao Zhang.
Muxserve: flexible spatial-temporal multiplexing for multiple llm serving.
In Proceedings of the 41st International Conference on Machine Learning, pages 11905–11917, 2024.
- [23]
Biao Sun, Ziming Huang, Hanyu Zhao, Wencong Xiao, Xinyi Zhang, Yong Li, and Wei Lin.
Llumnix: Dynamic scheduling for large language model serving.
In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pages 173–191, 2024.
- [24]
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric Xing, et al.
Lmsys-chat-1m: A large-scale real-world llm conversation dataset.
In The Twelfth International Conference on Learning Representations.
- [25]
Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng.
Wildchat: 1m chatgpt interaction logs in the wild.
In The Twelfth International Conference on Learning Representations.
- [26]
Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, and Ricardo Bianchini.
Splitwise: Efficient generative llm inference using phase splitting.
In 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA), pages 118–132. IEEE, 2024.
- [27]
Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang.
$\{$DistServe$\}$: Disaggregating prefill and decoding for goodput-optimized large language model serving.
In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pages 193–210, 2024.
- [28]
Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gulavani, Alexey Tumanov, and Ramachandran Ramjee.
Taming $\{$Throughput-Latency$\}$ tradeoff in $\{$LLM$\}$ inference with $\{$Sarathi-Serve$\}$.
In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pages 117–134, 2024.
- [29]
Youhe Jiang, Fangcheng Fu, Xiaozhe Yao, Guoliang He, Xupeng Miao, Ana Klimovic, Bin Cui, Binhang Yuan, and Eiko Yoneki.
Demystifying cost-efficiency in llm serving over heterogeneous gpus.
arXiv preprint arXiv:2502.00722, 2025.
- [30]
Li Zhang, Youhe Jiang, Guoliang He, Xin Chen, Han Lv, Qian Yao, Fangcheng Fu, and Kai Chen.
Efficient mixed-precision large language model inference with turbomind.
arXiv preprint arXiv:2508.15601, 2025.
- [31]
Yilong Zhao, Shuo Yang, Kan Zhu, Lianmin Zheng, Baris Kasikci, Yang Zhou, Jiarong Xing, and Ion Stoica.
Blendserve: Optimizing offline inference for auto-regressive large models with resource-aware batching.
arXiv preprint arXiv:2411.16102, 2024.
- [32]
Youhe Jiang, Fangcheng Fu, Xiaozhe Yao, Taiyi Wang, Bin Cui, Ana Klimovic, and Eiko Yoneki.
Thunderserve: High-performance and cost-efficient llm serving in cloud environments.
arXiv preprint arXiv:2502.09334, 2025.
- [33]
Jasper Dekoninck, Maximilian Baader, and Martin Vechev.
A unified approach to routing and cascading for llms.
In Forty-second International Conference on Machine Learning, 2025.
- [34]
Harikrishna Narasimhan, Wittawat Jitkrittum, Ankit Singh Rawat, Seungyeon Kim, Neha Gupta, Aditya Krishna Menon, and Sanjiv Kumar.
Faster cascades via speculative decoding.
In The Thirteenth International Conference on Learning Representations, 2025.
- [35]
Zhuohan Li, Lianmin Zheng, Yinmin Zhong, Vincent Liu, Ying Sheng, Xin Jin, Yanping Huang, Zhifeng Chen, Hao Zhang, Joseph E Gonzalez, et al.
$\{$AlpaServe$\}$: Statistical multiplexing with model parallelism for deep learning serving.
In 17th USENIX Symposium on Operating Systems Design and Implementation (OSDI 23), pages 663–679, 2023.
- [36]
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro.
Megatron-lm: Training multi-billion parameter language models using model parallelism.
arXiv preprint arXiv:1909.08053, 2019.
- [37]
Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al.
Gpipe: Efficient training of giant neural networks using pipeline parallelism.
Advances in neural information processing systems, 32, 2019.
- [38]
ETH-EASL.
Scratchpad, 2025.
- [39]
Ralph E Steuer and Eng-Ung Choo.
An interactive weighted tchebycheff procedure for multiple objective programming.
Mathematical programming, 26:326–344, 1983.
- [40]
Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han.
Awq: Activation-aware weight quantization for on-device llm compression and acceleration.
Proceedings of Machine Learning and Systems, 6:87–100, 2024.
- [41]
Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Livia Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, et al.
Sglang: Efficient execution of structured language model programs.
Advances in Neural Information Processing Systems, 37:62557–62583, 2024.
- [42]
You Peng, Youhe Jiang, Chen Wang, and Binhang Yuan.
Hexgen-text2sql: Optimizing llm inference request scheduling for agentic text-to-sql workflow.
arXiv preprint arXiv:2505.05286, 2025.
- [43]
Youhe Jiang, Fangcheng Fu, Xupeng Miao, Xiaonan Nie, and Bin Cui.
Osdp: Optimal sharded data parallel for distributed deep learning.
arXiv preprint arXiv:2209.13258, 2022.
- [44]
Guoliang He, Youhe Jiang, Wencong Xiao, Kaihua Jiang, Shuguang Wang, Jun Wang, Zixian Du, Zhuo Jiang, Xinlei Zhang, Binhang Yuan, et al.
Efficient pre-training of llms via topology-aware communication alignment on more than 9600 gpus.
arXiv preprint arXiv:2509.15940, 2025.
- [45]
Ran Yan, Youhe Jiang, Xiaonan Nie, Fangcheng Fu, Bin Cui, and Binhang Yuan.
Hexiscale: Accommodating large language model training over heterogeneous environment.
arXiv preprint arXiv:2409.01143, 2024.
- [46]
Xupeng Miao, Yujie Wang, Youhe Jiang, Chunan Shi, Xiaonan Nie, Hailin Zhang, and Bin Cui.
Galvatron: Efficient transformer training over multiple gpus using automatic parallelism.
arXiv preprint arXiv:2211.13878, 2022.
- [47]
Yujie Wang, Youhe Jiang, Xupeng Miao, Fangcheng Fu, Shenhan Zhu, Xiaonan Nie, Yaofeng Tu, and Bin Cui.
Improving automatic parallel training via balanced memory workload optimization.
IEEE Transactions on Knowledge and Data Engineering, 36(8):3906–3920, 2024.
- [48]
Yaniv Leviathan, Matan Kalman, and Yossi Matias.
Fast inference from transformers via speculative decoding.
In International Conference on Machine Learning, pages 19274–19286. PMLR, 2023.
- [49]
Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang, Xiaoxiang Shi, et al.
Specinfer: Accelerating large language model serving with tree-based speculative inference and verification.
In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3, pages 932–949, 2024.
- [50]
Xiaoxuan Liu, Lanxiang Hu, Peter Bailis, Alvin Cheung, Zhijie Deng, Ion Stoica, and Hao Zhang.
Online speculative decoding.
In Forty-first International Conference on Machine Learning.
- [51]
Surat Teerapittayanon and Bradley McDanel.
Branchynet: Fast inference via early exiting from deep neural networks.
In 2016 23rd international conference on pattern recognition (ICPR), pages 2464–2469. IEEE, 2016.
- [52]
Haseena Rahmath P, Vishal Srivastava, Kuldeep Chaurasia, Roberto G Pacheco, and Rodrigo S Couto.
Early-exit deep neural network-a comprehensive survey.
ACM Computing Surveys, 57(3):1–37, 2024.

## Appendix A Extended Related Work

Parallelism strategies. LLMs with huge memory and computational resource requirements typically rely on parallelization across multiple GPUs [35, 42, 43, 44]. There are three prevalent forms of parallelism: data parallelism (DP, i.e., model replication), tensor parallelism (TP) [36, 45, 46], and pipeline parallelism (PP) [37, 47].
DP replicates the model into multiple replicas, enabling parallel processing of requests. TP divides model weights and computationally intensive operations such as matrix multiplication across various GPUs, thereby splitting data scanning and computation to minimize LLM inference latency. PP divides the layers of a model into multiple stages. These stages are assigned to distinct GPUs for execution and they establish a pipeline. Only inter-layer activations are needed to be communicated between stages.

Speculative decoding and early-exit in LLM inference. Speculative decoding uses a lightweight draft model to generate token blocks, which a larger target model verifies—leveraging model heterogeneity to reduce computation and latency [48, 49, 50]. Similarly, early-exit networks add decision branches at intermediate layers, enabling inference to stop early when confidence is high—cascading computation within a single model [51, 52]. In contrast, we focus firmly on cascade model inference.

## Appendix B Simulator Design and Validation

Our simulator employs a round-robin strategy for request dispatching among multiple parallel models, and a first-come first-served strategy for per-model request processing. The single-GPU processing time is based on profiled characteristics like compute TFLOPS and memory bandwidth. The simulator also considers the phase-specific characteristics of LLMs. The prefill phase is compute-bound, so its batched processing capacity is determined by the sum of the individual latencies. In contrast, the decoding phase is memory-bound, and its batched processing capability is defined by a single latency value. This distinction has been validated in several studies (e.g., DistServe [27], Splitwise [26]).

Inputs of the simulator. The simulator requires three fundamental inputs: (i) the distributions of input and output sequence lengths for each model type within the cascade; (ii) the request arrival rate corresponding to each model type within the cascade; and (iii) the resource allocation designated for each model type within the cascade.

Example. Consider a workload distribution $\mathcal{W}$ that routes 100, 70, and 30 requests to model types 1, 2, and 3 respectively within the cascade, with corresponding GPU allocations of 2, 4, and 2 units. In this configuration, we record the distributions of input and output sequence lengths for each subset of requests (100, 70, and 30 respectively) as input files to the simulator, configure the request arrival rates and resource allocations according to the specified parameters, and execute the simulation. Subsequently, the simulator undergoes iterative execution to identify the optimal parallelism strategy based on the provided input files, request arrival rates, and resource allocation constraints.

**Table 1: Simulator accuracy across parallelism configurations on Llama3-70B model under a workload with average input and output lengths of 1600 and 16. Errors are absolute percentage errors.**
| Config (DP,TP,PP) | Real (req/s) | Estimated (req/s) | Abs. % Error |
| --- | --- | --- | --- |
| (1, 4, 1) | 0.21 | 0.219 | 4.29% |
| (1, 2, 2) | 0.26 | 0.280 | 7.69% |
| (1, 1, 4) | 0.27 | 0.287 | 6.30% |
| (2, 1, 2) | 0.33 | 0.347 | 5.15% |
| (2, 2, 1) | 0.40 | 0.408 | 2.00% |
| (2, 4, 1) | 0.41 | 0.437 | 6.59% |
| (2, 2, 2) | 0.55 | 0.559 | 1.64% |

Batching strategy in our simulator. The simulator’s internal batching strategy is continuous batching, which iteratively batches request tokens to fully utilize the current resources. The GPU’s memory limit constrains the maximum batch size for continuous batching.

Queuing mechanism. Our simulator maintains an individual queue for each model. Once there is free memory on the GPU (one request has finished), the model will fetch the next request in the queue for processing.

Different parallelism. Tensor and pipeline parallelism both split the computation workload of a single model across multiple devices. For pipeline parallelism, the simulator models communication overhead by profiling the relationship between estimated communication volume and observed latency. For tensor parallelism, the simulator assumes that each operator’s computation cost ideally scales down by a factor of $1/N$ when split across $N$ GPUs, and then adjusts this ideal cost using a speed-up coefficient $K(N)$ obtained from micro-benchmarks to account for communication and synchronization overhead. All profiling is performed offline before scheduling begins.

Simulator evaluation. We present the accuracy of our simulator with real-time experiments in [Table 1](https://arxiv.org/html/2506.04203v2#A2.T1). The table presents examples of our throughput estimation for the Llama3-70B model under a workload with average input and output lengths of 1600 and 16, respectively. The notation (1,2,2) indicates a DP degree of 1, TP degree of 2, and PP degree of 2. Although the estimations are not perfectly accurate, they are sufficiently reliable (with estimation errors within 2%–7%) for selecting optimal configurations.

## Appendix C Routing Solver in Latency-Constrained Case

The routing solver can also optimize system response quality under a user-specified latency budget by solving

$$ $\arg\min_{\theta}\left[-\,Q(\theta)\;+\;\nu\,\frac{\max\{0,\,L(\theta)-L_{\max}\}}{\,z^{\star}_{\mathrm{lat,\,max}}-z^{\star}_{\mathrm{lat,\,min}}\,}\right],$ $$

where $z^{\star}_{\mathrm{lat,\,min}}$ and $z^{\star}_{\mathrm{lat,\,max}}$ are the best (minimum) and worst (maximum) achievable latencies, $L_{\max}$ is the allowable latency budget, and $\nu>0$ scales the penalty. The same routing–deployment alternation, deployment solver, and convergence procedure are reused unchanged.

## Appendix D Complete Bi-level Optimization Formulation

Problem setup and notation.
We consider a cascade with $C$ model types/stages indexed by $\{1,\dots,C\}$ and labeled $\mathcal{C}=\{c_{1},\dots,c_{C}\}$, where $c_{i}$ denotes the $i$-th model type.
The routing strategy is denoted by $\theta$, parameterized by thresholds $\mathcal{H}=\{h_{1},\dots,h_{C-1}\}$, with $\Theta$ the feasible set of routing strategies.
The GPU resource allocation is $\mathcal{F}=\{f_{1},\dots,f_{C}\}$, where $f_{i}\in\mathbb{Z}_{+}$ is the number of GPUs assigned to model type $i$, subject to a total budget $N\in\mathbb{Z}_{+}$.
The parallelism plan is $\mathcal{S}=\{\mathrm{DP}_{i},\,\mathrm{TP}_{ij},\,\mathrm{PP}_{ij}\}_{i,j}$, where $\mathrm{DP}_{i}$ denotes the number of data-parallel replicas and, for each replica $j$, $\mathrm{TP}_{ij}$ and $\mathrm{PP}_{ij}$ denote its tensor- and pipeline-parallel degrees.
Given routing $\theta$ and deployment $(\mathcal{F},\mathcal{S})$, the estimated p95 latency is $L(\theta,\mathcal{F},\mathcal{S})$, and the system quality is $Q(\theta;\tilde{\mathcal{I}})$ estimated by a judger using a subsampled workload $\tilde{\mathcal{I}}$.
For Chebyshev-style normalization of quality, we use quality anchors $z_{1}^{\star}$ (utopia/best achievable quality, e.g., all requests at $c_{C}$) and $z_{2}^{\star}$ (nadir/worst credible quality, e.g., all requests at $c_{1}$).
A user-specified quality requirement is $q_{\min}$, and $\mu>0$ is a penalty weight.

Bi-level formulation.
The routing is optimized by a single scalar objective that penalizes quality shortfall, normalized by the utopia–nadir range, while the deployment is optimized under the GPU budget and parallelism feasibility:

$$ $\theta\in\operatorname*{arg\,min}_{\theta^{\prime}\in\Theta}\Bigg[L\!\big(\theta^{\prime},\,\mathcal{F}^{\star},\,\mathcal{S}^{\star}\big)\;+\;\mu\,\max\!\left\{0,\;\frac{q_{\min}-Q(\theta^{\prime};\,\tilde{\mathcal{I}})}{\,z_{1}^{\star}-z_{2}^{\star}\,}\right\}\Bigg],$ $$

$$ $(\mathcal{F}^{\star},\mathcal{S}^{\star})\in\operatorname*{arg\,min}_{\mathcal{F},\,\mathcal{S}}L(\theta^{\prime},\mathcal{F},\mathcal{S})\quad\text{s.t.}\quad\sum_{i=1}^{C}f_{i}\leq N,\qquad\sum_{j=1}^{\mathrm{DP}_{i}}\mathrm{TP}_{ij}\mathrm{PP}_{ij}=f_{i}\ \ (i{=}1,\dots,C),\qquad$ $$

$$ $f_{i},\ \mathrm{DP}_{i},\ \mathrm{TP}_{ij},\ \mathrm{PP}_{ij}\in\mathbb{Z}_{+}.$ $$

Tractability and solution strategy.
Because the problem couples routing, resource allocation, parallelism, heterogeneous LLM workloads, and user-specific quality requirements, a monolithic solve is intractable.
We therefore adopt a bi-level strategy: The deployment problem is solved as a MILP with latency values obtained from resource allocation and parallelism strategy optimization; the routing solver solves the Chebyshev-guided penalty problem.
The two phases are executed iteratively, with the routing solver updating $\theta$ and the deployment solver resolving $(\mathcal{F}^{\star},\mathcal{S}^{\star})$ accordingly, and termination declared once the routing objective stabilizes under a prescribed horizon.

Interpretation.
The bi-level problem decomposes into routing and deployment subproblems that are solved iteratively.

Deployment solver (deployment under resource/feasibility constraints).
For a fixed routing $\theta^{\prime}$, the deployment solver selects the latency-optimal deployment by choosing GPU allocations and parallelism plans subject to the budget and structural constraints:

$$ $(\mathcal{F}^{\star},\mathcal{S}^{\star})\in\operatorname*{arg\,min}_{\mathcal{F},\,\mathcal{S}}L(\theta^{\prime},\mathcal{F},\mathcal{S})\quad\text{s.t.}\quad\sum_{i=1}^{C}f_{i}\leq N,\qquad\sum_{j=1}^{\mathrm{DP}_{i}}\mathrm{TP}_{ij}\mathrm{PP}_{ij}=f_{i}\ \ (i{=}1,\dots,C),\qquad$ $$

$$ $f_{i},\ \mathrm{DP}_{i},\ \mathrm{TP}_{ij},\ \mathrm{PP}_{ij}\in\mathbb{Z}_{+}.$ $$

This solver captures both hardware limits (GPU budget $N$) and parallelism feasibility.

Routing solver (routing, Chebyshev-guided optimization).
Given the current deployment $(\mathcal{F}^{\star},\mathcal{S}^{\star})$, the routing solver updates the routing strategy (i.e., $\theta$) by minimizing a single scalar objective that balances latency and a normalized quality shortfall:

$$ $\theta\in\operatorname*{arg\,min}_{\theta^{\prime}\in\Theta}\Bigg[L\!\big(\theta^{\prime},\,\mathcal{F}^{\star},\,\mathcal{S}^{\star}\big)\;+\;\mu\,\max\!\left\{0,\;\frac{q_{\min}-Q(\theta^{\prime};\,\tilde{\mathcal{I}})}{\,z_{1}^{\star}-z_{2}^{\star}\,}\right\}\Bigg].$ $$

Here, $(z_{1}^{\star}\!-\!z_{2}^{\star})^{-1}$ provides Chebyshev (utopia–nadir) normalization for scale stability, and $\mu>0$ sets the severity of penalizing $Q(\theta^{\prime})<q_{\min}$. For sufficiently large $\mu$ (when the target is feasible), any minimizer is quality-compliant and the routing objective effectively reduces to minimizing latency among feasible routings.

Coupling and procedure.
The routing solver’s $\theta$ determines the workload distribution seen by each model type within the cascade (and hence the optimal deployment plan for the deployment solver), while the deployment solver’s $(\mathcal{F}^{\star},\mathcal{S}^{\star})$ determines the latency used by the routing objective (and hence the optimal routing strategy for the routing solver). Alternating updates continue until the routing objective stabilizes under a prescribed termination horizon (e.g., best-so-far objective unchanged for $K$ consecutive iterations).

## Appendix E Case studies on Model Deployment Plans and Routing Strategies

Case study on resource allocation and routing strategies.
[Table 2](https://arxiv.org/html/2506.04203v2#A5.T2) demonstrates the case study of thresholds, processing ratios and allocated resources for different model types across different testing cases.

**Table 2: Case study of the thresholds ($h_{1}$, $h_{2}$), processing ratios ($p_{1}$, $p_{2}$, $p_{3}$), and allocated resources ($f_{1}$, $f_{2}$, $f_{3}$) for each model type within the cascade across different testing cases. (90, 1) denotes testing on Trace 1 with an average quality requirement of 90.**
|  | $h_{1}$ | $h_{2}$ | $p_{1}$ | $p_{2}$ | $p_{3}$ | $f_{1}$ | $f_{2}$ | $f_{3}$ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| (90, 1) | 99 | 91 | 100% | 94% | 50% | 4 | 8 | 20 |
| (85, 1) | 74 | 64 | 100% | 62% | 21% | 4 | 12 | 16 |
| (80, 1) | 69 | 25 | 100% | 54% | 11% | 6 | 14 | 12 |
| (80, 2) | 61 | 18 | 100% | 31% | 3% | 8 | 16 | 8 |
| (80, 3) | 32 | 0 | 100% | 23% | 0% | 18 | 14 | 0 |
| (70, 3) | 10 | 0 | 100% | 5% | 0% | 24 | 8 | 0 |

Case study on parallelism strategies. [Table 3](https://arxiv.org/html/2506.04203v2#A5.T3) presents a case study on parallelism strategies for each model type within the cascade across different test cases.

**Table 3: Case study of the parallelism strategies for each model type within the cascade ($s_{1}$, $s_{2}$, $s_{3}$) across different testing cases.**
|  | Parallelism Strategies |
| --- | --- |
| (90, 1) | $s_{1}$: (DP=4), $s_{2}$: (DP=2, TP=4), $s_{3}$: (TP=4, PP=3), (TP=8) |
| (85, 1) | $s_{1}$: (DP=2, TP=2), $s_{2}$: (DP=6, TP=2), $s_{3}$: (DP=2, TP=8) |
| (80, 1) | $s_{1}$: (DP=6), $s_{2}$: (DP=5, TP=2), (TP=4), $s_{3}$: (TP=4, PP=3) |
| (80, 2) | $s_{1}$: (DP=6), (TP=2), $s_{2}$: (DP=8, TP=2), $s_{3}$: (TP=8) |
| (80, 3) | $s_{1}$: (DP=10), (DP=4, TP=2), $s_{2}$: (DP=2, TP=4), (DP=3, TP=2), $s_{3}$: - |
| (70, 3) | $s_{1}$: (DP=16), (DP=4, TP=2), $s_{2}$: (DP=4, TP=2), $s_{3}$: - |

**Table 4: End-to-end throughput results evaluating Cascadia against Sarathi-Serve.**
| Trace | Ours | Sarathi-Serve | Speedup | %Improvement |
| --- | --- | --- | --- | --- |
| Trace 1 | 0.2529 req/s | 0.1913 req/s | 1.322 | +32.20% |
| Trace 2 | 0.4659 req/s | 0.2385 req/s | 1.953 | +95.35% |
| Trace 3 | 0.6406 req/s | 0.3977 req/s | 1.611 | +61.08% |

## Appendix F Comparison with Sarathi-Serve

We evaluated Sarathi-Serve under the same experimental setup as SGLang, as described in §[4.1](https://arxiv.org/html/2506.04203v2#S4.SS1), using traces 1–3 with an average quality requirement of 90. We used Sarathi-Serve’s vLLM implementation (its most efficient variant) and tuned the chunk size to be optimal for each case. As shown in [Table 4](https://arxiv.org/html/2506.04203v2#A5.T4), our system achieves up to 1.95$\times$ higher throughput and averages a 1.64$\times$ speedup across traces.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Why do you try to minimize the maximum latency across models, L, in the MILP, when the latency of a query will be the sum of the latencies of the models that it passes through?

2. Why do you consider separate thresholds for each model when determining the routing strategy even though the same input is passed through the models until a satisfactory output is obtained (if the input is the same then the threshold on the output score should also be the same for all models)?

3. $L(\theta)$ in line 263 appears to be non-differentiable. If that is indeed the case, please clarify how $\theta$ is updated to converge to the minima of the optimization problem.

4. Please provide a citation for the baseline CascadeServe in Section 4.

5. How is query complexity measured when looking for a shift in workload characteristics in Section 4.4? 

6. How does CascadeServe handle distribution shifts? Does it not make any changes under distribution shift?

7. Can you quantify the scheduling overhead (line 476) in terms of additional latency (and not just throughput) when re-scheduling under online workloads? For e.g. if there is a spike in the P95 latency during the rescheduling window then that would not be a good thing.
soundness: 3
strengths: 1. The joint formulation of resource allocation and adaptive inference via model cascades is a novel approach to cost-efficient LLM inference and is a very important problem that needs to be solved before cascades can be deployed in real world systems. The paper also considers heterogeneity in model and workload characteristics which are important considerations in real settings.

2. The paper proposes a viable solution to the problem via bi-level optimization that helps to find an appropriate deployment and routing strategy and shows clear improvements over baselines in experiments
summary: The paper presents CASCADIA, an efficient cascade serving system for large language models that jointly optimizes resource allocation and request routing across hierarchies of model sizes, enabling fast, high-quality, and cost-effective LLM inference. By formulating cascade serving as a bi-level optimization problem and dynamically adapting deployment strategies, CASCADIA achieves significantly lower latency and higher throughput compared to single-model and existing cascade baselines, while maintaining answer quality across diverse workloads.
weaknesses: 1. Some of the details of the approach are not clearly explained. I have added several questions below around points that were not clear to me.

2. While the bi-level optimization itself doesn't seem to be taking too long to solve in online settings (Section 4.4), the latency of re-allocating the models/changing the parallelization may be high.

3. The approach does not consider prefix caching even though the traces used in the experiments do contain multi-turn conversations and prefix caching in such settings and may affect the latencies of both the baselines and Cascadia. Even if it is difficult to incorporate prefix caching in the optimization formulation, I believe when running the traces prefix caching should be enabled to see if it causes the results to deviate significantly from what is expected after solving the optimization problem.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 3
questions: 1. Maybe instead of using GPT-4, fine tune a BERT for cascading?
soundness: 2
strengths: Stengths:

1. Efficient serving multiple LLM to balance accuracy and latency is an important topic.

2. The proposed cascading method intuitively can help the multi-model serving system.

3. Extensive experiments show the performance.
summary: This paper introduces CASCADIA, a novel cascade serving framework designed explicitly to schedule request routing and deploy model cascades for fast, quality-preserving LLM serving. CASCADIA employs a bi-level optimization method: at the deployment level, it uses a mixedinteger linear program to select resource allocations and parallelism strategies based on LLM information and workload characteristics; at the routing level, it applies a Chebyshev-guided method to iteratively co-optimize the routing strategy and the system deployment produced by the deployment level. 

It uses mixed-integer linear programming (MILP) to determine the optimal deployment plan given a routing strategy. It balances response latency and output quality. Within each cascade stage, CASCADIA supports various parallelism strategies (e.g., tensor and pipeline parallelism), which allows it to automatically select the optimal strategy based on model size, incoming workload, and routing decisions.
weaknesses: 1. The main concern is on the real-time efficiency and cost. LLM serving is an online process. If using GPT-4 to judge the small model response, runs GPT-4 takes a few seconds and the cost is expensive. 

2. Time to first token is also very long. For simple prompt, it also needs to wait until GPT-4 finishes the judge.

3. The baselines are insufficient. BERT-based router [1, 2, 3] that directly routes prompt to multiple LLMs can be compared.


[1] https://github.com/vllm-project/semantic-router

[2] Tensoropera router: A multi-model router for efficient llm inference

[3] RouteLLM: Learning to Route LLMs with Preference Data

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: What is the typical/expected number of iterations required to achieve stable solutions? If the number of iteration tends to be huge, it can lead to non-negligible overheads and compromise the efficiency gains.
soundness: 3
strengths: 1. This paper studies efficient and effective LLM serving, which is critical problem for a wide range of real-world applications.
2. This paper introduces Cascadia with rigorous and solid technical developments.
3. Cascadia achieves up to 4x lower latency deadlines and 5x higher system throughputs, which are impressive.
summary: This paper proposes Cascadia, a novel cascade serving framework for efficient & effective LLM serving. Cascadia employs a bi-level optimization strategy to compute the optimal serving strategy which includes the deployment strategy (i.e., the resource allocation and parallelism strategies) as well as the routing strategy (i.e., which user requests should be consumed by which models). Extensive experiments on real-world datasets demonstrate the effectiveness of the proposed approach.
weaknesses: 1. In Algorithm 1, Cascadia relies on iteratively optimizing both deployment and routing strategies. It remains unclear if the bi-level optimization is theoretically optimal or not. Such guarantees could be critical in practical scenarios.
2. LLM routing is another well-studied technique aiming for efficient & effective LLM serving, which is under-discussed in this paper. Authors may want to discuss and compare to this line of work to better position the contribution of this paper. Several example references are as follows,

[1] Ong, Isaac, et al. "RouteLLM: Learning to Route LLMs from Preference Data." The Thirteenth International Conference on Learning Representations.  
[2] Ding, Dujian, et al. "BEST-Route: Adaptive LLM Routing with Test-Time Optimal Compute." Forty-second International Conference on Machine Learning.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. The routing quality depends on GPT-4o as the judge. If we swap in a weaker/open-source judge (e.g. Llama-3-70B with an arena prompt), does CASCADIA still satisfy the same 𝑞 min, or does it start over-routing to bigger models? A small sensitivity table would help.
2. Rescheduling policy. How exactly do you detect a “significant” workload shift before re-running the scheduler, and do you use any hysteresis to avoid oscillating between plans?
3. Can the deployment MILP be warm-started from the previous solution to keep solve time low when only the routing thresholds change slightly?
soundness: 3
strengths: - Real problem, 2026-relevant: serving fleets of heterogeneous LLMs under SLO and quality constraints is exactly what people are doing.
- The bi-level loop (MILP for deployment + Chebyshev for routing) is a clean way to expose the coupling between “how many requests go to 70B” and “how many GPUs the 70B should get.”
- Strong empirical numbers: up to 4× lower latency SLOs and up to 5× higher throughput than single-model; 1.7–2.5× over CascadeServe; wins also hold for Llama cascades.
- Removing parallelism search or using uniform GPU allocation degrades performance notably, so the system is not just “a nicer scheduler,” it’s actually using its degrees of freedom.
- Online re-scheduling: handling trace1 → trace2 → trace3 shifts and still beating baselines makes the system more believable for production.
summary: The paper introduces CASCADIA, a cascade-serving system for LLMs that jointly optimizes (i) GPU/resource deployment of multiple models (including DP/TP/PP choices) and (ii) threshold-based routing across the cascade to meet a target latency–quality trade-off. It uses a bi-level optimization loop: a MILP picks per-model allocations/parallelism given a routing pattern, and a Chebyshev-guided routing solver adjusts thresholds to satisfy a user quality floor. On DeepSeek and Llama cascades, CASCADIA outperforms single-model serving and CascadeServe, reporting up to 4× tighter SLOs and 5× higher throughput, with ablations showing that both resource allocation and parallelism search are necessary.
weaknesses: - Routing quality is based on GPT-4o-as-a-judge. That’s fine for the paper but less fine for on-prem/air-gapped setups; we don’t see how robust the method is to weaker judges.
- Incremental vs existing cascade work: CascadeServe, AutoMix, and even 2025 routing+speculative papers are moving in the same direction.
- No cost/energy accounting, they motivate with resource efficiency and even cite energy papers, but do not report $/req or some energy unit /req, that’s what operators would care about.
- If the quality distributions used in the routing solver are mis-estimated (domain shift, very hard math/coding workloads), the Chebyshev objective could over-route to large models and wipe out the latency gains.


Decision: Accept (Poster)