Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUs
Authors: YOUHE JIANG
Fangcheng Fu
Xiaozhe Yao
Guoliang HE
Xupeng Miao
Ana Klimovic
Bin CUI
Binhang Yuan
Eiko Yoneki
Venue: ICML 2025 poster (2025)
Keywords: Distributed Machine Learning System; Generative Inference of Foundation Model.
Primary area: optimization->large_scale_parallel_and_distributed

Abstract: Recent advancements in Large Language Models (LLMs) have led to increasingly diverse requests, accompanied with varying resource (compute and memory) demands to serve them. However, this in turn degrades the cost-efficiency of LLM serving as common practices primarily rely on homogeneous GPU resources. In response to this problem, this work conducts a thorough study about serving LLMs over heterogeneous GPU resources on cloud platforms. The rationale is that different GPU types exhibit distinct compute and memory characteristics, aligning well with the divergent resource demands of diverse requests. Particularly, through comprehensive benchmarking, we discover that the cost-efficiency of LLM serving can be substantially optimized by meticulously determining GPU composition, deployment configurations, and workload assignments. Subsequently, we design a scheduling algorithm via mixed-integer linear programming, aiming at deducing the most cost-efficient serving plan under the constraints of price budget and real-time GPU availability. Remarkably, our approach effectively outperforms homogeneous and heterogeneous baselines under a wide array of scenarios, covering diverse workload traces, varying GPU availablilities, and multi-model serving. This casts new light on more accessible and efficient LLM serving over heterogeneous cloud resources.

Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUs
Youhe Jiang" 12
Fangcheng Fu"
Xiaozhe Yao Guoliang He Xupeng Miao Binhang Yuan
Ana Klimovic
Bin Cui
Eiko Yoneki
Abstract
Sbort
Long
Sbortt Long Out
Long Long
J-Month Traces
6-Hoou Traces
Recent advancements in Large Language Models (LLMs) have led to increasingly diverse requests, accompanied with varying resource (compute and memory) demands to serve them. However, this in turn degrades the cost-efficiency of LLM serv- ing as common practices primarily rely on ho- mogeneous GPU resources. response this problem, this work conducts thorough study about serving LLMs over heterogeneous GPU re- sources on cloud platforms. The rationale is that different GPU types exhibit distinct compute and memory characteristics, aligning well with the divergent resource demands of diverse requests. Particularly, through comprehensive benchmark- ing, discover that the cost-efficiency of LLM serving be substantially optimized by metic- ulously determining GPU composition, deploy- ment configurations, and workload assignments. Subsequently, we design scheduling algorithm via mixed-integer linear programming, aiming at deducing the most cost-efficient serving plan un- der the constraints of price budget and real-time GPU availability. Remarkably, fectively outperforms homogeneous and heteroge- neous baselines under wide array of scenarios, covering diverse workload traces, varying GPU availablilities, and multi-model serving. This casts new light on more accessible and efficient LLM serving over heterogeneous cloud resources.
100
Req, Ratio
75
75
50
25
25
Time Slots
Time Slots
Figure 1: The real-world workload traces from the Swiss AT Center comprise over 500,000 traces collected over one month. We categorize the workload types based on input and output token lengths (longer than 512 and 128 are char- acterized as long).
1. Introduction
Large Language Models (LLMs), including GPT-4 (Ope- nAI, 2024), Gemini (Reid et al., 2024), Llama3 (Dubey et al., 2024), Claude (Anthropic, 2024), Mixtral (Jiang et al., 2024), and DeepSeck-V?. (Liu et al., 2024a), have demon- strated unprecedented performance across a wide range of real-world applications (GitHub, 2024; Jeon & Lee, 2023; Peng al., 2023), such chatbots, education, and health- care, profoundly impacting human lives. In this context, enhancing the cost-efficiency of LLM serving is crucial for democratizing access to these cutting-edge technologies.
approach ef-
Currently, predominant practices utilize homogeneous GPU resources to deploy LLMs and serve the incoming re- quests (Li et al., 2023; Kwon et al., 2023; Agrawal et al., 2024b). However, with the broadening application domains, serving LLMs is facing progressively varying request pat- terns, driving the serving workloads dynamic and diverse- a phenomenon referred to as workload heterogeneity (Sun et al., 2024; Zhao et al., 2024b). This contradiction makes the use of homogeneous GPU resources unsuitable.
'qqual contribution 'Department of Computer Science, Univer- sity of Cambridge, Cambridgeshire, UK Department of Computer Science and Engineering, The Hong Kong University of Science and Technology, Hong Kong, China 'Department of Computer Science, Peking University, Beijing, China 'Department of Com- puter Science, ETH Zurich, Zürich. Switzerland Department of Computer Science, Purdue University. West Lafayette, Indiana, US. Correspondence to: Binhang Yuan <biyuan huk. Eiko Yoneki <ciko.
To be specific, the requests be served have varying input and output token lengths, exemplified by the real- world LLM serving traces at the Swiss Al Center shown in Fig- ure 1. Such differences can exhibit significantly divergent resource (compute and memory) demands across different types of workloads, owing to the distinct characteristies of the two phases of inference- the prefill phase is compute-
Proceedings of the 4 International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Demystifying Cost-Efficiency in LLM Serving over Hetcrogencous GPUs
Table I: GPU Specifications and Pricing
Num. of GPUs
30
4090
GPU
Peak
Memory Access
Memory
Price
A40
Type
FP16 FLOPS
Bandwidth
Limit
(per GPU)
20
A6000
A6000
91 TELOPS
960 GB//s
48 GB
0.83 S/h
L40
10-
A40
TELOPS
696 GB//s
48 GB
0.55 $/h
A
L40
181 TFLOPS
864 GB/s
48 GB
0.83 $/h
H100
A100
312 TELOPS
1555 GB/s
80 GB
1.75 $/h
H100
1979 TILOPS
3.35 TB/s
80 GB
2.99 $/h
24-Hour Period
4090
83 TFLOPS
1008 GB/s
24 GB
0.53 $/h
Figure 2: The number of different types of GPUs on Vast.ai during 24-houu period.
bounded as it processes imput prompts in a single step, while the decoding phase is memory-bounded as it generates sub- sequent tokens step by step (Zhong et al., 2024; Patel et al.,
2024). Therefore, when using homogeneous GPU resources, is hard to fit the varying resource demands well.
parallelisms) and different workloads have their preferable resource needs. As a result, to improve resource utiliza- tion, necessitates assigning requests to more suitable replicas while balancing the burden across all replicas.
On the contrary, the heterogeneity in resource demands presents unique opportunity to enhance the overall serving efficiency by leveraging different GPU types. As shown in Table I, various GPU types offer diverse compute and memory capabilities, making them well-suited for process- ing different types of workloads. Motivated as such, we try to explore two questions: Can serving LLMs over het- erogeneous GPU resources achieve better cost-effiiiency than homogeneous GPU resources? If yes, how can enhance the cost-efficiency? To this end, this work makes two technical contributions correspondingly.
The second contribution is to design a brand new LLM serv- ing framework following the benchmarking, which aims at maximizing the cost-efficiency of LLM serving over hetero- geneous GPU resources in cloud platforms.
Given the three factors above, a straightforward approach is to rent the most suitable GPUs for each workload type and assign requests accordingly. Nevertheless, this is im- practical for two reasons. For one thing, due to the high demand for cloud GPUs, although cloud platforms (e.g.. Vast.a,, RunPod, and AWS) offer variety of GPU types, they usually have limited quantities of each type. We present the availability of different GPU types on Vast.ai over a 24- hour period in Figure 2. For another, the user-defined price budget is often constrained, making it impractical to always allocate sufficient GPUs for each workload demand.
The first contribution is a comprehensive benchmarking of LLM serving over various GPU types, which offers a de- tailed understanding of cost-efficiency with heterogeneous GPU resources. Based on the benchmarking results, we reveal three key factors that are vital to the cost-efficiency:
GPU composition (i.e., the number and types of GPUs that make up a heterogeneous cluster) is essential for opti- mizing the cost-efficiency LLM serving. Different GPU types exhibit varying characteristics (e.g., computational capabilities, memory bandwidths, and memory capaci- ties), making them more suitable for distinct workloads and model types. Given the varying types of incoming workloads, we need to strategically optimize GPU com- position to improve resource utilization, reduce latency, and enhance overall serving performance.
Consequently, we formulate scheduling algorithm based on mixed-integer linear programming (MILP), which takes account of both real-time GPU availability on cloud platforms and the user-defined price budget, while co- optimizing how to rent GPUs from the available pool (GPU composition), how to deploy the models over the rent GPUs (deployment configuration), and how to dispatch the work- loads among the model replicas (workload assignment). We further incorporate practical heuristics and binary search mechanism, as well as extend our approach to the multi- model scenario, improving scalability and solving efficiency for large-scale clusters.
Deployment configurations (i.e., how many model repli- cas to deploy and the parallelism strategy for each) are necessary for maximizing overall system performance. The optimal configurations is influenced by the model, workload, and GPU type, so using a unique deployment configuration for all replicas is impractical. Therefore, we should adaptively optimize the deployment configurations that the system efficiency can be improved.
We empirically evaluate our framework by comparing with both homogeneous and heterogeneous baselines across variety of scenarios, covering diverse workload traces, varying GPU availabilities, and multi-model serving. The results demonstrate that, within the same price budget, our approach can achieve up to 41 % and on average 20% higher throughput, or reduce the serving latency by up to 54% and on average 20%.
Workload assignment (i.e., allocating incoming work- loads to GPUs) becomes crucial as different replicas are deployed with varying configurations (i.e., resources and
2

---

Demystifying Cost-Efficiency in LLM Serving over Hetcrogencous GPUs
2. Background
node optimizations by refining request ordering: SarathiS- erve (Agrawal et al., 2024b) optimizes batching through prefill chunking to mitigate interference between the pre- fill and decoding stages; and Vidur (Agrawal et al., 2024a) develops an accurate simulator for deployment tuning. In contrast, our method is dedicated to achieving heteroge- neous, cost-efficient serving in cloud environments.
LLM inference phase. The inference process in LLMs consists of two main phases: the prefill phase and the de- coding phase. During the prefill phase, the model processes the input prompt to compute the key-value (KV) cache and generates the first token in single step. In contrast, the decoding phase uses the last generated token and the KV cache as inputs to generate subsequent tokens in a token-by- token manner. Generally, the prefill phase is compute-bounde while the decoding phase is memory-bound.
3. Observation and Opportunity
Given a user with specified budget estimation (in S/h) rent- ing GPUs from the cloud for serving certain workload traces, our objective is to deliver a comprehensive serving plan that maximizes the cost-efficiency of the user's serving system. In this section, we first benchmark the cost-efficiency per- formance of various workload types across different GPU types, model types, and deployment configurations. Then, we present our key observations and opportunities.
Workload heterogeneity. LLMs are designed to support diverse range of applications, and the inference work- loads associated with these applications often exhibit het- erogeneity in input and output token lengths, which is called workload heterogeneity. Different workload types ex- hibit varying characteristics in terms of compute and mem- ory demands. For example, requests from the WildGPT dataset (Zhao al.), with average input and output token lengths of 496 and 510 respectively (classified short input and long output), typically require more memory resources to handle the memory-boumn decoding phase. Conversely, requests from the Azure-Trace dataset (Patel et al., 2024; Azure, 2024), with average input and output token lengths of 2455 and 18 respectively (classified as long input and short output), generally demand more compute resources to manage the compute-boun prefill phase. Therefore, appro- priately allocating resources based on workload demands is critical for optimal performance.
Benchmark settings. We subsample nine workload types from the ShareGPT (Zheng et al.), WildGPT (Zhao et al.), and Azure- datasets (Patel et al., 2024). These workloads are characterized by average input token lengths of {2455, 824, 496} and output token lengths of (510, 253, 18). Each combination reflects distinct work- load characteristics. For example, {2455, 18) (long input, short output) represents compute- inteesiie workloads, while (496, 510} (short input, long output) represents memory- intensive workloads. Based on these workload types, we evaluate two models, Llama3-8B and Llama3-70B, on six commonly used cloud GPUs (A6000, A40, L40,, A100, H100, and 4090) with different deployment configurations. The benchmarking metrics include request throughput per unit cost (i.c., throughput divided by GPU cost) and the total cost associated with various latency percentiles (e.g.. p5, p10, p15, ... p95, p100). The total cost for each latency percentile is calculated by multiplying the latency time by the GPU cost. These metrics serve as indicators of cost efficiency. The GPU costs are demonstrated in Table 1.
Heterogeneous LLM serving. Recent research has ex- plored various approaches for deploying LLM serving in heterogeneous GPU environments to achieve cost-efficient solutions (JIANG et al.; Miao et al., 2024; Griggs et al., 2024; Zhao et al., 2024a; Mei et al., 2024; Borzunov al., 2023; Yan et al., 2024). HexGen introduces asymmetric partitioning and advanced scheduling techniques to deploy generative inference in decentralized and heterogeneous set- tings. Me'lange frames GPU allocation as a cost-aware bin-packing problem, optimizing cost efficiency for I.I.M services by effectively leveraging heterogeneous GPU types. Helix formulates the problem of heterogeneous GPU and max-flow problem,
Observation-1: Heterogeneous GPUs are well-suited for managing model and workload diversity in LLM serving. Figure and Figure 11 present the benchmark results for the Llama3-70B and Llama3-8B models across various GPU types and workload types. The observations can be summa- rized as follows: (i The H100 and A100 GPUs (data center GPUs) perform well on compute- inntenive workloads with the Llama3-70B model, as both GPUs have high computa- tional power to handle intense computational tasks. (ii) The A40, A6000, and L40 GPUs (workstation GPUs) excel in memory-intensive workloads with the Llama3-708 model. Workstation GPUs offer on average 1.2x higher memory bandwidth and 1.8x greater memory capacity per unit price than data center GPUs, making them more cost-efficient for memory-intensive intensive workloads that often underutilize H100
network connection optimization utilizing mixed-integer linear programming to determine the optimal model deployment. However, existing works typically optimize performance within a predefined hetero- geneous cluster, and fail to consider GPU availability and user-defined budget constraints on cloud platforms. In ad- dition, they are generally unaware of the workload hetero- geneity, and only consider uniform workload assignment.
LLM serving optimization. There are several related works focusing on the optimization of LLM serving (Li et al., 2023; Yu et al., 2022: Kwon et al., 2023). QLM (Patke et al., 2024) focuses on SLO-aware serving and multi-
3

---

Demystifying Cost-Efficiency in LLM Serving over Hetcrogencous GPUs
2455, Out:
Aug
2435, 253
Aur
2435, 510
In: 2455. Out
2455, Out:
Out: O1t:
42-11
2641
46-4
18-0-
20-01
20-0
1641
40-42-
50:05
11-01-
37.40
Throughpen Per Dollar (reeis)
16:1
9043
18-47
TB-09
28-02
52.42
25-00-
SE41
48-41
18-02
48-02-
18:40
310
(5)
58-12
42-42
22-42
sc-1-
26414
18-41
18-13-
16-00
55-42
42-40
66-90
Avg In: 824, Out:
Aug 924, Out 253
Aorg In: 924, Our:
Ave 925, Out
Aug 12; Out 253
Avg In: 824, Out:
MC-40-
15.01-
454
26.41
15-02
1E-42
12:40
18-2
45-42-
45
Different GPU Types
Different GPU Types
Difterent GPU Types
P5-100 Latencies
P5-100 Latencies
PS-100 Latencies
Figure 3: Benchmarked results for Llama3-70B model with different GPU types on different workload types. The left three columns represent the throughput results, x-axis represents different GPU types, y-axis represents throughput per unit price (i.e., throughput divided by GPU cost). The right three columns represent the latency results, x-axis represents the P5-100 latency results (P5-P100 Latencies means from left to right, the sticks represent P5 Latency, P10 Latency, P15 Latency ...). y-axis represents total price (i.e., each latency time multiplied by GPU cost). Results for Llama3-88 model are demonstrated in Appendix A.
4080-(18&
4090 (L.24)
(1.42
(LL.)
40-1
1140
5041
21-41
3640
13-0--
15:00-
16:0
26:0
Througlouu Per Dollar (2eqis)
(1...)
140 (1.24)
L40 (1,42)
L40
1,40 (.40-12.1.4
L40-22A1))
(2.12222)
S0-1--
7041
4600
[1-01
11.0
2000
160
(1,00(1,A1)
(100-(1222)
11100-(11.0
(1100(2.1.)
(1.00.2,2,1
11-1-
1041
Ex-G
teve
Differeat Condies
Different Coufges
Diffesent Configs
Different Confies Aug Im:40,,
Ditfereas Confles Aug In:4i6,
Different Condgs
Different Condgs
Difierzot Confies Avg In:24,, Ouit253
Dufforent Configs Avg In44 Che:510
Aug
Aug 1455, Cui255
Aug
Out51n
Aug In:466, Ose510
Ang In
Figure 4: Throughput results for Llama3-70B model with different deployment configurations on different workloads. The three-element array represents the DP, TP, and PP degrees. Full benchmarking results are listed in Appendix B.
and A100 GPUs due to memory constraints. (iii) The 4090 GPUs (consumer GPUs) deliver excellent performance with the Llama3-8B model. As smaller models require signifi- cantly less compute and memory, and the consumer GPUs offer superior memory bandwidth per unit price, approxi- mately 1.9x higher than that of the A100 and H100 GPUs. Overall, our experimental results demonstrate that select- ing the most appropriate GPU type for specific workloads and models can enhance the cost-efficiency performance of LLM serving by up to 2.27 These findings underscore the necessity of aligning GPU types with model and workload demands to maximize both performance and cost-efficiency.
workloads, and GPU types. Figure 4 presents the bench- mark results of various deployment configurations across different models, workloads, and GPU types. The obser- vations be summarized as follows: (i) Optimal con- figurations vary by workload type for given GPU type. For instance, on H100 GPUs serving Llama3-70B, ten- sor parallelism (TP) (Shoeybi et al., 2019) is most effec- tive for compute- and -entoryinteesivee workloads (e.g.. (2455, 510}), while higher degree data parallelism (DP, i.e., model replication) performs better for less demanding workloads (e.g., (496, 18)). (ii) Optimal configurations vary by GPU type for given workload type. For instance, in compute-intensive workloads (e.g.. (2455, 18)). the L40 GPUs achieve the best performance using pure pipeline par-
Observation-2: Optimal deployment configurations are crucial for maximizing cost-efficiency across models,

---

Demystifying Cost-Efficiency in LLM Serving over Hetcrogencous GPUs
allelism (PP) (Huang al., 2019), while the H100 GPUs excel with combination of DP and TP. (iii) Optimal con- figurations also depend model type. For instance, with Llama3-8B models, DP consistently outperforms model parallelism (i.e., TP and PP). Since the Llama3-8B model has lower memory requirements and can run on a single GPU without model parallelism, increasing the number of model replicas (i.e., raising the DP degree) enhances the sys- tem's parallel processing capability, thereby improving cost efficiency. Overall, our experimental results demonstrate that selecting the most effective deployment configurations can improve system performance by up to 2.61 x. These findings highlight the need for optimized deployment con- figurations to maximize cost-efficiency in LLM serving.
ployment for cost-efficient LLM serving. Existing sys- tems typically assume a homogeneous GPU cluster for LLM serving or focus on optimizing performance within a predefined heterogeneous cluster. However, adjusting the heterogeneous GPU composition within the serving clus- ter to align with specific workload demands offers a more cost-efficient alternative. Based on prior observations, we propose optimizing LLM serving by customizing the de- ployment of heterogeneous GPU types meet workload requirements This includes determining the optimal hetero- geneous GPU composition (observation-1) 1), selecting the most effective deployment configurations (observation-2), and implementing the most appropriate workload assign- ment (observation-3). Ultimately, our aim to deliver a comprehensive LLM serving plan that meets user require- ments, adapts to cloud environment constraints, and maxi- mizes cost-efficiency (constraints).
The workload assignment should be co- optimized with the heterogeneous GPU composition and deployment configurations. Effective workload assign- ment is critical for managing workload heterogeneity and achieving optimal performance. It serves two key objectives: (i) Directing requests to GPU types and deployment config- urations that best match their resource demands to enhance serving efficiency. As noted in observation-I and -2, even with an optimal GPU composition and deployment setup, performance may degrade if workloads are assigned to un- suitable GPUs or configurations. This misalignment can lead to inefficiencies and reduced overall performance. (ii) Balancing workloads across GPUs to prevent overloading or underutilization, thereby maximizing resource utiliza- tion. Workload balancing essential for cost-efficient GPU utilization. In some cases, achieving full GPU capacity requires assigning workloads to suboptimal GPUs or config- urations. While this may not be ideal for individual requests, it ultimately improves overall system performance.
Observation-3:
4. Scheduling Algorithm
4.1. Problem Formulation
Given the LLMs to be served, a set of heterogeneous work- loads, a user-defined budget, and GPU availability on the cloud, we seek a cost-efficient serving plan comprising: (I) GPU composition, i.e., selecting the type and num- ber of GPUs rent from the cloud while meeting budget and resource requirements; (2) deployment configurations, i.e., organizing the rented GPUs into serving groups, each responsible for serving one model replica, and determin- ing their parallelism strategies; and (3) workload assign- ment, i.e., determining the allocation of incoming work- loads across model replicas. Our objective is to minimize the overall makespan for processing all incoming workloads. The resulting plan must ensure that the user obtains the most cost-efficient LLM serving solution under the specified bud- getary and resource constraints.
Appropriate resource scheduling is crucial under limited resources and budget constraints. Allo- cating workloads to the most suitable GPUs is a straight- forward strategy for cost-efficient LLM serving. However, real-world deployments often face resource availability and budget limitations: (i) On cloud service platforms, GPU availability fluctuates, leading to shortages during peak pe- riods. For instance, A40 availability on RunPod and Vast.ai can range from 3 to 16 and 0 to 32, depending on the time. (ii) Budget constraints may prevent users from selecting the optimal GPU for each workload, necessitating compro- mises in resource allocation. These constraints often force workload assignments to suboptimal GPUs, reducing per- formance and efficiency. To mitigate these challenges, an effective scheduling algorithm is essential. It should ac- count for the user's 's budget constraints and real-time GPU availability, enabling efficient and adaptive LLM serving even under constrained conditions.
Constraints:
4.2. Simple Example
Experiment setup. We begin by assuming three GPU types, (4, l2, la), each with two units available. The hourly rental prices for these types are 4, 2, and 2 $/h, respectively. We consider two workload types {wi, w2)), which arrive simul- taneously with 80 total requests for wi (A) = 80) and 20 total requests for W2 (A2 20). We denote by Cr the throughput (in requests per second) of GPU type t on work- load w. If each GPU serves one model replica, the through- puts are C1,1 = 1.0, C1 = 1.2, C2.1 = 0.9, C2 = 0.9, Ca = 0.3, and C222 = 0.5. Note that C=,1 and C²a.2 vary with model parallelism. In Cases 1 and 2, we assume the workload is assigned each GPU in proportion to its processing rate, the wide throughput for each workload is the sum of individual-GPU rates. In Case 3, we allow workload-awe aware assignment for further optimization.
Opportunities: Optimization of heterogeneous GPU de-
5

---

Demystifying Cost-Efficiency in LLM Serving over Hetcrogencous GPUs
Case I: GPU composition. We compare two compositions under the same budget of 8 $/h, where each GPU serves one
Table 2: Symbols used in MILP.
model replica. Composition I consists of 1 x t1, 1 x l2, and 1 x tg. achieving total throughput of (2.2, 2.6) rps (wi, W2), with a processing time of 44.05 Composition 2 consists of 1 x t1 and 2 x 12, achieving throughputs of
Symbol
Description
N
number of GPU types number of workload types set of feasible configs type GPUs allocated maximum available type n GPUs rental price of type n GPUs memory limit of type n GPUs user-defined total price budget
W
C
da
(2.8,3.0 3.0) rps on (Wi, W2), with processing time of 35.24 s. Thus, modifying the GPU composition within the same budget results in 20% speedup.
an
Pn
mn
B
Case 2: Deployment configuration. Focusing composi- tion 2. we compare two ways to organize these three GPUs. Configuration assigns each GPU to serve a single model replica, resulting in a processing time of 35.24s. Config- uration applies TP to the two l2 GPUs, changing their combined throughput to 2.4 rps on wi and 1.5 rps on W2, reducing the overall processing time to 30.94s. s. Thus, modi- fying the deployment configuration improves the processing time by approximately 14%.
Ve
GPU composition of config parallelism strategy config e price cost of config c
8e
Og
he
throughput of config e on workload w config e
Te
assignment of workload
Ve
whether config e is used
T
makespan processing all workloads total requests of workload w
fw
Case 3: Workload assignment. Finally, we retain the same composition and TP-based configuration but optimize workload assignment. Specifically, we assign 15% of w1 and 100% of 22 to the replica with 4,, and 85% of wi to the replica with TP on 2 x t2. With this assignment, the overall completion time is reduced to 28.67 s. Thus, adjusting the workload assignment results in an additional 8% reduction in processing time.
( A vector n. = [dn (c) indicating exactly how many GPUs of each type n are used in configuration c. (ii An array 8, = {f1, egy used in configuration c. The array length S repre- sents the total number of pipeline stages, and the element L, represents the TP degree of the s-th stage. The sum- mation of all to should be equal to the total GPU count of configuration i.e., E to = EN= (c) (iii) A cost Oe Em= (dn((d)()) x Pr) indicating the total price required for configuration c. (iv) throughput he. indi- cating the rate at which configuration c process workload type w, which is obtained through a one-time profiling! ! By optimizing the configurations, we can obtain the GPU composition and deployment configurations mentioned in $4.1.
indicating the parallelism strat-
This step-by-step example demonstrates the joint optimiza- tion of GPU composition, deployment configuration, and workload assignment for optimal performance. A detailed processing time calculation for each case is in Appendix C.
4.3. MILP Formulation
In this section, introduce mixed-integer linear program- ming (MILP) formulation to find serving plan, i.c., GPU composition, deployment configurations and loak as- signment, that minimizes the overall processing time. An overview of the symbols is shown in Table 2.
Workloads and assignment. Let there be W workload types, indexed by (1,2, W). Each workload w must be fully served (i.e., 100% coverage). We allow
fractional assignment: a fraction Fc,w E [0. 1] of work- load w may be processed by configuration c. Concretely, Eeeo Le,w = 1, w = 1,,..... W. We also introduce an integer variable ye E {0, 1,. indicating how many copies of configuration are chosen (activated). ye O. then Fee, must be zero for all w. By co-optimizing the workload assignment (the fractions Ze,ww) with the activated configurations (y.), we determine the final workload as- signment as described in $4.1.
Let there be N types of GPUs, indexed by n E {1, 2. ........' NV. We denote the decision on how many GPUs of each type use (i.e., GPU composition) by vector D [dd, d2. .... dx where each d, > 0 represents the number of GPUs of the n-th type. These variables are subject to availability constraints encoded by vector A = [at, 022 ], ], such that 0- < d, S an, Yn = 1.2.....N N. Each GPU type n has price Pn (e.g., 1.75 $/h for A100, 2.99 $/h for H100), and memory limit mn (e.g., 48 GB for L40, 80 GB for H100).
Budget and GPU constraints. A valid configuration set must also satisfy the following constraints: (i the allocated number of GPUs for each type must not exceed the available number, i.e., 0 < Ecec (d, (c) x xye) S an, Vn = 1,..., N: (ii the total cost of all chosen configurations must be within
Configurations. We consider a set C of feasible configu- rations (i.c.. deployment configurations). Each config- uration e E C represents the serving plan for a single model replica, which is characterized by (Ve, Se, Oe, he:
'The detailed procedure of the one-time profiling is demon- strated Appendix I..
6

---

Extension to multiple LLM serving. Our MILP formula- tion can be easily adapted to scenarios involving multiple LLMs, such as simultaneously serving both Llama3-8B and Llama3-70B models. To accommodate this, we introduce model-type dimension to the decision variables and con- straints. This ensures that workload assignments, memory requirements, and other constraints are optimized for each (3) model type. The objective remains to minimize the overall Demystifying Cost-Efficiency in LLM Serving over Hetcrogencous GPUs
the user-defined budget B, i.e., Eeecloe x We) B.
layer partitioning based on memory allocation. See Ap- pendix D for details.
Optimization objective. We define a makespan variable T 1 0 to represent the overall completion time. For a con- figuration e, if it is instantiated Ve times and processes frac- tions Fe of workload w, each replica provides a through- put of howw- Let fu be the total number of requests for workload w. Consequently, the total effective throughput for is He x he,w and the time required on e is given by To 1.=-p - Feww-/ Since all chosen configurations run in parallel, the system completes once the slowest configu- ration finishes. Thus, we have T. < T for all c E C. Our optimization objective is to minimize T.
Binary search. To address the long computation times associated with the MILP solver for large-scale problems, we introduce binary-search-on-T method to accelerate the search process. Rather than directly minimizing the makespan T. we iteratively check whether a valid serving plan exists for different candidate values of T, based on reasonable lower and upper bounds. For full explanation, refer to Appendix F, and we evaluate the effectiveness of this binary search method in $5.2.
Ve-he..
MILP formulation. The problem can be summarized as the following Mixed-Integer Linear Program (MILP):
arg min'
s.t.
Ecec Fc,w 1, Vu, (Assignment Constraint)
(2)
Ewew Fe ST, Ve, (Makespan)
ye . hew
New Syo, Ve,w (Activation Coupling)
(4) makespan T. while also taking into account GPU avail- ability, budget constraints, and other constraints across all (5) model types. For detailed explanation of the formulation,
E cEC (oo x.) S B, (Budget Constraint)
Σ (da(c) x me)= S on, Vn, (GPU Avail.)
(6) please refer to Appendix E. We demonstrate the evaluation of our method for multiple model serving in $5.2.
Yc E 10,1,2.....
This formulation determines which configurations are used (ge) and how the workload fractions (Ir,w) are distributed, subject to memory limit, price budget, and GPU availability constraints, in order minimize the makespan T. Note that d,, (c) is an integer; enumerate all feasible integer combinations {c) in a precomputation step. In contrast, ze is continuous variable, and the solver relies on branch-and-boumt to systematically narrow the feasible region and converge to an optimal fractional assignment.
5. Experiments
5.1. Experimental Setup
Environments. Our experiments are conducted using two types of data center servers H1 0100 and A100, three types of work station servers A4 RTX A6000 and L40, and one type of consumer server RTX 4090. In data center servers, GPUs are linked by NVLink (300 GB/s), while in workstation/consumeer servers, GPUs are linked by PCIe (60 GB/s). Servers with inter-connection are connected via Ethernet with a bandwidth of 5 Gb/s. All experiments are conducted with vLLM (Kwon al., 2023).
Complexity analysis. The number of binary activation vari- ables Ye grows combinatorially with the number of feasible configurations (C). In this worst case [C] can be the order of (a, + 1). Since MILP solvers (e.g.. g.g branch-and- bound) have worst-case running time exponential in the number of binary variables, the theoretical worst-case time complexity scales as 0 (poly([C), W, x 211)), where the polynomial factor accounts for the overhead of processing each node in the search tree (e.g., solving continuous relax- ations of the MILP). As a result, the solution time escalates rapidly with the number of candidate configurations.
Baselines. We compare our method, which uses heteroge- neous cloud resources, against various homogeneous setups: Heterogeneous setups: We rent GPUs from Vast.ai, cloud provider offering range of GPU types. The rentals are based on real-time GPU availability on the cloud, For our experiments, we randomly selected four different GPU availabilities (shown in Table 4 in Appendix H) under varying price budgets of 15, 30, and 60 $/h.
Other constraints and heuristics. We introduce additional constraints and heuristics reduce the search space. Con- cretely, we enforce a memory constraint to eliminate config- urations with insufficient GPU memory, and a connectivity constraint to exclude those involving disconnected GPUs. Additionally, we refine the configuration search by restrict- ing TP to single machines and enabling non-uniform PP
Homogeneous setups: We rent H1 GPUs (representa- tive data center GPUs), RTX A6000 GPUs (workstation GPUs), and RTX 4090 GPUs (consumer GPUs) under different price budgets, with each GPU type representing a homogeneous baseline. For example, a budget of 60 S/h can rent up to 20 H1 GPUs Note that we fine-tune the deployment configurations and workload assignments us-
7

---

Demystifying Cost-Efficiency in LLM Serving over Hetcrogencous GPUs
ing our scheduling algorithm optimize the performance of each homogeneous baseline.
(Avail E)
(Avail 4)
RIX AS000 (Hoomo)
Ours (Avail 2)
RTX 4050 (Tome)
20100 (TFomo)
Ours (Aveil 3)
Models and traces. Our evaluation is conducted on Llama3- 8B and Llama3-70B models. And we follow prior work to generate workload traces based on real-world data. Our test- ing traces are subsampled from three sources: real workload traces collected over one month from the Swiss Al Center, the WildChat dataset, and the production traces Azure-Trace. Each trace comprises multiple workload types, with their ratios shown in Table 5 in Appendix I.
24
16
8
8
Trace 1. Budget 605h
Trace 1. Budget 30S/h
Trace 1, Budget 158h
Throughput (tegs
32
18%
16
109
24
12
16
0
TRAce Budget 605-h
Trace Budget 30$/h
Trace Budget 158h
Evaluation metrics. We focus on the overall system throughput and vairous percentile latencies (i.e., p10, p90. p100). P90 latency represents the maximum response time within which 90% of all requests are completed.
24
12
54%%
55%
16
8
8
Trace 3. Budget 605/h
Traco 3. Budget 305/h
Trace 3. Budget 15S/
5.2. Experimental Results
Figure 5: End-to-end throughput results on Llama3-70B model with different setups. We further demonstrate the Llama3-8B results in Appendix J.
End-to-end system performance. We evaluated our method across various traces, cloud GPU availability scenar- ios, price budgets, model types, and homogeneous baselines. Experimental results in Figure and Figure 6 show that our method improves system throughput by up to 41% % (25% on average) while reducing percentile latencies by up to 54% (20% on average). In traces 1 and 2, H100 (Homo) achieves the best performance among all baselines. our plan, the
Ours (Awail 4)
RTX
RTX A6000
11100
60
55
72
Latency (8)
45
40
52
30
15
10
GPU composition depends on the budget. With a high bud- get (60 $/h), data center GPUs are preferred, making up approximately 51 % of our rented resources for request pro- cessing. In contrast, with low budget (15 $/h), workstation GPUs are favored due to their lower cost. In trace 3, A6000 (Homo) demonstrates the highest performance among all baselines. In this scenario, our plan primarily relies on work- station GPUs, which constitute approximately 93% of the rented resources for request processing. Additionally, as shown in Figure 15 in Appendix J, the 4090 (Homo) delivers the best performance among all baselines for the Llama3-8B model. In this case, our plan prefers consumer GPUs, which form the majority of our rented resources and handle 53% of overall request processing.
P10:100
Latencies
P10:100 Latencies
P10:100 Latencies
Trace
Trace 2
Trace 3
Figure 6: End-to-end latency results on Llama3-70B model with different setups.
HexxGnn
HexGen (Optimal Composition)
Ours (Avail 4)
12
9%
Throughpu 8
8
35%
Trace 3,5/h
Trace
Trace 305/h
8
15%
8
a tJ c
30%
16%
6
27%
33%
Trace 155h
Trace 155th
Trace 155th
Comparison with HexGen. We also compare our method with the state-of-the-art heterogeneous serving framework, HexGen (JIANG et al.). Since HexGen schedules workloads based on a fixed GPU composition, we evaluate it using two setups: (i) a uniform composition, where six GPU types are evenly allocated within the budget, and (ii) the optimal composition used by our method. As shown in Figure 7, HexGen with a uniform composition suffers up to 35% and on average 29% performance degradation due to suboptimal GPU allocation. Even with the optimal composition, our method outperforms HexGen by up to 18% and on average 14%, benefiting from workload-awand scheduling.
Figure 7: Ours vs. HexGen. The first and second bars in each picture represent HexGen using a uniform and optimal GPU composition.
comparing our system with Helix (Mei et al., 2024). Specifi- cally, we compare our method against Helix's single-cluster case (the optimal case reported in their paper) under price budget of $15 per hour on the AzureTrace dataset (i.e., trace 2). While Helix optimizes heterogeneous LLM serving us- ing max-flow and MILP scheduling, our method explicitly considers workload heterogeneity and GPU composition op- timization, resulting in greater cost efficiency. Experimental
Compare with Helix. We conduct additional experiments
8

---

Demystifying Cost-Efficiency in LLM Serving over Hetcrogencous GPUs
Uniform GPU Composibon
Utifom Depleyment Configuration
MILP
MILP
Rule-hased Request Assignment
Curr (Avvil 4)
Binary Search
Binary Search
Throughya (10g5)
Search Time (s)
115
118
80
Makospan maan
15
45
12
Trace Budget 305/h
Traco Budget 155/h
Trace Budget 305/
T Budget 155/h
10
8
16
24
32
48
0
40
80
120
160
Num. of GPUs
Search Time (s)
Figure 8: Ablation study of Llama3-70B on traces and 2.
Figure 9: Algorithm scalability and efficiency.
results show that our system outperforms Helix by 25-35%.
Ours (Avail 1)
RTX 4090
RTX A6000
H1000
Throughnut (regis
Table 3: Performance comparison of Helix and our method.
96
20%%
48
%69
35%
72
36
Llama-30B
Llama3-70B
48
24
Helix
8.49 req/s
5.72 req/s
24
12
Ours
11.43 req/s (35%1)
7.13 req/s (25%1)
Trace 1, Budget 605/h
Trace 1, Budget 30S/h
Ablation studies. We assess the impact of each optimiza- tion target in our scheduling algorithm by systematically disabling them. Three baselines are considered: (i Uniform GPU composition: GPUs are rented uniformly across six types within the given budget. This evaluates the perfor- mance gains from optimized heterogeneous GPU compo- sition. (ii) Uniform deployment configuration: Instead of optimizing deployment per model replica, TP is uniformly applied across all replicas. This measures the impact of deployment configuration optimization. (iii) Rule-based request assignment: Requests are assigned using a Round- Robin strategy based on real trace arrivals, assessing the workload assignment. As
Figure 10: End-to-end experiments on multiple model types (Llama3-8B and Llama3-70B) with different sctups.
8B model, while the remaining 20% are assigned to the Llama3-70B model. As shown in Figure 10, our method outperforms homogeneous baselines, achieving up to 35% (average: 23%) performance gain. In the 60 $/h case, our scheduling algorithm allocates 70% of computing resources to Llama3-706 and 30% to Llama3-8B. In the 30 $/h case, the allocation shifts to 77% and 23%. Our algorithm bal- ances resource allocation based on model demands, enabling efficient multi-model serving in heterogeneous clusters.
benefit of heterogeneous-avvzz shown in Figure 8, disabling heterogeneous GPU compo- sition reduces throughput by up 27% (average: 20%), deployment optimization by up to 34% (average: 33%), and workload assignment by up to 32% (average: 29%). These results highlight the necessity of each optimization for high- performance LLM serving in heterogencous environments.
System performance price budget. We compare system with homogeneous baselines under different price budgets. The performance gap narrows as the budget in- creases due to cloud resource limits, which is reasonable since we assume an unlimited number of GPUs for our homogeneous baselines. See Appendix K for details.
Algorithm efficiency. We evaluate two strategy search methods from $4: (i) MILP and (ii) binary search. As shown in Figure 9, the left plot illustrates their scalability, while the right plot depicts algorithm performance during the search process. Compared to MILP, which exhaustively explores all combinations of heterogeneous GPU compositions, de- ployment configurations, and workload assignments, the binary search method, enhanced with feasibility checks us- ing knapsack approximation, achieves approximately 4x reduction in search time. This improvement comes with only marginal differences in algorithm performance, with deviations of less than 1 %.
Additional discussion and experiments. We provide fur- ther discussion and experiments on online scheduling for dynamic workloads, as well as on the trade-offs between cost-efficiency and request latency, in Appendix M.
6. Conclusion
This paper aims to address the questions of why and how heterogeneous cloud resources can be utilized for cost- efficient L.M serving. Specifically, we benchmark the cost-efficiency of LLM serving over heterogeneous GPUs, following which, a novel scheduling algorithm is developed. Experimental results demonstrate that our approach outper- forms existing works substantially.
Multi-model extension. We further evaluate our system in a multi-model serving scenario (discussed in $4.3), as- suming that 80% of the requests are assigned to the l.ama3-

---

Demystifying Cost-Efficiency in LLM Serving over Hetcrogencous GPUs
Impact Statement
He, G. and Yoneki, E. Cuasmrl: Optimizing gpu sass schedules via deep reinforcement learning. In Proceed- ings of the 23rd ACM/IEEE International Symposium on Code Generation and Optimization, CGO '25, PP. 493-506, New York, NY, USA, 2025. Association for Computing Machinery. ISBN 9798400712753. doi: 10. 00.1145536994433376694 URL https: : /doi. org 10. 1e..:..1113199910 3708943.
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Acknowledgment
This work is supported by the HKUST startup grant R9895 from CSE; RGC-ECS project 26218024; RGC-NSFC project CRS HKUST601/24. This work was supported as part of the Swiss AI Initiative by grant from the Swiss National Supercomputing Centre (CSCS) on Alps.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, Measuring massive multitask language understanding. In International Conference on Learning Representations.
Hu, C., Huang, H., Xu, L.., Chen, X., Xu, J., Chen, S., Feng, H., Wang, C., Wang, S., Bao, Y., et al. Inference without interference: Disaggregate Ilm inference for mixed down- stream workloads. arXiv preprint arXiv:2401.11188 2024.
References
Agrawal, A., Kedia, N., Mohan, J., Panwar, A., Kwatra, N., Gulavani, B. S., Ramjee, R., and Tumanov, A. Vidur: A large-scale simulation framework for Ilm inference. Proceedings of Machine Learning and Systems, 6:351-
Huang, Y., Cheng, Y., Bapna, A., Firat, O., Chen, D., Chen, M., Lee. H., Ngiam, J., Le, Q. V., Wu, Y., al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. Advances in neural information processing systems, 32, 2019.
366, 2024a.
Agrawal, A., Kedia, N., Panwar, A., Mohan, J., Kwatra, N., Gulavani, B., Tumanov, A., and Ramjee, R. Taming (Throughput-Latency } tradeoff in (LLM} inference with (Sarathi-Serve), In 18th USENIX Symposium Operat- ing Systems Design and Implementation (OSDI 24). pp. 117-134, 2024b.
Jeon, J. and Lee, S. Large language models in education: A focus on the complementary relationship between hu- man teachers and chatgpt. Education and Information Technologies, 28 (8(12):157733111
2023.
Anthropic. The claude model family: Opus, sonnet, haiku, 2024. URL https:
ant hwt.::onnn..tttiiiiii..ooop com
Jiang. A. Q. Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., Casas, D. d. I., Hanna, E. B., Bressand, F,, et al. Mixtral of experts. arXiv preprint 2411:22401.04088, 2024.
57 627 /Mdde/
Azure. Azure public dataset, 2024. URL URL htttt:/t////// //gith ub. httmm//Arrre//arreeulceeeeeee
JIAGG, Y., Yan, R., Yao, X., Zhou, Y., Chen, B., and Yuan, B. Hexgen: Generative inference of large language model over heterogeneous environment. In Forty-first Interna- tional Conference on Machine Learning.
Borzunov, A., Baranchuk, D., Dettmers, T., Riabinin, M.,
Belkada, Y., Chumachenko, A., Samygin, P., and Raffel, C. Petals: Collaborative inference and fine-tuning of large models. In Proceedings of the ó1st Annual Meeting of the Association for Computational Linguistics ( VVllue 3: System Demonstrations), pp. 558-568, 2023.
Jiang, Y., Fu, F., Yao, X., Wang, T., Cui, B., Klimovic, A., and Yoneki, E. Thunderserve: fiig--performacce and cost-efficient 1lm serving in cloud environments. arXiv preprint arXiv:2502.09334 2025a.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The Ilama herd of models. arXiv preprint arXiv:2407.21783, 2024.
Jiang, Y., Yan, R., and Yuan, B. Hexgen-2: Disaggregated generative inference of llms in heterogeneous environ- ment. arXiv preprint arXiv:2502.07903. 2025b.
GitHub. The world's most widely adopted ai developer tool, 2024. URL https: wtwpw/www..oo..atteeeeeeeeeeeeee features/ copilot.
Kwon, W., Li, Z., Zhuang, S., Sheng. Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Sym- posium on Operating Systems Principles, pp. 611-626, 2023.
Griggs, T., Liu, X., Yu, J., Kim, D., Chiang, W.-L,, Che- ung, A., and Stoica, I. M elange: Cost efficient large language model serving by exploiting gpu heterogeneity. arXiv preprint arXiv:2404. 14527, 2024.
10

---

Demystifying Cost-Efficiency in LLM Serving over Hetcrogencous GPUs
Li, H., Fu, F., Ge, H., Lin, S., Wang, X., Niu, J., Miao, X., and Cui, B. Hetu v2: A general and scalable deep learning system with hierarchical and heterogeneous sin- gle program multiple data annotations. arXiv preprint arXiv:2504 20490, 2025.
Parke, A., Reddy, D., Jha, S., Qiu, H., Pinto, C., Narayanaswami, C., Kalbarezyk, Z.,, and Iyer, R. Queue management for slo-oriented large language model serv- ing. In Proceedings of the 2024 ACM Symposium on Cloud Computing, pp. 18-35, 2024.
Li, Z., Zheng, L., Zhong, Y., Liu, V., Sheng, Y., Jin, X., Huang, Y., Chen, Z., Zhang. H., Gonzalez, J. E., et al. (AlpaServe): Statistical multiplexing with model paral- lelism for deep learning serving. In 17th USENIX Sympo- sium on Operating Systems Design and Implementation (OSDI 23). pp. 663-679, 2023.
Peng, C., Yang, X., Chen, A., Smith, K. E., PourNejatian, N., Costa, A. B., Martin, C., Flores, M. G., Zhang, Y., Magoe, T., et al. A study of generative large language model for medical research and healtheare. NPJ digital medicine, 6(1): 2023.
Peng. Y., Jiang, Y., Wang, C., and Yuan, B. Hexgen-tex(2sql: Optimizing 11m inference request scheduling for agentic text-to-sql workflow. arXiv preprint arXiv:2505.0522 055288 2025.
Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., et al. Deepseek- v3 technical report. arXiv preprint arXiv:2412.19437, 2024a.
Qiao, Y., Anzai, S., Yu, S., Ma, H., Wang, Y., Kim, M., and Xu, H. Conserve: Harvesting gpus for low-latency and tightthroughput large language model serving. arXiv preprint w:Xiv:2410.01222 2024.
Liu, Y., He, H., Han, T., Zhang, X., Liu, M., Tian, J., Zhang, Y., Wang, J., Gao, X., Zhong, T., et al. Understanding Ilms: A comprehensive overview from training to infer- ence. arXiv preprint arXiv:2401.02053: 2024b.
Qin, R., Li, Z.., He, W., Zhang, M., Wu, Y., Zheng, W., and Xu, X. Mooncake: Kimi's ss kvcache-centric architecture for Ilm serving. arXiv preprint urXiv:2407.000
Liu, Z., Wang, J., Dao, T., Zhou, T., Yuan, B., Song, Z., Shrivastava, A., Zhang, C., Tian, Y., Re, C., et al. Deja Contextual sparsity for efficient Ilms at inference time. In International Conference on Machine Learning, pp. 22137-22176. PMLR, 2023.
2024.
Reid, M., Savinov, N., Teplyashin, D., Lepikhin, D., Lilli- crap, T., Alayrac, J.-b., Soricut, R., I Lazaridou, A., Firat, O., Schrittwieser, J., et al. Gemini 1.5: Unlocking multi- modal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05532 2024.
Mei, Y., Zhuang. Y., Miao, X., Yang, J., Jia, Z., and Vinayak, R. Helix:
Distributed serving of large language models via max-flow on heterogeneous gpus. arXiv preprint arXiv:2406.00 01.0015566 566, 2024.
Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., and Catanzaro, B. Megatron-Im: Training multi- billion parameter language models using model paral- lelism. arXiv preprint arXiv:1909.0800 080000,,003,.... 2019.
Miao, X., Shi, C., Duan, J., Xi, X., Lin, D., Cui, B., and Jia, Z. Spotserve: Serving generative large language models on preemptible instances. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, pp. 1112-1127, 2024.
Stojkovic, J., Zhang, C., Goiri, f. Torrellas, J., and Choukse, E. Dynamollm: Designing Ilm inference clusters for performance and energy efficiency. arXiv preprint 2024.
Oh, H., Kim, K., Kim, J., Kim, S., Lee, J., Chang, D.-s., and Sco, J. Exegpt: Constraint- aware resource schedul- ing for IIm inference. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Vol- ume 2, pp. 369-384, 2024.
Sun, B., Huang, Z., Zhao, H., Xiao, W., Zhang. X., Li, Y.. and Lin, W. Llumnix: Dynamic scheduling for large language model serving. In 18th USENIX Symposium Operating Systems Design and Implementation (OSDI 24), pp. 173-191, 2024.
Wang. Y., Chen, Y., Li, Z., Kang, X., Tang, Z., He, X., Guo, R., Wang, X., Wang. Q., Zhou, A. C., et al. Burstgpt: A real-world workload dataset to optimize Imm serving systems. arXiv preprint arXiv:2401 176444444, 2024a.
OpenAI. Openai gpt-4o, 2024. URL https:// rm. openai. com docs/ / / hwwe/ooeelllle secoodlls/geeg (pppte/44s/pp/oooo 4o.
Patel, P., Choukse, F., Zhang, C., Shah, A., Goiri, f., Maleki, S., and Bianchini, R. Splitwise: Efficient generative Ilm inference using phase splitting. In 2024 ACM/IEE 51st Annual International Symposium on Computer Architec- ture (ISCA), pp. 118-132, IEEF, 2024.
Wang. Y., Chen, Y., Li, Z., Tang, Z., Guo, R., Wang, X., Wang, Q., Zhou, A. C., and Chu, X. Towards efficient and reliable Ilm serving: A real-world workload study. arXiv preprint arXiv:2401. 2024b.

---

Demystifying Cost-Efficiency in LLM Serving over Hetcrogencous GPUs
Wu, B., Zhong, Y., Zhang, Z., Huang, G., Liu, X., and Jin, X. Fast distributed inference serving for large language models. arXiv preprint arXiv: 05920, 2023.
Zheng, L., Chiang, W.-L.,, Sheng, Y., Li, T., Zhuang, S., Wu, Z.., Zhuang. Li, Z.,, Lin, Z., Xing, E., al. Lmsys-chat- Im: A large-scale real-world Ilm conversation dataset. In The Tweifth International Conference on Learning Representations.
Yan, R., Jiang, Y., Tao, W., Nie, X., Cui, B., and Yuan, B. Flashflex: Accommodating large language model training over heterogeneous environment. arXiv preprint arXiv:2409.01 143, 2024.
Zhong, Y., Liu, S., Chen, J., Hu, J., Zhu, Y., Liu, X., Jin, X., and Zhang, H. (DistServe): Disaggregating prefill and decoding for goodput- poodput-optimized large language model serving. In 18th USENIX Symposium on Operating Sys- tems Design and Implementation (OSDI24) 24), pp. 193-210, 2024.
Yu, G.-I,, Jeong, J. S., Kim, G.-W., Kim, S., and Chun, B.- G. Orca: A distributed serving system for (Transformer- Based] generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI
Zhou, Z., Wei, X., Zhang, J., and Sun, G. (PetS): A unified framework for | [Parameter-Efficient] transformers serving. In 2022 USENIX Annual Technical Conference (USENIX ATC 22), pp. 489-504, 2022.
22), pp. 521-538, 2022.
Zhang. J., Huang, H., Zhang, P. Wei, J., Zhu, J., and Chen, Sageattention2: Efficient attention with thorough outlier smoothing and per-thread int4 quantization. In Interna- tional Conference on Machine Learning (ICML), 2025a.
Zhang, J., Wei, J., Zhang, P., Xu, X., Huang, H., Wang, H., Jiang. K., Zhu, J., and Chen, J. Sageattention3: Microscal- ing fp4 attention for inference and an exploration of 8-bit training, XrXiv preprint arXiv:2505. 1594, 2025b.
Zhang, J., Wei, J., Zhang, P., Zhu, J., and Chen, J. Sageatten- tion: Accurate 8-bit attention for plug-and-play inference acceleration. In International Conference on Learning Representations (ICLR), 2025c.
Zhang. J., Xiang, C., Huang, H., Wei, J., Xi, H., Zhu, J., and Chen, J. Spargeattn: Accurate sparse attention accelerat- ing any model inference. In International Conference Machine Learning (ICML), 2025d.
Zhang, J., Xu, X., Wei, J., Huang, H., Zhang. P., Xiang, C., Zhu, J., and Chen, J. Sageattention22+:: A more efficient implementation of sageattention2. arXiv preprint arXiv:2505.21136. 2025e.
Zhao, J., Wan, B., Peng, Y., Lin, H., and Wu, C. Llm-pq: Serving Ilm on heterogeneous clusters with phase-aware partition and adaptive quantization. the 29th ACM SIG- PLAN Annual Sympo-sine Principles and Practice of Par- allel Programming [Pb0PP44)0223/2024400000000 Edinburgh), 2024a.
Zhao, W. Ren, X., Hessel, Cardie, C., Choi, Y., and Deng, Y. Wildchat: Im chatgpt interaction logs in the wild. In The Twelfth International Conference on Learning Representations.
Zhao, Y., Yang, S., Zhu, K., Zheng, L., Kasikci, B., Zhou, Y., Xing, J., and Stoica, I. Blendserve: Optimizing offline inference for auto-regressive large models with resource- aware batching. arXiv preprint 1..Xi:241111611000 2024b.
12

---

Demystifying Cost-Efficiency in LLM Serving over Hetcrogencous GPUs
A6000
2455, Our: 18
Ang In: 2455, Out 253
Avr In: 2435, Out: 510
In: 2455. Out 18
2455, Out: 253
2415, Out
22 1000
2E-00-
28-05
10:00
10:004
K0-41-
16-0
11-45-
31-47
Throughpan Per Dollar (rea/i)
4E:01
41
LC:
KB:44
1E-08
25:49
15-01-
Avg.
Out
5E-49
Out:
45:07
In:
46-01-
[E:42
In: 495,
(5)
10
4i 81-
28:02
$2:45
70 10
100
Il: 02-00-
18-07
16-0--
52.40
32 10p
12 100
6e-90
Avg In: 824, Out:
Arg In 924, Out
Avr In: 824, Our: 310
Ave In: Out
Aug 12; Out 253
Avg In: 824, Out: $10
66-02
10-40-
35-40
(5401-
45.04
i6 18-49-
1b.07
15:00-
12:42-
t: 81-
25-0
45-01-
65:03
te:
Different GPU Types
Different GPU Types
Different GPU Types
P5-100 Latencies
P5-100 Latencies
P5-100 Latencies
Figure 11: Benchmarked results for Llama3-8B model with different GPU types on different workload types.
A. Benchmarking Results for Llama3-8B
We demonstrate the benchmark results for Llama3-8B model in Figure 11.
B. Benchmarking Results of Different Deployment Configurations for Remaining GPUs
We demonstrate the benchmark results for different deployment configurations in Figure 12 and Figure 13.
C. Simple Example
Experiment setup. We begin by assuming three GPU types, {f1, f2, (a), each with two units available. The hourly rental prices for these types are 2, and 2 S/h. respectively. We consider two workload types {w,, W2), which arrive simultaneously with 80 total requests for wi (A1 80) and 20 total requests for w2 (A2 20). We denote by Cr the throughput (in requests per second) of GPU type
workload If each GPU serves one model replica, the throughputs are C1,1 1.0, C 1.2, C2 = 0.9, C2 0.9, C3,1 = 0.3, and C3.2 = 0.5. Note that C,,,1 and C=.2 vary with model parallelism. In Cases 1 and 2, we assume the workload is assigned to each GPU in proportion to its processing rate, so the system-wide throughput for each workload the sum of individual- rates. In Case 3, we allow workload-awee aware assignment for further optimization.
Case 1: GPU composition. We compare two compositions under the same budget of 8 $/h, where each GPU is responsible for serving one model replica. Composition I uses 1 f,, 1 12, and ta. This setup achieves a total throughput of (1.0+0.9+ + = 2.2 rps on wi and (1.2-0.9+0.3 (80/2.2 + 20/2.6) 44.05 s. Composition 2 uses 1 t and 2 x 12. for throughputs of (1.0 + 0.9 + 0.9) 2.8 on w1 and (1.2 + 0.9 + 0.9) = 3.0 on w2, so (80/2.8 + 20/3.0) Rs 35.24s. s. this case, changing the GPU composition under the same price budget results in 20% speedup.
2.6 w2, giving processing time of (>1/C=,1
Case 2: Deployment configuration. Focusing composition 2, compare two ways to organize these three GPUs. Configuration keeps all GPUs in purely DP style (i.e., each GPU is responsible for serving one model replica), summing up to 2.8 on wi and rp on w2, matching the 35.24s s above. Configuration 2 applies TP to the two to GPUS, which changes their combined rate, e.g., to rps on w, and 1.5 on W2. Together with the single t, GPU (1.0rps on Wi and 1p2 on W2), the total throughput becomes (3.4, 2.7) rps for (W1, W2). The corresponding time (80/3.4 + 20/2.7) x 30.94s. s. In this case, changing the deployment configuration results in an improvement in the overall processing time of roughly 14%.
Case 3: Workload assignment. Finally, we keep the same composition and Ta-based configuration but allow workload- aware assignment. Concretely, we assign:
Replica (f:): 15% of w1, 100% of w2,
Replica (TP on 2 x t2): 85% of wi.
13

---

Demystifying Cost-Efficiency in LLM Serving over Hetcrogencous GPUs
(1AA1)
(1.2.4)
(1,42))
(1.1.)
16.01
16.0
15.01
15.0
15.0
9642
4E40
TE:0
6142
3041
10:2
4E-22
Throughgus Per Dollur (reqk)
L40-0.2A)
L.00-1144221
L40-11,1.8)
L40-22114)
L40--2,431)
140-22221
Xc-11
1140
20:00
TE41
43:00
21:01
(6.1
(cal
ska
24-1
804F
((00-(1A.))
H1000-12222)
H100-C.1A)
H100-22112)
H100-22.11
4641
I14)
1041
12.00
4241
1e42
2540
4140
11:0
Difffonen Configs Aug In:3455, Oui:18
Difforent Configs
Differemt Configs
Difforence Configs In:496, OuE
Different Configs In:496, Out253
Differont Configs Avg In: Oui:51
Differenc Configs In:824, Out
Diéferent Configs In:R24, Out253
Differont Configs In:924, Ou:510
Aug In:455,, Oui:23
Avg 14:2555, 11:
4590 (1,8.3)
4080 (.244)
4500 (1.42)
4090 (1.1.))
16-11
48-07
71-41
20-42
45422
21-91
40-00
2/90
CL18)
L/0
LA0
20-41- N0-0t-Snnnn
12.00
52-2
12.00
22.0
[E:1
2E.00
18-01-
31-07
02.00
11100-1.2.2)
H100--21122
(1100-222111
I7:01
27:00
10-22
$E-00
IT:31
11-2-
18:0
Latencies 49:
Laneocies Aug In:244, Out
Latencies
Aug In:4455, Out:18
Aug I4:4555, Out:25
Aug 14:2455,
Aug In:495, CuE225
In:824, Out
OutSin
Figure 12: Throughput and latency results for Llamm3-70B model with different deployment configurations on different workloads.
Under these fractions, 4 processes 12 requests of wi 1.0 and 20 requests of w2 1.2 rps, while the TP-based replica handles 68 requests of w1 2.4 rps. By balancing the load and routing the workload to the preferable replica, i... the one with relatively higher throughput for specific workload, we reduce the overall completion time from 30.94 to maz(0.855A, ,, /C1,, + A2/Ci,1) = mar(6882.44 12/1 + 20/1.2) = 28.67 s. In this case, changing the workload assignment results in an additional improvement in the overall processing time of approximately 8%.
rps.
This step-by-step example (also illustrated in Figure 14) shows how all three factors GPU composition, deployment configuration, and workload assignment must be jointly optimized achieve the best performance.
D. Other Constraints and Heuristics
We enforce two additional constraints to minimize the overall search space and speed up the search process: i We perform an early memory check on each configuration, which ensures that the sum of GPU memories in configuration is sufficient for model replica, i.e., Σm (d,, (c) x m,)) 2 My. where M, represents the least memory required for serving one model replica (e.g., 140 GB for 1-amma--7BB model). Configurations that violate this constraint will be eliminated from further evaluation; (ii we enforce a connectivity constraint within each configuration. If certain GPUs lack interconnection (e.g.. they located in different data centers), those combinations do not appear in each configuration Additionally, two heuristic methods to facilitate the deployment configuration search: (i) we only adopt TP within a single machine containing multiple GPUs, as TP typically requires high intra-machine communication bandwidth (c.g., PCIe, NVLink) for efficient deployment; (ii) we support non-uniform pipeline layer partitioning for PP, and determine the partition based on the total memory allocated for each stage. For instance, if there are total of 24 layers and the GPU memory allocated for each stage is 1:2, then we allocate 8 and 16 layers to the first and second stages.
14

---

Demystifying Cost-Efficiency in LLM Serving over Hetcrogencous GPUs
A4H-224
AMH-(1442
254
20-11
1b00
154H
W.a
Thooughpa Per Dollar (reqis)
47-08
K0:00
CE-30
(1,8.)
(1.2.4)
(1,42)
(1.1.8)
SC41
16-11
12:0:
tover
45-11
21-07
X041
20:11
2041
[b-l
12-12
s041
1p-11
3540
2241
A100-(1AA1)
A100-12.2)
A100-011A)
A100-2.122
A100-2,11)
2641
40-1
4E-04
4C41
3541
12-10
1541
45-1
IE:0
5541
X441
20-11
Difforent Confags
Ditforent
Differont
Difforenz Configs
Difforent Contas
Differont
Duflecent Configs
Difforent Configs
Ditfereat Contless
M4 (LA.))
A1D (2.24)
A40 U.A.)
A/9 (1.1.)
2041
26.01
10-41-
1641-
15.01
A000-1442
Total Price (3)
20-4
16-01
18:31
30:41
21-4-
16-01
11.31
11.41
1041
16-01
2042
10--2
11411
PS-100
PS-
PS-100
In:4455,
In:4455, Out255
In:2555, Oun:10
In:096,
OuE253
Im:844,
Ouu530
Figure 13: Throughput and latency results for Llama3-70B model with different deployment configurations on different workloads.
E. Extend to Multiple LLM serving
The previous MILP formulation assumes single LLM serving with multiple model replicas. However, cloud services typically involve multiple LLM serving with varying sizes, c.g., Llama3-8B and Llama3-70B models. To integrate multiple LLM serving plan search into our MILP, we introduce the following extended MILP formulation.
Let there be M model types, indexed by m E (1,,...., each type has its own memory requirement. The MILP formulation can be extended
arg min T
(8)
Eeecm Towrm = Vu Wm: s.t. Vm: Ewew Rew.mifo. S T, AC Cy.
(9)
Nemmhe..umm
Tow.m I No,m: VCE Cm, Vw Wm.
M Σ m=1 Exse cece.. (oo. Vem) < B,
(10)
Σ m=1 Eeso (dolc, m) x yom)) S an. Vn, Ne.m 1,,.......
(11)
(12)
In this extended MILP formulation, we introduce an additional model-type dimension to every relevant variable and constraint. Consequently, the problem now accommodates multiple model types (each with own workload set, throughput
profiles, memory requirements, etc.) within unified optimization framework. The objective remains the same minimizing
15

---

Demystifying Cost-Efficiency in LLM Serving over Hetcrogencous GPUs
Workload Type
Workload Type 2
Tensor Parallelism
GPU Type?
GPU Type 2
GPU Type
S
R1
2013
R1
R1
R1
103
R2
R2
00:00
R3
R3
Naive Serving Plan
Optimize Compusition (Case
Optimize Configuration (Case 2)
Optimize Assignment (Case
Figure 14: Illustration of simple example.
Algorithm 1 Binary Search on T
Input: T. T finitial bounds}
Input: 7 (tolerance)
Output: Approximate minimal feasible makespan while (T- > 7 do
T T+T
if FEASIBILITYCHECK(( is true then
T + T (If feasible, try smaller }
else
T î (If infeasible, increase T)
end if end while return? T
the overall makespan while jointly enforcing GPU availability, budget, and other constraints across all model types. This ensures that the chosen configuration set and workload assignments meet the demands of every model type while adhering to the total GPU and budget limits.
F. Binary Search
For large numbers of model, workload and GPU types, it might take hours for the MILP solver to provide relatively good solution. To expedite the search process, incorporate the binary-seareh-on-T approach into existing MILP formulation. Specifically, we transform the previous "minimize T" problem into sequence of feasibility checks: for given candidate T, we ask whether a valid serving plan exists that completes all workloads in at most T, subject to budget and GPU constraints. If yes, we can try smaller 7'; if no, we must increase T.
Binary search. The lower bound of the makespan, T, is identified as the best possible time if infinite GPUs were available with no budget limit (e.g., using the fastest configuration for each workload type). The upper bound, T, is the worst-case scenario (e.g., using the slowest feasible configuration serve all workloads). During the binary search loop, if the difference between the lower and upper bounds exceeds certain tolerance (e.g., second), i.c., T-T 2 т, calculate 2
/ T+T and check its feasibility. If a pair (Ie,w> or (Ie, me Me.m) (in the extended case) satisfies all constraints in Fe.w.m T, Ve Cm. we update T -r. Otherwise, we update T. No.N T or Enew... Pee,ur
$4.3 or $4.3, with Vwew New
When the loop concludes, the value of T (or T) represents the minimal feasible makespan within the specified tolerance. Note that the feasibility check can be further approximated using knapsack approximation, which makes the binary search approach more efficient for handling large-scale MIIP problems. We outline the binary search process in Algorithm 1.
Other optimizations for speeding up MILP. For extremely large-scale MILP problems (e.g., dozens of model and workload types with hundreds of GPUs), we introduce several optimizations, such as pruming configurations, providing good starting point, and early stopping based on the lower bound, as detailed in Appendix G. The experimental results presented in $5
16

---

Demystifying Cost-Efficiency in LLM Serving over Hetcrogencous GPUs
Ours (Avail 2)
RTX 4090 (Hame)
H100 (Homo)
Ouri (Avail 4)
RTX A6000 (Homo)
200
100
13%
150
75
%
Throughput (reqis)
100
50
50
25
0
0
Trace Budget 30S/h
Trace Budget 15$h
200
150
75
100
50
25
0
Trace 2, Budget 30S/h
Trace 2, Budget 15$/h
Figure 15: End-to-end experiments on Llama3-8B model with different setups.
Ours (Avail
RTX 6000 (Homo)
RTX 4090 (Home)
(Homo)
750
600
Makespan
450
300
150
0
5
10
15
20
25
30
35
40
45
50
55
60
Price Budgets (S/h)
Figure 16: System performance v.s. price budget.
demonstrate the efficiency, effectiveness, and scalability of our scheduling algorithm.
G. Other Optimizations for Speeding up MILP
For large numbers of GPUs and model types, it might take hours for the MILP solver to provide a relatively good solution. To expedite the search process, we introduce three optimizations to minimize the search space without sacrificing the effectiveness of our scheduling results: (i) for each model type, we prune configurations that are clearly dominated. For example, anfigurations with high degrees of model parallelism are retained for Llama3-70B, which requires substantial memory for model serving, but are pruned for Llama3-8B to prevent excessive communication overhead; (ii) pre-estimate the resource requirements for each model type based on incoming workloads and their memory demands, and proportionally allocate resources to provide good starting point for the MILP solver, thereby expediting the search process; (iii) we establish a theoretical lower bound for the makespan by analyzing the minimum possible processing time across all feasible configurations, which enables the implementation of early stopping criteria during optimization, i.e., the search process stops when it finds a solution that is very close to this lower bound. The minimum possible makespan occurs when all workloads are assigned the most efficient configuration without considering resource constraints.
H. Real Time GPU Availabilities
We randomly selected four real-time GPU availabilities on the cloud, shown in Table 4.
I. Workload Type Ratios for Each Trace
We demonstrate the workload type ratios for the three traces in Table 5.
17

---

Demystifying Cost-Efficiency in LLM Serving over Hetcrogencous GPUs
Table 4: Real time GPU availabilities on cloud platform.
GPU Avails
4090
A40
A6000
1,40
A100
H100
Avail 1
16
12
8
12
6
8
Avail 2
32
8
16
16
7
12
Avail 3
32
16
8
8
32
8
Avail 4
24
24
24
16
4
8
Table 5: Workload type ratios for subsampled traces from the Swiss Al Center (Trace 1), Azure-Trace (Trace 2), and
WildGPT dataset (Trace 3). Workloads 1-9 correspond to the nine workload types shown Figure from left to right.
Workloads
1
2
4
5
6
7
8
9
Trace I (%)
33
7
8
7
27
6
6
3
3
Trace 2 (%)
22
5
21
5
5
19
6
12
Trace 3 (%)
4
1
3
20
27
25
15
J. End-to-end Experiment Results for Llama3-8B Model
The end-to-end experiments on Llama3-8B model with different setups are shown in Figure 15.
K. System Performance vs Price Budget
We further evaluate our system's 's performance compared homogeneous baselines under various price budgets. As shown in Figure 16, as the price budgets increase (from 5 S/h 60 S/h), the performance gap between approach and the homogeneous setups narrows from approximately 30% to 15%. This is primarily due to the limited availability of cloud resources. In homogeneous baselines, we assume unlimited number of GPUs, allowing performance to scale linearly with the price budget. However, in cloud-based scenarios, resource restrictions prevent such linear scaling. When larger price budgets are applied, unsuitable GPUs for the current workload may be rented they are the only available options, further limiting performance scalability.
Table 6: Performance of different configurations.
Diff Configs
Real
Estimated
H100 (2,4)
0.56 req/s
0.60 req/s
H100 (4,2)
0.44 req/s
0.47 req/s
H100 (4,2) (cross machine)
0.42 req/s
0.44 req/s
L40 (2,4)
0.42 req/s
0.46 req/s
L40 (4,2)
0.21 req/s
0.22 req/s
L40 (4,2) (cross machine)
0.18 req/s
0.19 req/s
H100+A100 (4,2) (cross machine)
0.48 req/s
0.52 req/s
L. One-Time Profiling
We employ one-time profiling strategy that captures the following components. This approach referred to the profiling method used Vidur (Agrawal et al., 2024a): (1) Inference prefilling latency: We profile the latency for a single transformer layer across varying TP degrees, different workload types, and various GPU types; (2) inference decoding latency: We profile the decoding latency for single transformer layer under similar variations in TP degrees, workload types, and GPU types; (3) pipeline communication latency: We measure the communication latency between different GPUs across various workload types. Using these measurements, the per-request latency for any configuration is estimated by combining the TP costs (both communication and computation) of all layers-wwe which may be served by different GPUs and at varying TP degrees-wind with the PP communication cost. Note that our heuristics, as discussed in Section 4.3 and Appendix D, largely reduce the profiling space, e.g., TP is employed only intra-machine. When estimating throughput, the prefill and decoding
18

---

Demystifying Cost-Efficiency in LLM Serving over Hetcrogencous GPUs
phases are treated separately: (1) The prefill phase is compute-bound, and its batched processing capacity is determined by the sum of the individual latencies; (2) the decoding phase is memory-bound, with its batched processing capability defined by single latency value. This distinction has been validated in several studies (Zhong et al., 2024; Patel et al., 2024).
Table 6 demonstrates examples of our cost estimation under long-input, short-ouiput workload (i.e., workload 1 in Figure 4), In Table 6, the notation (2,4) indicates that the TP degree is and the PP degree is 4. The estimation errors range from 4% to 7%. Although the estimations are not perfectly accurate, they are sufficiently reliable for selecting the optimal configurations.
M. Discussion
Online replanning. Online scheduling for dynamic workloads is orthogonal to the approach presented in this work. However, accommodating dynamic workloads could be achieved through the implementation of replanning mechanism analogous to the one proposed in DistServe (Zhong et al., 2024). Concretely, the system could (1) monitor the real-time time composition of incoming workloads, (2) track GPU resource availability within the cloud environment, and (3) upon detecting a significant shift in workload distribution, (e.g., an increase in the proportion of certain workload types) the scheduling algorithm could be executed again, incorporating recent historical data to produce an updated serving plan.
Table 7: Performance changes in workload and GPU drop.
Workload Change
GPU Drop
Before
26.89 req/s
26.89 req/s
After
23.70 req/s (13%1)
20.80 req/s (29%.L)
Replanning
29.61 req/s (25%1)
22.85 req/s (10%1)
As shown in Table 7, we test the workload surge in short output requests in AzureTrace dataset (i.e., trace 2) with price budget of $30 per hour. Before surge, the optimal GPU composition is f20%, 65%, 15%} for datacenter, workstation, and consumer GPUs, achieving 26.89 req/s. After workload change, the throughput degrades to 23.7 req/s. In this case, replanning (shifting allocation to {63%. 23%, 14%}) boosts throughput to 29.61 req/s. We also test the case when GPU drop happens (4 H100s down), the throughput falls from 26.89 to 20.80 req/s. In this case, replanning raises throughput to 22.85 req/s.
Trade-offss between prioritizing cost-efficiency and request latency. Prioritizing cost-efficiency typically involves using fewer resources (i.e., lower budgets), which can lead to slightly higher response latencies. In contrast, prioritizing latency often requires utilizing more resources (i.c.. incurring higher costs). We acknowledge that optimizing for cost efficiency may result in a slight increase in latency. However, inference tasks typically do not require extremely low latency; meeting a predefined latency threshold is usually sufficient. In resource-limited scenarios, where systems are naturally under- provisioned, emphasizing throughput can also indirectly improve latency by reducing queuing delays. Our experimental results in Figure 6 demonstrate that our method achieved the lowest P99 latency among all compared baselines.
19


Reviews:
[{"review_id": "mvskNg4uQe", "reviewer": "Reviewer_nHPD", "summary": "This paper focuses on the cost efficiency of LLM services on heterogeneous GPUs, proposing ways to improve efficiency by optimizing GPU composition, deployment configurations, and workload allocation.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> 1. Specific shortcomings of existing methods.\n\nExisting methods require a heavy redesign of the scheduling algorithms or demand significant additional system development to achieve similar cost optimization.\n\n**Compare with HexGen and Helix.** Both approaches (1) fail to consider workload heterogeneity and the presence of mixed workloads during scheduling, and (2) their scheduling algorithms are designed for fixed heterogeneous cluster configurations, missing opportunities for GPU composition optimization.\n\n**Compare with Melange.** Melange does not consider the impact of parallelism strategies or cloud resource constraints on deployment performance. Its method fails to utilize different parallelism strategies or different GPU types to handle various workloads.\n\n> 2. Adapting to dynamic tasks and preemption uncertainty.\n\nOnline scheduling to adapt to workload changes and GPU drops is an interesting idea that can easily be integrated into our current solution:\n\n**Online replanning.** We implement an online replanning mechanism analogous to the one proposed in DistServe (OSDI'24). Concretely, the system could (1) monitor the real-time composition of incoming workloads, (2) track GPU resource availability within the cloud environment, and (3) upon detecting a significant shift in workload distribution, the scheduling algorithm could be executed again, incorporating recent historical data to produce an updated serving plan.\n\n|| Workload Change|GPU Drop|\n|-|-|-|\n|Before|26.89(req/s)|26.89|\n|After|23.70(13%↓)|20.80(29%↓)|\n|Replanning|29.61(25%↑)|22.85(10%↑)|\n\n**Experimental Results.** (1) We test the workload surge in short output requests in AzureTrace. Before surge, the optimal GPU composition is {20%, 65%, 15%} for datacenter, workstation, and consumer GPUs, achieving 26.89 req/s. After workload change, the throughput degrades to 23.7 req/s. In this case, replanning (shifting allocation to {63%, 23%, 14%}) boosts throughput to 29.61 req/s. (2) We also test the case when GPU drop happens (4 H100s down), the throughput falls from 26.89 to 20.80 req/s. In this case, replanning raises throughput to 22.85 req/s. Additionally, the rescheduling and model-weight reloading phases can be completed within 1–2 minutes, which is significantly shorter than the hourly timescale of real-world workload changes.\n\n> 3. Vast.ai platform has small GPU availability.\n\nGPU shortages are a well-documented challenge in cloud-based tasks (SkyPilot NSDI'23). This issue is not unique to Vast.ai—similar GPU availability limitations are observed across many widely-used platforms, including FluidStack, DataCrunch, and RunPod. Even major cloud providers, such as Google Cloud, face GPU quota constraints, as noted in Table 4 of Helix (ASPLOS'25), with quotas being limited and varying across regions.\n\n> 4. Compare other scheduling methods.\n\n**Scheduling algorithm comparison.** We further compare our method with a heuristic scheduling method by replacing the plan optimization component of MILP with population-based mutation and selection (i.e., genetic algorithm). In a 48-GPU cluster, the heuristic method requires 115 seconds to achieve comparable performance to our MILP formulation, which only requires 30 seconds.\n\n**End-to-end system comparison.** We compared our method with HexGen in our paper, here, we further compare our system with Helix (ASPLOS'25), which optimizes heterogeneous LLM serving using max-flow and MILP scheduling, under a price budget of $15 per hour on the AzureTrace dataset. Our method explicitly considers workload heterogeneity and GPU composition optimization, resulting in greater cost efficiency.\n\n||Llama-30B|Llama3-70B|\n|-|-|-|\n|Helix|8.49(req/s)|5.72|\n|Ours|11.43(35%↑)|7.13(25%↑)|\n\n> 5. Other factors that may affect the cost.\n\n**Network Latency impact.** We have incorporated network latency considerations into our MILP formulation. For instance, (1) the GPU-to-GPU communication latency is used to determine the pipeline communication cost, and (2) tensor parallelism with high communication volumes is prioritized to use intra-machine networks.\n\n**Energy consumption.** Our focus is optimizing the cost efficiency of cloud-based LLM serving; optimizing energy consumption is orthogonal to our paper’s scope. However, we acknowledge that energy consumption is a critical metric and could be considered as a future direction.\n\n**Computational complexity.** Thank you for mentioning this. We will include a detailed discussion of computational complexity of the MILP formulation in Section 4.3 of the updated draft: “the theoretical worst-case time complexity scales as: O(poly(|C|,W,N)×2^|C|), the polynomial factor accounts for the overhead of processing each node in the search tree”.\n\n> 6. Scheduling across data centers.\n\nWe have already used GPUs across data centers in our heterogeneous experiment setup, since Vast.ai provides different GPUs in different data centers (e.g., A40 and A100 GPUs reside in Australia and New Jersey, US).", "claims_and_evidence": "When discussing related work, the paper mainly emphasizes that other methods do not consider GPU heterogeneity and user budget constraints, but does not analyze in depth the specific shortcomings of existing methods, e.g., whether pre-existing methods can achieve similar cost-optimization results by adjusting their strategies.", "methods_and_evaluation_criteria": "The paper assumes that task assignments are all computed before the task is executed, while in real cloud environments, task arrivals are dynamic and the paper does not provide an online scheduling strategy that adapts to dynamic changes in tasks.", "theoretical_claims": "Yes", "experimental_designs_or_analyses": "The paper's experiments are mainly based on the Vast.ai platform, which has small GPU availability and the experimental results may be difficult to generalize to large-scale cloud environments.", "relation_to_broader_scientific_literature": "The paper only compares a few heterogeneous LLM scheduling methods and lacks comparative analysis with a wider range of scheduling optimization methods (e.g., reinforcement learning methods, heuristic scheduling methods)", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths:\nThe organization of this paper is good.\n\nWeaknesses:\n1. The research problem of the paper focuses too much on the cost-effective optimization of heterogeneous GPUs, but other factors that may affect the cost (e.g., network latency, energy consumption, computational complexity of scheduling algorithms, etc.) are under-explored.\n\n2. The paper fails to address more complex scheduling challenges in cloud environments, such as scheduling GPU resources across data centers and uncertainty due to preemptive instances.", "other_comments_or_suggestions": "No", "questions_for_authors": "No", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kF2GLRQD8y", "reviewer": "Reviewer_79GJ", "summary": "This paper investigates cost-efficient LLM serving on heterogeneous GPUs. It benchmarks various GPU types and develops a MILP-based scheduling algorithm to optimize deployment configurations. The study shows that leveraging heterogeneous GPUs improves cost efficiency compared to homogeneous setups.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> 1. Workload spikes and dynamic GPU availability fluctuations.\n\nOnline rescheduling to adapt to workload changes and GPU drops is an interesting idea that can easily be integrated into our current solution. We introduce this approach and present some preliminary experimental results.\n\n**Solution: Online replanning.** To address the reviewer's concern, we implement an online replanning mechanism analogous to the one proposed in DistServe (OSDI'24). Concretely, the system could **(1)** monitor the real-time composition of incoming workloads (mentioned in Section 3), **(2)** track GPU resource availability within the cloud environment, and **(3)** upon detecting a significant shift in workload distribution, (e.g., an increase in the proportion of certain workload types) the scheduling algorithm could be executed again, incorporating recent historical data to produce an updated serving plan.\n\n| Status| Workload Change| GPU Drop|\n|------------|-------------------------|--------------------|\n| Before| 26.89 req/s| 26.89 req/s|\n| After| 23.70 req/s (13%↓)| 20.80 req/s (29%↓) |\n| Replanning| 29.61 req/s (25%↑)| 22.85 req/s (10%↑)|\n\n**Replanning results.** **(1)** We test the workload surge in short output requests in AzureTrace. Before surge, the optimal GPU composition is {20%, 65%, 15%} for datacenter, workstation, and consumer GPUs, achieving 26.89 req/s. After workload change, the throughput degrades to 23.7 req/s. In this case, replanning (shifting allocation to {63%, 23%, 14%}) boosts throughput to 29.61 req/s. **(2)** We also test the case when GPU drop happens (4 H100s down), the throughput falls from 26.89 to 20.80 req/s. In this case, replanning raises throughput to 22.85 req/s. Additionally, the rescheduling and model-weight reloading phases can be completed within 1–2 minutes, which is significantly shorter than the hourly timescale of real-world workload changes.\n\nWe will integrate this discussion into Section 6 of our updated draft.\n\n> 2. The trade-offs between prioritizing cost-efficiency and request latency.\n\n**Trade-offs.** We acknowledge that there are trade-offs between optimizing cost-efficiency and latency. **(1)** Prioritizing cost-efficiency typically involves using fewer resources (i.e., lower budgets), which can lead to slightly higher response latencies. **(2)** In contrast, prioritizing latency often requires utilizing more resources (i.e., incurring higher costs).\n\n**Our focus.** Despite the trade-offs, our work mainly focuses on the cost-efficiency for two reasons. **(1)** Some inference tasks do not require extremely low latency; meeting a predefined latency threshold (e.g., reading/speaking speed) is usually sufficient. And more importantly, **(2)** in resource-limited scenarios, where systems are naturally under-provisioned, emphasizing throughput can also indirectly improve latency by reducing queuing delays. Our experimental results in Figure 6 demonstrate that our method achieved the lowest P99 latency among all compared baselines.\n\nWe will integrate the above discussion into Section 6 of the updated draft.\n\n> 3. Why adding budget constraints to existing LP solvers (e.g., Melange) is challenging?\n\nExisting heterogeneous frameworks require substantial redesign of their scheduling algorithms or significant additional system development to achieve cost optimization comparable to our approach.\n\n**Compare with Melange.** **(1)** Melange assigning each workload to a *single* GPU type, overlooking the impact of parallelism strategies on resource allocation. As shown in Section 3, tuning parallelism strategies is crucial, and incorporating this dimension significantly expands the search space and demands additional system development. **(2)** Melange assumes unlimited resources, yet in practice, GPU availability and budget constraints are real issues that necessitate evaluating workloads across *multiple* GPU types to find a globally optimal solution—further expanding the search space and rendering Melange’s approach impractical.\n\n**Compare with HexGen and Helix.** Both approaches **(1)** fail to consider workload heterogeneity and the presence of mixed workloads during scheduling, and **(2)** their scheduling algorithms are designed for fixed heterogeneous cluster configurations, missing opportunities for further performance improvements achievable through GPU composition optimization.\n\n> 4. Essential References Not Discussed.\n\nQLM (SoCC 2024) focuses on SLO-aware serving and multi-node optimizations by refining request ordering; Sarathi Serve (OSDI 2024) optimizes batching through prefill chunking to mitigate interference between the prefill and decoding stages; and Vidur (MLSys 2024) develops an accurate simulator for deployment tuning. \n\nOur work has a different goal—it is dedicated to achieving heterogeneous, cost-efficient serving in cloud environments. We will integrate the discussion of these references into Section 2 of the updated draft.", "claims_and_evidence": "Claim: Heterogeneous GPU compositions improve cost-efficiency over homogeneous deployments.\nEvidence: Benchmarking results demonstrate that different GPU types align well with varying workload demands, leading to improvement in cost-efficiency.\n\nClaim: Optimal deployment configurations significantly affect performance.\nEvidence: Experimental results show that tuning deployment strategies (e.g., tensor parallelism vs. data parallelism) improves performance.\n\nClaim: The proposed MILP-based scheduling algorithm outperforms existing heterogeneous serving frameworks.\nEvidence: Comparisons with HexGen show up to 18% improvement in throughput.", "methods_and_evaluation_criteria": "The study evaluates its approach on real-world and synthetic workload traces, including Llama3-8B and Llama3-70B models. The study lacks a discussion on the impact of fluctuating real-time workloads.", "theoretical_claims": "The paper does not introduce new theoretical models but formulates GPU scheduling as a MILP problem.", "experimental_designs_or_analyses": "While the evaluation is thorough, there are several weaknesses remain:\nThe paper does not address how the system handles sudden bursts of traffic or fluctuating real-time workloads.\nWhile the proposed method improves cost-efficiency, it lacks details on handling conflicts between prioritizing throughput and latency.\n\nThe paper does not clearly articulate why adding budget constraints to existing LP solvers is challenging, given prior work like Melange.", "relation_to_broader_scientific_literature": "This is a growing area but novelty with respect to existing literature is think or not clear.", "essential_references_not_discussed": "QLM (SoCC 2024): Addresses SLO-aware LLM serving and multi-node optimizations.\nSarathi Serve (OSDI 2024): Implements prefill chunking for hybrid execution batches.\nVidur (MLSys 2024): Solves the problem of deployment tuning", "other_strengths_and_weaknesses": "see rest of the review.", "other_comments_or_suggestions": "How does the approach handle workload spikes and dynamic GPU availability fluctuations?\nWhat are the trade-offs between prioritizing cost-efficiency and request latency?\nUnder what scenarios does the MILP-based scheduling significantly diverge from existing heterogeneous scheduling frameworks?", "questions_for_authors": "Please see above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zI3WcKV4E2", "reviewer": "Reviewer_2VxX", "summary": "The paper investigated a highly practical problem of co-optimizing GPU composition, deployment configurations, and workload assignment in a heterogeneous GPU environment under a budget constraint to maximize Large Language Model (LLM) serving efficiency. The authors proposed a Mixed Integer Linear Programming (MILP) formulation to optimize the makespan for processing all workloads. Experimental results showed that the proposed method outperforms both homogeneous and heterogeneous baselines across a wide range of scenarios.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> 1. How the throughput h is profiled or defined?\n\nWe agree with the reviewer that, because $h$ is recursively defined, it is impractical to profile every possible configuration exhaustively. However, in practice, this can be solved by employing a one-time profiling strategy that captures the following components. (This approach is referred to the profiling method used in Vidur MLSys'24.) \n- Inference prefilling latency: We profile the latency for a single transformer layer across varying tensor parallelism (TP) degrees, workload types, and GPU types.\n- Inference decoding latency: We profile the decoding latency for a single transformer layer under similar variations in TP degrees, workload types, and GPU types.\n- Pipeline communication latency: We measure the communication latency between different GPUs across various workload types.\n\nUsing these measurements, the per-request latency for any configuration is estimated by combining the TP costs (both communication and computation) of all layers—which may be served by different GPUs and at varying TP degrees—with the pipeline parallelism (PP) communication cost. (Note that our heuristics, as discussed in Section 4.3 and Appendix D, largely reduce the profiling space, e.g., TP is employed only intra-machine.)\n\nAdditionally, when estimating throughput, the prefill and decoding phases are treated separately: **(1)** The prefill phase is compute-bound, and its batched processing capacity is determined by the sum of the individual latencies; **(2)** the decoding phase is memory-bound, with its batched processing capability defined by a single latency value. This distinction has been validated in several studies (DistServe OSDI'24, Splitwise ISCA'24).\n\n|Diff Configs|Real (req/s)|Estimated (req/s)|\n|-|-|-|\n| H100(2,4)|0.56|0.60|\n| H100(4,2)|0.44|0.47|\n| H100(4,2)(cross machine)|0.42|0.44|\n| L40(2,4)|0.42|0.46|\n| L40(4,2)|0.21|0.22|\n| L40(4,2)(cross machine)|0.18|0.19|\n| H100+A100(4,2)(cross machine)|0.48|0.52|\n\nThe table demonstrates examples of our cost estimation under long input and short output workload (i.e., workload 1 in Figure 4), the notation (2,4) indicates that the TP degree is 2 and the PP degree is 4. Although the estimations are not perfectly accurate, they are sufficiently reliable (estimation errors within 4%-7%) for selecting the optimal configurations.\n\nWe will integrate the details of the one-time profiling into the updated version of our paper.\n\n> 2. Workload distribution absent from MILP formulation.\n\nThanks for the reviewers' detailed and in-depth understanding of our paper. Our workload assignment method ensures that every workload type is completely allocated across all configurations, as enforced by the constraint: $\\sum_{c \\in C} x_{c,w} = 1, \\forall\\,w$. Although these variables are normalized to sum to 1, they can be scaled by the actual workload counts (e.g., 500 requests for type 1300 requests for type 2) during implementation. \n\nTo avoid misunderstadning, we will revise the formulation in the updated version of our paper: \"Let $f_w$ be the total number of requests for workload $w$. The time required on $c$ is given by $T_c = \\sum_{w=1}^{W} \\frac{x_{c,w} \\cdot f_w}{y_c \\cdot h_{c,w}}$.\"\n\n> 3. Integrating more heterogeneous baseline.\n\n||Llama-30B|Llama3-70B|\n|-|-|-|\n|Helix |8.49 req/s|5.72 req/s|\n|Ours |11.43 req/s (35%↑)|7.13 req/s (25%↑)|\n\nWe conduct additional experiments comparing our system with Helix (ASPLOS'25). Specifically, we compare our method against Helix under a price budget of $15 per hour on the AzureTrace dataset. While Helix optimizes heterogeneous LLM serving using max-flow and MILP scheduling, our method explicitly considers workload heterogeneity and GPU composition optimization, resulting in better cost efficiency. Experimental results show that our system outperforms Helix by 25–35%.\n\nWe will integrate the baseline into our updated paper.\n\n> 4. Additional Suggestions.\n\nThanks for the reviewers' detailed review. We will solve the following mistakes and update our paper:\n- We will update Figure 3, 4, 11, 12, 13 using \"Throughput Per Dollar\".\n- We will explain that the x sticks represent P5 Latency, P10 Latency, P15 Latency, etc.\n- \n    - We will update Figure 4, 12, 13 using \"Different Deployment Configurations\" instead of \"Different GPU types\".\n    - We will clarify that different bars within Figure 4 represent different deployment methods, and the optimal method varies according to workload, model type, and GPU type.\n- We will correct m and M to w and W.\n- We will check all the cited papers and update the references to the published ones, e.g., sarathi-serve (OSDI'24), HexGen (ICML'24), WildChat (ICLR'24) etc.", "claims_and_evidence": "- The paper claims that LLM serving efficiency can be significantly influenced by heterogeneous GPU composition, deployment configuration, and workload assignment policy. This claim is well supported by the evidence in Figures 3 and 4, as well as the example provided in Section 4.2.\n\n- The paper claims that the proposed method significantly outperforms both homogeneous and heterogeneous baselines across a wide range of scenarios. This claim is generally supported by the experimental results.", "methods_and_evaluation_criteria": "The method proposed in this paper has several significant problems.\n\n- In the formulated MILP problem, a deployment configuration $c$ is defined by a tuple $(v_c, s_c, o_c, h_{c,w})$, with $h_{c,w}$ defined as the \"throughput of config $c$ on workload $w$\", and $v_c$ indicating the GPU composition. However, I find this questionable, as the throughput should be affected by the GPU composition. This would mean $h_{c,w}$ is recursively defined. Here, it seems that the authors are treating the throughput as invariant to the GPU composition, which is not realistic.\n\n- Also for the $h_{c,w}$, solving the MILP problem requires it to be known in advance. This is a significant limitation, as measuring this throughput requires non-trivial effort and resources, especially considering the vast number of possible configurations and workloads.\n\n- The MILP formulation aims to optimize the makespan for processing all workloads. However, the makespan is affected by the distribution of workloads, a factor entirely absent from the MILP formulation.", "theoretical_claims": "The paper did not make any theoretical claims or proofs.\n\n\n##", "experimental_designs_or_analyses": "The experimental design is generally well-structured, and the results are thoroughly analyzed. The authors compare their system's end-to-end performance against the primary baseline, HexGen, and conduct ablation studies to assess the effects of GPU composition, deployment configurations, and request assignment policies. In addition, they evaluate the effectiveness of their proposed MILP problem heuristic. Methods for extending to multi-model cases are also discussed.\n\nHowever, HexGen is currenty the only baseline used for comparison. Introducing more baselines would provide a stronger support for the paper's performance claim.", "relation_to_broader_scientific_literature": "The paper is mainly related to heterogeneous Large Language Model (LLM) serving, specifically exploring how a heterogeneous GPU configuration can reduce cost budget while improving the LLM inference efficiency.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "- The paper's overall presentation is generally clear and well-organized.\n- This paper focuses on a highly practical problem, exploring the challenges of optimizing LLM serving efficiency in a heterogeneous GPU environment under budget constraints.\n- Despite the questionable workload distribution setup, the experimental results are generally well-presented and thoroughly analyzed.", "other_comments_or_suggestions": "I have some very specific comments and suggestions for the authors to consider, listed below:\n\n- The y-axis label \"Throughput\" for the left subplots in Figure 3 may mislead readers, as it doesn’t reflect cost efficiency. I would suggest \"Throughput Per Dollar\" instead.\n- Also in Figure 3, the x-axis label \"P5-P100 Latencies\" for the right subplots needs a better explanation. Exactly what percentiles are being referred to here?\n- The Figure 4 is difficult to interpret.\n  - I recommend explicitly stating what the bars in each subplot represent. The current x-axis labels seem incorrect and should likely be \"Different Deployment Configurations\" instead.\n  - Additionally, Observation 2 from the paper is not apparent in this figure.\n- In the \"Workloads and Assignment\" paragraph of Section 4.3, the symbols $m$ and $M$ appear to be errors and should be corrected to $w$ and $W$, respectively.\n- The reference format requires substantial revision. Several cited works, published in conference proceedings, are inaccurately listed as arXiv preprints, which is misleading and needs correction.", "questions_for_authors": "See Comments.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The following issues have been pointed out in this paper:  
- Lack of consideration for dynamic workloads and preemptible GPU instances  
- Limited baseline comparisons  
- Theoretical and modeling assumptions  
- Trade-off between latency and cost-efficiency  
- Presentation issues and missing references  

However, the following points are evaluated positively, and based on an overall assessment, the recommendation is Weak Accept:  
- Timely and practically relevant problem (LLM serving cost-efficiency on heterogeneous GPUs)  
- Solid MILP-based optimization formulation addressing real-world deployment constraints  
- Extensive and realistic experiments using modern LLMs and real workload traces  
- Introduction of an online replanning mechanism for dynamic adaptation  

That said, neither the reviewers nor the Area Chair strongly recommend acceptance of this paper.
Decision: Accept (poster)