HexGen-2: Disaggregated Generative Inference of LLMs in Heterogeneous Environment
Authors: YOUHE JIANG
Ran Yan
Binhang Yuan
Venue: ICLR 2025 Poster (2025)
Keywords: Distributed Machine Learning System; Generative Inference of LLM.
Primary area: infrastructure, software libraries, hardware, systems, etc.

Abstract: Disaggregating the prefill and decoding phases represents an effective new paradigm for generative inference of large language models (LLM). This approach offers some significant system advantages, such as eliminating prefill-decoding interference and optimizing resource allocation. However, it is still an challenging open problem about how to deploy the disaggregated inference paradigm across a group of heterogeneous GPUs, which can be an economic alternative of the deployment over the homogeneous high performance GPUs.
Towards this end, we introduce HexGen-2, a distributed system for high throughput and cost-efficient LLM serving on heterogeneous GPUs following the disaggragated paradigm. Built on top of HexGen, the core component of HexGen-2 is a sophisticated scheduling algorithm that formalizes the allocation of disaggregated LLM inference computations and communications over heterogeneous GPUs and network connections as a constraint optimization problem. We leverage the graph partitioning and max-flow algorithm to co-optimize resource allocation, parallel strategies for distinct inference phases, and the efficiency of inter-phase key-value (KV) cache communications. We conduct extensive experiments to evaluate HexGen-2, i.e., on OPT (30B) and Llama-2 (70B) models in various real-world settings, the results reveal that HexGen-2 delivers up to a 2.0$\times$ and on average a 1.3$\times$ improvement in serving throughput, reduces the average inference latency by 1.5$\times$ compared with state-of-the-art systems given the same price budget, and achieves comparable inference performance with a 30% lower price budget.

Published as a conference paper at ICLR 2025
HEXGEN-2: DISAGGREGATED GENERATIVE INFER- ENCE OF LLMs IN HETEROGENEOUS ENVIRONMENT
Youhe Jiang, Ran Yan; Binhang Yuan Department of Computer Science and Engineering The Hong Kong University of Science and Technology
ABSTRACT
Disaggregating the prefill and decoding phases represents an effective new paradigm for generative inference of large language models (LLM),, which elimi- nates prefill-dcoddng interference and optimizes resource allocation. However, it is still an open problem about how deploy the disaggregated inference paradigm across a group of heterogeneous GPUs, which can be an economical alternative deployment over homogeneous high-pe reee-performacce GPUs. Towards this end, we introduce HEXGEN-2, distributed system for efficient and economical LLM serving on heterogeneous GPUs following the disaggregated paradigm. Built on top of HEXGEN, the core component of HEXGEN-2 is a scheduling algorithm that formalizes the allocation of disaggregated LLM inference computations and com- munications over heterogeneous GPUs and network connections as constraint optimization problem. We leverage the graph partitioning and max-flow algo- rithms co-optimize resource allocation, parallel strategies for distinct inference phases, and the efficiency of inter-phase key-value (KV) cache communications. We conduct extensive experiments evaluate HEXGEN-2, i.e., on OPT (30B) and LLAMA-2 (70B) models in various real-world settings, the results reveal that HEXGEN-2 delivers up 2.0x and on average a 1.3 x improvement in serving throughput, reduces the average inference latency by 1.5x compared with state- of-the-art systems given the same price budget, and achieves comparable inference performance with 30% lower price budget.
INTRODUCTION
Large Language Models (LLMs), such as OPT (Zhang et al. 2022). LLAMA TTuvron et al. 2023 GPT (OpenAi 2024), GEMINI (Reid et al. 2024). CLAUDE Anthropic 2024 and MIXTRAL (Jiang et al. 2024a) have shown exceptional performance across various advanced applications. However, deploying the generative inference service for such LLMs can be costly, typically requiring sub- stantial number of homogeneous, high beigh-peffrmance GPUs to meet the service demands, such as first token latency and generation throughput. In this paper, we explore an alternative solution that de- ploys the most advanced disaggregated generative inference paradigm over a set of heterogeneous GPUs to provide an efficient and economical LLM service.
Disaggregated inference is currently the most efficient framework for serving the generative in- ference requests of LLMs (Zhong et al. 2024 Patel et al., 2024). By splitting the prefill phase (compute-bounded) bounded) and decoding phase 10-Oobdddddd across different GPUs, the disaggrega- tion significantly reduces interference between different requests and enables more flexible parallel configurations for the two phases. When compared with colocating the prefill and decoding compu- tations, the disaggregated approach optimizes resource usage and enhances the scalability and effi ciency of the LLM inference service. Recent efforts (Jiang et al.) 2024b] Griggs al.) 2024 Zhao et al. 2024: Miao et al.) (024) have shown that serving LLMS with heterogencous GPUs can be economical alternative to deploying over homogeneous
performaaee GPUs. Heterogeneous deployments offer significant opportunities to reduce inference service costs by leveraging the wide availability of diverse GPU types across commercial and private computing platforms. Note that
Nvidia typically releases new GPU generations every 24 months, e.g., Turing in 2018, Ampere in
Equal contribution.
Correspond to Binhang Yuan (bi Soannnust.... hk).

---

Published as a conference paper at ICLR 2025
2020, Hopper in 2022, and Blackwell scheduled for Q4 2024; but one particular version of GPU general remains in use for a much longer period
The wide availability of heterogeneous GPU pools presents significant opportunities to adapt the most advanced disaggregated inference paradigms. However, effectively adapting the disaggregated paradigm to this heterogeneous setting is much harder to implement than to ask for. Traditional implementation of co-locating prefill and decoding phases only leverage standard tensor model par- allelism (Narayanan al., 2021) and pipeline parallelism (Huang et al.) 2019) for LLM inference, where only the activations are communicated. In the disaggregated paradigm, transferring the key- value (KV) cache between prefill and decoding model replicas introduces significant data movement, potentially creating a communication bottleneck that must be carefully managed in a heterogeneous setting. Additionally, the flexibility of parallel configurations among prefill and decoding model replicas also introduces new complexity in the esterogeneity-awan scheduling.
Towards efficiently adapting the disaggregated paradigm under the heterogeneous setting, we iden- tify two types of new challenges and opportunities that previous heterogeneity-awarc scheduling approaches (Jiang et al. 2024b) fail to integrate:
Accommodate the computation flexibility in disaggregated paradigm. a heterogeneous setting, each GPU type has distinct peak FLOPS, HBM memory bandwidth, and HBM memory limit, even making optimal computation allocation for the colocating paradigm a difficult problem. The disaggregated paradigm adds further complexity, as the prefill and decoding phases have different resource requirements and favor specific parallel strategies depending on varying LLM inference workloads, such as arrival rates and input/output sequence lengths.
Accommodate additional KV cache movement over heterogeneous connections. GPU com- munication bandwidth also varies widely, from different NVLink and PCIe generations within a server to InfiniteBand(1BB) ROCE, TCP, and Ethernet connections among different servers. Along with communication demands from parallel strategies within each model replica, disag- gregated inference requires extensive KV cache transmissions, which are especially sensitive to bww-bandwidth links. Therefore, an effective scheduling algorithm is essential to manage commu- nication across heterogeneous GPU connections and minimize costs.
In order to overcome these challenges, we propose HEXGEN-2, a disaggregated LLM inference system that coordinates distributed LLM inference computations and communications over set of GPUs with different computation capabilities and heterogeneous network connections. Our contri- butions are summarized as:
We formulate the scheduling problem of allocating disaggregated LLM inference computations over set of heterogeneous GPU devices constraint optimization problem. To solve this problem efficiently, we propose a sophisticated scheduling algorithm that employs com- bination of graph partitioning and max-flow algorithm coordinate the resource allocations and parallelism plans for the prefill and decoding phases of LLM inference. Concretely, the graph par- titioning algorithm partitions the available GPUs into multiple model serving groups, where each group should be dedicated to serving a prefill or decoding model replica; and the max-flow algo- rithm guides the iterative refinement of the graph to optimize model placement.
Contribution 1:
Contribution 2:
We implement HEXGEN-2, a heterogeneous LLM inference system that facilitates tensor model parallelism and pipeline parallelism with disaggregated paradigm. HEXGEN-2 al- lows the two phases of LLM inference to be split onto separate GPUs with different parallel plans, effectively eliminating prefill-decoding interference and boosting inference performance.
We evaluate HEXGEN-2 through extensive experiments, where we compare HEXGEN-2's system efficiency across various LLM inference workloads with HEXGEN on sev- eral heterogeneous settings and DISTSERVE on standard homogeneous setting. We conduct these comparisons on the popular LLM models OPT (30B) and LLAMA-2 (70B). We show that given the same budget in terms of cloud service fees, HEXGEN-2 can choose to achieve up to a 2.0x and on average a 1.3x higher serving throughput or reduce the average inference latency by 1.5x. Additionally, when given only 70% of the budget, HEXGEN-2 can still maintain a comparable level of inference service compared to the homogeneous baseline.
Contribution 3:
For example, Tesla K80 GPUs, released in 2006, are still available on AWS as p2. instances
2

---

Published as a conference paper at ICLR 2025
2 PRELIMINARY
LLM generative inference. Given the input request, the LLM inference process typically contains two phases: prefill and decoding. The prefill phase processes the request to compute the KV cache and generates the first token for the response in a single step. The decoding phase then takes the last input token and KV cache as inputs to generate subsequent tokens by one token at each step. The distinct characteristics of both phases lead to differing GPU resource utilization: the prefill phase is compute- bound, whereas the decoding phase is HBM memory I/O-bound. Naive imple- mentation of the inference engines colocates the two phases on the same group of GPUs, despite their distinct computational characteristics. Two standard strategies are applied to parallelize the LLM inference computation: tensor model parallelism and pipeline parallelism. Tensor model par- allelism (TP) (Narayanan al., (22T)) distributes inference computations across multiple GPUs by partitioning the weight matrices of transformer layers both row-wisely and column-wisely, each layer's output activations are aggregated through two Al Reduce operations. Pipeline parallelism (PP) (Huang et al., 2019) divides the model into multiple stages, each assigned to specific GPU or group of GPUs for execution, the inter-layer activations are communicated between stages.
Inference serving goal. There are two essential metrics to evaluate LLM serving: throughput and inference latency. Throughput refers to the number of tokens system can generate within a specified time period. Inference latency is the time required to complete each inference request from start to finish. We assess system performance on inference latency using service level objective (SLO) attainment, which gauges the proportion (e.g.. 99%) of requests fulfilled within a time frame predefined by the SLO. This SL. adjusted to various multiples of single device execution latency (termed as SLO scale) to measure performance under different degrees of SLO stringency.
Batching. Due to the computational difference of the prefill and decoding phases, integrating batch- ing strategies leads to varying performance outcomes. As shown in Figure I] in the prefill phase, a small batch size quickly saturates the GPU's computation capacity batched tokens reaches 2048, no further improvement in throughput is observed but the prefill la- tency escalates with batch size. Conversely, in the decoding phase, where the system bottleneck lies in scanning the LLM parameters, the throughput increases linearly as the total number of batched tokens rises, highlighting the effectiveness of batching in this phase for performance enhancement. The current state-of- oifti--ffthh-aate art LLM serving system employs a batching optimization called continuous batching (Yu et al. 2022), which batches the prefill of new requests with the decoding of ongoing requests to enhance GPU utilization. However, this leads to severe prefill-decddnng interference. Adding a single prefil job to a batch of decoding requests significantly slows down both processes, with the slowdown intensifying as the prefill length increases.
Once the total number of
Throughen (hkmee))
220
Prefill Phase
10 x11²
Decode Phose
Requests
176-
152
88
4-
Prefill Model Replica
Decode Replica
ranser
Butch Size
0000
Figure 1: Effects of batching different phases (LLAMA-2 (7B) inference with an input length Figure 2: Illustration of disaggregated paradigm. of 512 on single A100 GPU).
Disaggregated architecture. As the two phases in LLM inference have distinct characteristics, recent efforts (Zhong et al. 2024 Patel et al. 2024: Jin et al.) 2024] Qin et al.) 2024: Hu et al.) 2024] propose disaggregated inference architecture that splits the two phase in separate hardware resources. In the disaggregated inference architecture (See Figure 2). there are two types of model replicas: prefill model replica is responsible for taking the incoming request, generating the first token and KV cache; decoding model replica takes the generated token and KV cache as inputs, and generates the subsequent tokens. This separation enhances LLM serving by: (1) Eliminate the prefill-deooiig interference; (2) Allow prefill and decoding model replicas to use different batch- ing and parallelism strategies - Prefill replicas benefit from tensor model parallelism and smaller batches to reduce per-request latency, while decoding replicas perform better with larger batches to
3

---

Published as a conference paper at ICLR 2025
286-0
00:9
Devices
Super Nirdes
Initial Graph Partition
Secondary Graph Partition
13
Swap inter
Swap to inta
Sourc
Fell Murid
Slock
Sink Goord.
Cound
Dequard
Foruns
0
Complete
Cache
Sotflemec Edge
Undentiized Edge
Balanced Edge
Max-flow Directed Graph
Max-flow Guided Edge Swap
Figure Illustration of each scheduling step.
maximize throughput. (3) Accommodate varying LLM serving workloads by adjusting resource al- locations between the two phases, e.g., the coding workload characterized in (Patel et al. (2024) with typically longer prefill and shorter decoding sequence lengths requires more resources for prefill to optimize performance. As prefill and decode model replicas operate independently, it is crucial to transmit the KV cache from the prefill to the decode model replicas. Given the large volume of KV cache in LLM serving, current implementations necessitate a banhidhhd communication link to facilitate the transmission of the KV cache. (Patel et al. (224) utilize InfiniteBand (IB) for inter- node KV cache transmission, while (Qin et al.
2024) deploy their system on GPU clusters equipped 2024) collocate prefill and decode model replicas
with RDMA network cards, and (Zhong et al.
GPUs within the same node to expedite KV cache transmission via NVLink. We also include the discussion of disaggregation versus chunked prefill in Appendix D
SCHEDULING ALGORITHM IN HEXGEN-2
The core technique component in HEXGEN-2 is scheduling module that efficiently allocate the heterogeneous GPUs to serve prefill or decoding model replicas. In this section, we formulate the scheduling problem and introduce our solution.
3.1 PROBLEM FORMALIZATION
To support LLM serving with the disaggregated paradigm under heterogeneity, the scheduling al- gorithm should determine four essential allocations: (1) the group partition, i.e., how to partition the GPUs to multiple groups, where each responsible for serving one model replica; (2) the group type, i.e., whether group serves prefill decoding model replica. (3) the parallel strategy for
each model serving group, i.e., the combination of TP and PP under the heterogeneous setting (Jiang eet al. 2024b); (4) the KV cache communication strategy among prefill and decoding model replicas. We term a solution to these four components as a model placement strategy.
Given the exponential search space, determining the optimal model placement is NP-hard prob- lem. To solve the problem, we adopt a two-phase search algorithm, which can be summarized as:
Graph partition: Given set of heterogeneous GPU devices D, the first phase ($3.2) aims to partition them into multiple model serving groups, and determines the group type.
Max flow: Based the outputs from the first phase, the second phase ($3.3) find the current optimal parallel strategies for prefill and decoding model replicas, and generates the optimal KV cache communication strategy among them.
Iterative refinement: We iteratively repeat the two-phase algorithm to find the optimal model placement strategy (3,44) that maximizes the end-to-end system performance.
4

---

Published as a conference paper at ICLR 2025
3.2 FIRST PHASE: GRAPH PARTITION
The first phase of our scheduling algorithm aims to partition the GPU devices D into multiple model serving groups and determine whether each group is a prefill or decoding model replica. We first formulate the GPU device set D as a global graph G = (D, E), with each GPU d E D representing a graph node, and the GPU memory limit my defined as the node weight. The communication link ed,d' E E between GPU and d', Vd, d' E D, is defined as the graph edge, with communication bandwidth 3a. as the edge weight. Then, we partition the formulated graph G into partition P = {p1 (pI...PKy PK] ). where pke denotes the k-th model serving group, and determine the type for each group. Concretely, there are three steps in the first phrase:
Step (i) - Initial partition: We first partition the global graph into multiple model serving groups based on edge weights (bandwidths), and balance the node weights (memory capacities) across groups. We leverage the spectral partitioning method (Alpert & Yao (995) to partition the graph G into K groups, which uses the eigenvectors of the Laplacian matrix to guide partitioning and minimize inter-group edge weights. The group size K is determined by dividing the cluster's 's total memory by the estimated memory required for single model replica (detialed in Appendix A). Then we adopt the Kernighan-Lin algorithm (Kernighan & Lin 1970) to iteratively refine the par- tition P by swapping node pairs between groups, which further reduces edge weights and balances node weights (memory capacities) across groups. FFiurre 3 demonstrates the process. Note that we balance memory rather than compute capacity avoid potential OOM issues and provide solid starting point for further optimization.
Step (ii) Coarsen & secondary partition: We then determine the group type, where the graph is coarsened and partitioned again to determine the model replica type for each group. Note that coarsen is a common operation that merges nodes and edges to simplify graph partition (Hendrick- son et al. 1995). Here, the coarsening operation merges graph nodes (GPUs) within the same group (model replica) into super nodes, which ensures the graph only includes relationships among the super nodes. The coarsened graph is then partitioned to distinguish between prefill and decoding model replicas. As illustrated in Figure 3] the four super nodes are divided into two parts: the two super nodes on the left are designated as prefill model replicas, while the two on the right are desig- nated as decoding model replicas. Different from initial partition, the secondary partition focuses on maximizing inter-partition edge weights (i.e., the edge weights between prefill and decoding model replicas) to support frequent KV cache communications between different group types.
Step (iii) Projection: Once we allocate the super nodes into prefill and decoding model replicas, we need to apply project operation, i.e., the reverse operation of the coarsen operation described in step (ii), to recover the GPU information within each super node. Note that after the projection, we can leave the problem of determining the optimal parallel strategies for each prefill or decoding model replica based on the GPU information within each super node during the second phase.
3.3 SECOND PHASE: MAX-FLOW
The second phase of our scheduling algorithm determines the parallel strategies within each super node and KV cache communication strategies between each super node. We leverage max-flow, as promising method, to formulate the disaggregated inference pradiagm. Taking the partitioned graph from the first phase input, we transform it into a directed graph with compute nodes and network connections. We define the source and sink of the directed graph to be the coordinator node h, which responsible for request dispatching and completion. Formally, we define:
Compute nodes. The prefill and decoding model replicas are defined as compute nodes C, with 6, E C denoting a prefill model replica and 6; E C denoting a decoding model replica. For each compute node /2/5, E C, we force it connect with two other nodes in the graph, named //th/55 /8 and ofnet /8.ut. The capacity of the directed edge (6""/8" potht/85 number of requests this node can process within a certain time period T (c.g., 10 minutes). We adopt the inference cost model from HEXGEN (Jiang et al., 2024b) and detail the node capacity estimation in AApendixx A] To optimize capacity, the optimal paraliel strategy should be selected for each node. As discussed in $2, given the distinct computational characteristics of different phases, their optimal parallel strategies also vary. For prefill model replicas, we aim to determine the aatency-optinal parallel configurations, as they are computation-intenae enhance efficiency. In contrast, for decoding model replicas, we aim to deduce the throughput- optimal parallel configurations, since this phase is memory I/O-bound and benefits from batching more requests. Based these considerations, iterate through all possible model parallelism
represents the maximum
and batching does not
5

---

Published as a conference paper at ICLR 2025
combinations for each model replica and select the optimal one. For compute node óz/6,. the amount /8("1) should be no larger than its maximum capacity.
of flow that passes through (d /6"", oftht
Network connections. A node in the directed graph might be connected with any other nodes, while only a subset of those connections are valid. A valid connection should satisfy one of the following criteria: (1) the connection is from coordinator node h to compute node @i, we represent the connection with directed edge (source, of");; ); (2) the connection from 6; to coordinator node h, we represent the connection with directed edge (doue, sink);; (3) the connection is from a compute node Φb to another compute node o;, we represent the connection with directed edge (Øgge, (or)) The edge capacity equals the maximum number of requests this connection can process within the time period T. Note that for connection type (3), between any two prefill and decoding model replicas Φ and 6, with an edge connection, each GPU containing the j-th layer within Φ should transmit its KV cache to the matching GPU housing the j-th layer within 8, The edge capacity is determined by the collective performance of all GPU-to-GPU transmission connections, as each connection is responsible for portion of the KV cache transmission. The estimation of edge capacity is detailed We only permit flow to pass through valid network connections, and the transmitted flow should not exceed the maximum capacity of the connection.
in inpppeddix
After constructing the directed graph, we run preflow-puu push algorithm (Cheriyan & Maheshwari, 1989) to get the max flow between source and sink node, with one unit of flow representing one request that can pass through a compute node or network connection per unit time. This algorithm continuously pushes the maximum allowable flow up the edge's ' capacity to maximize the flow through the direct connection. The generated flow assignments between compute nodes o; and ò; are used guide the KV cache communication. The communication frequency set to be proportional to these flow values to follow the max flow of the directed graph without creating bursts, as illustrated in(Figure 3] However, the algorithm may not fully utilize edge capacities as flows within the directed graph are interdependent; upstream and downstream edges can restrict total flow, preventing the full utilization of sigher-capacity edges due to bottlenecks or imbalanced capacities. For instance, a low capacity on the edge (0out, (8th)) can restrict the flow on edge (6.th 8out) from reaching node capacity. Therefore, iteratively refining the directed graph is essential.
3.4
ITERATIVE REFINEMENT
$.3 presented how we obtain the max flow for given graph partition; now we introduce how we can iteratively refine the graph partition maximize the flow. We refine the graph iteratively based on edge swapping, which is common approach for optimizing graph partition (Hendrickson et al., 1995 Vaishali et al.) 2018), and we further propose max-flow guided edge swap operation, which uses mlx-flow outputs to guide the iterative refinement of the graph.
The preflow-push algorithm mentioned in provides the detailed flow assignments necessary to analyze edge utilization (Waissi] 1994). By comparing the flow through each edge with its capacity, we can identify bottleneck and underutilized edges. Bottleneck edges are defined as those where the flow reaches capacity limits, preventing the directed graph from achieving a higher overall flow. Underutilized edges are those where the flow falls short of capacity and could accommodate more data flow. As long these imbalances exist, attempt to swap edges. Therefore, we implement local swaps of edges guided by the max-flow outputs to form a new graph partition, as illustrated in Figure 3] This swap operation is essential in terms of: (i balancing the inter- and intra-group edge weights to maintain high intra-group capacities while enabling efficient inter-group KV cache communicating; and (ii) adjusting the node and edge weights across gutrus to optimize re- source allocation. After the swaps, we rerun the two-phase algorithm to obtain the optimal model placement strategy and max flow of the new graph partition. We then refine the partition again. This iterative process continues until no further improvements can be made. Evaluation in $5.3 higghlitts the necessity of our design, the max flow guided edge swap overcomes local minima and acceler- ates optimization compared with other approaches. To better illustrate each phase of our scheduling algorithm, we provide detailed analysis Appendix C) and a case study in Appendix
SYSTEM IMPLEMENTATION
HEXGEN-2 is distributed system designed to support efficient LLM inference service under the disaggregated paradigm in heterogeneous environments. HEXGEN-2 uses a task coordinator to handle the dispatch of incoming I.LM inference requests, which is based on open-source im-

---

Published as a conference paper at ICLR 2025
Henegmmon Se
Hicserase Selig
Betmeraree Seiag
Hiarsnoa Stlay
Figure 4: Communication bandwidth (Gbps) matrix for different settings. Homogeneous setting contains 8xH100 GPUs with budget of 29.5 8/h; heterogeneous setting contains 2xH100, AXA100, 4xL40 and 8 A6000 GPUs with budget of 28.8 8/h: heterogeneous setting 2 contains 3xH100 and A100, 6xL40 and A6000 GPUs with a budget of 26.9 8/h; heterogeneous setting contains 6x A100 and A6000, 12xL40 GPUs with budget of 27.1 $/h; heterogeneous setting
contains H100 and 9xA100 GPUs with a budget 26.3 $ 8/h; heterogeneous setting 5 contains AxA100, Lx0 and 10xA6000 with 70% budget of 20.5 $/h.
plementation of decentralized computation coordination Yao 2023) that utilizes libP2P (LibP2P) 2023) to establish connections among the work groups in a peer-to-peer network. All parallel com- munications HEXGEN-2 are implemented using NVIDIA Collective Communication Library (NCCL) (NVIDIA 2024), and all required communication groups for different parallelism plans are established in advance to avoid the overhead associated with constructing NCCL groups. HEXGEN- for KV cache communication to enable overlapping between computation and communication. Furthermore, HEXGEN-2 integrates popular features for optimizing LLM inference such continuous batching (Yu et al., 2022), FlashAtten-
2 utilizes asynchronous NCCL SendRecv/CudaMee
tion (Dao et al. 2022 Dao
2024), Paged PagedAAtention (Kwon et al., 2023), and supports open-source
I.MMs such OPT (Zhang al. 2022) and LLAAMA (Touvron al. (023)
EVALUATION
To evaluate the design and implementation of HEXGEN-2, we ask the following essential questions:
What the end-to-end performance comparison in terms of throughput and latency between HEXGEN-2 2 and the state-of-the catte-of-the-deee oof-the-ar art homogeneous or heterogeneous generative inference systems?
How effective is our scheduling algorithm in terms of finding the optimal assignment of the infer- ence workflow compared with existing methods?
5.1 EXPERIMENTAL SETUP
Distributed environment. We rent GPUs from RunPod (RunPod) 2023). a GPU cloud provider with services for various GPUs, and perform evaluation in the following setups:
Homogeneous setup: We rent one on-demand instance equipped with 8xNVIDIA H100-80G
GPUs, with a budget of $29.52/hoon to represent the standard homogeneous case.
We utilize four types of GPUs: H100, A100, I.40, and A6000, to con- struet five different heterogeneous cluster setups, where the first four settings use similar budget as the homogeneous setting, while the last setting use a 70% budget of the homogeneous settings. The detailed configuration is illustrated in Figure 4
Heterogeneous setups:
We measure the communication bandwidth between each pair of GPUs via NCCL for all above men- tioned environments. As shown in Figure 4] the heterogeneous environments demonstrate notable bandwidth limitation and heterogeneity.
LLM inference workloads. To evaluate the performances in different LLM inference workloads, we run four different types of workloads: heavy prefill with light decoding (HPLD), heavy prefill with heavy decoding (HPHD), light prefill with heavy decoding (LPHD), light prefill with light decoding (LPLD). Prefill requests that have more than 512 tokens are categorized as heavy, others light, and decoding requests with more than 128 tokens are categorized heavy (Hu et al. 2024). We generate these workloads using samples from the Azure Conversation dataset (Patel
al. 2024
7

---

Published as a conference paper at ICLR 2025
Heatres22
HoiGes
DonSevedonm
138
418
633
495
316
356
66
423
216
190
76
Online
118
342
512
320
242
43-
230
160
235
138
Thmughpst
162
157
434
305
61
102
415
212
191
Online Heizoo3
154
407
604
355
264
238
232
192
45
235
395
171
146
HPLLD: Hutero
HPHO: Hotom-
LPHE>
LPL.T,
Online Hetrro
Figure 6: Throughput results to evaluate HEXGEN-2 LLAMA-2 (70B). Each row corresponds to a particular heterogeneous setting. The first four columns demonstrates the offline inference results on different LLM workloads. The last column represents the online inference results.
Hention:2
HoxGee
629
1389
1663
1265
928
1052
Throughput (tokend)
H2M
925
601
679
LPLD;
1535
411
1242
1645
1211
M27
225
741
1182
566
488
HPLD. Hetwo-4
EPHD, Hater-4
LPHD Henaro
Hetzoo
Onlinc Hetero-4
Figure 7: Throughput results to evaluate HEXGEN-2 OPT (30B).
Online and offline testing. We test two different arrival rates: In the online setting, we scale the average arrival rate 75% of the cluster's peak throughput to prevent request bursts that could cause system outages due to out-of-memory (OOM) errors, Figure 5) illustrates the distribution of input and output lengths in our trace. In the offline setting, we permit requests to arrive at rate that fully utilizes the cluster, testing all four types of workloads (HPLD, HPHD, LPHD, LPLD).
Models. We evaluate HEXGEN-2 on OPT (30B) Zhang et al. 2022) and LLAMA-2 (70B) Touvron et al."
2023) models, both are representative and popular open-source transformer models, to study the system performance on models of different sizes.
Froqquancy
50
1000 2000 3000 4000
150
300
450
We carefully select state-of-the-arr ap- proaches baselines. To understand end-to-end performance, we compare HEXGEN-2 with DIST- SERVE (Zhong al., 2024) the state-of-the-art approach under the homogeneous setting, which enhances LLM serving by disaggregating prefill and decoding computations across different GPUs. allowing different resource allocation and par- allelism for each phase. And HEXGEN (Jiang et al. (024b) as the state-of-the-art approach under heterogeneous settings, which is a distributed inference engine that efficiently manages I.I.M in- ference across heterogeneous environments, leveraging asymmetric parallelism with scheduling algorithm to optimize resource allocation. To understand the efficiency of the proposed scheduling algorithm, compare its convergence with the truncated variant of our scheduling algorithm and genetic algorithm.
Baselines.
Input Length
Output Length
Figure Request traces for online testing.
Evaluation metries. For offline serving, report the average decoding throughput, measured as the number of tokens generated per second. For online serving, we additionally report the SLO attainments detailed in
8

---

Published as a conference paper at ICLR 2025
Hoxtran
Distierve
87-
63
55
60-
SLO
Scale
Scale
Scale
Sade
Figure 8: Latency results in online experiments.
5.2 END-TO-END EXPERIMENTAL RESULTS
End-to-end performances. Figure and Figure demonstrate the end-to-end throughput results of HEXGEN-2 compared with HEXGEN with different models, workloads, and heterogeneous settings, and DISTSERVE in the homogeneous setting. Given the same price budget, HEXGEN-2 outper- forms its counterparts in almost all cases. In fact, compared with HEXGEN, HEXGEN-2 achieves up to a 1. 1.5x and, on average, a 1.4x increase in serving throughput. Compared with DISTSERVE, HEXGEN-2 achieves up a 2x and, on average, 1.3x higher serving throughput. We also demon- strate the latency results of HEXGEN-2 compared with HEXGEN in different heterogeneous settings and with DERVE in the homogeneous setting. As shown in Figure 8] HEXGEN-2 achieves on average a 1.5x lower latency deadlines than its counterparts. Specifically, analyzing the scheduling results under different heterogeneous settings and LLM workloads, we find that: (1) our schedul- ing approach prioritizes tensor model parallelism for prefill model replica to minimize latency and hybrid parallelism for decoding model replica to maximize throughput: (2) the scheduled result also employs pipeline parallelism to reduce the inter-machine communication over limited bandwidth, and avoid ultra-low cross data center communication; (3) relatively more resources are assigned for prefill and decoding in the HPLDD and LPHD) workloads to balance the resource demands for differ- ent phases; (4) our approach always schedules KV cache communications through high-bandwidth links such as NVLink and PCIe prevent them from becoming system bottlenecks. We also com- pare GEN-2 with the state-of-the-art LLM serving platform VLLM in in/Appendix F. and demon- strate the performance of HEXGEN-2 2 in the homogeneous setup in Appendix G
Cost efficiency. To evaluate cost- efficiency in terms of serving throughput between homogeneous and heterogeneous setup, we reduce the budget in the heterogeneous set- ting by 30%. As shown in Figure 9 HEXGEN-2 2 in heterogeneous setting 5 still reveals similar performance
HexGon-2
Ditterve-Homo
Thrugggqu (1oka13))
162
11.1x
443
11.0k
121
11.3x
81
223-
254-
40
110
169
127
LPLD
LPHD
HPLD
HPHD
Figure Throughput results with 70% price budget. to DISTSERVE in the homogeneous setting, and even outperforms it by 30% in some specific workloads. We believe that this is strong evidence to illustrate that a heterogeneous system such as HEXGEN-2 is capable of managing heterogeneous GPUs to provide more economical LLM inference services without compromising service quality.
5.3 EFFECTIVENESS OF THE SCHEDULING ALGORITHM
To evaluate the effectiveness of our scheduling algorithm, we compared its convergence behavior with some truncated variants, which disables the max-flow guided edge swap operation mentioned in b3.4|by replacing it with random swap operation, and with the genetic algorithm. The genetic algo- rithm, designed to optimize model deployment, uses a population-ban split, and swap operations to iteratively refine GPU groupings (Jiang al. 2024b). In our compari- son, we replaced the group generation step in the graph partition phase and the iterative refinement phases of our algorithm with the genetic algorithm enable HEXGAN-2 with this method. We benchmarked heterogeneous setting across all four types of workloads. Figure 10 and Figure 11 illustrate the convergence curves and experimental results. Our scheduling algorithm identifies opti- mal assignments for all scenarios within 90 to 120 seconds, which significantly outperforms both the truncated variant and the genetic algorithm, finds assignments that deliver on average 1.8x higher serving throughput and converges much faster, while the others get stuck in local minima. Addi- tionally, we verified that in all cases, the estimated serving throughput closely aligns with the actual throughput. Our scheduling algorithm also scales effectively with larger clusters, we demonstrate the experimental results in Appendix H
approach involving merge,
Thhe placements chosen by HEXGEN-2 for online experiments be found in Appendix

---

Published as a conference paper at ICLR 2025
Edge Swap
Algoritim
HPII
269
515
551
F-iimtal
126-
386
488
257
125
123
175
162
Derraions
Reralioss
Peraioos
Figure 10: Convergence comparison of our proposed search strategy, our strategy without edge swap, and genetic algorithm, where all run 15 times.
HexDen-2
HocGen-2 we Edge Swap
HesCloo22 (Gesstiic (GeseticAlgrriinn Algarilum)
Theosgip (eleza))
484
632
406
500
003
327
11.7%
11
498
11.7k
368
10.2x
11.7x
249
373
235
171
108
LPHD
HPHD
Throughput comparison in heterogeneous setting I among HEXGEN-2, HEXGEN-2 without edge swap, and HEXGEN-2 empowered by genetic algorithm.
Figure 11:
6 RELATED WORKS
LLM inference serving and disaggregated inference paradigm. There are plenty of recent
searches focused on optimizing LLM inference and serving (Li et al. 2023 Kwon et al. 2023
Agrawal al. [2024] [Liu al. [2023]
[W et al., [223] Zhou et al., 2022 Yu et al. 2022 Among
them, vLLM (Kwon et al. 2023) proposes paged-attention to improve the memory efficieccy of
the system. Orca ( Yu al. [2022] introduces continuous batching to improve inference through-
put. AlpaServe (Li et al. 2023) adopts model parallelism to optimize LLM serving performance.
al. (2024) introduces a chunked-prefiln approach and piggybacks decoding requests to improve hardware utilization. Deja Vu (Liu et al. (2023) predicts contextual sparsity on- fhe-fly and uses an asynchronous and hardware-awar implementation to enhance L.M inference. On the other hand, many very recent works have been produced using disaggregated paradigm. Splitwise (Patel et al., 2024) splits the prefill and decoding phases onto separate machines to op- timize hardware utilization. DistServe (Zhong et al. (0224) further implements distinct parallel strategies for different phases. TetriInfer (Hu et al. (024) partitions prompts into fixed-size chunks and adopts two-level scheduling algorithm to improve the performance of disaggregated inference. Mooncake (Qin et al. 2024) features KV cache-centric disaggregated architecture that enhances inference by fully leveraging the underutilized resources of GPU clusters, excelling in long-context scenarios. These works further confirm the effectiveness of the disaggregated architecture.
SARATHI (Agrawal
Heterogeneous GPU computing. Recent efforts have investigated diverse approaches deploy- ing LLMs in heterogeneous environments. LLM-PQ (Zhao et al.) 2024) supports adaptive model quantization and phase-aware partitioning to boost LLM serving efficiency on heterogeneous GPU clusters. Helix (Mei et al.) 2024) formulates heterogeneous GPUs and network connections as discover highly optimized strategies for serving LLMs. HexGen (Jiang et al., 2024b) proposes asymmetric paral- lelism and an advanced scheduling algorithm to deploy generative inference in decentralized and heterogeneous environments. Mélange (Griggs et al., (024) formulates the GPU allocation task as a cost-aware bin packing problem and optimizes cost efficiency for LLM services by leveraging heterogeneous GPU types. Note that our work shares similar objective and but is the first to adapt the disaggregated inference architecture for heterogeneous environments.
maxflow problem, and adopts a mixed integer linear programming algorithm
7 CONCLUSION
We explore the potential of implementing disaggregated inference framework in heterogeneous environments with devices of diversified computational capacities connected over a heterogeneous network. Toward this end, we propose HEXGEN-2, a generative inference framework that incor- porates disaggregated architecture alongside an efficient scheduling algorithm tailored for such deployments. Our empirical study suggests that, given the same budget, HEXGEN-2 can outper- form state-of-the-art homogeneous and heterogeneous inference frameworks by up to 2.0x and on average 1.3x in serving throughput, and reduces the average inference latency by 1.5x. Addition- ally, HEXGEN-2 maintains competitive inference performance relative to leading frameworks with a 30% lower price budget. We believe that such an effort from HEXGEN-2 to provide efficient economical LLM inference could potentially democratize the usage of generative Al.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENT
This work is supported by the HKUST startup grant R9895 from CSE; RGC-ECS project 26218024; RGC-NSFC project CRS HKUST601/24; HKUST-WeBank Joint Lab project WEB24EG01-E; HKUST-Huawei Joint Lab project 24250360Y043.
REFERENCES
Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S Gula- vani, Alexey Tumanov, and Ramachandran Ramjee. Taming throughput-latency tradeoff in llm inference with sarathi-serve. arXiv preprint arXiv:2403.0233 02.00000.. 2024.
Charles J Alpert and So-Zen Yao. Spectral partitioning: The more eigenvectors. the better. In Proceedings of the 32nd annual ACM/IEEE design automation conference, pp. 195-200, 1995.
Anthropic. The claude model family: Opus, sonnet, haiku, 2024. URL /www/cdh/ nthropic.com de 8ba
Cl claude_3.
Joseph Cheriyan and SN Maheshwari. Analysis preflow push algorithms for maximum network flow. SIAM Journal Computing, 1866::005-10866 1989.
Tri Dao. FlashAttemtion Faster attention with better parallelism and work partitioning. In Inter- national Conference on Learning Representations (ICLR), 2024.
Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. In Advances in Neural Information Process- ing Systems (NeurIPS), 2022.
Tyler Griggs, Xiaoxuan Liu, Jiaxiang Yu, Doyoung Kim, Wei-Lin Chiang, Alvin Cheung, and Ion Stoica. M. elange: Cost efficient large language model serving by exploiting gpu heterogeneity. arXiv preprint arXiv:2404.J /4527, 2024.
Bruce Hendrickson, Robert W Leland, et al. A multi-level algorithm for partitioning graphs. SC, 95 (28):1-14, 1995.
Cunchen Hu, Heyang Huang, Liangliang Xu, Xusheng Chen, Jiang Xu, Shuang Chen, Hao Feng, Chenxi Wang, Sa Wang, Yungang Bao, et al. Inference without interference: Disaggregate Ilm inference for mixed downstream workloads. arXiv preprint 11X11 2024.
Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. Advances in neural information processing systems, 32, 2019.
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bam- ford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral experts. arXiv preprint arXiv:2401.04000 2024a.
Youhe Jiang, Ran Yan, Xiaozhe Yao, Yang Zhou, Beidi Chen, and Binhang Yuan. Hexgen: Gener- ative inference of large language model over heterogeneous environment. In Forty-first Interna- tional Conference on Machine Learning, 2024b.
Yibo Jin, Tao Wang. Huimin Lin, Mingyang Song, Peiyang Li, Yipeng Ma, Yicheng Shan, Zhengfan Yuan, Cailong Li, Yajing Sun, et al. P/d-serve: Serving disaggregated large language model at scale. arXiv preprint 1.XY::2408.08147. 2024.
Brian W Kernighan and Shen Lin. An efficient heuristic procedure for partitioning graphs. The Bell system technical journal, 49(2):291-307, 1970.
Woosuk Kwon, Zhuohan Li. Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Prin- ciples, pp. 61 1626, 2023.
11

---

Published as a conference paper at ICLR 2025
Zhuohan Li, Lianmin Zheng, Yinmin Zhong, Vincent Liu, Ying Sheng, Xin Jin, Yanping Huang, Zhifeng Chen, Hao Zhang, Joseph E Gonzalez, et al. (AlpaServe): Statistical multiplexing with model parallelism for deep learning serving. In 17th USENIX Symposium Operating Systems Design and Implementation (OSDI 23), pp. 663-679, 2023.
LibP2P. A modular network stack, 2023. URL
Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivastava, Ce Zhang, Yuandong Tian, Christopher Re, et al. Deja vu: Contextual sparsity for efficient Ilms at inference time. In International Conference on Machine Learning, pp. 22137-22176. PMLR, 2023.
Yixuan Mei, Yonghao Zhuang, Xupeng Miao, Juncheng Yang, Zhihao Jia, and Rashmi Vinayak. Helix: Distributed serving of large language models via max-flow on heterogeneous gpus. arXiv preprint arXiv:2406.0/ 01566, 2024.
Xupeng Miao, Chunan Shi, Jiangfei Duan, Xiaoli Xi, Dahua Lin, Bin Cui, and Zhihao Jia. Spot- serve: Serving generative large language models preemptible instances. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, pp. 1112-1127, 2024.
Deepak Narayanan, Mohammad Shocybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vi- jay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, et al. Efficient large-scale language model training gpu clusters using megatron-Im. In Proceed- ings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1-15, 2021.
NVIDIA. Nvidia collective communications library (ncel) documentation, 2024. URL https:
7/developer: nwidia. com ncc/n
OpenAl. Openai gpt-4o, 2024. URL
got-4o
Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, fnigo Goiri, Saeed Maleki, and Ri- cardo Bianchini. Splitwise: Efficient generative Ilm inference using phase splitting. In 2024 ACM/IEE 51st Annual International Symposium on Computer Architecture (ISCA), pp. 118- 132. 2024.
Ruoyu Qin, Zheming Li, Weiran He, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xin- ran Xu. Mooncake: Kimi's eveache-centric architecture for Ilm serving. arXiv preprint 2rXiv:2407.00079. 2024.
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean- baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gem- ini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024.
RunPod. Runpod documentation, 2023. URL https: /ocp.. runpod. io/
Hugo Touvron, Louis Martin, Kevin Stone. Peter Albert, Amjad Almahairi, Yasmine Babaci, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.092888 2023.
S Vaishali, MS Atulya, and Nidhi Purohit. Efficient algorithms for graph partitioning problem. In International Workshop on Frontiers in Algorithmics, pp. 29-42. Springer, 2018
Gary R Waissi. Network flows: Theory, algorithms, and applications, 1994.
Bingyang Wu, Yinmin Zhong, Zili Zhang, Gang Huang, Xuanzhe Liu, and Xin Jin. Fast distributed inference serving for large language models. arXiv preprint arXiv:2305.05920 2023.
Xiaozhe Yao. Open Compute Framework: Peer-to-Peer Task Queue for Foundation Model Inference Serving. September 2023.
com
amework
12

---

Published as a conference paper at ICLR 2025
Gyeong-In In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung- Gon Chun. Orca: A distributed serving system for (Transformer-Basee generative models. In 16th USENIX Sympo- sium Operating Systems Design and Implementaation (OSDI 22), pp. 521-538, 2022.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang. and Luke Zettlemoyer. Opt: Open pre-trained transformer language models, 2022.
Juntao Zhao, Borui Wan, Yanghua Peng, Haibin Lin, and Chuan Wu. Llm-pq: Serving Ilm on heterogeneous clusters with phase-aware partition and adaptive quantization. arXiv preprint arXiv:2403.01136. 2024.
Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. (DistServe): Disaggregating prefill and decoding for goodput-optimized large language model serving. In 18th USENIX Symposium Operating Systems Design and Implementation (OSDI 24), pp. 193-210, 2024.
Zhe Zhou, Xuechao Wei, Jiejing Zhang, and Guangyu Sun. {PetS}: A unified framework for PPrramete-Efffiieent transformers serving. In 2022 USENIX Annual Technical Conference (USENIX ATC 22), pp. 489-504, 2022.
13

---

Published as a conference paper at ICLR 2025
Table 1: Modeling the generative inference cost and limit.
Deseription
Predil Formulation
Dreade Cost Formulation
24"
12
24h 11
Computation cost
d.y/ce Cr
max
max
R.
h. Map
TP communication
[du,]] Pead'
.46.
make didi
[d.p) (3,00)
Bu H Pugs
by 1o2
communication
mnn acd,y. idi.
Rew
Memory limit
12 Pagee
2b, (of 6t
Boye
X 4. +
Ah (Af +z))) Mt 11
Id.,l
[diy]
26 Burg
KV cache communication cost
We formulate the computation cost, tensor parallel (TP) communication cost, key-value (KV) cache communication cost, momory limit of the j-th stage 6-th pipeline, the pipeline parallel (PP) communicatin cost between the j-th (j+1))thh stages the i-th pipeline for a particular inference tusk & E T. Here, d the GPU device, my is the GPU memory hundwidth, oa is the lensur cure computation power, at and A a² the latency and bandwidth between device and d', di,; is the set of GPUs serves the j-th stage the i-th pipeline that holds lc.; transformer layers, by is the batch size, of is the sequence length of the input prompt. et" is the number of output tokens, H is the size hidden dimension transformer bluck, Bugo denotes the number bytes for precision inference compulation (e.g.. Bayee (FP16) = 2).
Table 2: GPU Deployment, Strategy, and Type.
LLAMA-2 (708)
Heterngenenus Setting
Hederogeneous Setting 3
GPU Configuration
Strategy
Type of Instance
GPU Configuration
Strategy
Type of Instance
1xH1001XXA100
TP=LPPP=
Pretil Instance
2xA100
TF=1,PP=2
Pretili Instance
2xA10022AA6000
TP=2,PP-2
Pretill Instance
21L4043466000
TP=1.PE
Prefill Instance
4xL40
TP=1
Prefill Instance
4KL40
PF=1
Pretill Instance
1sH100-1xA100
TP=1,PP=2
Decode Instance
4GA100
TP-2,PP-2
Decade Instance
24A100-24A6000
TP=2,PP-2
Decude Instance
2141043066000
TP=1,PP=S
Decode Instance
4xL40
TP-2,PP-2
Decule Inslunce
4xL40
TP=2,PP-2
Decade Instance
Heterogeneous Setting 2
Heferogeneous Setting 4
GPU Configuration
Sirategy
Type of Instance
GPU Confgurratioo
Strategy
Type of Instance
1<H100=IxA100
TP=1,PP-2
Prefill Instance
1x1100-xAA100
TP=1,PP=2
Prefill Instance
A4TAD42xA6000
TP=2
Prefill Instance
2xA 100
PP=2
Prefill Instance
2x1110022A100
TP=2,PP-2
Decole Instance
211100244100
TP=2,PP-2
Decude Instance
4XL10HXA6000
TP=1,PP=2
Docode Instance
4XA100
Decode Instance
OPT (30B)
Heterogeneous Setting
Heterogeneous Setting 4
GPU Configuration
Strategy
Type of Instance
GPU Configuration
Strategy
Type Instance
1s1110=1xA1100
TP=1,PP-2
Prefill Instance
1x11100
TP=1,PP=1
Prefill Instance
2XA100
TP=2.PP=1
Prenil Instance
IxA100
TP=1.PP=1
Pretill Instance
2XLA0+1AA6000
TP=L.P
Prefill Instance
IxA100
TP=11PP=1
Pretill Instance
2XL40+1XA6000
PP=
Pretill lostance
IXAT00
TP=1.P==1
Prefill Instance
1<H100~1xA100
1P-1
Decode Instance
2HH100
TP-2,PP-1
Decode Instance
2xA100
TP=1,PP-2
Decole Instance
2A110
TP=1,PP22
Decude Instance
IAA6000
TP=1
Decude Instance
2AA100
TP=1.P--2
Decude Instance
TP=1
Decode Instance
2AA100
1P=1
Decode Instance
A GENERATIVE INFERENCE COST ESTIMATION
Node capacity estimation. To estimate the generative inference cost, we adopt the cost model from HEXGEN (Jiang et al., (024b) and summarize the computation costs, communication costs, and memory consumption constraints in Table I The inference latency for a single request is calcu- lated by summing the total computation and communication costs. We determine the capacity of the compute- bound prefill node, where batching more requests does not enhance system through- put, by dividing the predefined time period by the latency. Conversely, for the memory I/O-dound decoding node, which benefits from batching, we calculate its capacity by dividing the product of the maximum available batch size and the time period by the latency.
Edge capacity estimation. For connection types (1) and (2) mentioned in $3.3 the edge capacities are equal to the product of the predefined time period and the connection bandwidth, divided by the transmission size of request. For connection type (3), the edge capacity is equal to the time period divided by the estimated KV cache communication cost in Table I As mentioned in $3.3 the edge capacity of connection type (3) is determined by the collective performance of all GPU-to-GPP transmission connections, as each connection is responsible for a portion of the KV cache transmis- sion. To optimize it, given the parallel configurations of the prefill and decoding model replicas,
14

---

Published as a conference paper at ICLR 2025
adjust the pipeline stage order of both phases to minimize the overall KV cache communication cost, which in turn determines the edge capacity.
Memory requirement estimation for single model replica. The memory cost model in Table 1 estimates the memory required for a single model replica. To determine the total memory require- ment for single model replica, assume batch size of 32 concurrent requests (i.e., b, 32). Thus, the total memory requirement is calculated as: model parameter size + 32 KV cache size per request.
B HEXGEN-2 SCHEDULING RESULTS
We list the model serving group partitions and types generated by HEXGEN-2 the online experi- ments for each heterogeneous setting in Table 2
C SCHEDULING ALGORITHM ANALYSIS
The scheduling algorithm aims to optimize the deployment of large language model (LLM) infer- ence workloads on heterogeneous GPU cluster. The optimization involves the following essential phases:
Graph partition. The initial partition focuses on creating memory-balanced groups and optimiz- ing the capacity within each group. The secondary partition determines group type (i.e., prefill or decoding), focusing on maximizing inter-type communication bandwidth for efficient KV cache transfer.
Max-flow. This phase determines optimal parallel strategies for each group and determines optimal inter-type KV cache communication paths based on the max-flow outputs.
Iterative refinement. This phase continuously adjusts partitions and strategies based on workload demands until no further improvements can be made.
The upper bound for graph partitioning indicates the optimal utilization of heterogeneous compu- tation power and connections. The theoretical upper bound of the graph partition phase is achieved when the cluster partitioned into groups with balanced memory capacities and optimized pro- cessing capabilities, and the groups are assigned types (i.e., prefill decoding) in a manner that maximizes inter-type communication bandwidth for key-value (KV) cache transfers.
The upper bound for max-flow indicates the maximum possible data flow within the cluster: The theoretical upper bound of the max flow phase determined by the maximum possible data transfer rate of the entire system. This upper limit is achieved when the system fully utilizes the inter-type network bandwidth for KV cache transfers and optimizes the processing capabilities of the prefill and decoding model replicas.
Based on our scheduling algorithm, the optimization will iteratively narrow the gap between the current allocation and the theoretical upper bounds, where the iterative refinement process addresses the limitations inherent in each phase. The challenges in reaching upper bounds lie in two aspects:
In the graph partition phase, creating an ideal graph partition in a single iteration is challeng- ing since this phase lacks critical information (e.g., parallel strategy and KV cache communica- tion path) from subsequent phases. Without these insights, the initial graph partitioning cannot guarantee an ideal utilization of the heterogencous cluster, leading potential communication bottlenecks and workload imbalances.
The max flow phase operates within the constraints set by the graph partition. The max-flow algorithm cannot achieve the theoretical maximum flow if the preceding graph partition results in sus--than-ppimall grouping. Limited inter- group communication bandwidth and unbalanced node capacities prevent the system from fully utilizing the network's 'ss data transfer capabilities.
Iterative refinement. The iterative refinement phase is crucial in bridging the gap toward the upper bounds. It continuously evaluates and adjusts groupings, fine-tunes parallel configurations and recalculates optimal KV cache communication paths based on updated partitions, This approach allows the algorithm to:
15

---

Published as a conference paper at ICLR 2025
Rebalance trade-off's for graph partition. Balance intra-group resource optimization with inter- type communication efficiency for optimized resource utilization.
Enhance max-flow potential. Balance overutilized and underutilized edges within the formulated flow network for optimized data flow efficiency.
D DISAGGREGATION AND CHUNKED PREFILL
Chunked prefill (Agrawal et al. 2024) is a method that divides input tokens into smaller chunks, which are then processed in continuous batch. This approach simplifies scheduling by treating all nodes uniformly and enhances computational efficiency during decoding, improving machine uti- lization. However, this approach may not result in significant performance gains across all workload types. We evaluate chunked prefill using vLLM (Kwon et al., (023) on one H100 GPU serving the OPT-30B model. Experimental results demonstrate that HPLD and LPLD workloads, chun- ked prefill brings an approximately 20% throughput improvement, while it only brings around 5% throughput gains on HPHD and LPHD workloads. Therefore, choose disaggregation, which enables different batching strategies, resource allocations, and parallel approaches for each phase, providing greater flexibility in handling various types of workloads.
E CASE STUDY: SCHEDULING ALGORITHM ANALYSIS ON SMALL CLUSTER
In this section, we provide study of our scheduling algorithm relatively small size hetero- geneous cluster with 4 H100s and 4 A100s for better understanding of our scheduling algorithm. The detailed procedures are listed bellow.
E. PHASE 1: GRAPH PARTITION
The graph partition phase aims to find the group construction and type mentioned in 3.1
Step 1: initial partition. Step divides the GPUs into multiple independent groups based min- imizing inter-group communication bandwidth and balancing the memory capacity of each group. After step the cluster is divided into four groups gl-4, and the construction of each group gl: two H100, g2: two H100, g3: two A100, and g4; two A100.
Step 2 & 3: coarsen & secondary partition & projection. This step aims to distinguish the type for each group (prefill or decoding). In the small case, gl and g3 are determined to be the prefill model replicas, and g2 and g4 are determined to be the decoding model replicas.
F.2 PHASE 2: MAX-FLOW ALGORITHM
The max-flo max-flow algorithm aims to fine the parallel strategy and KV cache communication path men- tioned $3.1
Step 1: find the optimal parallel strategies for prefill and decoding groups. This step deter- mines the latency- and throughput-optimal parallel strategies for prefill and decoding model repli- cas. After searching, gl and g3 (prefill model replicas) use parallel strategy of (TP=2, PP=1) (latency-optimal while g2 and g4 (decoding model replicas) use a paralicl strategy of (TP=1, PP-2) (throughput-optimaa))
Step 2: find the optimal KV communication path. We run preflow- push algorithm to get the max flow of the cluster. The generated flow assignments used guide the KV cache communication. In the small case, gl (prefill model replica) communicates with g2 (decoding model replica), and g3 (prefill model replica) communicates with g4 (decoding model replica).
16

---

Published as a conference paper at ICLR 2025
E.3 PHASE 3: ITERATIVE REFINEMENT
The iterative refinement phase aims at co-optimizes the four objectives (group construction, group type, parallel strategy and KV cache communication path) in the first and second phases.
Iterative refinement using swap operation. We use max-flow guided edge swap to iterative refine the graph partition until no further improvements can be made. For instance, for workloads with light prefill and heavy decoding (LPHD) needs, the algorithm would attempt to allocate more resources to decoding model replicas. In the small case with LPHD workloads, one H100 from gl (prefill model replica) is swapped into g2 (decoding model replica) and one A100 from g3 (prefill model replica) swapped into g4 (decoding model replica) for enhancing the decoding ability of the system and maximizing the system throughput. The iterative refinement will optimize the plan for any given LLM inference workload accordingly given the workload characteristics.
In this small case, the output of our scheduling algorithm is the same the output that is derived through exhaustive search.
F COMPARE HEXGEN-2 WITH VLLM
In this section, we conduct additional experiments to compare HEXGEN-2 with state-of-the-art LLM serving platform. We evaluated vLLM using the same homogeneous experimental setup described rent 8 H100 GPUs from the RunPod platform and test vLLM with the Llama2-1 70B model using samples from the Azure Conversation dataset. As demonstrated in Table 3] HEXGEN-2 achieves up to 2.1 x and average 1.5x higher serving throughput compared with vLLM in our testbed.
in $5.1 Specifically,
Table 3: Comparison between different frameworks with different setups.
Setting
System
HPLD
HPHD
LPHD
LPL.D
Online
Heterogeneous Setting
HRXGEN-2
157 tokens/s
448 tokens's
689 tokens/s
570 tokens/s
350 tokens/s
Heterogeneous Setting
HEXGEN
123 tokens/s
375 tokens's
tokens/s
407 tokens/s
259 tokens/s
Homogeneous Setting
DISTSERVE
128 tokens/s
368 tokens/s
553 tokens/s
291 tokens/s
251 tokens/s
Homogeneous Setting
VLLM
97 tokens/s
437 tokens/s
563 tokens/s
270 tokens/s
256 tokens/s
G CASE STUDY: HOMOGENEOUS SYSTEM COMPARISON
In this section, we compare HEXGEN-2 with DISTSERVE and HEXGEN in homogeneous setup.
Experimental setup. To compare the runtime of HEXGEN-2 with DISTSERVE and HEXGEN, we rented H100 GPUs from the RunPod platform and tested serving throughput on the OPT-30B model using the four types of LLM inference workloads (HPLD, HPHD, LPHD, LPLD) described in $5.1
Compare with DISTSERVE. We found that for certain inference workloads, the scheduling results of HEXGEN-2 and DISTSERVE differ. For example, with the HPLD workload, HEXGEN-2 fa- vors replicating more model replicas to enhance the system's parallel processing, while DISTSERVE prefers model parallelism to distribute the computation of a single model replica across multiple GPUs. Experimental results demonstrate that HEXGEN-2 outperforms DISTSERVE in certain cases due to better scheduling results while delivering comparable performance when the scheduling out- comes are the same.
Compare with HEXGEN. HEXGEN-2, with optimized scheduling in a disaggregated architec- ture. minimizes interference between the prefill and decoding phases of LLM inference. It selects appropriate parallelism and batching strategies for each phase, resulting in improved inference per- formance compared to HEXGEN in homogeneous environment.
H CASE STUDY: SCHEDULING ALGORITHM SCALABILITY
In this section, we conduct additional experiments to evaluate the scalability of our scheduling algo- rithm. The results are shown below.
17

---

Published as a conference paper at ICLR 2025
Table 4: Throughput comparison a homogeneous cluster.
HEXGEN-2
DISTSERVE
HEXGEN
HPLD
365 tokens/s
302 tokens/s
277 tokens/s
HPHD
683 tokens/s
692 tokens/s
505 tokens/s
LPHD
758 tokens/s
774 tokens/s
533 tokens/s
LPLD
730 tokens/s
553 tokens/s
545 tokens/s
Table 5: Algorithm convergence time across different cluster sizes.
Ngpus
Time (min)
64
4.03
128
7.93
192
21.66
256
28.44
320
47.77
Experimental results demonstrate that scheduling algorithm scales polynomially and shows po- tential for addressing larger and more complex heterogeneous scheduling problems.
18


Reviews:
[{"review_id": "W7r6wrSD6l", "reviewer": "Reviewer_cEk2", "summary": "The paper introduces the disaggregated LLM servings into the heterogeneous environment. The deployment is formulated as an optimization problem, a graph max flow problem. Results show improvement compared to the homogeneous methods.", "questions": "See above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The idea of scheduling among heterogeneous GPUs is intuitive. And the design provides insights. Using graph partitioning and modeling the query processing as max-flow problem is valid.\n\n- The paper is well-written and generally easy to follow.", "weaknesses": "- Perhaps the paper can provide some analysis about the optimization problem. \n  - Is there an upper bound of the performance (from the perspective of graph partitioning and max flow problem, respectively) your approach can reach? How is the performance your method achieves compared to this upper bound?\n\n- The main contribution seems to be formulating the optimization problem while some parts of the solutions are not that new.\n\n- Some details can be clarified in the paper.\n  - Which GPU is used in Figure 1?\n  - What is the outcome (objective) of the graph partitioning? How to determine which group is prefill group and which one is decoding group? Are the results optimal?\n \n- Miscellaneous\n  - carefully use \\cite and \\citep for citation.\n  - Capitalize the first letter after the colon.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GpgRlKviBu", "reviewer": "Reviewer_cxpP", "summary": "The paper introduces HexGen2, a distributed LLM inference framework targeting heterogeneous GPU clusters. The framework disaggregates prefilling and decoding tasks onto different GPUs, ensuring that the two phases do not interfere with each other and that they can be parallelized in different ways when doing so is beneficial.", "questions": "- Is HexGen2 built on an existing runtime, or was it developed from scratch? Specifically, how does HexGen2's runtime compare to those of DistServe and HexGen? Understanding this would be helpful for better assessing the end-to-end performance comparison with the other two frameworks.\n- In the evaluation settings, could you explain your motivation for selecting a 70% lower budget as the target for the evaluation scenario? How did you arrive at this percentage?\n- Do you have any insights or, preferably, evaluation data regarding which additional heterogeneous clusters HexGen2 would perform well with, besides those already evaluated? Does HexGen2 support any type of cluster, or are there specific restrictions regarding GPU types, interconnects, or CUDA architectures?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- LLM inference on heterogeneous GPUs is a critical and timely issue. Separating the prefill and decode stages when serving LLMs in diverse clusters represents a novel approach.\n- Evaluation is thorough\n- Very clear presentation", "weaknesses": "- The evaluation does not include an ablation study that compares HexGen2's runtime with those of DistServe and HexGen. Understanding HexGen2's performance in terms of throughput and latency compared to DistServe and HexGen in a homogeneous setting would be valuable.\n- In the introduction, the authors assert that disaggregated inference is the most efficient framework for serving large language models (LLMs) without providing proof or citations. Is this assertion already widely accepted by the community, or is there still an ongoing debate regarding the advantages of disaggregation versus chunked prefills? It would be beneficial for the authors to clarify this—at least in the related work section—and to explain their reasoning for choosing the disaggregation approach over chunked prefills.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RCfNxtAKCx", "reviewer": "Reviewer_iocs", "summary": "This paper presents HexGen-2, a framework designed for serving language models on heterogeneous clusters with disaggregating prefill and decode computation on different devices. HexGen-2 first used a multi-round partition to divide all devices into multiple groups, and then introduced a maxflow algorithm to decide how to dispatch and route the workload on each device partition.\nThe authors did experiments on 5 different heterogeneous clusters, as well as 4 different workload pattern. Results show that even under a lower price budget, HexGen-2 matches the performance of the state-of-the-art disaggregated serving platform, DistServe, if not surpassing.", "questions": "Some questions that could help make the paper more clear is already mentioned in the Weakness section. In addition to that, there are some other concerns:\n- According to Figure 4, the network topology is globally heterogeneous but locally homogeneous: to any destination device $u$, the bandwidth $(u_1,v),(u_2,v)$ from the same type of GPU $u_1$, $u_2$ is always the same. In this case, the Graph Partition algorithm seems an overkill to me: what if we consider to directly merge devices by the GPU types they belongs to, and then bipartition GPU types according to the inter-type bandwidth? Showing some nontrivial examples generated by HexGen-2 could also explain the importance of the graph partition algorithm.\n- The context of serving with heterogeneous cluster lacks detailed introduction. As an important benchmark baseline, as well as the system that this work is built on top of, HexGen itself is not well introduced and explained. For example, the author mentioned the \"genetic algorithm\" in ablation study of scheduling algorithm but never explained it. Adding more detail to section 2 and 5 about the background of heterogeneous serving and the two specified baseline could help improve the paper's self-completeness.\n- The benchmark baseline is too weak. On Llama-70B, vLLM, which does not even support disaggregation, reported a throughput of 25 req/s ($\\approx$400 tok/s) on **4 H100-80GB** GPUs with the \"prefill-heavy\" workload (https://blog.vllm.ai/2024/09/05/perf-update.html). Given that their prefill token's average length is only around 400 tokens, I compare it with the \"LPLD\" benchmark: In this way, HexGen-2 uses 2x more budgets, but only with a maximum throughput around 550 (1.4x higher). Adding more experiments with state-of-the-art LLM serving platform for the homogeneous setup could better show the predominance of HexGen-2 against using homogeneous clusters with the same budget.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- The paper is the first to combine disaggregated serving together with heterogeneous devices, which is an important direction. This enhances the paper's novelty.\n- The approach of using Max-Flow to solve the scheduling problem is interesting and efficient.", "weaknesses": "- The detail of the paper's method is not well motivated, neither well explained (to clarify physical node with abstract graph `nodes` in the algorithm, I use `this font` for abstract graph nodes):\n  - In Graph Partition, the motivation of partitioning with two steps is not well explained. Specifically, it is confusing that why the first round minimizes the edge cost, while the second round maximizes the cost. Besides, why the `node` weight should be balanced (i.e., the computation capacity of each partition should be roughly the same)?\n  - In Graph Partition, step 2, the concept of \"partition\" and \"replica\" causes confusion. Does each partitioning output group correspond to a replica? Is there multiple replicas for prefill, or only one replica for prefill, and one fore decide?\n    - It seems to me that there are multiple prefill replicas (according to Max-Flow \"for prefill model replicas, ...\"). However, in this case, maximizing the total edge cost is not reasonable because the edges between prefill replicas are meaningless.\n    - Does the algorithm consider the balance between node weights of (all) prefill replica(s) and that of (all) decode replica(s)? I think this is important because it avoids one part having too few devices to compute.\n  - In Graph Partition, there is an input argument for the targeted group size $K$. How is this argument defined and set?\n  - In Max-Flow, the author estimated the communication cost by dividing the total communication volume to the bandwidth. However, there exists a case that the bandwidth of some `node`s are shared, because these GPUs belong to the same physical node. On the other side, when both the send and receive side are on different physical nodes, the communication can be parallelized. Using the collective performance is not accurate.\n  - In Max-Flow, the author mentioned that each \"model replica\" finds the optimal parallelism strategy themselves. Is this model replica inherited from Graph Partition? If so, what is the motivation of the projection?\n  - In Max-Flow, what is the search space of the parallelism strategy? For example, when a model replica has two type of GPUs (e.g. one A100 and 4 V100), what is the parallelism strategy?\n- The evaluation is not convincing to me. The baseline is too weak (see Question section for more details);", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SvLV3031AL", "reviewer": "Reviewer_Rq11", "summary": "Traditional LLM serving frameworks co-locate the execution of prefill and decode stages, leading to prefill-decode interference.\nDisaggregated inference is a more efficient LLM serving approach that reduces such interference and allows more flexible scheduling of the two stages.\nThis paper proposes HexGen-2, a disaggregated LLM inference framework that coordinates distributed LLM inference over a heterogeneous set of GPUs and network connections.\nThe core of HexGen-2 is a two-level scheduling algorithm that leverages graph partitioning and max-flow algorithm to determine the resource allocation and parallelization plan for the prefill and decode stage.\nEvaluations have shown that HexGen-2 could achieve better serving throughput and lower average latency compared to SOTA works.", "questions": "Thank you for submitting to ICLR 2025! I think this paper tries to tackle the important problem of scheduling heterogeneous GPU and network resources for disaggregated LLM serving.\nDespite the good evaluation results, I have a few comments for the paper and it would be great if the authors could address them.\n\nThe current explanation for the core scheduling algorithm is confusing and unclear.\nThere is no clear intuition on why the algorithm should be designed at these two levels.\nThere is also no clear guidance on what the objective is during each phase of the algorithm.\nFor example, in the first step of the first phase: graph partition, is balancing the node weights (memory capacity) the overall objective at this step?\nThe node weight in the global graph seems to be determined by GPU memory but not GPU compute, why is that the case?\nWhy is the coarsen step necessary in Step 2?\nMy understanding is that a prefill or decode replica could have multiple super nodes, is that right?\n\nIn addition, in the second phase, why is it required that each compute node needs to be connected to two other nodes in the same graph?\nHow are latency-optimal configuration and throughput-optimal configuration for prefill and decode replicas respectively related to the max-flow algorithm used?\nIn iterative refinement, what is the physical meaning of swapping edges?\nI think the scheduling algorithm should also have easy-to-follow examples on the side to clearly give intuitions to readers.\nThe current Figure 3 is hard to understand even after reading the entire Section 3.\n\nI am also wondering how the current algorithm compares to the optimal allocation and parallelization plan.\nDespite the NP-hardness of the problem, the optimal plan should be solvable by just brutal forcing all possible plans in small cases such as 4 H100 and 4 A100 GPUs.\nIn Section 5.3, it says \"Our scheduling algorithm identifies optimal assignments for all scenarios within 90 to 120 seconds\". \nHow is optimality defined here?\nIs the algorithm always guaranteed to find the optimal solution given enough search time?\n\nAlso, how much overhead would the algorithm incur if running on a cluster with, for example, hundreds of GPUs?\nIt may be hard to rent such a large number of GPUs for experiments, but evaluating algorithm overhead should be possible.\n\nIn the evaluation, it says HexGen-2 is compared against DistServe under the homogeneous setting with 8 H100 GPUs.\nWhy is DistServe appearing in all heterogeneous settings in both Figure 6 and Figure 7?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The problem of scheduling heterogeneous resources over disaggregated LLM inference makes sense.\n2. The framework shows good improvement over prior works on serving throughput and latency.", "weaknesses": "1. Lack of intuition and lack of clear and detailed explanation for the core scheduling algorithm.\n2. Did not compare the scheduling decision with the optimal solution in small cases.\n3. Lack of justification on scalability of the framework.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)