HyGen: Efficient LLM Serving via Elastic Online-Offline Request Co-location
Authors: Ting Sun
Penghan Wang
Fan Lai
Venue: NeurIPS 2025 poster ()
Keywords: LLM Inference
Request Scheduling
Machine Learning Systems
Primary area: infrastructure
TL;DR: Colocating online and offline LLM inference requests in the same inference engine.

Abstract: Large language models (LLMs) have facilitated a wide range of applications with distinct service-level objectives (SLOs), from latency-sensitive online tasks like interactive chatbots to throughput-oriented offline workloads like data synthesis. The existing deployment model, which dedicates machines to each workload, simplifies SLO management but often leads to poor resource utilization. This paper introduces HyGen, an interference-aware LLM serving system that enables efficient co-location of online and offline workloads while preserving SLOs. HyGen incorporates two key innovations: (1) performance control mechanisms, including a latency predictor to estimate batch execution time and an SLO-aware profiler to quantify latency interference, and (2) SLO-aware offline scheduling policies that maximize serving throughput and prevent starvation. Our evaluation on production workloads shows that HyGen achieves up to 3.9-5.8× throughput gains over online and hybrid serving baselines, while ensuring latency SLOs. The code of HyGen is publicly available at https://github.com/UIUC-MLSys/HyGen.

HyGen: Efficient LLM Serving via Elastic Online-Offline Request Co-location
Ting Sun"l, Penghan Wang" Fan Lai'
Siebel School of Computing and Data Science, University of Illinois Urbana-Champaign
2 Department of Computer Science, Purdue University
suntcrickdgmail com, xang61994purdue. edu, fanlai@illioois.o.mm
Abstract
Large language models (LLMs) have facilitated a wide range of applications with distinct service-level objectives (SLOs), from latency-sensitive online tasks like interactive chatbots to throughput-oriented offline workloads like data synthesis. The existing deployment model, which dedicates machines to each workload, simplifies SLO management but often leads to poor resource utilization.
This paper introduces HyGen, interference-awee enables efficient co-location of online and offline workloads while preserving SLOs. HyGen incorporates two key innovations: (1) performance control mechanisms, including a latency predictor to estimate batch execution time and an SLO-aware profiler to quantify latency interference, and (2) SLO-aware offline scheduling policies that maximize serving throughput and prevent starvation. Our evaluation on production workloads shows that HyGen achieves up to 1.95.88 throughput gains over online and hybrid serving baselines, while ensuring latency SLOs. The code of HyGen is publicly available wttttps:// //tpt/ggttttttt com/ MLSU
LLM serving system that
Introduction
Large language models (LLMs) have emerged as transformative tools across diverse domains, handling both latency-critical online requests (e.g., chatbot interactions 132 39 and throughput- oriented offline tasks (e.g., document summarization (18). Online serving demands low and stable response times, measured by Time First Token (TTFT) and Time Between Tokens (TBT), while offline tasks prioritize high throughput and resource utilization, often processing large batches with relaxed latency constraints. The disparity in these requirements has led most production deployments to segregate online and offline serving onto separate clusters to avoid interference [41 [1] 69]
However, real-world L.M workloads demonstrate significant temporal variations in request load. Our analysis of production traces shows that in addition to the popular diurnal request arrival patterns, online request rates vary by up 3x within minutes (Section 3). To meet latency requirements under such bursty loads, service providers have to provision GPU resources for peak demand, as minute-level resource scaling is often impractical due to infrastructure complexity and engineering overhead
15 64 66
leading to substantial underutilization during off- fef-peak hours.
This resource overprovisioning suggests an opportunity to improve resource efficiency via hybrid coviing-co-locatingg online and offline workloads on the same inference engine instance. By opportunistically padding online requests with offline requests during periods of low online load, the system could maintain high GPU utilization while preserving latency guarantees for online requests. However, realizing this opportunity requires addressing several fundamental challenges.
"Equal contribution.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

First, LLM services exhibit diverse latency requirements across applications [32]. Interactive chatbots require consistent response times with both low initial latency (TTFT) and smooth token generation (TBT), while batch processing tasks prioritize throughput over latency. These requirements often manifest in different statistical metrics- frrm strict P99 latency bounds to mean performance targets- making it difficult to establish unified resource-sharing policies.
Second, LLM workloads are inherently unpredictable in both their arrival patterns and resource demands, Request arrival rates exhibit both diurnal patterns and unpredictable short-term fluctuations. This variability is further complicated by uncertainty in resource demands- input sequences vary widely in length, and the number of output tokens can hardly be predicted until generation completes.
Third, co-locating online and offline workloads introduces interference. For instance. offline requests may delay time-sensitive online requests; using large batch sizes to improve the throughput of offline requests increase the latency of online requests. Effectively managing this interference while ensuring latency guarantees demands meticulous orchestration of resource sharing.
This paper presents HyGen, interference-aware LLM serving system that elastically co-locates online and offline workloads. HyGen introduces several key techniques: (1) a latency predictor that accurately estimates the execution time of different request batches, (2) an interference-aware profiler that quantifies the performance interference of co-location, and (3) an adaptive scheduler that maximizes offline throughput while maintaining strict latency guarantees for online requests.
Our evaluation on production workloads shows that HyGen improves serving throughput by 3.87- 5.84 over existing advances (2), while guaranteeing strict SL.O compliance. To summarize, this paper makes the following contributions:
Key insights on the feasibility and benefits of co-locating online and offline LLM workloads, derived from systematic characterization of production traces.
statistical latency prediction model that accurately captures the relationship between batch composition and execution latency, accounting for quadratic complexity in prefill and linear scaling decode phases.
A novel scheduling system that dynamically co-locates online and offline workloads, formulated as a constrained optimization problem that maximizes throughput with prefix sharing while preserving strict latency SLOs and providing theoretical guarantees for fairness.
Experimental evaluation real-world workloads demonstrating up to 5.84x throughput improve- ment over state-of-the-ree alternatives.
2 Related Work
LLM Inference Optimization.
Recent advances in LLM inference through kernel optimization [12
61 , compilation frameworks 33 59
and scheduling algorithms 30
31 55
have substantially
improved LLM serving performance. Among these, a particularly important development is predictive scheduling, which aims to estimate request difficulty [13 38 or generation length [16
23 27
Contrasting these efforts, HyGen uniquely predicts batch execution time, providing precise control over request interference during co-located workloads. Additionally, the concept of request prefix sharing has been explored by [17 [17 (1), where shared prefixes across requests optimize resource use. HyGen takes this a step further by applying a prefix sharing maximization strategy to opportunistically schedule offline requests, utilizing residual capacity from a primary, ALO-bound online workload. Moreover, HyGen introduces a fairness aware extension to the standard prefix- -eefix-sharing maximization (PSM) method, addressing the issue of starvation often encountered in naive designs.
In parallel, the increasing demand for offline LLM inference has driven two key trends. First, platforms like Huggingface Accelerate (25), DeepSpeed ZeRO-Inferenee 44, and FlexGen [51] enable efficient inference on commodity hardware through memory offloading across GPUs, CPUs, and disk. Second, cloud providers, including OpenAl's Batch API [4), have launched specialized services optimized for throughput rather than latency, processing millions of requests daily for large-scale data processing tasks.
Workloads Co-location. In data center environments, co-locating latency-sensitive applications with batch applications has been explored improve resource utilization [8] 9 42 68 However,

---

88
Users
Querics Per Minute
submit anline requists
aubmit offlina requeats
400
Online Request Queue
Offline
2
Request
update index
Queue
200
Online Selector: FCFS. VTC...
Offinne Selector: FCFS, PSM...
select add requests
18:15
18:25
18:35
18:45
18:55
19:05
19:15
select add
offiie requests
Section 4.2: predict execution latency
Queries Per Second
15
10
Latency Predictor
Continuous
Batch
dispatch tasks
18:44
18:45
18:46
6PU
GPU
Workers
Figure 1: Request rate varies significantly Microsoft Azure's LLM service over one- hour and two-minute periods.
Figure 2: HyGen Overview. Online and offline requests are processed asynchronously, with offline requests op- portunistically scheduled to respect latency budgets.
these approaches generally overlook the unique characteristics of LLM inference. In the context of LLM inference, several works have investigated the co- co-location various model types and tasks. For instance, Punica 77). S-LORA [49], and dLoRA (58) batch requests from different LoRAs, while MuxServe [14] multiplexes resources across multiple LLMs. contrast, HyGen focuses on batching online and offline requests, addressing distinct optimization problems that not interchangeable.
Background and Motivation
This section first introduces the background of LLM serving deployment (Section then illustrates how these characteristics motivate our system designs (Section 3.2)
3.1
LLM Serving
Large-scale inference clusters consist of multiple serving instances, with router intelligently direct- ing incoming requests to the most suitable instances [53] 655) Each instance, which could based on architectures like vLLM (29] SGLang (71). typically employs iteration-level scheduling [63 and chunked prefill (2). This setup enables decode requests to perform an additional decoding step, while prefill requests are limited to fixed token budget. These two types of requests are processed together within single iteration, optimizing resource utilization.
LLM serving deployment can be categorized into online serving and offline serving scenarios. Online serving targets real-time user interactions, such as chatbots, code assistants, and interactive applications [35] This interactive nature often requires a short Time to First Token (TTFT) as well as short Time Between Tokens (TBT). Offline serving prioritizes throughput over latency, measured in queries per second (QPS) or tokens per second (TPS). For example, OpenAl's Batch API processes requests with relaxed latency requirements (up to hours) significantly lower
costs compared to standard APIs 40). Applications include model benchmarking [2] 34 document data cleaning
processing 10 67
36 70], and data synthesis
3.2 Motivation and Challenges
L.MM serving systems face critical resource utilization challenge due to the highly variable nature of their workloads. Figure F reports analysis of Microsoft Azure's LLM service [4], showing that

---

request rates can fluctuate saamaaticall--vryyiiii up to 3x within minutes while following broader diurnal patterns. This variability creates an inherent tension in resource provisioning: serving clusters must be sized to handle peak loads, leading to resource underutilization during off-peak periods.
This load variability suggests an opportunity to improve resource utilization by co-locating online requests with offline requests. For each serving instance in the cluster, during periods of low online traffic, it can opportunistically schedule offline tasks harvest idle resources. While recent methodologies in simultaneous batching of prefill and decode requests set the premise for dynamic request co-location
21 doing so at scale introduces several fundamental challenges:
Diverse Latency Requirements: Applications and even requests of an application have distinct latency requirements, For example, paid users require strict latency SLOs while free users accept more relaxed guarantees. How respect diverse latency requirements in flight?
Massive Uncertainties: L.M serving faces temporal uncertainty-onlee
requests arrive in unpredictable bursts with varying urgency levels as well as resource demand uncertainty due to unpredictable output lengths. How to perform efficient scheduling in the wild?
3. Request Interference: Co-locating online and offline workloads introduces performance inter- ference. Large batches of offline requests can cause severe head-of-line blocking, delaying the processing of time-sensitive online requests. Worse, batching requests of long inputs with short in- teractive queries can elongate the latency of all requests in the batch by an order of magnitude (2). How to account for interference in co-locating requests?
The HyGen Design
4.1 Overview
HyGen introduces novel approach to integrate online and offline requests while maintaining strict latency guarantees. As shown in Figure 2 HyGen employs a dual-queue architecture that separates latency-sensitive and throughput-oriente variable workloads while remaining compatible with existing scheduling policies within each queue. We note that HyGen functions as an instance-level scheduler, receiving requests from an upstream system-level router (e.g., Preble (53)). As a result, both the request concurrency and scheduling overhead at each instance are inherently bounded. HyGen's two-phase scheduling operates as follows (see Appendix A. for the asynchronous two-queue workflow and message passing details):
requests. This design accommodates diverse SLOs and
The online phase prioritizes latency-sensitive requests, forming an initial batch using established policies such as Pees--Com-FFirt-SSeee (FCFS) [29 or fairness request scheduling [50]. We introduce priority- based preemption mechanism that protects online request performance by selectively preempting offline requests. Currently, HyGen preserves execution state for preempted requests, while its architecture supports various preemption mechanisms-includnng state discarding, preservation, and swapping- categorized by InferCept O.
The offline phase uses our latency predictor to allocate remaining capacity to throughput-oriented requests. This predictor accurately estimates the latency impact of each potential offline request addition, determining either decode request latency costs or maximum chunked prefill lengths that fit within the available latency budget without violating online SLOs.
Algorithm I formalizes our scheduling approach, with implementation details and complexity analysis (O()) where is the number of requests) provided in Appendices A. |and|A.A Appendix C presents a novel cluster serving paradigm that addresses the longstanding tradeoff between SLO compliance and resource utilization based on HyGen.
4.2 Performance Control Mechanisms
Co-locating online and offline requests under diverse SLO requirements requires precise control over resource allocation. The key challenge lies in accurately estimating the latency impact of scheduling decisions to ensure SL.O compliance. This section introduces latency predictor for estimating batch exccution latency and an SLO-aware profiler for translating the estimates into scheduling decisions.

---

Algorithm HyGen SIO-aware scheduler
1:
function SLO
2: Input: running requests It request queue Q
3: latency budget I, chunk size memory budget
4: Output: batched requests B
5: B 1-1 {
6:
for R.decode do freg +- PREDICTOR preddcct DECODE) freg I t or PHASE = ONLINE then
7:
// predict latency of the decoding request
8:
9:
// schedule request if it is: 1. online, or 2. offline and enough latency budget left
10:
t t treg B 3 BU [(r, 0, breq)) for E R.prefill UQ do
12:
// try schedule a prefilling waiting request I/ get the max number of tokens allowed under memory and latency budget litreg PREDICTOR max max Baaaamaaaaaaaaaaaaaa tokens(t, c, m, r)
TRY_SCHEDULE:
14:
16:
if 0 then
schedule request t t t-treg
18:
C c-1
20:
+- mn- GET BIET_NUMBBRR NUM_BLOCKS()
21:
B +- BU ((r, brog))
22:
else
PHASE ONLINE and R + then PERFORM PREEMPTION(E, m) // preempt request with lower priority
24:
25:
goto TRY_ [RY_SCHEDULE // try to schedule again
26:
else
27:
break
28:
return B
Latency Predictor. The design of our latency predictor is guided by three key requirements. First, must provide fast inference support real-time scheduling decisions. Second, it needs be robust across varying workload patterns to maintain reliable performance. Third, it should be adaptable to different hardware configurations accurately capture their unique performance characteristics.
The execution time an L.M serving batch is primarily determined by two distinct processing stages with different computational patterns. The prefill stage exhibits quadratic complexity due to attention computations, with latency growing quadratically with input sequence length. The total load of this stage depends on both the number of requests and their individual sequence lengths. contrast, the decode stage shows linear scaling, with computational requirements growing proportionally with the number of tokens. We can model the batch execution time below to capture these characteristics:
Thatch f(Sp, Sa, S., S2, NNp, Na)
(1)
where Sp and Sa represent the total number of prefill and decode tokens in the batch, respectively. The quadratic terms (S3 and 52 account for non-linear scaling effects, particularly in the prefill phase. Np and Na represent the number of prefill and decode requests in the batch, respectively.
We employ linear regression as the prediction model because of efficiency and effectiveness. Train- ing data for the model is collected by systematically profiling target hardware across diverse batch compositions, varying the number of requests different phases, sequence length distributions, and total batch sizes. The linear model enables rapid evaluation of varying batch compositions during scheduling, while its simple feature set ensures stable predictions across varying conditions, Further discussion on the implementation and expandability of the LR predictor can be found in Appendix B
SL.--aware Profiling. The latency predictor provides accurate latency estimates for filling offline requests. Our Sl-O-aware profiler leverages latency budget to ensure SL.O compliance scheduling. The profiler first analyzes the given combination of workload and SLO to establish viable latency
5

---

budget ranges. Given that larger batch sizes and longer inputs will increase latency, the profiler test-runs latency budgets within the range to check their compliance with the given SLO and employs binary search to decide an upper limit that meets the overall SL.O for online requests. During deployment, this latency budget is used as the batch latency limit in the two-phase scheduling process (Section [ to ensure SLO compliance. This profiling enables three key capabilities: (1) determines appropriate latency thresholds that maintain SLO compliance for various workloads and limitations (e.g.. power constraints [4T). (2) It provides flexible adaptation by adjusting budgets based on changing workload characteristics and performance requirements. (3) It establishes a robust foundation for hybrid scheduling by accounting for both online and offline workload patterns.
4.3 SLO-aware Offline Scheduling Policies
After scheduling online requests, our offline scheduling policy repurposes the residual capacity to maximize throughput while ensuring faimess [50]. To further optimize the serving throughput for offline requests, HyGen employs an SLO-aware Prefix Sharing Maximization (PSM) strategy. Prefix sharing is a widely adopted technique for reusing the KV cache of shared input prefixes between requests [17 69 731.
Prefix Sharing Maximization Strategy. Our PSM strategy organizes offline requests into prefix tree following the structure of a Trie tree with each leaf node representing a request, capturing prefix sharing characteristics of all offline requests. The priority of each request is determined by the Depth-First Search (DFS) order of the prefix tree, where requests with the greatest prefix sharing potential are scheduled together. Subsequently, HyGen performs SLO-aware offline scheduling (Section using this order to maximize prefix cache reuse, reducing redundant computation and improving throughput. The prefix tree structure also ensures fast insertion and deletion in runtime scheduling. A formalized algorithm is in Appendix [A.2] For example, consider system that can process two offline requests per batch with the following request queue: (What is MI., How to code, What is AI, How to debug). Under traditional FCFS scheduling, requests are processed in arrival order: (What is ML., How to code), (What AI, How to debug), resulting in no prefix sharing opportunities. In contrast, PSM's prefix- aware scheduling reorders requests as: (What is ML, What is Al), (How to code, How to debug), enabling KV cache reuse through shared prefixes.
The PSM strategy demonstrates strong extendability. Under certain scenarios, the vanilla PSM strategy may lead to starvation for requests with minimal prefix-sharing potential. Consider a request queue: (What is ML, What is AI, How to code, What is DL). When new requests arrive with similar prefixes (What LLM, What is DMN). naive prefix-sharing policy would continuously prioritize requests sharing the What is prefix, potentially starving the How to code request indefinitely. This issue can be mitigated by an extended version of our PSM policy, combining maximum prefix sharing with request freshness by using utility ratio to ensure balance between efficiency and fairness. Based on the utility ratio, a new offline request would be selected from the DFS order of the prefix tree the most stale request from a self-balanced binary search tree sorted by freshness. A detailed algorithm is in Appendix AA.3] These enhancements can make the PSM strategy more practical for real-world deployments, retaining its efficiency while improving fairness and adaptability.
5 Performance Evaluation
5.1 Evaluation Setup
Implementation and Testbeds. We implement HyGen on top of vLLM 29] 57 and Sarathi 46 with 1,300 lines of additional code. We evaluate HyGen on three server configurations: one with 4 NVIDIA A100 GPUs (40GB VRAM each), one with NVIDIA A40 GPUs (48GB VRAM each). and one with I NVIDIA A5000 GPU (24GB VRAM). All servers have 64 CPU cores, 256GB DDR4 RAM, and a 1.5TB NVMe SSD.
Models and Workloads. For end-to-end end evaluation, we use Llama2-74 7B [56] and Qwen- 14B (6] models on A 1000 and A40 GPUs, respectively. Online workloads are based on the conversation trace from Azure LLM inference trace 2023 [41), a one-hour production trace with real-world requests and timestamps. We randomly sampled the trace to achieve the desired QPS that suits our hardware serving capacity. Specifically, within time duration of T seconds, we would sample T x Q requests

---

Sarathi
HyGen
Sarathi++
TBT
Liamaz-78
(x)
0.06
0.06
02
0.04
0.02
0.02
0.3
0.5
0.00
0.1
0.2
0.3
0.4
0.5
0.00
0.05
0.10
0.15
0.20
025
0.0.0 1.05
0.10
0.15
0.20
0.2
0.0
0.05
0.07
0.1
0.125
0.15
0.6
Qwen- 149
(5)
0.075
0.15
0.10
0.025
0.05
0.2
0.00%.11
6.12
0.14
0.16
0.18
0.00.00
0.04
0.06
0.08
0.0050
0.075
0.100
0.125
6.05
0.10
0.15
0.2
Telerance
Telerance
Tolerance
Talerance
Figure 3: HyGen respects latency requirements in co-locating requests.
Sarathi
Sarathi-offline
HyGen Mean TIFT
HyGen
Mean TBT
P99 TBT
P99 TTFT
Uamm22 7B
Total TPS
4000
4000
4000
4000
2000
2000
2000
2000
0.2
0.3
0.4
0.3
0.05
0.30
0.15
0.20
0.25
0.10
0.15
0.20
0.25
0.05
0.075
0.1
0.125 0.15
Owen-1 148
TPS
500
500
500
500
250
250
250
250
8.32
0.14
0.16
0.18
0.02
0.04
0.06
0.08
8.050
0.075
0.100
0.125
6.05
0.10
0.15
0.20
Tolerance
Tolerance
Tolerance
Tolerance
Figure 4: HyGen improves serving throughput under varying SLOS.
to suit a desired QPS Q. For offline workloads, we use arXiv summarization [I]] a dataset for long document summarization. In ablation studies, we evaluate HyGen across different model scales ranging from Sheared-L J...M--2.....
(60). Mistral-7B 26 to Yi-34B (62]. The Mooncake trace 44
further used the online trace, providing industrial request length distributions and arrival patterns, while CNN/DailyMail [22] 47) and MMLU [20] 21] are used as offline traces in the ablation studies. For interference evaluation, we focus on Time to First Token (TTFT) and Time Between Tokens (TBT), including their mean and 99th percentile (P99) values. Throughput is measured in tokens per second (TPS) and queries per second (QPS).
Baselines. For pure online inference, we use Sarathi 2 our baseline. For pure offline serving, we use Sarathi-offine to evaluate the maximum offline serving capacity, where an optimal chunk size is profiled for offline workload to maximize throughput. The hyperparameter search of Sarathi- offline achieves ~ 12% throughput gain compared the default setup, ensuring optimal baseline performance for fair comparison. We then compare HyGen with two baselines for interference and throughput evaluation, respectively: (1) Sarathi++: We implement our online-firs scheduling policy on Sarathi to support hybrid serving, including the request management and preemption handling policies introduced Section 4.1 (2) HyGen To evaluate the throughput benefit our HyGen design, we further improve Sarathi+ + + to an SLO-aware serving system, HyGen*. Besides inheriting the serving policies from S HyGen serves offline requests at a specific offline QPS to control overall interference. The offline QPS is profiled using similar design with the HyGen profiler to guarantee bounded SLO interference.
5.2 End-to-end Performance
HeGen respects latency requirements in co-locating requests. We evaluate HyGen under four SLO metrics (mean TBT, P99 TBT, mean TTFT. and P99 TTFT) with varied interference tolerance ratios. 3 shows that HyGen controls interference and guarantees to meet specific SL.Ss across our settings. Compared with Sarathi++, an SI.O-unaware system that yields the same result for all metrics and tolerance ratios, HyGen shows efficient SLO-aware latency control.

---

Liama2-78 78
Qwen-148
Prefix Sharing Maximizatioo FCPS
HyGen profiler 5LO
HyGen
1.0
1.0
T6T
082
0.5
0.5
10
0,0
0.0
0.0
0.10
0.15
0.20
0.25
Absolute Error (5)
Absolute Percentage Eror
Mean TDT Tolerance
Mean TDT
Figure SLO-aware pro- Maximization improves filer contributes to Hy- serving throughput.
Figure 6: Prefix Sharing
Figure 5: HyGen latency predictor achieves high accuracy for batch latency prediction.
Gen's s improvements.
Sarathi HyGen
HyGen
Hy brid
Online
Offline
HyGen
(6)
4000
Total TPS
1000
2000
500
0.90.5
0.00
Time (s)
(a) SLOs
(b) Throughput
Figure 8: HyGen dynamically controls
Figure 9: HyGen meets SLOs and achieves higher
throughput according to online workload.
throughput for Yi-34B model using TP=2, PP=2.
HyGen improves serving throughput. Figure 4 shows the offline throughput of HyGen for various metrics and tolerance ratios. Through efficient request co-location, HyGen improves overall serving throughput by up to 3.87 compared to pure online serving. Under the same SL.,, HyGen consistently achieves higher throughput compared to HyGen*, yielding up 5.84x offline throughput gain. Furthermore, HyGen achieves up to 84.3% total throughput compared to Sarathi-ofline, a pure offline serving system whose high throughput benefits from an optimal chunk size profiled for offline requests only. This verifies that our fine-grained latency predictor and SLO-aware profiler designs achieve higher serving efficiency compared their simplified counterparts.
5.3 Performance Breakdown
Accuracy of latency predictor. We evaluate the accuracy of our latency predictor on Llama2-7B and Qwen- using Azure LLM Inference trace mixed with arXiv summarization dataset. Figure shows that our latency predictor achieves mean absolute percentage error of only 1.78% and 1.07%, confirming its high accuracy.
Impact of prefix sharing. To test HyGen's compatibility with prefix sharing, we conducted a simulation experiment using Azure LLM Inference the online trace and MMLU [20 21 the offline dataset on a Llama2-7B model. In our simulation, we deducted the shared prompt prefix length between consecutive offline requests to simulate prefix sharing. Figure ( shows that HyGen yields up to 4x offline throughput gain with its prefix sharing maximization scheduling policy.
Impact of SLO-aware profiler. To demonstrate the effect of HyGen's SLO-aware profiler, we compared it with simple strategy that sets the desired mean TBT SLO as the batch latency budget. Figure 7 shows the performance gap between individual batch latencies and overall mean TBT, illustrating how the SLO-aware profiler bridges this gap for controlled SLO in hybrid serving.
Breakdown by time. Figure shows a temporal throughput breakdown of HyGen. At runtime, HyGen dynamically adjusts offline throughput based online workload and overall latency budget, batching offline requests more aggressively during online QPS troughs and reducing offline throughput during online bursts, harnessing compute resources in an adaptive manner.

---

Sarathi
HyGen
Sarathi+-
Mean TBT
P99 TBT
Mean TIFT
P99 TTFT
0.04
0.06
Time (5)
0.04
10
0.02
0.00
0.5
1.0
1.5 QPS
2.0
2.5
0.00
0.5
1.0
1.5
2.0
2.5
0.5
1.0
1.5 QPS
2.0
2.5
0.5
1.0
1.5
2.0
2.5
Figure 10: HyGen meets SLO under various online QPS settings.
Sarathi HyGen
Sarathi++
Sarathi HyGen
HyGen
Sarathi
HyGen
SLO
Mean TBT (3)
2 0.75
0.06
0.02
959 T8T
0.04
0.01
X 0.25
0.00
0.00
Mean TBT Tolerance
0.2
0.3
0.5
0.00
01
Mean TBT Tolerance
0.2
0.3
0.5
0.06
P99 TBT Tolerance
0.075
0.155
0.125 0.150
0,050 0.075
0.100 0:125 0.150 P99 T8T Tolerance
(a) SLOs
(b) Throughput
Figure 11: HyGen is able to meet multiple SL.Ss simultaneously.
Figure 12: HyGen meets SLOs and achieves higher throughput with CNN/DailyMai offline dataset.
5.4 Ablation Studies
Impact of model parallelisms. To evaluate HyGen's effectiveness in distributed inference, deployed the Yi-34B model [62 on a server with NVIDIA A40 GPUs using tensor-parallelism (TP) [52] and pipeline-parallelise (PP) 5 24 37 with degree 2 for each dimension. Using Azure LLM Inference and arXiv summarization workloads, Figure 9 shows that HyGen maintains its ability to meet SLOs and achieves higher offline throughput (up 1.89x) than the baseline.
Impact of SLO requirements. We further evaluate HyGen's ability to meet stringent SLOs under varying online QPS settings on the four aforementioned metrics, each with 5% interference tolerance. Figure shows that HyGen meets stringent SLOS for all metrics. We further demonstrate HyGen's ability to meet multiple SLOS at the same time. By testing HyGen with a fixed P99 TTFT interference ratio (8%) and mean TBT interference ratios ranging from 10% to 50%, Figure 11 shows that at lower mean TBT tolerance, HyGen's performance is bounded by mean TBT SLOs; after reaching the fixed P99 TTFT SLO, mean TBT stops increasing in order to keep P99 TTFT under control.
Impact of models and datasets. We further evaluate HyGen's adaptability on two experi- ments: The first using Mistral-7B model (26) with Mooncake trace [4], a trace containing request servers, the online trace, and arXiv summarization
lengths and timestamps taken from as the offline trace: The second experiment uses Llama2-7B model with Azure LLM Inference trace
and replaced the offline dataset with CNN/DailyMail summarization dataset [22 47 Figure [3] shows the varying request arrival rates of Mooncake trace over one-hour and ten-minute periods, further demonstrating the fluctuating nature of LLM services. Figure 12 and Figure 14show show that HyGen achieves superior performance than its counterparts under these settings.
Impact of hardware testbeds. To further evaluate HyGen's effectiveness under different hardware configurations, memory limitations and model sizes, we further conducted experiments on A5000 GPU with 24 GB VRAM and able to guarantee SLO attainment and achieve higher throughput, with up to 2.18x offline throughput gain and 1.30x overall throughput gain compared to the baseline.
model [60]. Figure 15 shows that HyGen is
Impact of predictor accuracy. tested HyGen's robustness using several pre-trained LR latency predictors with varying prediction accuracy taken from other workloads and tested them on Azure LLM Inference trace and arXiv summarization dataset. Figure shows how predictor accuracy (measured in mean absolute percentage error) affects offline throughput under the same P99 TBT

---

Sarathi
Sarathi
Sarathi --- lyyGen
HyGen
Quazio Par
2 0.05
7500
20
0.01
5000
2500
0990
0824
89.4
0944
Dk.11
0.0 0.05
5.30
5:15
0.20
322
6:05
0,10
0:35
0:23
0.23
Tolerance
TBT
(a) hour
(b) 10 minutes
(a) SLOs
(b) Throughput
Figure 13: Request rate varies in Moonshot
Figure 14: HyGen meets SLOSs and achieves
Mooncake's 's LLM service one-hour and ten-
higher throughput for Mooncake trace.
minute periods.
Sarathi HyGen
SLD
Sarathi HyGen
HyGen
HyGen
HyGen!
HyGen HyGen*
0.02
3000
Offine TPS
2000
0.03
2000
2000
1000
0.0%
5:15
0.25 Talerance
0.25
9..55
Hean TBT Tolerance
0.15
0.25
0.35
0.25
Predictor MAPE
0.33
0.20
33
Online OPS
2.5
(a) SLOs
(b) Throughput
Figure 16: HyGen is bust to predictor accu- racy.
Figure 17: HyGen dy- namically adjusts offline throughput.
Figure 15: HyGen meets SLOs and achieves higher throughput on A5000 GPU and Sheared- LLaMA-2.7B model.
SL.O. HyGen remains robust across different accuracy settings. Also, our LR-based latency predictor is lightweight for training. with only ~m5 training time for over 80,000 samples on CPUs. HyGen's lightweight latency predictor also only incurs ~8ps runtime latency per iteration on our experiment CPU, guaranteeing efficient runtime scheduling.
Impact of online arrival rate. Figure shows the effect of online QPS on offline throughput with 5% P99 TBT tolerance. As online load increases, HyGen adjusts offline throughput based on the system's is residual serving capacity while maintaining higher throughputs. Understandably, a high online arrival rate limits the headroom for co-location as it approaches system serving capacity,
6 Conclusion
This paper introduces HyGen, an LLM serving system that enables efficient co-location of online and offline workloads. We employ control mechanisms predict and manage interference impacts, and a scheduling policy to opportunistically schedule offline serving. Evaluation on production workloads demonstrates that HyGen improves serving throughput by 3.87-5.84x while maintaining strict latency SLOs.
Acknowledgements
We thank the anonymous reviewers for their constructive and insightful feedback. This work was supported in part by grants from Cisco and Google, and by award from NVIDIA Academic Program. also utilized the Delta system at the National Center for Supercomputing Applications (NCSA) through allocation CIS240236 from the ACCESS program.
References
[1] Reyna Abhyankar, Zijian He, Vikranth Srivatsa, Hao Zhang, and Yiying Zhang. Infercept: Efficient intercept support for augmented large language model inference. arXiv preprint arXiv:2402.018863 2024.
[2] Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gulavani, Alexey Tumanov, and Ramachandran Ramjee. Taming throughput-latency tradeoff
10

---

in Ilm inference with sarathi- In Proceedings of 18th USENIX Symposium on Operating Systems Design and Implementation, 2024, Santa Clara, 2024.
[3] Loubna Ben Allal, Anton Lozhkov, and Daniel van Strien. Cosmopedia: how create large-
scale synthetic data for pre- traiiiin Large Language Models huggingface.co. https
//muggingaace.co/bllo..
cosmopedia 2024. [Accessed 25- 10-2024].
[4] Reza Yazdani Aminabadi, Samyam Rajbhandari. Ammar Ahmad Awan, Cheng Li. Du Li, Elton Zheng, Olatunji Ruwase, Shaden Smith, Minjia Zhang, Jeff Rasley, et al. Deepspeed- inference: enabling efficient inference of transformer models unprecedented scale. In SC22: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 15. IEEE, 2022.
[5] Sanjith Athlur, Nitika Saran, Muthian Sivathanu, Ramachandran Ramjee, and Nipun Kwatra. Varuna: scalable, low-cost training of massive deep learning models. In Proceedings of the Seventeenth European Conference on Computer Systems, pages 472-487, 2022.
[6] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16600 2023.
[7] Lequn Chen, Zihao Ye, Yongji Wu, Danyang Zhuo Luis Ceze, and Arvind Krishnamurthy. Punica: Multi-tenant lora serving. Proceedings of Machine Learning and Systems, 6:1-13, 2024.
[8] Ruobing Chen, Haosen Shi, Yusen Li, Xiaoguang Liu, and Gang Wang. Olpart: Online learning based resource partitioning for colocating multiple latency-critical jobs commodity computers. In Proceedings of the Eighteenth European Conference on Computer Systems, pages 347-364, 2023.
[9] Shuang Chen, Christina Delimitrou, and José F Martínez. Parties: Qos-aware resource parti- tioning for multiple interactive services. In Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems, pages 107-120, 2019.
[10] Xinyun Chen, Petros Maniatis, Rishabh Singh, Charles Sutton, Hanjun Dai, Max Lin, and Denny Zhou. Spreadsheetcoder: Formula prediction from semi-structured context. In International Conference on Machine Learning, pages 1661-1672. PMLR, 2021.
[11] Arman Cohan, Franck Demoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. A discourse-aware attention model for abstractive summarization of long documents. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 615-621, New Orleans, Louisiana, June 2018. Association for Computational Linguistics.
[12] Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing 3 35:1444441111555 16359, 2022.
[13] Dujian Ding, Ankur Mallick, Chi Wang, Robert Sim, Subhabrata Mukherjee, Victor Ruhle, Laks VS Lakshmanan, and Ahmed Hassan Awadallah. Hybrid Ilm: Cost-efficient and quality- aware query routing. arXiv preprint 2xXiv2244414612 2024.
[14] Jiangfei Duan, Runyu Lu, Haojie Duanmu, Xiuhong Li, Xingcheng Zhang, Dahua Lin, Ion Stoica, and Hao Zhang. Muxserve: Flexible spatial-temporal multiplexing for multiple Ilm serving. In Forty-first International Conference on Machine Learning, 2024.
(15) Yao Fu, Leyang Xue, Yeqi Huang, Andrei-Octavia Brabete, Dmitrii Ustiugov, Yuvraj Patel, and Luo Mai. LarreriesLLLM,:LLLLLLttt In 18th USENIX Symposium Operating Systems Design and Implementation (OSDI 24), pages 135-153, 2024.
serverless inference for large language models.
[16) Yichao Fu, Siqi Zhu, Runlong Su, Aurick Qiao, Ion Stoica, and Hao Zhang. Efficient Ilm scheduling by learning to rank. arXiv preprint arXiv:2408.1 15792, 2024.
11

---

[17] In Gim, Guojun Chen, Seung-seor seob Lee, Nikhil Sarda, Anurag Khandelwal, and Lin Zhong. Prompt cache: Modular attention reuse for low-latency inference. Proceedings of Machine Learning and Systems, 6:325-338, 2024.
[18] Grand View Research. Large language model (Ilm) market size, share & trends analysis report by component, by application, by enterprise size, by end-use, by region, and segment forecasts, 2023 2030. Grand View Research, 2023.
[19] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00052, 2023.
[20] Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. Aligning ai with shared human values. Proceedings of the International Conference on Learning Representations (ICLR), 2021.
[21] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the Interaational Conference on Learning Representations (ICLR), 2021.
[22] Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. Advances in neural information processing systems, 28, 2015.
[23] Cunchen Hu, Heyang Huang, Liangliang Xu, Xusheng Chen, Jiang Xu, Shuang Chen, Hao Feng, Chenxi Wang, Sa Wang, Yungang Bao, al. Inference without interference: Disaggregate Ilm inference for mixed downstream workloads. arXiv preprint 24xv:2401111181 2024.
[24] Yanping Huang. Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, Hy- oukJoong Lee, Jiquan Ngiam, Quoe V Le, Yonghui Wu, al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. Advances in neural information processing systems, 32, 2019.
[25] Hugging Face. Hugging face accelerate. GitHub repository, 2025. Accessed: 2025-01-01.
[26] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023.
[27] Yunho Jin, Chun-Feeg Wu, David Brooks, and Gu-Yeor Wei, S3: Increasing gpu utilization during generative inference for higher throughput. Advances in Neural Information Processing Systems, 36: 18:11881518021 2023.
[28] Jordan Juravsky, Bradley Brown, Ryan Ehrlich, Daniel Y Fu, Christopher Ré, and Azalia Mirhoseini. Hydragen: High-throughput I1m inference with shared prefixes. arXiv preprint arXiv:2402.05009, 2024.
[29] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pages 611 1211666,222 2023.
[30] Bin Lin, Tao Peng, Chen Zhang, Minmin Sun, Lanbo Li, Hanyu Zhao, Wencong Xiao, Qi Xu, Xiafei Qiu, Shen Li, et al. Infinite-lmm: Efficient Iim service for long context with distattention and distributed kvcache. arXiv preprint arXiv:2401.0266 2024.
[31] Chaofan Lin, Zhenhua Han, Chengruidong Zhang, Yuqing Yang, Fan Yang. Chen Chen, and Lili Qiu. Parrot: Efficient serving of (LL-M- applications with semantic variable. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pages 929 9019-995,222 2024.
[32] Jiachen Liu, Zhiyu Wu, Jae- Won Chung, Fan Lai, Myungjin Lee, and Mosharaf Chowdhury. Andes: Defining and enhancing quality-of-experinee llmebased text streaming services. arXiv preprint 2xXhv:2404.1628. 2024.
12

---

[33] Zixuan Ma, Haojie Wang, Jingze Xing, Shuhong Huang, Liyan Zheng, Chen Zhang, Huanqi Cao, Kezhao Huang, Mingshu Zhai, Shizhi Tang, Intelligen: Instruction-level auto- tuning for tensor program with monotonic memory optimization. In Proceedings of the 23rd ACM/IEEE International Symposium on Code Generation and Optimization, pages 107-122, 2025.
[34] Meta-Team. The llama 3 herd of models, 2024.
[35] Microsoft. GitHub Copilot Your AI pair programmer - github.com. httpp:://itthub.com chmb.comm features/copilot] 2023. [Accessed 28- 10-2024).
[36] Avanika Narayan, Ines Chami, Laurel Orr, and Christopher Ré. Can foundation models wrangle your data? Proc. VLDB Endow. 16(4):738-746, December 2022.
[37] Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R Devanur, Gregory R Ganger, Phillip B Gibbons, and Matei Zaharia. Pipedream: Generalized pipeline parallelism for dnn training. In Proceedings of the 27th ACM symposium on operating systems principles, pages 1-15, 2019.
[38] Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E Gonzalez, M Waleed Kadous, and Ion Stoica. Routellm: Learning route Ilms from preference data. In The Thirteenth International Conference on Learning Representations, 2024.
[39] OpenAl. Introducing chatgpt. https://opeeaa..co/nnnneee/aaaas 2022. [Accessed
20- 10-2024].
[40] OpenAl. Batch api, 2024.
[41] Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Iñigo Goiri, Saeed Malcki, and Ricardo Bianchini. Splitwise: Efficient generative I1m inference using phase splitting. In 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA), pages 118-132 IEEE, 2024.
[42] Tirthak Patel and Devesh Tiwari. Clite: Efficient and qos-aware co-location of multiple latency- critical jobs for warehouse scale computers. In 2020 IEEE International Symposium High Performance Computer Architecture (HPCA), pages 193-206. IEFE, 2020.
[43] Archit Patke, Dhemath Reddy, Saurabh Jha, Haoran Qiu, Christian Pinto, Chandra Narayanaswami, Zbigniew Kalbarczyk, and Ravishankar Iyer. Queue management for slo- oriented large language model serving. In Proceedings of the 2024 ACM Symposium on Cloud Computing, pages 18 18-35, 2024.
[44] Ruoyu Qin, Zheming Li, Weiran He, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. Mooncake: Kimi's kveache-centric architecture for Ilm serving, arXiv preprint arXiv:2407.00077 2024.
[45] Haoran Qiu, Weichao Mao, Archit Patke, Shengkun Cui, Saurabh Jha, Chen Wang, Hubertus Franke, Zbigniew Kalbarczyk, Tamer Başar, and Ravishankar K Iyer. Power-aware deep learning model serving with (u-Serve]. In 2024 USENIX Annual Technical Conference (USENIX ATC
24), pages 75 95-93, 2024.
[46] Sarathi-Serve Project. Sarathi-serve: A low-latency and high-throughput serving engine for lmms. GitHub repository, 2024. Accessed: 2025-01-01
[47] Abigail See, Peter Liu, and Christopher D Manning, Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Anmual Meeting of the Association for Computational Linguistics (Volume 1; Long Papers), pages 1073-1083, 2017.
[48] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The aparsely-gated mixture-of-experts layer. In International Conference on Learning Representations (ICLR), 2017.
[49] Ying Sheng, Shiyi Cao, Dacheng Li, Coleman Hooper, Nicholas Lee, Shuo Yang, Christopher Chou, Banghua Zhu, Lianmin Zheng, Kurt Keutzer, al. Slora: Scalable serving of thousands of lora adapters. Proceedings of Machine Learning and Systems, 6:296-311, 2024.
13

---

[50] Ying Sheng, Shiyi Cao, Dacheng Li, Banghua Zhu. Zhuohan Li, Danyang Zhuo, Joseph E Gonzalez, and Ion Stoica. Fairness in serving large language models. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pages 965-988, 2024.
[51] Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang. Flexgen: High-throughput generative inference of large language models with single gpu. In International Conference on Machine Learning, pages 31094-3111 31116. PMLR, 2023.
[52] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-Im: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.0880 08053, 2019.
[53] Vikranth Srivatsa, Zijian He, Reyna Abhyankar, Dongming Li, and Yiying Zhang. Preble: Efficient distributed prompt scheduling for Ilm serving. arXiv preprint arXiv:2407.00025 2024.
[54] Jovan Stojkovic, Chaojie Zhang, fñigo Goini, Josep Torrellas, and Esha Choukse. Dynamollm: Designing I1m inference clusters for performance and energy efficiency. arXiv preprint arXiv:2408.00741 2024.
[55] Ting Sun, Penghan Wang, and Fan Lai. DiSCo: Device-server collaborative LLM-bbased text streaming services. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Findings of the Association for Computational Linguistics: ACL 2025, pages 14259-14277, Vienna, Austria, July 2025, Association for Computational Linguistics.
[56] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv2230.00928. 2023.
[57] vLLM Project. vllm: Easy, fast, and cheap Ilm serving with pagedattention. GitHub repository, 2023. Accessed: 2025-01-01.
[58] Bingyang Wu, Ruidong Zhu, Zili Zhang, Peng Sun, Xuanzhe Liu, and Xin Jin. {dLoRA): Dynamically orchestrating requests and adapters for (LoRA) (LLM} serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pages 911 911-927, 2024.
[59] Mengdi Wu, Xinhao Cheng. Shengyu Liu, Chunan Shi, Jianan Ji, Man Kit Ao, Praveen Vellien- giri, Xupeng Miao, Oded Padon, and Zhihao Jia. Mirage: [Multi-Level superoptimizer for tensor programs. In 19th USENIX Symposium on Operating Systems Design and Implementation (OSDI 25), pages 21-38, 2025.
[60] Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. Sheared Ilama: Accelerating language model pre-training via structured pruning. arXiv preprint vrXiv:23/0.06690 2023.
[61] Zihao Ye, Lequn Chen, Ruihang Lai, Wuwei Lin, Yineng Zhang, Stephanie Wang, Tianqi Chen, Baris Kasikci, Vinod Grover, Arvind Krishnamurthy, et al. Flashinfer: Efficient and customizable attention engine for Ilm inference serving. arXiv preprint arXiv:250I.0100 01005, 2025.
[62] Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangeheng Zhu, Jianqun Chen, Jing Chang, et al. Yi: Open foundation models by 01. ai. arXiv preprint arXiv:2403.04653 04.400000455 2024.
[63] Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. Orca: A distributed serving system for Trransformer-Based generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pages 521-538. 2022.
[64] Minchen Yu, Rui Yang, Chaobo Jia, Zhaoyuan Su, Sheng Yao, Tingfeng Lan, Yuchen Yang, Yue Cheng, Wei Wang, Ao Wang, al. [ammbda scale: Enabling fast scaling for serverless large language model inference. arXiv preprint a7Xiv:2502.09922, 2025.
14

---

[65] Yifan Yu, Yu Gan, Nikhil Sarda, Lillian Tsai, Jiaming Shen, Yanqi Zhou, Arvind Krishnamurthy. Fan Lai, Hank Levy, and David Culler. Ie-cache: Efficient large language model serving via in-context caching. In SOSP. ACM, 2025.
[66] Shaoxun Zeng, Minhui Xie, Shiwei Gao, Youmin Chen, and Youyou Lu. Medusa: Accelerating serverless Ilm inference with materialization. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1, pages 553-668,70 2025.
[67] Haopeng Zhang, Xiao Liu, and Jiawei Zhang. Summlt: Iterative text summarization via ChatGPT. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10644-10657, Singapore, December 2023. Association for Computational Linguisties.
[68] Wei Zhang, Zhiyu Wu, Yi Mu, Banruo Liu, Myungjin Lee, and Fan Lai. Tempo: Application- aware Ilm serving with mixed slo requirements. arXiv preprint arXiv:2504.2006 2025.
[69] Yilong Zhao, Shuo Yang, Kan Zhu, Lianmin Zheng, Baris Kasikei, Yang Zhou, Jiarong Xing, and lon Stoica. Blendserve: Optimizing offline inference for auto-regressive large models with resource-aware batching. arXiv preprint arXiv:2411.16100 2024.
[70] Lianmin Zheng, Wei-Lin Chiang. Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing. Hao Zhang, Joseph E. Gonzalez, and lon Stoica. Judging llm-as-a-judge with nntheech and chatbot arena, 2023.
[71] Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph Gonzalez, et al. Sglang: Efficient execution of structured language model programs. arXiv preprint arXiv:2312.07104, 2024.
[72] Liyan Zheng, Haojie Wang. Jidong Zhai, Muyan Hu, Zixuan Ma, Tuowei Wang, Shuhong Huang, Xupeng Miao, Shizhi Tang, Kezhao Huang, et al. (EINNET): Optimizing tensor programs with (Derivation-Basse transformations. In 17th USENIX Symposium on Operating Systems Design and Implementation (OSDI 23), pages 739-755, 2023.
[73] Zhen Zheng, Xin Ji, Taosong Fang, Fanghao Zhou, Chuanjie Liu, and Gang Peng. Batchllm: Optimizing large batched Ilm inference with global prefix sharing and throughput-orientel token batching. arXiv preprint arXiv:2412.03594, 2024.
15

---

Limitations
While HyGen demonstrates substantial improvements in throughput and latency compliance through co-locating online and offline LLM workloads, several limitations remain. First, our approach assumes stable performance predictions from the latency predictor, which may degrade under highly dynamic or adversarial inputs. Second, HyGen focuses a single model co-location scenario; extending support to heterogeneous models or multi-tenant environments could introduce additional interference patterns. Lastly, our evaluation is limited to specific production workloads- gennddaiiiininnngggggggg to other LLM architectures or serving frameworks may require further adaptation and tuning.
Broader Impact
This paper proposes HyGen, LLM serving system for efficient co-location of online and offline requests. Through efficient co-location and SL.O control mechanisms, HyGen improves resource utilization and system serving throughput. We believe that the deployment of HyGen will help any kind of L.LM service providers by improving serving throughput and providing L.M service with a wider range of options (online/ofiline). Since our paper provides efficient serving system for LLM applications without modification to the structure or the outputs of the model, there are no possible negative societal impact that needs be mentioned in our paper as far as we are concerned.
Code and Dataset Licenses
Codebase. HyGen's implementation is based on 57 and Sarathi-Serve(40) both using 2p0 License.
Datasets. We list the license of used datasets as follows:
arXiv summarization dataset [| Apache-2.0 License;
Azure LLM Inference trace [6)] CC-BY-4.0;
MMLU dataset 20 21: MIT License.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our abstract and introduction claim the key innovations and results of our proposed system, HyGen. We have included detailed explanation of our design in Section 4 and presented comprehensive experimental results in Section 5
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We have provided detailed discussion of the limitations in this paper. We have also conducted holistic ablation studies in Section 5.4 to reflect on each factor of our evaluation setup, including hardware configurations, models, datasets, traces, etc.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
17

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: Our paper does not include major theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We have provided detailed explanation of our evaluation setup for each experiment in Section 5 including testbeds, models, datasets, workloads, etc.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
18

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
The code of HyGen is publicly available at hatpss://iithub.com// COUUUCC MLSys/HyGen
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/nips.cc/pullic/uuueee huabbi//guiddes/odeeuuutti
Codeeubmii ssi sesoonolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
We have included the evaluation setup and details for each experiment in Section
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Our main metrics include SLO compliance and throughput gain, which have both been well-defined in previous literature and related works. We have included detailed metrics and data for our experimental results Section 5
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
19

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have included the testbeds and hardware configurations in Section 5.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: We have reviewed the NeurIPS Code of Ethics and confirm that our research complies with the Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have provided relevant discussion in the "Broader Impact" section.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
20

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Our paper presents an efficient serving system for request co-location LLM serving, which does not involve further training or modification to the model. Our paper does not involve scraped datasets.
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We have provided explicit information and citation of each asset used in this
paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
21

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, ppaperswithode. wttpeecoomaatii datasetss has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The codebase of HyGen is publicly available at htlttts://github.com MLSys/HyGen HyGem with detailed documentation.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
22

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMS as any important, original, non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
23

---

Appendix
A
Algorithms
A.1
return B HyGen Two-phase Scheduling Algorithm
This section gives detailed and formalized demonstration of the two-phase scheduling algorithm introduced in Section [.] In each scheduling step, the workflow invokes the SLO-aware scheduling process Algorithm twice (line 13 and line 18) to form hybrid batch with online and offline request co-location while respecting latency and memory limits. To reduce the scheduling overhead of HyGen, we employ message queue for asynchronous communication between the main process and the offline scheduler. After each scheduling step, the main process sends the metadata of batched requests the message queue. The offline scheduler first calculates the expected status of each request based on scheduling decisions from the previous batch, and then runs scheduling simulation to generate offline request scheduling decisions using our latency predictor and the profiled latency budget. The offline scheduling decisions are then sent back to the main process using the message queue and used for the next scheduling step. To support pipeline parallclization, scheduling history archive of K steps is kept by the offline scheduler for pipeline parallelization degree K, in order to have a holistic view of every request running in each pipeline stage at the time.
Algorithm 2 HyGen wwo-phase scheduler
1: global Qon (online request queue), Qos (offline request queue)
2: global Rox (online request running list), Rofs (offline request rumning list)
3: global Qeend. Qreee (message queues)
4: global L (latency budget), M (memory budget), C (chunk size), Mosy (offline memory)
5: function ASYNC_SCHEDULLER
6: while True do
7: scheduled requests < Q Quend-getten berbookge(lo
8: UPDATE REQUEST (PDATE_REQUEST (SSSUSSSTSSSSS,,,, Roa, Ross)
9:
batched requests B +- (} latency budget
L
11:
memory budget m <- GET_FREES MEM__RY Moss
chunk size f-
13:
B + BU SLO AWARE SCHEDULE(Ron Qoni t, c. m)
i'm < Moss then
15:
PREEMPT PREEMPT_OFFLINERR OFFLINE(RR.) m, Moff)
m -- m Moff
17:
B BU A.L__AWRR SWARE_SCHEDULECRE Qoff. t, c, m)
18:
Qreew.Send(() B,, 6, m)
19:
function SCHEDULER
20:
Output: batched requests B
21:
scheduled requests B, latency budget 4, memory budget m <- Qrec (e(toocee(bl
22:
if REQUEST ARRIVAL then
23:
UPDATE BUDGET(B, m)
24:
ift or m then
25:
PREEMPT CEEDMPPTUUTTI__FTTCCE t,, m)
26:
Sendd
27:
24

---

A.2 SLO-Aware Prefix Sharing Maximization Algorithm
This section details the SLO-aware prefix sharing maximization algorithm design in HyGen. We construct a prefix tree Tр to capture prefix sharing characteristics among offline requests. During scheduling, offline requests are selected in the DFS order of the prefix tree, and deleted once being scheduled. Additionally, running requests keep their original DFS order in future scheduling process, effectively utilizing prefix sharing.
Algorithm 3 Prefix sharing aware offline scheduler
Construct prefix tree Tр
2: function CREFIX_SHAARIEE SOENNNGOFFLINELEC SCHEDULE
3: Input: running requests Rt, latency budget i, remaining chunk size c, memory budget m
4: Output: batched requests B
5: B
B 0
6: for do
7:
r.state = DECODE then
8:
trey +- PREEDICTORRprccict( DECODE)
9:
ift > >teg then
10:
break
11:
t treg
12:
B BU [(r, O, tree))
13:
else if r.state = PREFILL then
beg PREDICTOR.geL_maa_peiinee if > then
m)
15:
t 1--ttre treg
17:
C c c-1
m + m- GET NUM BLOCKS())
19:
B + BU ((r, I, freg))
else
21:
break
22:
while T, empty do
23:
Tp pe-_get_nex_requesste
24:
1, 1,breg t- PREDICTOR e, m, r)
25:
if ( then
26:
t treg
27:
C c-
28:
m - m GET NETTNUU_BBB BLOCK(()))(
29:
B - BU ((r, 1,trg)
30:
ceermoverrequstt//
31:
else
32:
break
33:
return B
25

---

A.3 Extended SLO-Aware Prefix Sharing Maximization Algorithm
This section details the extended version of the SLO-aware prefix sharing maximization algorithm. enhanced with fairness-aware scheduling. For offline requests, we construct prefix tree Tp for prefix sharing, and self- balancing BST T for request freshness. A utility ratio between 0 and is used to balance the chance between these two data structures. During scheduling, the extended prefix sharing maximization algorithm retrieves waiting offline requests from either 7 or T,, based on the utility ratio. The selected request is then deleted from both data structures to ensure synchronization. This solution balances prefix sharing and fairness without disrupting the DFS order of the prefix tree while avoiding possible starvation.
Algorithm Prefix-sharing-awan offline scheduler
1: Construct prefix tree T and self-balanced BST T
2: function PREFIX_ PREFIX_SHARING SHARING LIA.RIIGGOOFFLIIEEEEE.. LORFOOOEE
SCHEDULE
3: Input: running requests Rt latency budget I, remaining chunk size c, memory budget m.
utility value u
4: Output: batched requests B
5:
6: for do
7:
r.state = DECODE then
8:
trey t- PREDIICOOR DECODE)
9:
if > breg then
break
11:
1. + 1 kreg. B +- BU ((r, 0, breg))
else if r.state PREFILL then
13:
1, treq &- PREDICTOR. max coee-_ffflll(t,, c, m, r)
if then
15:
t t Ereg + c L, m + m- GET NUM BLOCKS(). B + BU ((r, I, fre))
else
17:
break
while T, empty do
19:
rand < RAANOM_NNMBBRRO.
20:
rand < u then
21:
f7.geet_next_reqee
22:
else
23:
Ty.get next request()
24:
I, titre + PREDICTOR. get max prefillt c, m, r)
25:
if > 0 then
26:
t t t-ter c c 1, m + m- GET HET_NUUM_LOON BLOCKS(), B + BU ((r, 1, treg))
27:
Tp )
28:
Ty,rovve_rrqueste
29:
else
30:
break
31:
return B
26

---

A.4 Complexity Analysis of the Two-Phase Scheduling Algorithm
In this section, we present an analysis of the computational complexity of HyGen's two-phase scheduler. The time complexity of its core components is as follows:
Latency Prediction: O(1) inference using a pre-trained LR model.
PSM: The initial construction of the prefix tree is O(N where N is the number of offline requests and L is the average number of tokens. Each insertion or deletion costs OL)) In implementation, getting the next request the DFS order only takes O since the DFS order is put in pre-processed list derived from the prefix tree and can be synced up with the prefix tree asynchronously.
PSM with Fairness: In the fairness-aware PSM algorithm, a self-balancing BST is used for picking the stalest request. Each lookup, insertion, or deletion takes (log n). In implementation, the requests can be kept in an FCFS queue, which syncs up with the BST asynchronously to guarantee correctness, so that each lookup still only takes O(1) time.
Overall, for the two-phase scheduling using existing policies, the time complexity remains ((n) as asynchronous updates can be performed for advanced policies like faimess-aware PSM.
B Further Discussion of the Latency Predictor
Effective co-location in HyGen hinges latency predictor that is both highly accurate and compu- tationally lightweight, as it enables real-time scheduling decisions without introducing significant overhead. To this end, we employ linear regression (LR) model, which provides inference in constant time, O()), making it ideal for a dynamic serving environment, This section details the model's formulation and its adaptability to the complexities of real-world deployment scenarios.
Model Formulation. The execution time of a serving batch, Touch, is primarily determined by the computational patterns of its prefill and decode stages. We model this relationship function of the batch's 's composition:
Thunch (Sp, Sa, Sp. Np, Na)
where Sp, and Sa represent the total number of tokens in the prefill and decode phases, respectively, and Np and Na are the corresponding request counts. The quadratic term Sp is crucial for capturing the non-linear scaling of the self-attention mechanism, which dominates the computational cost of the prefill stage. In contrast, the decode stage, which processes one token per request at time, exhibits linear scaling with the batch size (Na). The model's coefficients are pre-trained on data gathered by systematically profiling the target hardware and LLM across diverse set of batch compositions, lightweight process that ensures applicability any deployment environment.
Robustness Through System-Levvll Design. While linear model offers unparalleled efficiency, its accuracy in the face of dynamic system conditions (e.g., GPU temperature variations, resource contention) is critical consideration. HyGen ensures robustness not through more complex model, but through synergistic system design. The SLO-aware profiler (Section 4.2) first establishes macro-level operational latency budget, which implicitly captures the system's 's current performance characteristics. The LR predictor then operates at a micro-level, making fine-grained decisions on batch composition within this pre-calibrated budget.
This two-level approach effectively decouples the real-time scheduling decision from low-level hardware variability. As shown in Figure 16 this design allows the system to maintain robust performance and meet SI.Os even in scenarios with predictor error rates exceeding 20%. Furthermore, the feature set is extensible; environmental factors like hardware load can be readily integrated into the model if required for specific use cases.
Adaptability to Modern Model Architectures. The feature set defined in Equation is sufficiently general to generalize across various modern LLM architectures without modification. For instance, for Koxture-of-Experee (MoE) models [48), where fixed number of experts are activated per token, the resulting computational cost scales linearly with the total number of tokens processed and effectively modeled by the Sp and S₂ features. Similarly, in hybrid architectures that combine
27

---

linear-compn complexxt components (e.g., Mamba (19] with muadratic-oopleen attention, our model naturally captures both computational pattern;; the linear cost reflected in the learned coefficient for Sp. while the quadratic cost of the Transformer blocks is captured by the Sy term. This adaptability underscores the predictor's 's robust design, ensuring its relevance as L.MM architectures continue to evolve.
C Taming the Throughput-Latency Tradeoff in LLM Serving with HyGen
Traditional instance scaling solutions address bursty workloads by launching new instances, which of seconds to several minutes cold-start delays [5] 64 66). To handle these delays, providers often keep standby instances online, leading to wasted resources during off-peak periods.
In contrast, HyGen optimizes resource utilization by running offline workloads on idle resources and reallocating them to online requests real-time, within single inference iteration. This eliminates cold-start delays while ensuring high resource utilization.
HyGen complements instance scaling solutions by automating the transition between online and offline workloads, reducing the need for manual intervention. While instance scaling manages large load fluctuations. HyGen ensures efficient resource use during low-traffic periods, optimizing overall system performance in fixed-size clusters.
28


Reviews:
[{"review_id": "sdOSU7ogx3", "reviewer": "Reviewer_KDZQ", "summary": "The paper introduces HyGen, an efficient LLM serving system that co-locates online  and offline workloads on the same GPU cluster. The authors investigate different features of the two workloads, i.e., latency-sensitive for online queries and throughput-oriented for offline workloads, and design different mechanism for managing the workloads. Specifically, the method leverages a control mechanism to predict and quantify latency, as well as an offline scheduling policy that maximize throughput.", "questions": "I understand that you might consider sharing the code after the paper is accepted. However, since ML sys papers are typically hard to evaluate without checking implementation details, would you be able to share the source code?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\n1. The proposed method integrates online and offline requests while maintaining strict latency guarantees. It can adapts to diverse latency requirements.\n\n2. The authors propose a novel control mechanism that estimates batch execution time and quantifies interference to ensure latency guarantees.\n\n3. The authors design an adaptive scheduler that maximizes offline throughput. It leverages a Prefix Sharing Maximization Strategy that transforms the offline requests into a prefix tree for capturing prefix sharing characteristics of the offline requests, thus reuse KV caches and improve efficiency.\n\n4. The performance of the approach achieves 3.87–5.84× throughput gains over baselines while maintaining strict latency SLOs. \n\n5. The evaluations are comprehensive.\n\n\nWeakness:\n\n1. This paper does not consider the overhead of shift between different workloads.\n\n2. The method is designed to optimize workloads for a single model instance. However, real-world deployments often serve multiple model replicas simultaneously.\n\n3. Typo: line 52: \"while guaranteeing strict SLO compliance and .\"\n\nSuggestions:\n\nThe authors might strengthen their evaluation by evaluating the impact of workload shifting on throughput.", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for recognizing our method as **novel**, our results as **strong**, and our evaluation as **comprehensive**. We appreciate the insightful questions and address them below.\n\n## **Concern 1: Overhead of Workload Shifting**\n\n> This paper does not consider the overhead of shift between different workloads.\n>\n> The authors might strengthen their evaluation by evaluating the impact of workload shifting on throughput.\n\n**HyGen is designed for minimal overhead during workload shifts, at both the micro (task-switching) and macro (workload-adaptation) levels.**\n\n1.  **Zero Runtime Task-Switching Overhead:** The switch from offline to online processing is nearly instantaneous, completing **\"within a single iteration step\"** (**Lines 628-632**). Our scheduling algorithm's low computational complexity, formally analyzed in **Appendix A.4**, ensures this preemption incurs negligible overhead.\n\n2.  **Minimal Macro-Level Adaptation Overhead:** HyGen adapts to new or changing workloads with extreme efficiency:\n    *   **Inherent Robustness:** The system is inherently robust to workload variations. **Fig. 14** shows that HyGen's performance remains stable even when predictor accuracy varies, proving its resilience to distributional shifts without needing reconfiguration.\n    *   **Trivial Retraining:** For entirely new models or tasks, retraining our lightweight latency predictor is trivial. As stated on **Line 299**, training on over 80,000 samples takes **only ~15ms on a CPU**. This makes the one-time cost of adapting to a new workload practically zero.\n\n## **Concern 2: Applicability to Multi-Replica Deployments**\n\n> The method is designed to optimize workloads for a single model instance. However, real-world deployments often serve multiple model replicas simultaneously.\n\n**HyGen scales seamlessly to multi-model-replica deployments as it functions as a modular, instance-level scheduler managed by a system-level router.**\n\n1.  **Scalable by Design (Architecture):** HyGen operates as a modular, **instance-level scheduler**, intended to run on each node of a large cluster. As detailed in **Sec. 5.1 (Line 227)**, a system-level router (e.g., Preble, as cited) distributes requests across these independent instances. This standard, decentralized architecture **has no central bottleneck** and scales horizontally simply by adding more nodes. Our experiments accurately model the workload and deployment complexity any single instance would face in such a deployment.\n\n2.  **Proven Effectiveness (Evidence):** Our evaluation already provides strong evidence for this scalability. The experiments using **Tensor and Pipeline Parallelism (Sec. 5.4, Fig. 9)** demonstrate that HyGen effectively manages the complex scheduling logic required in multi-GPU environments. This capability is foundational for and directly applicable to multi-replica production scenarios.\n\n\n## **Concern 3: Typo Correction**\n\n> Typo: line 52: \"while guaranteeing strict SLO compliance and .\"\n\nThank you for catching this. We will correct it in the final version.\n\n## **Concern 4: Code Availability**\n\n> I understand that you might consider sharing the code after the paper is accepted. However, since ML sys papers are typically hard to evaluate without checking implementation details, would you be able to share the source code?\n\n**We agree on the importance of implementation details and have provided the AC with a private, anonymized link to our development codebase, per NeurIPS policy.** We are enthusiastic about contributing our work to the community upon acceptance and believe a public release accompanying a peer-reviewed publication is the best way to ensure the work is presented completely and accurately. The publicized code will be thoroughly documented and well-structured, and we’re actively making efforts towards a public release version. We appreciate your understanding."}, {"review_id": "P5fg81spoC", "reviewer": "Reviewer_znBi", "summary": "This work proposes the HyGen, an interference-aware LLM serving system that enables efficient co-location of online and offline workloads while preserving service-level objectives. The two major contributions are: \n- performance control mechanisms: a latency predictor to estimate the batch execution time and a profiler to estimate latency interference\n- SLO-aware offline scheduling policies: maximize throughput and prevent starvation\n\n\nThe author starts with characterizing the different demands of online serving (low and stable response time) and offline serving (high throughput and better resource utilization). The author states that previous systems often segregate online and offline serving into different clusters.\nThe author argues that resource efficiency can be optimized by co-locating online and offline workloads on the same GPU cluster, because the real-world LLM workloads demonstrate up to 3x variation in request rates within minutes, leading to the need to provide more GPU resources for peak demand, but underutilize the resources during off-peak hours.", "questions": "- In section 4.2, when calculating the batch execution time, why do you add the quadratic scaling term, when the decode stage shows linear scaling?\n- Is the prefix sharing maximization strategy similar to the one used in the SGLang framework’s radix tree structure for prefix caching? Have you compared using block-level prefix caching, similar to vLLM? It would be interesting to have some ablation studies of using different prefix caching strategies.\n- In the experiments, I am wondering why HyGen can improve over the baselines in pure online and pure offline settings? What are the key designs that achieve that?", "limitations": "- The work has the goal of improving the serving of LLMs in large clusters, but the scale of the experiments seem quite limited (with only two nodes with 4 GPUs each)\n- In general, I find this work well-written and the motivation be clear and well-conveyed, the solutions seem conceptually simple but I do not doubt it involves a lot of engineering and optimization to make it work", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\n\n- The author addresses an important question in LLM serving in contemporary AI infrastructure, tackling the underutilization of GPU resources by intelligently utilizing both online and offline serving, this line of research could have potentially big impact for service providers of LLM serving\n- The author makes clear and comprehensive analysis of the challenges of building a hybrid serving system, summarizing the specific requirements to meet the demands of the current uses\n\nWeaknesses\n\n- The key to jointly handling online and offline requests lie in accurately estimating the latency for each request, but there is no experimental evaluation of the effectiveness linear regression model for latency prediction, also whether more advanced modelling can improve its robustness. Since it plays an important role in the proposed pipeline, more experimental details is necessary.", "was_revised": true, "final_justification": "The author clarified the points I mentioned in the review further, and I will maintain my initial assessment.", "rebuttal": "We sincerely thank the reviewer for their positive assessment and insightful questions. We are encouraged that the reviewer finds our work **\"well-written\"**, our motivation **\"clear and well-conveyed,\"** and recognizes the potential impact of our research. Below we address the questions and concerns point-by-point.\n\n## **Concern 1: Latency Predictor Effectiveness**\n\n> The key to jointly handling online and offline requests lie in accurately estimating the latency for each request, but there is no experimental evaluation of the effectiveness linear regression model for latency prediction, also whether more advanced modelling can improve its robustness.\n\n**Our latency predictor is highly accurate, as proven in our evaluation, and the system is robust even to significant prediction errors.**\n\n1.  **High Accuracy Proven:** As evaluated in **Fig. 5 (Sec. 5.3)**, our predictor is highly effective. The results show a mean absolute percentage error (MAPE) of only **1.78% and 1.07%** on a mixed production workload.\n2.  **Robustness to Inaccuracy:** While more complex models are possible, our system is not brittle. As shown in **Fig. 14**, HyGen strictly meets P99 SLOs even when we artificially increase the predictor's error to over 20%. This demonstrates that extreme accuracy is not required, making our lightweight linear model a practical and sufficient choice under runtime dynamics.\n\n## **Concern 2: Modeling of Batch Execution Time**\n\n> In section 4.2, when calculating the batch execution time, why do you add the quadratic scaling term, when the decode stage shows linear scaling?\n\n**The quadratic term is essential for accurately modeling mixed batches containing requests in the *prefill* stage, which has quadratic complexity.** While the *decode* stage is linear, a batch often contains a mix of requests. As stated in **Sec 4.2 (Lines 156-158)** and established in prior work [1,2,3], the prefill stage's attention mechanism has quadratic complexity w.r.t. input length. Our model (**Eq. 1**) includes the quadratic term `S²p` to correctly account for these prefill requests.\n\n[1] Attention Is All You Need. NeurIPS '17.\n[2] On The Computational Complexity of Self-Attention. ALT '23.\n[3] MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse. NeurIPS '24.\n\n## **Concern 3: Novelty of Prefix Sharing Strategy**\n\n> Is the prefix sharing maximization strategy similar to the one used in the SGLang framework’s radix tree structure for prefix caching? Have you compared using block-level prefix caching, similar to vLLM? It would be interesting to have some ablation studies of using different prefix caching strategies.\n\n**Our novelty is not the caching mechanism itself, but its SLO-aware application and fairness-aware extension in a hybrid serving context.** Our scheduling policy is orthogonal and complementary to low-level caching architectures like SGLang's RadixAttention or vLLM's Automated Prefix Caching. Our contributions are:\n\n1.  **SLO-Aware Co-location:** We are the first to apply a PSM-style strategy to opportunistically schedule offline requests using only the **residual capacity** from a primary, SLO-bound online workload.\n2.  **Fairness Extension:** We introduce a practical extension (**Lines 207-213**) that incorporates request freshness to **mitigate the starvation problem** inherent in naive PSM, making it deployable in production.\n\nWe agree that an ablation study on the interplay between our policy and different caching implementations would be a fascinating follow-up.\n\n## **Concern 4: Clarification on Performance Gains**\n\n> In the experiments, I am wondering why HyGen can improve over the baselines in pure online and pure offline settings? What are the key designs that achieve that?\n\n**To clarify, HyGen's significant throughput gain is achieved in a hybrid setting over a pure online baseline, demonstrating the value of co-location, not by outperforming specialized pure-mode systems.** You are correct that a specialized system is optimal for a single task.\n\n1.   The `Sarathi` (pure online) and `Sarathi-offline` baselines in **Fig. 4** represent the performance upper bounds for their respective tasks.\n2.   HyGen's goal is to **productively utilize the idle resources** of the pure online system. The reported gain (3.87-5.84×) is achieved by our hybrid system **over the `Sarathi` pure-online baseline**, demonstrating the value of our approach.\n\n## **Concern 5: Scalability and Experiment Design**\n\n> The work has the goal of improving the serving of LLMs in large clusters, but the scale of the experiments seem quite limited (with only two nodes with 4 GPUs each)\n\n**Our experimental scale is appropriate because HyGen is designed as an instance-level scheduler, a standard architecture for large-scale serving, and our results already prove its effectiveness in complex, distributed settings.**\n\n1.  **Scalable by Design (Architecture):** HyGen operates as a modular, **instance-level scheduler**, intended to run on each node of a large cluster. As detailed in **Sec. 5.1 (Line 227)**, a system-level router (e.g., Preble, as cited) distributes requests across these independent instances. This standard, decentralized architecture **has no central bottleneck** and scales horizontally simply by adding more nodes. Our experiments accurately model the workload and deployment complexity any single instance would face in such a deployment.\n\n2.  **Proven Effectiveness (Evidence):** Our evaluation already provides strong evidence for this scalability. The experiments using **Tensor and Pipeline Parallelism (Sec. 5.4, Fig. 9)** demonstrate that HyGen effectively manages the complex scheduling logic required in multi-GPU environments. This capability is foundational for and directly applicable to multi-replica production scenarios."}, {"review_id": "XsS11i4f37", "reviewer": "Reviewer_cK9i", "summary": "This paper proposes HyGen, an efficient serving system for Large Language Models that addresses the issue of low resource utilization in current LLM deployments through elastic online-offline request co-location. Its core methodology involves precise performance control using a latency predictor and an SLO-aware profiler, coupled with SLO-aware offline scheduling policies designed to maximize throughput and ensure fairness.", "questions": "1. Regarding Figure 1: How did the authors obtain the request rate data for Microsoft Azure's LLM service? \n2. The paper's organizational structure and presentation need significant optimization. Specifically, Section 3, which covers background and motivation, could be condensed or moved to the appendix. In contrast, the explanation of the proposed framework and Algorithm 1 in Section 4 is overly vague and confusing, urgently requiring clearer elaboration. More crucially, the content from Appendix A.1 and A.2 should be moved into the main body of the paper, as they constitute essential components of the core methodology.\n3. The architectural diagram (Figure 2) in the paper requires more detailed explanations. For instance, the diagram mentions \"check violation\" under \"Section 4.2,\" but there is no corresponding explanation or detailed description of this mechanism in the main text, leaving readers confused about its meaning.\n4. What are the specific novelties or differences of the \"Prefix Sharing Maximization Strategy\" presented in this paper compared to prior work? These distinctions should be clearly identified and elaborated upon.\n5. Please provide a specific and clear explanation for the phrase \"at a specific offline QPS\" in Line 240.", "limitations": "The paper seems to lack formal guarantees or theoretical analysis of the scheduling policies. Under what conditions might the SLO guarantees fail?", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The paper focuses on an interesting and important research direction: how to efficiently co-locate online and offline LLM workloads.\n2. The experimental evaluation is relatively comprehensive, providing a thorough investigation into the effectiveness of the proposed method.\n﻿\n﻿\nWeaknesses:\n1. Limited Evaluation Scope and Generalizability\n- The paper claims evaluation on \"production workloads\" but doesn't specify the diversity of LLM models tested. Does HyGen work equally well across different model architectures (GPT, LLaMA, etc.) and sizes? The performance characteristics may vary significantly.\n- No discussion of how the system performs under different hardware configurations (different GPU types, memory constraints, etc.)\n﻿\n2. Overhead and Complexity Analysis\n- The paper introduces a latency predictor and SLO-aware profiler but doesn't appear to quantify their computational overhead. These components themselves consume resources and may impact the very metrics they're trying to optimize.\n- Missing analysis of the system's bootstrapping phase - how long does it take for the predictor to become accurate? What happens during this warm-up period?\n﻿\n3. Workload Assumptions and Limitations\n- The system appears to assume predictable workload patterns. How does it handle sudden spikes or highly variable request patterns?\n- No discussion of handling failures or degraded performance scenarios. What happens when predictions are wrong?\n﻿\n4. Comparison Methodology Concerns\n- Lack baseline fairness. Were the baselines properly optimized?\n- Missing comparison with other state-of-the-art hybrid serving systems beyond \"online and hybrid serving baselines\"\n﻿\n5. Practical Deployment Challenges\n- No discussion of deployment complexity or operational overhead. How difficult is it to tune the system parameters in practice?\n- Missing analysis of how the system handles model updates or version changes, which are common in production environments", "was_revised": true, "final_justification": "Thank you for your rebuttal and the additional experiments provided. While I appreciate the efforts to address some concerns, several critical issues remain unaddressed or insufficiently resolved, which are essential to evaluating the robustness and practicality of HyGen.\n \nFirst, regarding the evaluation scope and generalizability: the inclusion of more models (e.g., Sheared-LLaMA-2.7B) and an additional GPU (A5000) is a step forward, but questions about performance across broader architectures (e.g., GPT variants) and detailed hardware constraints (e.g., memory limitations under varying loads) still lack concrete analysis.\n \nMore importantly, key concerns about overhead and complexity—specifically the computational costs of the latency predictor and SLO-aware profiler, as well as the bootstrapping/warm-up phase of the predictor—remain entirely unaddressed. These are critical to understanding whether the system’s components introduce bottlenecks that undermine its claimed efficiency.\n \nAdditionally, there is no discussion of how HyGen handles sudden workload spikes, variable request patterns, or failure scenarios (e.g., when latency predictions are inaccurate). Such scenarios are common in production, and their absence weakens claims of practical applicability.\n \nThe comparison methodology also remains problematic: there is no clarification on whether baselines were properly optimized, nor additional comparisons with state-of-the-art hybrid serving systems beyond the mentioned \"online and hybrid baselines.\"\n \nFinally, practical deployment challenges—including parameter tuning difficulty and handling model updates—are still unaddressed, leaving uncertainties about the system’s real-world feasibility.\n \nIn summary, while some clarifications (e.g., \"check violation\" in Figure 2, the \"Prefix Sharing Maximization\" novelties) are helpful, the core issues related to overhead, workload robustness, comparison rigor, and deployment practicality need more thorough analysis to justify the work’s validity.\n\nSo i will keep my original score.", "rebuttal": "We thank the reviewer for their valuable feedback and for recognizing our work as an **\"interesting and important research direction\"** with a **\"relatively comprehensive\"** experimental evaluation. We believe the concerns raised are primarily due to misunderstandings, which we have addressed point-by-point below. We have also included **new experimental results** on additional hardware to further strengthen our claims of generalizability.\n\n## **Concern 1: Limited Evaluation Scope and Generalizability**\n\n> Does HyGen work equally well across different model architectures... and sizes?\n>\n> No discussion of how the system performs under different hardware configurations...\n\n**HyGen is generalizable across diverse models and hardware, as demonstrated in the paper and confirmed with new experiments.**\n\n1.  **Paper's Evidence:** Our evaluation already included diverse models (**Llama2-7B, Qwen-14B, Yi-34B, Mistral-7B**) on two distinct hardware platforms (**4x A100 & 4x A40 GPUs**), as detailed in **Sec. 5.1 & 5.4**. The consistent results in our figures validate HyGen's robustness.\n\n2.  **New Results:** We conducted new tests on an **NVIDIA A5000 GPU (24 GB VRAM)** with a **Sheared-LLaMA-2.7B model**. The results below confirm HyGen achieves significant throughput gains (**up to 2.18× offline and 1.30× total**), reinforcing its adaptability.\n\n| Mean TBT Tolerance | Offline TPS HyGen* | Total TPS HyGen* | Offline TPS HyGen | Total TPS HyGen | Offline Throughput Gain | Total Throughput Gain |\n| :--- | :--- | :--- | :--- | :--- | :--- |:--- |\n| 5% | 214.84 | 1969.84 | 468.59 | 2223.59 | **2.18×** | **1.13×** |\n| 10% | 366.35 | 2121.35 | 678.55 | 2433.55 | **1.85×** | **1.15×** |\n| 15% | 519.48 | 2274.48 | 987.93 | 2742.93 | **1.90×** | **1.21×** |\n| 20% | 667.59 | 2422.59 | 1247.69 | 3002.69 | **1.87×** | **1.24×** |\n| 25% | 811.96 | 2566.96 | 1432.88 | 3187.88 | **1.76×** | **1.24×** |\n| 30% | 909.68 | 2664.68 | 1600.49 | 3355.49 | **1.76×** | **1.26×** |\n| 35% | 1009.28 | 2764.28 | 1831.82 | 3586.82 | **1.81×** | **1.30×** |\n\n## **Concern 2: Overhead and Complexity Analysis**\n\n> The paper introduces a latency predictor and SLO-aware profiler but doesn't appear to quantify their computational overhead.\n>\n> ...how long does it take for the predictor to become accurate? What happens during this warm-up period?\n\n**HyGen has a low-cost, one-time bootstrapping and negligible runtime overhead.**\n\n1.  **Bootstrapping:** There is no \"warm-up\" period. The process is a **one-time, pre-deployment step** that is extremely lightweight, requiring **only ~15ms on a CPU** for over 80,000 samples (**Line 299**).\n2.  **Runtime Overhead:** We measured prediction overhead just **~80μs per scheduling iteration** on a CPU, **three orders of magnitude smaller** than a typical chunked batch step. This overhead can be **fully hidden** by asynchronous scheduling (e.g., in recent of vLLM/SGLang versions), where scheduler logic runs on the CPU concurrently with GPU execution. We will clarify this in the final version. We included algorithm complexity analysis to analyze its negligible overhead in Appendix A.4 too.\n\n> No discussion of deployment complexity or operational overhead. How difficult is it to tune the system parameters in practice?\n>\n> Missing analysis of how the system handles model updates or version changes, which are common in production environments\n\n**HyGen is designed for simple deployment and minimal operational overhead.**\n\n1.  **Automatic Tuning & Trivial Updates:** Deployment is straightforward. The key parameter (latency budget) is set **automatically** by a one-time profiler (**Sec. 4.2**), requiring no manual tuning. For model updates, this lightweight profiling is simply re-run. Training on 80k+ samples takes **only ~15 ms on a CPU (Line 299)**, making adaptation cost trivial.\n\n2.  **Zero-Overhead Runtime Adaptation:** HyGen adapts to workload shifts instantly. The switch from offline to online processing completes **\"within a single decoding step\"** (**Lines 628–632**), due to our asynchronous design. **Fig. 14** demonstrates the system's robustness to prediction errors and workload variations, maintaining SLOs without reconfiguration.\n\n## **Concern 3: Workload Assumptions and Limitations**\n\n> The system appears to assume predictable workload patterns. How does it handle sudden spikes or highly variable request patterns?\n\n**HyGen is expressly designed for unpredictable, bursty workloads.**\n1.  We evaluate our system using **two public, bursty production traces**: the **Microsoft Azure trace (Patel et al.; Sec. 3.2, Fig. 1)** and the **Kimi Mooncake trace (App. B, Fig. 13 & 16)**.\n2.  **Fig. 8** shows HyGen dynamically throttling offline throughput in direct response to unpredictable online bursts.\n3.  The preemption mechanism is built precisely for this purpose (**Sec. 4**, **App. D**).\n\n> No discussion of handling failures or degraded performance scenarios. What happens when predictions are wrong?\n\n**HyGen is inherently robust to prediction errors and system degradation via its closed-loop design.**\n\n1.  **Prediction Errors:** The system is not brittle. The profiler establishes a conservative latency budget (**Alg. 1, Lines 15–16**). **Fig. 14** shows that even with predictor error (MAPE) **>20%**, HyGen maintains high throughput while *strictly* meeting the P99 SLO.\n2.  **System Degradation:** For issues like thermal throttling, HyGen adapts automatically without any tuning. If system performance degrades, batch latencies increase, automatically reducing the available latency budget for the *next* scheduling iteration and ensuring SLOs are always met.\n\n## **Concern 4: Comparison Methodology Concerns**\n\n> Lack baseline fairness. Were the baselines properly optimized?\n>\n> Missing comparison with other state-of-the-art hybrid serving systems beyond \"online and hybrid serving baselines\"\n\n**Our baselines are SOTA and meticulously designed for a rigorous and  fair evaluation.** While other hybrid systems (e.g., MuxServe, Punica) solve orthogonal problems, our comparison focuses on single-model, online/offline co-location:\n1.  **SOTA Foundation:** We chose **Sarathi (OSDI '24)** as our foundation. Sarathi's chunked prefill is used to support numerous recent LLM advancements (e.g., EMNLP '24 Memorize Step by Step, ASPLOS '25 POD-Attention, SwiftKV, PrefillOnly), making it a widely-used SOTA system.\n2.  **Optimized \"Ceiling\" Baseline:** We compared against a fully-optimized pure-offline system (`Sarathi-offline`) representing max throughput (**Fig. 4**). HyGen reaches **84.3%** of this theoretical ceiling while serving an online workload under strict SLOs.\n3.  **Fair Isolation Baselines:** To ensure a fair, apples-to-apples comparison, we enhanced Sarathi with our own **online-first preemption mechanism**, thus isolating the benefit of our SLO-aware scheduler. To ablate our dynamic scheduling algorithm, we created **`HyGen*`** that uses a static offline rate. The large performance gap between HyGen and `HyGen*` (**Fig. 4 & 9**) cleanly quantifies the gains from our dynamic, SLO-aware approach alone.\n\n## **Specific Questions**\n\n> How did the authors obtain the request rate data for Microsoft Azure's LLM service?\n\nAs cited in **Line 106**, this trace is publicly available (Patel et al., ISCA 2024). Our use of this trace is detailed in **Sec. 5.1 (Line 222)**.\n\n> The paper's organizational structure and presentation need significant optimization... the explanation of the proposed framework and Algorithm 1 in Section 4 is overly vague... content from Appendix A.1 and A.2 should be moved into the main body.\n\nThank you for the feedback. We agree Section 4 can be clearer. As moving the full pseudocode would exceed the page limit, in the final version, we will **enhance Sec. 4 by integrating the key architectural details from Appendix A.1**. We will add a description of our asynchronous, dual-queue scheduling workflow to the main text, clarifying how Alg. 1 is invoked and providing necessary system-level context.\n\n> The architectural diagram (Figure 2) requires more detailed explanations. For instance, the diagram mentions \"check violation\" under \"Section 4.2,\" but there is no corresponding explanation.\n\nThe \"check violation\" block refers to our core SLO compliance logic. The system predicts a potential batch's latency and **checks if it violates** the pre-set budget. This is the central idea of **Sec. 4.2**. We will add this explanation to the main text and **Fig. 2**'s caption.\n\n> What are the specific novelties or differences of the \"Prefix Sharing Maximization Strategy\"...?\n\nWe clarify that our novelty is not the Prefix Sharing Maximization (PSM) concept itself, which we cited as prior work (**Line 189**). Our contributions are the **SLO-aware application and extension** of PSM, which again highlights the effectiveness of our solution to augment existing deployments:\n1.  Our system integrates PSM to maximize throughput for offline tasks using only the **residual GPU capacity** available *after* satisfying strict online SLOs.\n2.  Our extended policy (**Lines 207-213**) introduces a **fairness mechanism** to mitigate request starvation, a critical issue for production deployments that is unaddressed by naive PSM.\n\n> Please provide a specific and clear explanation for the phrase \"at a specific offline QPS\" in Line 240.\n\nThis describes our **`HyGen*` baseline**. We profile the max **fixed, constant** offline QPS the system sustains without violating online SLOs. `HyGen*` serves requests at this static rate, contrasting with the full HyGen system, which **dynamically** adjusts its offline rate. We will rephrase for clarity."}]

Author rebuttal:

Meta-review: This paper proposes HyGen, an efficient serving system for large language models using online-offline request co-location. It uses a latency predictor and an SLO-aware profiler, coupled with SLO-aware offline scheduling policies designed to maximize throughput and ensure fairness.

Strengths:
1. This paper introduces a new and important problem, i.e., efficient co-location of online and offline workloads.

Weaknesses:
1. As suggested by cK9i, there are many questions regarding how this new framework applies to different models and compute resource constraints and how this new framework addresses some extreme edge cases.

These gaps persist despite the rebuttal and are essential to assessing HyGen’s broader applicability and reliability.

However, I believe this paper marks a significant step froward in this particular research direction.

For the remaining concerns raised by cK9i, see the author's final remarks.

P1. Our data-driven latency predictor, T_batch = f(Sp, Sd, S_p^2, S_d^2, Np, Nd), adapts flexibly to new architectures. For hybrid models (e.g., Mamba-Transformer), it learns a reduced S_p^2 coefficient (Transformer cost) while capturing Mamba’s linear cost via Sp, consistent with Flash-Linear-Attention GitHub repo's report. For MoE models, due to a fixed activated expert size per token, MoE-FFN costs are captured by Sp and Sd. This again highlights the flexibility and effectiveness of our design.

P2. The “warm-up” concern reflects a misunderstanding. Latency predictor and budget are fixed via one-time, pre-deployment profiling; No runtime stabilization and no SLO risk at startup. The system is immediately ready.

P3. HyGen uses a multi-layer strategy for traffic bursts. First, its latency budget is robust to in-distribution bursts, as it is derived from realistic traces containing such bursts. On spikes, online queues fill, offline requests are preempted/blocked (Fig. 8). Our evaluations on realistic workloads (Fig. 4), with large fluctuations (e.g., request load varies up to 8x in minutes), show HyGen outperforms prior methods. Second, it remains resilient to out-of-distribution events, maintaining SLOs even with 20% MAPE (Fig. 14). Third, for lasting workload shifts, it adapts using feedback from online deployment, asynchronously to serving. Predictor retraining is highly efficient (80k samples in 15 ms).

P4. Baselines were transparently optimized. The Sarathi-offline ceiling baseline tunes chunk size via hyperparameter search (Line 252), achieving 12% throughput improvement over default.

P5. Our problem scope is orthogonal to the cited hybrid systems. Multi-LoRA methods (e.g., Punica) batch requests from different LoRAs, while multi-model methods (e.g., MuxServe) multiplex models across space. Instead, HyGen batches online and offline requests, addressing distinct optimization problems that are not interchangeable.

P6. The profiler’s practicality comes from its data-driven design: it measures performance on target hardware under the actual online request distribution, capturing hardware and memory constraints. Our evaluations show that HyGen efficiently adjusts latency budgets when model, hardware, workload, or SLO changes (Figures 4, 8, 14, and our A5000 experiment replied in Concern 1), as noted in P3.
Decision: Accept (poster)