Title: HyGen: Efficient LLM Serving via Elastic Online-Offline Request Co-location
Venue: NeurIPS 2025
Accepted: yes
Keywords: LLM Inference

Request Scheduling

Machine Learning Systems
Abstract: Large language models (LLMs) have facilitated a wide range of applications with distinct service-level objectives (SLOs), from latency-sensitive online tasks like interactive chatbots to throughput-oriented offline workloads like data synthesis. The existing deployment model, which dedicates machines to each workload, simplifies SLO management but often leads to poor resource utilization. This paper introduces HyGen, an interference-aware LLM serving system that enables efficient co-location of online and offline workloads while preserving SLOs. HyGen incorporates two key innovations: (1) performance control mechanisms, including a latency predictor to estimate batch execution time and an SLO-aware profiler to quantify latency interference, and (2) SLO-aware offline scheduling policies that maximize serving throughput and prevent starvation. Our evaluation on production workloads shows that HyGen achieves up to 3.9-5.8× throughput gains over online and hybrid serving baselines, while ensuring latency SLOs. The code of HyGen is publicly available at https://github.com/UIUC-MLSys/HyGen.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes HyGen, an efficient serving system for Large Language Models that addresses the issue of low resource utilization in current LLM deployments through elastic online-offline request co-location. Its core methodology involves precise performance control using a latency predictor and an SLO-aware profiler, coupled with SLO-aware offline scheduling policies designed to maximize throughput and ensure fairness.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper focuses on an interesting and important research direction: how to efficiently co-locate online and offline LLM workloads.\n2. The experimental evaluation is relatively comprehensive, providing a thorough investigation into the effectiveness of the proposed method.\n﻿\n﻿\nWeaknesses:\n1. Limited Evaluation Scope and Generalizability\n- The paper claims evaluation on \"production workloads\" but doesn't specify the diversity of LLM models tested. Does HyGen work equally well across different model architectures (GPT, LLaMA, etc.) and sizes? The performance characteristics may vary significantly.\n- No discussion of how the system performs under different hardware configurations (different GPU types, memory constraints, etc.)\n﻿\n2. Overhead and Complexity Analysis\n- The paper introduces a latency predictor and SLO-aware profiler but doesn't appear to quantify their computational overhead. These components themselves consume resources and may impact the very metrics they're trying to optimize.\n- Missing analysis of the system's bootstrapping phase - how long does it take for the predictor to become accurate? What happens during this warm-up period?\n﻿\n3. Workload Assumptions and Limitations\n- The system appears to assume predictable workload patterns. How does it handle sudden spikes or highly variable request patterns?\n- No discussion of handling failures or degraded performance scenarios. What happens when predictions are wrong?\n﻿\n4. Comparison Methodology Concerns\n- Lack baseline fairness. Were the baselines properly optimized?\n- Missing comparison with other state-of-the-art hybrid serving systems beyond \"online and hybrid serving baselines\"\n﻿\n5. Practical Deployment Challenges\n- No discussion of deployment complexity or operational overhead. How difficult is it to tune the system parameters in practice?\n- Missing analysis of how the system handles model updates or version changes, which are common in production environments\n\nQUESTIONS:\n1. Regarding Figure 1: How did the authors obtain the request rate data for Microsoft Azure's LLM service? \n2. The paper's organizational structure and presentation need significant optimization. Specifically, Section 3, which covers background and motivation, could be condensed or moved to the appendix. In contrast, the explanation of the proposed framework and Algorithm 1 in Section 4 is overly vague and confusing, urgently requiring clearer elaboration. More crucially, the content from Appendix A.1 and A.2 should be moved into the main body of the paper, as they constitute essential components of the core methodology.\n3. The architectural diagram (Figure 2) in the paper requires more detailed explanations. For instance, the diagram mentions \"check violation\" under \"Section 4.2,\" but there is no corresponding explanation or detailed description of this mechanism in the main text, leaving readers confused about its meaning.\n4. What are the specific novelties or differences of the \"Prefix Sharing Maximization Strategy\" presented in this paper compared to prior work? These distinctions should be clearly identified and elaborated upon.\n5. Please provide a specific and clear explanation for the phrase \"at a specific offline QPS\" in Line 240.\n\nLIMITATIONS:\nThe paper seems to lack formal guarantees or theoretical analysis of the scheduling policies. Under what conditions might the SLO guarantees fail?", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-18"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes the HyGen, an interference-aware LLM serving system that enables efficient co-location of online and offline workloads while preserving service-level objectives. The two major contributions are: \n- performance control mechanisms: a latency predictor to estimate the batch execution time and a profiler to estimate latency interference\n- SLO-aware offline scheduling policies: maximize throughput and prevent starvation\n\n\nThe author starts with characterizing the different demands of online serving (low and stable response time) and offline serving (high throughput and better resource utilization). The author states that previous systems often segregate online and offline serving into different clusters.\nThe author argues that resource efficiency can be optimized by co-locating online and offline workloads on the same GPU cluster, because the real-world LLM workloads demonstrate up to 3x variation in request rates within minutes, leading to the need to provide more GPU resources for peak demand, but underutilize the resources during off-peak hours.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n- The author addresses an important question in LLM serving in contemporary AI infrastructure, tackling the underutilization of GPU resources by intelligently utilizing both online and offline serving, this line of research could have potentially big impact for service providers of LLM serving\n- The author makes clear and comprehensive analysis of the challenges of building a hybrid serving system, summarizing the specific requirements to meet the demands of the current uses\n\nWeaknesses\n\n- The key to jointly handling online and offline requests lie in accurately estimating the latency for each request, but there is no experimental evaluation of the effectiveness linear regression model for latency prediction, also whether more advanced modelling can improve its robustness. Since it plays an important role in the proposed pipeline, more experimental details is necessary.\n\nQUESTIONS:\n- In section 4.2, when calculating the batch execution time, why do you add the quadratic scaling term, when the decode stage shows linear scaling?\n- Is the prefix sharing maximization strategy similar to the one used in the SGLang framework’s radix tree structure for prefix caching? Have you compared using block-level prefix caching, similar to vLLM? It would be interesting to have some ablation studies of using different prefix caching strategies.\n- In the experiments, I am wondering why HyGen can improve over the baselines in pure online and pure offline settings? What are the key designs that achieve that?\n\nLIMITATIONS:\n- The work has the goal of improving the serving of LLMs in large clusters, but the scale of the experiments seem quite limited (with only two nodes with 4 GPUs each)\n- In general, I find this work well-written and the motivation be clear and well-conveyed, the solutions seem conceptually simple but I do not doubt it involves a lot of engineering and optimization to make it work", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces HyGen, an efficient LLM serving system that co-locates online  and offline workloads on the same GPU cluster. The authors investigate different features of the two workloads, i.e., latency-sensitive for online queries and throughput-oriented for offline workloads, and design different mechanism for managing the workloads. Specifically, the method leverages a control mechanism to predict and quantify latency, as well as an offline scheduling policy that maximize throughput.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The proposed method integrates online and offline requests while maintaining strict latency guarantees. It can adapts to diverse latency requirements.\n\n2. The authors propose a novel control mechanism that estimates batch execution time and quantifies interference to ensure latency guarantees.\n\n3. The authors design an adaptive scheduler that maximizes offline throughput. It leverages a Prefix Sharing Maximization Strategy that transforms the offline requests into a prefix tree for capturing prefix sharing characteristics of the offline requests, thus reuse KV caches and improve efficiency.\n\n4. The performance of the approach achieves 3.87–5.84× throughput gains over baselines while maintaining strict latency SLOs. \n\n5. The evaluations are comprehensive.\n\n\nWeakness:\n\n1. This paper does not consider the overhead of shift between different workloads.\n\n2. The method is designed to optimize workloads for a single model instance. However, real-world deployments often serve multiple model replicas simultaneously.\n\n3. Typo: line 52: \"while guaranteeing strict SLO compliance and .\"\n\nSuggestions:\n\nThe authors might strengthen their evaluation by evaluating the impact of workload shifting on throughput.\n\nQUESTIONS:\nI understand that you might consider sharing the code after the paper is accepted. However, since ML sys papers are typically hard to evaluate without checking implementation details, would you be able to share the source code?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-03"}