Title: Transcending Cost-Quality Tradeoff in Agent Serving via Session-Awareness
Venue: NeurIPS 2025
Accepted: yes
Keywords: Agent Serving

LLM Serving

LLM infrastructure

Large language models

inference optimizations
Abstract: Large Language Model (LLM) agents are capable of task execution across various domains by autonomously interacting with environments and refining LLM responses based on feedback.
However, existing model serving systems are not optimized for the unique demands of serving agents. Compared to classic model serving, agent serving has different characteristics:
predictable request pattern, increasing quality requirement, and unique prompt formatting. We identify a key problem for agent serving: LLM serving systems lack session-awareness. They neither perform effective KV cache management nor precisely select the cheapest yet competent model in each round.
This leads to a cost-quality tradeoff, and we identify an opportunity to surpass it in an agent serving system.

To this end, we introduce AgServe for AGile AGent SERVing.
AgServe features a session-aware server that boosts KV cache reuse via Estimated-Time-of-Arrival-based eviction and in-place positional embedding calibration, a quality-aware client that performs session-aware model cascading through real-time quality assessment, and a dynamic resource scheduler that maximizes GPU utilization. 
With AgServe, we allow agents to select and upgrade models during the session lifetime, and to achieve similar quality at much lower costs, effectively transcending the tradeoff. Extensive experiments on real testbeds demonstrate that AgServe (1) achieves comparable response quality to GPT-4o at a 16.5\% cost. (2) delivers 1.8$\times$ improvement in quality relative to the tradeoff curve.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents AGSERVE, a session‐aware agent serving system designed to “transcend” the traditional cost–quality tradeoff in LLM‐based agent workflows. By combining (1) in‐place RoPE calibration and an ETA‐based cache eviction policy to boost KV‐cache reuse under middle‐truncation, (2) a Quality Maintenance Module (QMM) with a Q-Judge for initial model selection and an R-Judge for runtime quality monitoring and migrations, and (3) a dynamic GPU scheduler for demand‐driven resource allocation, AGSERVE claims up to 83.5% cost savings versus GPT-4o at equivalent quality, 1.8× quality gain at fixed cost, 2.86× cache‐hit improvement, and 1.2× end-to-end speedup on real multi-GPU testbeds across four agent benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n1. Tailors cache and eviction policies to multi-round agent contexts, yielding substantial cache reuse and latency gains.\n\n2. Integrates caching, quality judgers, and resource scheduling under a coherent API, simplifying adoption.\n\n3. Thorough ablations isolate the benefits of in-place calibration, ECE vs. LRU eviction, Q-Judge/R-Judge overheads, and dynamic allocation across two testbeds (A6000 and A8000 clusters).\n\n\nWeaknesses\n1. All experiments target four programming-agent tasks on AgentBench. It remains unclear whether AGSERVE’s session-aware optimizations generalize to non-coding agents (e.g., dialogue, planning).\n\n2. While Q-Judge and R-Judge training details and accuracy/distribution tables are provided, the paper omits any analysis of misclassification trade-offs (false-positive vs. false-negative rates) or calibration curves that would clarify migration costs under varying thresholds.\n\n3. The ECE eviction decision algorithm is described in pseudocode, but its runtime complexity and decision-latency overhead under tight memory budgets are neither analyzed nor experimentally measured.\n\nQUESTIONS:\n1. Have you evaluated AGSERVE on any non-programming agent workloads to assess generality?\n\n2. Can you provide false-positive/false-negative curves for the R-Judge, and quantify migration overhead when misclassifications occur?\n\n3. What is the worst-case decision latency of the ECE algorithm as agent count and cache size scale?\n\nLIMITATIONS:\nExperiments are confined to four programming‐agent tasks, leaving AGSERVE’s effectiveness on dialogue, planning, or other agent workloads unverified.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe work proposes an agent serving system called AgServe, which jointly optimize KV cache reuse, model selection and resource allocation, reducing the agen serving cose while achieving higher quality results.\n\nThe author starts by arguing that the current serving systems are largely developed for serving generic LLMs and are not optimized for the agent workflows. The authors have identified the following agent serving characteristics:\n- predictable and high frequency interaction: the interactions with agents are light-weight (less tokens) and frequent\n- growing context and increasing difficulties: agents interact in multi-round fashion, making the context length grows rapidly\n- unique prompt formatting: agent has specific ways of formatting prompts\n\nThe main idea is to predict sessions to reduce inference latency and depending on the phase of the session, use different-sized models to reduce cost.\n\nThe major challenges to achieve this are:\nsession aware cache policy: current system’s reuse and eviction policies are not well-suited for agent serving\n- quality assessment: existing systems do not estimate the session lifetime and do not have the ability to dynamically assess model response quality and thus cannot choose the best model for each stage\n- resource allocation: the gpu usage of model switching is not optimized\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n- clear motivation and detailed problem analysis: the authors start by claiming that there does not exist a serving system that is optimized for serving LLM agents, then\nanalyzing the problems of using the generic LLM’s serving systems, making its motivation clear\n\n\nWeaknesses\n\n- The idea of introducing a Q-Judge and R-Judge is interesting, but I am having doubts that the mentioned training of Q-Judge / R-Judge is sufficient to accomplish their designed roles, by only training on Chatbot-Arena for such little time, also the only training on this dataset seems limited, leaving doubt that whether the trained Q-Judge and R-Judge is able to handle all kinds of tasks, or it would require task specific training\n\nQUESTIONS:\n- I am curious what the Q-Judge / R-Judge are practice? Are they also LLMs? How can you ensure these instances fulfill the designated tasks? E.g., the Q-Judge should select the most cost-efficient model for the session by assessing the task’s difficulty, but how can you be sure it chose the most cost-efficient model? Do you need to train them or provide additional context? Similarly, the R-Judge should evaluate the reasoning quality, how do decide the consistency in evaluation and how do you set the standard / threshold?\n- The author mentioned the customized CUDA kernels in chapter 7, what exactly does the kernels do and why do you need them? There is no explanation of that.\n- The ECE eviction policy seems reasonable, but it comes with additionally computation overhead, because it needs to make prediction? LRU follows the simple but universal rule of least recently used to evict cache and since LRU has been adopted by the two most popular inference framework vLLM and SGLang, I believe it has shown robustness in large scale caching. Whether ECE is truely beneficial needs more large scaled validations\n\nLIMITATIONS:\n- Generally I feel like the delivery of this paper needs further improvement to convey their message more cleaner; currently they keeps on mentioning new concepts, without previous definition or explanation, making the reading difficult. For example, in the beginning of section 7, the author mentioned different agens (AW, CG, KG, M2W), but you have to search through the whole paper and find the definition in the Appendix, or in the ECE Eviction Policy section, it states the formula for calculating the eviction cost, and the P is also not defined. There are other examples like this.\n- The strictness of R-Judge theta seems to be an important hyperparamter, as it greatly controls the decision of R-Judge, but how it is obtained and quantified seems unclear (judging from the section 5.2 and the Appendix B) \n- Model allocation seems to be too simplified, the RS profiles the nunimum number of GPUs required, but in practise, increasing the number of GPUs will lead to faster inference, this trade-off should be considered when allocating memory", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces AGSERVE, a novel system designed to overcome the cost-quality tradeoff in serving Large Language Model (LLM) agents. Unlike traditional LLM serving systems, AGSERVE is optimized for the unique characteristics of agentic workflows, such as predictable request patterns, increasing quality requirements over multi-round sessions, and unique prompt formatting including middle truncation. AGSERVE features a session-aware server for efficient KV cache management, a quality-aware client that employs real-time quality assessment for dynamic model cascading, and a dynamic resource scheduler to maximize GPU utilization. Experimental results demonstrate that AGSERVE achieves comparable response quality to GPT-4o at a significantly lower cost (16.5% in some cases) , delivers a 1.8x improvement in quality relative to the cost-quality tradeoff curve , and increases KV cache hit rates by up to 2.86x compared to existing methods.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n- The paper identifies and tackles the unique challenges of serving LLM agents, distinguishing it from traditional LLM serving systems that are not optimized for agent workflows. This focus on session-awareness, predictable request patterns, and growing context is a key strength.\n\n- AGSERVE offers a holistic solution by integrating a session-aware server with innovative KV cache management (ETA-based eviction and in-place positional embedding calibration), a quality-aware client for session-aware model cascading, and a dynamic resource scheduler. These components work together to transcend the cost-quality tradeoff.\n\n- The paper provides strong empirical evidence on real testbeds, showing that AGSERVE achieves comparable quality to GPT-4o at a substantially lower cost (16.5%), delivers a 1.8x quality improvement relative to the tradeoff curve, and significantly boosts KV cache hit rates (2.84x) while reducing latency.\n\n## Weaknesses\n\n- The evaluation primarily focuses on four specific agent implementations (AlfWorld, card game, knowledge graph, and Mind2Web) . While these cover different interaction patterns, the generalizability of AGSERVE's benefits across a wider range of LLM agent applications is not fully explored.\n\n- The in-house deployment of AGSERVE primarily uses Llama-3 (8B and 70B versions) as the adjustable models . While GPT-4o is used as an external API , the system's performance and optimizations might be less effective or require significant re-tuning for other LLM architectures or model families that are not from the same family and do not share the same tokens and dictionary .\n\n- The cost estimation for open-source models is based on retail cloud server rentals, and the paper acknowledges that \"the cost of open-source models could be significantly lower (30% to 60% lower) for self-operated data centers or long-term users\" . This simplification might not fully capture the true cost-efficiency advantages or disadvantages in all deployment scenarios, potentially skewing the cost-quality comparison.\n\n- The R-Judge training relies on a customized Chatbot-Arena dataset where human experts labeled responses. The paper notes the difficulty of annotating prompt difficulty for Q-Judge , and it's unclear if the \"human-selected preferences\" for R-Judge fully capture the nuances of agent task success or are prone to subjective biases.\n\n- While the paper mentions related works, some aspects of AGSERVE, like KV cache management and model cascading, could benefit from a more in-depth comparison with the very latest developments in those specific areas, beyond just general LLM serving systems. For instance, the discussion on cache reuse mentions CacheBlend , but a detailed comparative analysis of their respective strengths for agent serving is limited.\n\nQUESTIONS:\n- What are the specific heuristics or machine learning models used to predict the Estimated Time of Arrival (ETA) for sessions, which is crucial for the ECE eviction policy, and how robust are these predictions to unexpected agent behaviors or workload shifts?\n\n- Beyond the qualitative \"satisfactory rate drops as round increases\" observation, can a more quantitative characterization of the increasing task difficulty for LLMs as context grows be provided, perhaps in terms of specific metrics or a more detailed analysis of failure modes at later stages of agent sessions?\n\n- Given that the paper \"disregard[s] the sessions that trigger rule violations\"  when evaluating the performance of the RS Dynamic Allocation, how frequently do such violations occur in practice, what are their root causes, and how does AGSERVE handle them to maintain overall system stability and performance?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a session-aware system for LLM agent serving, addressing inefficiencies in KV cache management, model selection, and resource allocation. Key innovations include ETA-based KV cache eviction, in-place positional embedding calibration for middle-truncation reuse, quality-aware cascading via QMM, and dynamic GPU scheduling. Evaluated on AgentBench with agents (e.g., ALFWorld, Mind2Web) and models (Llama-3 8B/70B, GPT-4o), it achieves 83.5% cost reduction at GPT-4o quality, 1.8x quality at same cost, 2.84x cache hit rate, and up to 50% latency cut vs. baselines like vLLM and Llumnix++.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n- The paper is clearly presented and well written.\n- The paper proposes the session-aware kv cache strategy, the proposed eta eviction and calibration enhance reuse for multi-round agents, outperforming LRU with 2.84x hit rates. This could offer tailored advantages against existing prefix-focused systems, which optimize general inference but overlook agent-specific middle-truncation.\n- The paper proposes quality-aware cascading, the session-level upgrades via QMM (95.2% recall) minimize costs by preferring underkill, surpassing RouteLLM, and could achieve similar quality to GPT-4o with a 16.5% cost in AW.\n- The proposed system integrates cache, cascading, and scheduling for 65.6% multi-agent savings and provides throughout empirical evaluations on AgentBench and synthetic request traces from Poisson and Gamma distributions. Theoretical proofs has also been provided for in-place kv-cache calibration. \n\nWeakness:\n- The assumption of ece is uniform/burst, which may be vulnerable to the variability as noted in [1]. \n- In sec 7.4.2, the authors discuss the low overhead of QMM, but this may accumulate with long-input or long-context checks (different scenarios of prefilling/decoding bound), providing more details about the context-length, quality & efficiency tradeoff could make the claim stronger; \n- It is understandable that the study already provides throughout analyses on A6000 and recent AgentBench, but the scalability might be untested vs. heterogeneous LLMs and the fixed three-tier cascade lacks diversity; \n\n[1] Wang, Jiahao, et al. \"KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Provider.\" arXiv preprint arXiv:2506.02634 (2025).\n\nQUESTIONS:\n- The authors mentioned the long-input scenarios in sec 7.2. Could the authors elaborate a bit more on how to handle the straggler settings with the four agents?\n- Could the authors provide more cascade performance with more levels/non-Llama models, and potentially extended to large-scale serving involving more than two nodes?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-07"}