Transcending Cost-Quality Tradeoff in Agent Serving via Session-Awareness
Authors: Yanyu Ren
Li Chen
Dan Li
Xizheng Wang
Zhiyuan Wu
Yukai Miao
Yu Bai
Venue: NeurIPS 2025 poster ()
Keywords: Agent Serving
LLM Serving
LLM infrastructure
Large language models
inference optimizations
Primary area: infrastructure
TL;DR: agent serving framework with high KV cache hit rate and session-aware cascading

Abstract: Large Language Model (LLM) agents are capable of task execution across various domains by autonomously interacting with environments and refining LLM responses based on feedback.
However, existing model serving systems are not optimized for the unique demands of serving agents. Compared to classic model serving, agent serving has different characteristics:
predictable request pattern, increasing quality requirement, and unique prompt formatting. We identify a key problem for agent serving: LLM serving systems lack session-awareness. They neither perform effective KV cache management nor precisely select the cheapest yet competent model in each round.
This leads to a cost-quality tradeoff, and we identify an opportunity to surpass it in an agent serving system.

To this end, we introduce AgServe for AGile AGent SERVing.
AgServe features a session-aware server that boosts KV cache reuse via Estimated-Time-of-Arrival-based eviction and in-place positional embedding calibration, a quality-aware client that performs session-aware model cascading through real-time quality assessment, and a dynamic resource scheduler that maximizes GPU utilization. 
With AgServe, we allow agents to select and upgrade models during the session lifetime, and to achieve similar quality at much lower costs, effectively transcending the tradeoff. Extensive experiments on real testbeds demonstrate that AgServe (1) achieves comparable response quality to GPT-4o at a 16.5\% cost. (2) delivers 1.8$\times$ improvement in quality relative to the tradeoff curve.

Transcending Cost-Quality Tradeoff in Agent Serving via Session-Awareness
Yanyu Ren'
Li Chen?
Dan Li
Xizheng Wang'
Zhiyuan Wu
Yukai Miao
Yu Bai
TTinghua University
ZZhongguancun Laboratory
wu-zy25)@mailss .ammailsstsinhua.... edu. en tolidan@tsinghua. edu. {lichen, miaoyk, baiyu) edu.
{ryy23, wang-xz22,
Abstract
Large Language Model (LLM) agents are capable of task execution across vari- ous domains by autonomously interacting with environments and refining LLM responses based on feedback. However, existing model serving systems are not optimized for the unique demands of serving agents. Compared to classic model serving, agent serving has different characteristics: predictable request pattern, increasing quality requirement, and unique prompt formatting. We identify key problem for agent serving: LLM serving systems lack session-awareness. They neither perform effective KV cache management yet competent model in each round. This leads to a cost-quality tradeoff, and we identify an opportunity to surpass it in agent serving system.
precisely select the cheapest
To this end, we introduce AGSERVE for AGile AGent SERVing. AGSERVE features session- server that boosts KV cache reuse via Estimated- Time- of-Arrival-basee eviction and in- place positional embedding calibration, a quality- aware client that performs session- model cascading through real-time quality assessment, and a dynamic resource scheduler that maximizes GPU utilization. With AGSERVE, we allow agents to select and upgrade models during the session lifetime, and to achieve similar quality at much lower costs, effectively transcending the tradeoff. Extensive experiments real testbeds demonstrate that AGSERVE (1) achieves comparable response quality to GPT-4o at 16.5% cost. (2) delivers 1.8x improvement in quality relative the tradeoff curve.
Introduction
Large Language Model (LLM) agents are revolutionizing task execution by incorporating agentic workflows and refining responses through observational feedback from their surroundings [47 27
Gaining tremendous attention from both academia and industry, the application scenarios of L.LM--
based agents range from query agents that automatically extract information from large databases (39
to embodied agents that carry out household activities following user commands [45] 660.
Despite the rapid development of LLM agents, the serving systems powering them largely dependent on existing LLM serving paradigms, which are not optimized for agent workflows. In typical LLM serving process, the model takes in human instructions and performs complex reasoning to generate responses that satisfy human requirements. Compared to classic LLM serving, our analysis shows that agent serving exhibits distinct characteristics:
Predictable and High Frequency Interactions. Agents utilize lightweight plug-ins generate observations and initiate new requests at intervals of milliseconds [31] 29] after obtaining the response. Additionally, the responses to agent requests usually short (within 100 tokens). The generation latency is also short and relatively constant.
"Corresponding Author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

<Shared Prefix (SP)) You are household agent. Here example
User: You household agent.Here an example. You need to put a soap in the toilet. You see around you.
Non- Truncatable
<Task-Speoific Template (TST)>
( SP & TST
Truncated IH)
You need to put soap in the toslot. You around you.
User: You see nothing in cabinet
Agent: I sae nothing in cabinet 1, nead check it elsewhere. Action: go
Truncated
Thought: I need to put a soap in toilet, I need to find first. Action: cabinet 1
Agent
cabinct 2.
LLM
cabinet 2. Agent: I have already found the snap.
User:
Truncatable
You see nothing in cabinet
Action: pick
User: You have take the snap. What your next step?
(a) Session History Pattern
(b) Prompt after Truncation
Figure 1: A prompt example in agent serving. LLM interacts with the agent in multiple rounds. The shared prefix (SP) and task-specific template (TST) are non-truncatable. The interaction history (IH)
is truncatable. (b) shows the actual prompt after the first entry in IH is middle-truncated.
Growing Contexts and Increasing Difficulties. Agents engage in multi-round sessions, and their contexts grow rapidly [56] [55] by the number of rounds. The task difficulty for LLMs also increases with longer context.
Unique Prompt Formatting. Agents format prompts in unique patterns, such middle truncation instead of prefix truncation, as shown in [
Existing systems perform suboptimally in agent serving. Single-mode model approaches 66 18
38 tend to overkill tasks in the early stages and underperform in the later ones. Routing- methods (37 13 and model eascading systems 7 suffer from inaccurate model selection and frequent model migrations, respectively. Consequently, they inherit the inherent cost-quality tradeoff' from conventional LLM serving: achieving higher-quality responses requires interaction with larger models, which leads to increased costs esspecillyy when using commercial APIs.
Looking into agent serving characteristics, we identify an opportunity to transcend the cost-quality tradeoff. We discover that the session predictability can be exploited to reduce the inference latency, and that the model of different sizes can be applied in different segments of the session lifetime to reduce costs. Thus, we conclude that to transcend the cost-quality tradeoff, we can leverage session-awars KV cache reuse and eviction policies together with session-awar model cascading.
Therefore, we are motivated to design an agent serving system to improve agent serving efficiency and quality. However, realizing this vision requires addressing three key challenges:
C1 Session-Aware Cache Policy. Modern serving systems employ reuse and eviction policies to reduce the recomputation overhead across requests. However, existing serving systems enforce strict prefix- metti-naa for reuse [19] and adopt simple deast-Recently-
(LRU) eviction policies 24 62). Neither policy fits well in agent serving, which leads low KV cache hit rates and undermines efficiency in heavy agent-serving scenarios.
C2 Quality Assessment. Existing systems overlook session lifetime and lack mechanisms to dynamically assess model response quality under long-context conditions. As a result, they often overcommit large models too early, incurring extra costs, or underperform in later stages due to poor model choices. An effective agent- serving system must not only select good initial model to reduce early-stage migration, but also assess response quality in real time to inform upgrades.
C3 Resource Allocation. The model switching imposes dynamic and imbalanced resource demands across models. Static GPU allocation wastes resources, while existing systems do not adaptively provision GPUs based on workload shifts. Thus, the serving system must support fine ane-grained. dynamic resource allocation ensure high overall GPU utilization.
To address these challenges, we design AGSERVE, a session- aware cascading agent serving system. Specifically, to address C1, AGSERVE introduces in-place positional embedding calibration to enable high KV cache reuse under middle-truned truncation. It further improves cache efficiency via the novel cache eviction policy based on the estimated time of arrival (ETA). To address C2, AGSeRvE integrates a Quality Maintenance Module (QMM) to assess task difficulty and response quality in real time, allowing informed model upgrades throughout session lifetime. To address C3, SGREE implements a resource scheduler that dynamically allocates GPU resources based supply-demand statisties. AGSERVE also exposes set of APIs to facilitate flexible integration by agent developers. By jointly optimizing KV cache reuse, model selection, and resource allocation, AGSERVE transcends

---

the traditional cost- cust-quality tradeoff in agent serving. It achieves lower latency and reduced serving cost while maintaining comparable response quality.
We implement and depoyy AGSERVE, on real testbeds, and evaluate its performance using Agent- Bench [29]. Results show that, with similar quality to GPT-4o, AGSERVE reduces agent serving cost by 83.5%. At the same cost, AGSERVE can achieve 1.8x higher quality relative to the cost-quality tradeoff curve. For multi-agent workloads, AGSERVE reduces cost by 64% and achieves 1.6x quality compared to existing LLM serving systems. AGSERVE also raises KV cache hit rate by 2.84X and reduces the round duration by up to 50% with session-aware KV cache management. It also accelerates agent serving by 1.2x with the dynamic allocation strategy.
Our work is available at httttt:://gttheii
com/robin www.rroiiiennono en00 wwwpoiinnnenn//aeeeeeeeeee agserve
2
Understanding Agent Serving
2.1 LLM Agent Preliminary
LLM agents often interact with the model using a structured prompt consisting of a shared prefix, a task-specific template, and references to available tools, allowing them iteratively carry out the task. The shared prefix explains tools and gives few-shot examples (5]. while the task template gives the specific task. In each round, the agent requests LLM's chain of thought (CoT) [54] response, performs the action in it, and appends observations. Due to context length limits, LLM agents may truncate the interaction history but preserve the SP and TST maintain correct behavior.
2.2 LLM Serving Systems are Insufficient for Agent Serving
To better understand the motivation behind the design of AGSERVE, we first examine the key differences between classic LLM serving and agent serving.
We exhibit their workload differences in Fig 2 Mainstream LLM serving systems provide inference service of a certain model [24] 21.. They schedule and run the sequences in each node with no quality concerns. To our knowledge, there is currently no system specifically designed for agent serving. We argue that agent serving systems should go beyond conventional LLM serving systems since their task is to provide quality-guaranterd actions. They incorporate an additional transparent layer to choose an LLM from a pool, and then also perform request routing and scheduling.
Their behavioral differences lead to poor performance of existing systems.
B1
Agent serving exhibits frequent and regular request patterns in each round. As shown in Fig 3 a),
the token number grows like a staircase across rounds. The arrival time for the next request under the same session is predictable [50]. Compared with human-oriented LLM serving 44 Fig bb) reveals that agent requests have shorter output lengths and shorter request intervals. These imply that agent serving constitutes a heavy-prefill, light-decode workload. Existing prefill-decode disaggregation systems such as MoonCake 41 and DistServe (63] are optimized for decoodeintenssiv tasks, which brings little benefit for agent serving [21].
B2 Agent sessions continuously grow by appending new responses to IH, while enforcing middle truncation if needed. Existing systems employ prefix-based cache reuse [19], which assumes strictly matched prefixes. Middle truncation breaks the prefix pattern and renders techniques like Cached RiteedAtttntion [17] ineffective. In parallel, existing systems such as vLLM 24] and SGLang (62) apply LRU policy to manage cache eviction. LRU is unaware of session semantics and may evict session caches that are about to request, further incurring a low cache hit rate.
B3 As agent sessions progress, the task complexity increases with a growing demand for larger models. F captures this trend with dropping satisfactory rate one model. Smaller models tend to generate invalid actions or violate preset rules as shown in appendix SB Existing rout- ing (37] and cascading [ systems overlook this trend by treating each request independently, leading to fragile model choices and costly migrations.
3 AGSERVE Overview
AgServe Architecture. As presented in Fig 6 AGSERVE comprises three pieces: server, client, and scheduler. We describe the roles and submodules of each of these components as follows,
The Session-Aware Server (SAS, $ manages session caches and performs LLM inference. SAS maintains a Session ID-Sequence Table (SIST), which maps session IDs to their latest sequences. When a new request from an existing session arrives, SAS extracts its session cache from memory and performs in-place positional embedding calibration if needed. SAS maintains the ETA-based cache eviction (ECE) policy in case of GPU memory overloads.
3

---

Agent Serving
2540
Human
Sequence
2000
Action
Scheduling
2320-
History, Observation
Serving
3000
130
256
0004
Sequence Scheduling
(aJAgene Time
73 25.
Intervalls)
[b] Humanjagent serving pattem
Figure 3: Comparison of LLM and agent serving behaviors. PTIOT represent the prompt/output token numbers, tested on our A6000 testbed.
Agent
LLMs
Servers
Figure 2: LLM and Agent Serving workflows.
Satisfactory Rate
SAS Session Cache Management Module
SID:1
Block Table
Token
8B
Embedding
[9]
[3x"]
70B
[0]
[6.50-]
3 toks
Round
Rlttured Cache Space
Session Input
Figure 4: Agent serving satisfac-
tory rate drops round increases.
Figure 5: SAS session cache management module.
The Session Guard Client (SGC, $5) provides observations for the agent and ensures the serving quality throughout the session. The client monitors the lifetime of the session and determines the LLM adoption. To achieve this, SGC employs Q-Judge to select the most cost-efficient model for the session. also rums a daemon R-Judge to periodically check response qualities, deciding mitigation actions if quality issues arise.
The Resource Scheduler (RS, $6) assigns LLMs onto GPUs and dynamically adjusts the configurations based on the real-time demand-supply ratio across the system.
AGSERVE Workflow. When a new agent starts, the SGC assesses the task's 's difficulty with Q-Judge. SGC establishes a session with an SAS instance or external API that provides service for given model. Within the session, the agent then interacts with the LLM for multiple rounds.
Throughout the session, SGC monitors the reasoning quality using the R-Judge. If performance falls below pre-designated threshold, the SGC initiates retry requests migration to more powerful model instance. In the case of a migration, SGC terminates the current session and builds new session with SAS or API of more powerful model. Furthermore, the RS looks at the requests that each model handles and dynamically adjusts the GPU resource distribution across models.
4 Sessionizing KV Cache to Maximize Cache Hit in SAS
AGSERVE adopts Attentioo following the practices of vLLM and SGLang. We incorporate SIST into the block table as shown in Fig S SIST introduces session awareness to the KV cache by maintaining each session's 's block ID. Our two following innovations take advantage of SIST.
In-place KV Cache Calibration. Modern LLMs adopt Rotary Positional Embedding (RoPE, (46) to embed tokens. RachedAAttenton [I7] proposes decoupled KV cache. However, the decoupled cache consumes twice the space, one for the decoupled KV cache to preserve and one for the normal cache to decode. Instead, AGSERVE adopts in-place calibration to facilitate middle truncation. The in-place calibration multiplies the original key cache by et6p where 8p = p - is the opposite
AGSERVE
Session Guard Client
Session Aware Server
Ericton
Scheduler
Manager
LLM Executor
B-Audge Ancha Bust
place
Operator
Dynamic Modal
LUM Agents
O-Jedge Tash Or
Quality Qualty-Aosured Ansured Sendion Dancading
Marnsher
Session Cache Pool
Figure 6: AGSERVE Architecture and Workflow.

---

Time
0:03
0:05
0:06
0:07
Unexpected
0:08
0:09
Request SID
3
2
0
Behavior
SID
LRU time
SID
LRU time
SID
LRU time
SID
LRU time
SID
LRU time
SID
LRU time
SID
LRU time
LRU
00:03
3
00:04
0
00:05
00:06
2
00:07
00:08
3
00:09
(YLLM, SGLang..)
Cache Hit.1/7
00:02
00:03
00:04
00:05
00:06
00:07
00:08
00:03
00:05
00:05
00:07
4x
Miliss...
Miss...
Miss..
Miss...
Hit!
Miss...
Miss..
Improvemients
SID
ETA time
SID
ETA time
SID
ETA cime
SID
ETA time
S10
ETA time
SID
ETA time
SID
ETA time
ECE
00:05
0
06:05
00:06
00:08
00:08
00:07
2
00:10
(ours)
Cache Hit-4/7
00:06
00:08
00:09
00:09
00:20
00:08
00:09
00:11
00:13
00:13
Miss..
Hit!
* Hit!
Miss..
Hit!
Hit!
Mili....
Figure 7: An illustration of how ECE outperforms LRU. LRU policy decides the evicted session cache based the LRU time, while ECE is based on the ETA of the next request from the session. In this simplified demonstration, there are four sessions ongoing with no shared blocks. The SAS can only preserve three in the session cache. All sessions share the same arrival pattem, and thus, their request arrives at SAS in a round-robin fashion. LRU policy, adopted by vL..M and SGL ang. evicts the cache of sessions to arrive, causing a low cache hit rate. On the other hand, ECE achieves session cache hit rate of 75% in theory. We also consider an occasional unexpected behavior at 0:07, causing the reversed order of 0 and 3. ECE still performs robustly with consistent hit rate. Notably, the ECE policy does not change the ETA for occasional flapping. Taking all into consideration, ECE achieves 4x hit rate compared to LRU under this case. In more complicated case, ETA of different sessions from the processing time may differ, and sessions may share blocks.
number of truncated tokens. This operation calibrates the position embedding from etp9 to e In practice, SAS performs
Kr.2i = cos( 5,0, + sin(-5,0,) K,
Ky p',2i+ cos(- p. - sin(-6,,0)) Kn
to reuse the cosine-sine cache of the RoPE module, further reducing the computation time. ECE Eviction Policy. SAS needs evict KV caches due limited memory in online serving. To formulate the question, we assume that there are n sessions ranked by the arrival time of their pending requests sCO, SC1 .............................. SCn- SAS predicts the arrival time if the next request has not arrived yet. The cache size of session sc; is 4. We take two steps to review SAS's ECE eviction policy.
First, we assume that all session caches are of the same size. Inspired by the OPT strategy 40 in
CPU cache, we remove SC; in the order of n to 1. We prove its optimality in appendix $A.2
However, session cache sizes are different, meaning that evicting one session may not spare enough space for another session, and the above solution cannot achieve the global optimal. To tackle this, we introduce new symbol T(k) as the optimal total TTFT for only considering scn-k to 80n-
To spare enough space for the first k pending sessions, we need to decide on a set F. of session caches to evict. The eviction of sc; costs P. I(sG.) to recompute and affects the n i i 1 sessions behind, leading
(scc)) Thus, (n)) EieF, (n i 1)P 4 + (n D)Ds + T(n k). where Dz is the average decoding time consumption with requests handling in parallel. P is the prefilling overhead of one token relative to the average decoding time cost, considered as a penalty for evicting the session cache. For each k; SAS finds the Eg to minimize (n(n) with dynamic programming. and overlooks T(n k) to avoid excessive eviction. Finally, SAS finds the minimum Te (n) among all ks and evicts E Each SAS predicts the ETA upon the request trace of sessions and makes the eviction decision based on the ETA. We discuss the detailed algorithm and efficiency for ECE policy in appendix $D
penalty of (n
SAS does not swap caches to CPU, since such swapping introduces high overhead with long contexts and blocks other sequences in the batch. SAS schedules each sequence in a First-Come- First-Served manner instead of SGLang's 's radix approach to avoid starvation.
5 Exploiting Session-Level Cascading to Minimize Cost in SGC
SGC leverages the Q-Judge to assess task difficulty and establish sessions for agents with capable models. The Q-Judge (1(),)) takes the agent task I,, and the observation p as input. QJ(-) assigns a difficulty label among 0, 1, and 2 for each task. SGC leverages the R-Judge for quality monitoring.
5

---

5.1 Reducing Overkill and Underkill of the Task
Q-Judge inevitably faces deviation from the ground truth, causing underkill or overkill. In the case of underkill, SGC needs to migrate several times to reach the capable model, wasting SAS resources. If overkilled, SGC faces an irreversible cost. We observe that SGC prefers an underkill rather than an overkill. We optimize the loss function for Q-(-) as follows, with p. q as the ground truth and predicted probabilities, respectively, and g as the ground truth label.
[(, (.9 = Σ ((s))oo((((x))) + B. Σ [1s 9)² 1=1
O22
1- ((2))
in which 0;; 1 if xi > 9 (overkill) and az = < 1 if 9 (underkill), my is the value of the i-th label. The first element of the function is the traditional cross-entropy loss. The second element gives extra loss to the wrong labels if it is far from larger than the ground truth. We train the judge on a customized Chatbot-Aren dataset [8] Details are disclosed in appendix C
5.2 Rime Response Quality Monitoring and Issue Mitigation for Long Context
We observe four typical types of quality issues in agent serving, namely service failure, violation of preset rules, invalid actions, and low reasoning quality. The first three are easy to identify by algorithmic approaches. For the fourth issue, SGC calls the R-Judge classifier to evaluate the reasoning quality at a user-customized frequency of v.
Designing and Training R-Judge. The prompt length easily exceeds the context window of modern classifiers with over a thousand tokens. Thus, we only input selected elements from the context to reduce the workload. R-Judge (m e, takes in four parameters. m indicates the model size. It provides the R-Judge with a bias of the serving quality against the model size. r is the thought part of the CoT response. If the implementation does not adopt CoT, R-Judge takes the whole response as r. is the task of the agent. e is the latest actions and observations when AGSERVE sends the request. R.)(.) judges based on how well can help assist the task of in the condition of e, covering cases of evasive and redundant replies. 0 reflects the strictness of the judge. 0 = 0 is the strictest, which always considers the response as disqualified, and 1 allows all responses pass the test. Users can adjust this threshold based on their own quality requirements or workload characteristics.
We disclose the training details of R-Judge in appendix $C
Addressing Quality Issues. SGC adopts retry or migrate to mitigate quality issues. The retry strategy reruns the response generation, taking advantage of the randomness. However, if the response still faces quality issues, SGC forces service-upgrade migration. Migration routes the session to another serving instance. SGC implements two types of migration. The service-upgrade migration routes the session to larger LLM instance or API. The general migration decides the model option, with the possibility of a smaller model in case of service failures, such as timeouts or network issues.
SGC also supports restoration for certain agents. Restoration reverts the session to the last checkpoint, shortening the chat history to reduce reasoning complexity and improve performance.
6 Model Allocation across GPUs to Maximize Hardware Utility in RS
Efficient model allocation is crucial for optimizing resource usage in agent employs a dynamic allocation strategy to ensure high resource utilization.
agent serving systems. AGSERVE
Model Instances. We divide the models into two categories adjustables and presets. Adjustables fit in a single node, while presets must span across multiple nodes. RS confines adjustable instances to single node to minimize communication latency. For each model m, RS profiles the minimum number of GPUs required for inference, denoted as Wm- Each instance utilizes a multiple of w. GPUs. RS relies on manual operation for presets, since they have heavy initialization overhead. Dynamic Allocation. RS dynamically allocates models based on model demand and resource availability. RS tracks the frequency of inference requests across all instances for each model and calculates its demand by summing up the calling frequency. The scheduler determines the supply by the available tokens to preserve in the KV cache of all instances.
RS prioritizes models with high demand-te to--upply ratio, scaling up their instances while reducing instances for models with lower ratios. RS redistributes model instances across available nodes to optimize resource usage. RS consolidates instances of the same model wherever possible to leverage parallelism. For example, RS starts three instances, utilizing 1, 2, and 4 GPUs each 7 GPUs of the same device are allocated for model with wm = 1.

---

AGSERVE
Uame-ds
Usere yoa
G"I-4o
Gescatle
RouteLLK
Qualtyy
75
50
0.30
0.05
0.10
0.000
0.005
0,010
0.0155
(3)0
0.02
0.04
0.56
0.00
0.01
0.02
(a] AW
(b) GG
Averege
Iel KG
(d) M2W
Figure 8: The red line shows the cost-quality tradeoff in each agent. AGSERVE breaks the cost- quality tradeoff in all four agents, achieving comparable quality to GPT-4o at lower costs.
Evaluation
We implement AGSERVE in Python with SAS based on vLLM 24 We also implement customized CUDA kernels to support batched in-place KV cache calibration. We adapt four agents (AW.CG.KG.22W) with AGSERVE's API. We disclose their details in appendix $E.T We evaluate AGSERVE with these four agents and models of the 8B, 30B versions of Llama-3 and GPT-4o, due to generality of Ilama structure and our testhed capability(see appendix SD. Without explicit benefit from the model selection, our key findings are
With similar quality to GPT-4o, AGSERVE reduces serving cost by 83.5%.
At the same cost, AGSERVE achieves 1.8x quality relative to the cost-quality tradeoff curve.
Under multi-agent workloads, AGSERVE reduces cost by 64% and achieves 1.6x quality com-
pared to an adapted sfate-of-the (SoTA) LLM serving system.
AGSERVE achieves 2.86x cache hit rate with ECE policy than LRU, and reduces the round latency by up to 50% with in-place positional embedding calibration combined. AGSERVE accelerates agent serving by 1.2x with dynamic hardware resource allocation.
7.1 Testbed and Metrics
We evaluate the performance of AGSERVE based on AgentBench 29 over two testbeds. The first consists of two nodes, each equipped with four A6000 GPUs (48GB per GPU). The second comprises two nodes, each equipped with eight A800 GPUs (80GB per GPU), interconnected via PCIe.
Our evaluation focuses on three metrics of the sessions: end-to-end (e2e) latency, quality, and cost. Sessions that terminate due invalid actions other quality issues before task completion reaching the round limit are excluded from latency measurements.
We rank each agent serving system with quality score out of 100 concerning three aspects to follow the ranking mechanism of AgentBench,
25% for behaving normally until the task completed or the interaction rounds limit reached.
50% for the wellness of handling the task.
25% for completing the task in the fewest rounds possible, to benefit competitive model selection.
We disclose the details of the quality metric of each agent in appendix E
The cost includes two parts: the cost of nodes decided by the e2e latency of each agent task, and
the cost of OpenAl APIs. For open-source models, we compute the cost based on the testbed capacity,
selected strategy, and e2e latency. We provide details of model costs in appendix (E.3
To exhibit AGSERVE's performance, we enlist a VLLM-adapted cache-centric strategy, and a model routing strategy, namely RouteLLLM
33), as baselines for each experiment.
7.2 End-to-end Cascading Serving
For this evaluation, we disable dynamic allocation and run two instances the A6000 testbed, one running Llama-8B and the other running Llama-70B.
Baselines. We compare the cascading strategy of AGSERVE to the following strategies.
vLLM+. We adopt a single model throughout the serving process, like LLM serving systems. We pick one model among Llama-8B: Llama-70B. 70B, or GPT-4o each time, the same model choices in AGSERVE. We adopt prefix caching and a retry strategy to facilitate the approach.
Cascade. Cascade adopts three-layer cascading, but disables the R-Judge and Q-Judge. It always starts at the smallest model and only addresses explicit quality issues.
Ro. RouteLLM trains four routers on the same Arena dataset as AGSERVE. Since matrix factorization and similarity-weighte ranking route almost all requests the larger model, we adopt the BERT router with a preference for the smaller model at 0.8.
Evaluation Results. We exhibit the average cost-average quality curves of four agents in Fig We utilize Piccewise Cubic Hermite Interpolating Polynomial 3 to draw the tradeoff curve. As
7

---

PouteLLM
AGSERVE
Luumiix
296.0
257.1
72
79.2%6.6
22
64.1
7r1
135
15c
11
10
38.8
89
78.31
77.273.0
50.0
75.5
31.0
Latencyls)
(1st)011)
Quality
Latency(s)
Cost.o05!
Quality
Latencyls)
Quality (c) Gamma
015)
Latency(s) Costi
Quality
(a) Uniform
(b) Burst
(d) Poisson
Figure 9: AGSERVE'S performance in different distributions.
Table 1: Performance of routing models. S/M/L.
Table 2: Latency comparison between dynamic
represents small/medium/large labels.
and static allocation strategies. Average, P90, and P50 exhibit the average, 90th percentile, and median of the e2e latency, respectively.
M
Acc.
Low
High
GT
112
306
173
Average(s)
P90(s)
P50(s)
QJ
190
397
70.4% %
109
RI
374
N/A
185
57.9%
161
88
AGSERVE FSI
119.38
177.05
106.81
125.43
206 86
106.85
591
0
62.4%
182
FS2
242.35
295.86
237.93
displayed, AGSERVE breaks the cost-quality tradeoff in all four agents, achieving similar quality to GPT-4o with a 16.5% cost in AW. For the same cost, AGSERVE gets a 1.8x quality than the tradeoff curve. In comparison, without R-Judge, the Cascade strategy only significantly transcends the tradeoff in one of the four agents. On the other hand, RouteLLM performs below the cost-quality tradeoff in all four agents. We believe that this is largely due to two reasons. First, RouteLLM is designed for short inputs such as MMLU 200 and GSM8K (9), not for long-input agent serving. Second, RouteLLM sensitive the output of the routing model due a lack of
7.3 End-to-end Multi-Agent Serving
In this section, we evaluate the multi-agent performance of AGSERVE in distributed service scenario. We run the evaluation on our A800 testbed with TP size constrained 4 or below. Baselines. We compare AGSERVE with two baselines.
Llumnix++ [48). Llumnix is the SoTA LLM serving system to manage request allocation backed by vLL.M We extend Llumnix with support for Llama-3 and integrate Llumnix with AGSERVE'S QMM. Llumnix+ runs four 4-GPU instances, two each for Llama-8B and Llama-70B, since Llumnix requires each node
the same model.
RouteLLM. We adopt the same route setting in 7.2
Datasets. To our best knowledge, there is currently no real dataset on the traces of agent launching. We generate the traces of uniform and burst distributions. Similar LLM serving [6] 28 48). we also use Poisson and Gamma distributions to synthesize request traces. The trace only defines the launch time of each agent and does not affect any following interaction rounds the session.
Evaluation Results. As shown in Fig 9 Llumnix++ + has higher latency and lower quality in the high- frequency agent serving scenario. AGSERVE reduces cost by 64% and, in the meantime, achieves 1.6x quality compared to Llumnix++. RouteLLM routes most of the requests to GPT-4o and utilizes large and cheap computing resources to achieve low latency, and has high latency in bursts due to the rate limit. With much more expensive hardware rent, AGSERVE achieves comparable quality with only 81% RouteLLM costs, and can reach 40% cost with wholesale-price cloud service.
7.4
Ablation Studies and Microbenmarks
7.4.1
Performance of SAS Session Cache
We evaluate the performance of session caching on agent serving. We deploy a Llama-8B instance on one A6000 node. The temperature is set to 1, following the OpenAl API default.
Correctness. We compare AGSERVE against vLLM, and exhibit the latency and quality of four agents in Fig TO Latency is shown per request for CG due to its nondeterministic response and varying rounds. AGSERVE achieves up to 2x acceleration as AW. The quality scores between AGSERVE and vLLM show negligible discrepancy with no correctness concerns.
Efficiency. We evaluate the hit rate and TTFT for AW agents on Llama-8B SAS with limited memory and context window size. Fig exhibits 2.86x hit rate in the batch size of 8 and 2.80 2.14x,

---

ACSERVE
KLLM
50
100
15
4.-
10-
30-
1.0
Quality
Quality
(a) AW
(b)
CG
(c) KG
(d) M2W
Figure 10: Latency (in seconds) and quality of vLLM and AGSERVE SAS across different agents.
12 St
Dencity
ASSERVE
Number of Cazes
400
Accurecy: 75.80%
Correct
Whrong
TTFT(s)
QMM on QMM
200
6.1
Unsatislctorry
Satisfcctory
Rousing
Interaction
Figure 11: Cache hit Figure 12: KDE plot of Figure 13: R-Judge out- Figure 14: TTFT with
rate by block.
TTFT.
put against ground truth. and without QMM.
1.73x in the batch sizes of 7, 6, 5, respectively. Fig 12 shows the Kernel Density Estimation (KDE) plot of the TTFT. Benefiting from the high cache reuse by ECE eviction (peak around 0.06s) and in-place calibration (peak around 0.08s), TTFT of AGSERVE is significantly to the right of vLLMM, showing less prefilling latency.
7.4.2 Performance of SGC QMM
We evaluate QMM to ensure proper quality maintenance with low overhead.
Correctness of R-Judge. To knowledge, there is existing human preference data for agent serving. We ask experts to label the responses of different-sized models, spanning across all four agents. We depict the performance of R-Judge in Fig 13 with 95.2% recall of unsatisfiable results. We exhibit the false positive and false negative rate of the R-Judge when picking different 0 in Table 3 Correctness of Q-Judge. It is hard to label the difficulty of prompts. We utilize the dataset above and build loose restriction on the label given. For example, if the medium model response is satisfiable, is considered right label (RRL) and model trained by the cross-entropy (os)(CE) in Table Q-Judge achieves high accuracy and few predictions that are higher than the ground truth, as we require in $5.1
both small and medium. We compare Q-Judge((J) with
Efficiency. In real-world applications, AGSERVE makes the most use of the QMM's context windows by filling them up with session history. Fig 14 shows the influence of QMM on TTFT in the real application, when the average context length reaches 500 of the 512 window size. Still, QMM causes negligible overhead in normal interactions and an acceptable overhead when routing. We list their respective overhead in appendix $E.7.2
7.4.3
Performance of RS Dynamic Allocation
RS only conducts reallocation for a significant imbalance in the demand-supply ratio across models avoid excessive weight loading. We manipulate of the R-Judge to create such scene. We evaluate the performance of AGSERVE on one A800 node, against two fixed allocation configurations. FS1 allocates four GPUs to Llama- 8B and Llama-70B each; FS2 runs an 8-GPU Llama-70B instance. We disregard sessions that violate preset rules, accounting for roughly 6% of all sessions across FS1, FS2, and AGSERVE. We reveal the latency of all other sessions in Table AGSERVE reduces 14% P90 latency compared FSI. Compared FS2, an L.MM serving approach, AGSERVE achieves 49% average e2e latency. We provide case study in appendix $.7.3
Table 3:
False positive and false negative rates of the R-Judge under different
1
2
4
5
6
False Positive Rate
0.00%
28.09%
40.06%
41.22%
47.93%
57.55%
False Negative Rate
33.64%
17.60%
8,98%
3.94%
0.91%
0.00%

---

8 Limitations and Discussions
R-Judge Sensitivity. AGSERVE relies on R-Judge to perform cascading. When 0 = 0, AGSERVE falls to GPT-4o, and f = 1 degrades AGSERVE to Cascade as in $7.2 We show R-Jugge's 's comparable performance to humans in appendix $E.8 the possible cost of early migration. We acknowledge that a judge trained with more sufficient data may benefit agent serving better.
Node Routing. So'A L.LM serving frameworks, such SGLang 62] and MoonCake [41], decide the machine routing of each request by prefix matching. Due to the limited testhed size, AGSERVE binds each session to specific instance (i.e., the most idle one at first-round scheduling). We recognize that there are two future directions to uncover: (1) the migration of in-flight sessions to balance load; (2) to exploit the prompt length predictability as in Fig 4a) match the instance. Large Scale Agent Serving. Previous works [48] 42 suggest that a distributed scheduling out- performs in large-scale environments. We present theoretical simulation in appendix $Taabtut distributed AGSERVE for large-scale serving and show AGSERVE'S potential application. Broader Impacts. The technologies used in AGSERVE make agent serving cheaper and more efficient, and do not bring extra societal impact beyond that of existing agents and LLMs. We rely on external safeguard measures such as red-teaming to minimize negative impacts.
Related Work
Cache-centric LLM Serving, Etrly-stage works such as SpotServe 33 and AlpaServe (28 mainly focus on parallelism strategics. Orca [61] proposes continuous batching and makes decoding memory- bounded. vLLM 24 improves KV cache management with PagedAttention. Further, Llumnix 48 explores multi-instance LLM serving with virtual memory and request migration. Tetrilnfer 21 proposes the idea of scheduling sequences according length prediction, which is similar to the AGSERVE'S session routing strategy. While TetriInfer makes scheduling decisions by the memory usage at present or shortly afterwards, AGSERVE looks into resource usage function over time. Splitwise 38 and DistServe [63] disaggregate the prefill and decode in LLM inference. However, agent serving, a light-decode scenario, does not work well with disaggregation and its tools such as CacheGen (30). as we previously mentioned in $2.2 Quantization [16] 6 and kernel optimization
10 52 can also accelerate inference. CacheBlend 58) selectively recomputes the KV caches for Retrieval-Augmented Generation, which is orthogonal to AGSERVE.
Multiple-model serving. Tabi (53] proposes automatic cascading using confidence, followed by FrugalGPT 7 focusing on the financial sector. AGSERVE uses prompt engineering to overcome the generality problem of FrugaIGPT. HybridLLM B3 and RouteLLM 337 route requests to LLMSs
Like other traditional LLM serving systems, they consider requests from the same session indepen- dently, making the results extra sensitive to the router without quality assurance. OnlineCascade (36) proposes an approach to learn cascading strategies, which is orthogonal to AGSERVE. Speculative decoding (26) uses small models to mimic the large ones with target LLM's verification. AGSeRVE focuses the reasoning quality compared to speculative decoding rather than the responses' consis- tency with the large models and supports speculative decoding models in its hicrarchy.
LLM Agents. ReAct [59] retines LLM's response with multiple rounds of interactions, while LangChain [55 provides an interface for agent coding. Beyond the wide application of LLM
agents
33 45 29 recent works also propose multi-agent collaboration 49 32). Despite its rapid
development, we still see most LLM agents rely on a sole model [1] AGSERVE provides a set of APIs that wrap up the model cascading and quality maintenance process of agent serving.
10 Conclusion
We present AGSERVE, the first agent serving system designed to overcome the inherent cost-quality tradeoff in agent serving. AGSERVE addresses the unique demands of agent serving, including predictable request patterns, growing reasoning complexity, and dynamic resource requirements. combines session-aware KV cache management, quality-aware model cascading, and adaptive resource scheduling to support efficient, high-quality interactions across agent sessions. Experimental results show that AGSERVE significantly reduces latency and resource costs while maintaining service quality on par with state-of-the-ar LLM serving systems.
10

---

Acknowledgment
We thank anonymous reviewers for their constructive comments. Li Chen is the corresponding author. This work is supported by the National Natural Science Foundation of China under Grant U21B2022 and the Beijing Outstanding Young Scientist Program (No. JWZQ20240101008)
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
[2] Zhiqiang Shen Aidar Myrzakhan, Sondos Mahmoud Bsharat. Open-lIm-leaderboard:: From multi-choice open-style questions for Ilms evaluation, benchmark, and arena. arXiv preprint arXiv:2406.07545, 2024.
[3] Garrett Birkhoff, Martin H Schultz, and Richard S Varga. Piecewise hermite interpolation in one and two variables with applications to partial differential equations. Numer. Math, 1 11(3):232-256, 1968.
[4] Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. Freebase: a collaboratively created graph database for structuring human knowledge. In Proceedings the 2008 ACM SIGMOD International Conference on Management of Data, SIGMOD '08, page 1247-1250, New York, NY, USA, 2008. Association for Computing Machinery.
[5] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020.
[6] Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher M De Sa. Quip: 2-bit quantiza- tion of large language models with guarantees. Advances in Neural Information Processing Systems, 36, 2024.
[7] Lingjiao Chen, Matei Zaharia, and James Zou. Frugalgpt: How to use large language models while reducing cost and improving performance. arXiv preprint arXiv:2305.05 76, 2023.
[8] Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E Gonzalez, et al. Chatbot arena: An open platform for evaluating 11ms by human preference. arXiv preprint orXiv:2403.04132. 2024.
[9] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021.
[10] Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08661, 2023.
[11] Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances Neural Information Processing Systems, 153444641,,,,, 2022.
[12] Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36, 2024.
[13] Dujian Ding, Ankur Mallick, Chi Wang. Robert Sim, Subhabrata Mukherjee, Victor Ruhle, Laks VS Lakshmanan, and Ahmed Hassan Awadallah. Hybrid Hm: Cost-efficient and quality- aware query routing, arXiv preprint 244v:2404.146188 2024.

---

[14] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, and etc. herd of models, 2024.
The Ilama
[15] Orri Erling. Virtuoso, hybrid rdbms/graph column store. IEEE Data Eng. Bull., 35(1):3-8, 2012.
[16] Elias Frantar, Salch Ashkboos, Torsten Hoefler, and Dan Alistarh. Optq: Accurate quantization for generative pre-trained transformers, In The Eleventh International Conference on Learning Representations, 2022.
[17] Bin Gao, Zhuomin He, Puru Sharma, Qingxuan Kang, Djordje Jevdjic, Junbo Deng. Xingkun Yang, Zhou Yu, and Pengfei Zuo. Cost-efficient large language model serving for multi-turn conversations with cachedattention. In Proceedings of the 2024 USENIX Conference Usenix Anmual Technical Conference, USENIX ATC'24, USA, 2025. USENIX Association.
[18] Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. Model tells you what to discard: Adaptive kv cache compression for lims. arXiv preprint arXiv:2310.01801, 2023.
[19] In Gim. Guojun Chen, Seung-seob Lee, Nikhil Sarda, Anurag Khandelwal, and Lin Zhong. Prompt cache: Modular attention reuse for low-latency inference. P. Gibbons, G. Pekhimenko, and C. De Sa, editors, Proceedings of Machine Learning and Systems, volume 6, pages 325-338, 2024.
[20] Dan Hendrycks. Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, In International Conference on Learning Representations, 2021.
[21] Cunchen Hu, Heyang Huang, Liangliang Xu, Xusheng Chen, Jiang Xu, Shuang Chen, Hao Feng. Chenxi Wang, Sa Wang, Yungang Bao, Ninghui Sun, and Yizhou Shan. Inference without interference:
Disaggregate Ilm inference for mixed downstream workloads, 2024.
[22] Jacob Devlin Ming-Was Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of naacL-HLT, volume page 2, 2019.
[23] KVCache-Ar AL. Ktransformers: A flexible framework for experiencing cutting-edge Ilm inference optimizations, 2024. attps:///iithub.com//tttt wwwp///wwaccheeee www.eeeasttatttataaaaeeeoaoaaaee
[24] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pages 61 1666666,2200 2023.
[25] LangChain. Build an Agent. agents/ 2025. [Accessed 01-2025).
[26] Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding, 2023.
[27] Xinzhe Li. A survey on Ilmebased agentic workflows and IIm-profiled components. arXiv e-prints, pages arXiv-2406, 2024.
[28] Zhuohan Li, Lianmin Zheng, Yinmin Zhong, Vincent Liu, Ying Sheng, Xin Jin, Yanping Huang, Zhifeng Chen, Hao Zhang, Joseph E Gonzalez, et al. (AlpaServe): Statistical multiplexing with model parallelism for deep learning serving. In /7th USENIX Symposium on Operating Systems Design and Implementation (OSDI 23), pages 663-679, 2023.
[29] Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. Agentbench: Evaluating lmms as agents. arXiv preprint arXiv:2308.00 036888 2023.
12

---

[30] Yuhan Liu, Hanchen Li, Yihua Cheng, Siddhant Ray, Yuyang Huang, Qizheng Zhang, Kuntai Du, Jiayi Yao, Shan Lu, Ganesh Anamthanaray anna, Michael Maire, Henry Hoffmann, Ari Holtzman, and Junchen Jiang. Cachegen: Kv cache compression and streaming for fast large language model serving. Proceedings of the ACM SIGCOMM 2024 Conference, ACM SIGCOMM '24, page 38556, New York, NY, USA, 2024. Association for Computing Machinery.
[31] Zhiwei Liu, Weiran Yao, Jianguo Zhang. Liangwei Yang. Zuxin Liu, Juntao Tan, Prafulla K Choubey, Tian Lan, Jason Wu, Huan Wang, al. Agentlite: lightweight library for building and advancing task-oriented IIm agent system. arXiv preprint arXiv:2402. 15538, 2024.
[32] Zijun Liu, Yanzhe Zhang, Peng Li, Yang Liu, and Diyi Yang. Dynamic Ilm-agent net- work: An 11m- -an-ttent collaboration framework with agent team optimization. arXiv preprint arXivv2310.021177 2023.
[33] Kai Mei, Zelong Li, Shuyuan Xu, Ruosong Ye, Yingqiang Ge, and Yongfeng Zhang. Aios: Llm agent operating system, 2024.
[34] Xupeng Miao, Chunan Shi, Jiangfei Duan, Xiaoli Xi, Dahua Lin, Bin Cui, and Zhihao Jia. Spot- serve: Serving generative large language models on preemptible instances. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, ASPLOS '24, page 1112-1127, New York, NY, USA, 2024. Association for Computing Machinery.
[35] Philipp Moritz, Robert Nishihara, Stephanie Wang, Alexey Tumanov, Richard Liaw, Eric Liang, Melih Elibol, Zongheng Yang, William Paul, Michael I Jordan, et al. Ray: A distributed framework for emerging {AI} applications. In 13th USENIX symposium on operating systems design and implementation (OSDI 18), pages 561-577, 2018.
[36] Lunyiu Nie, Zhimin Ding, Erdong Hu, Christopher Jermaine, and Swarat Chaudhuri. Online cascade learning for efficient inference over streams. arXiv preprint 240.0:2402.. 2024.
[37] Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E. Gonzalez, M Waleed Kadous, and Ion Stoica. RouteLLM: Learning to route LLMs from preference data. In The Thirteenth International Conference on Learning Representations, 2025.
[38] Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, fñigo Goiri, Saced Maleki, and Ricardo Bianchini. Splitwise: Efficient generative Ilm inference using phase splitting. In 2024 ACM/IEEE 51 Annual International Symposium on Computer Architecture (ISCA), pages 118 132. IEEE, 2024.
[39] Rubén Pérez-Mercado, Antonio Balderas, Andrés Muñoz, Juan Francisco Cabrera, Manuel Palomo-Duarte, and Juan Manuel Dodero. Chatbotsql: Conversational agent support relational database query language learning. SoftwareX, 22: 201104,,,, 2023.
[40] Barton G Prieve and Robert S. Fabry. Min-an optimal variable-space page replacement algorithm. Communications of the ACM, 19(5):295-297, 1976.
[41] Ruoyu Qin, Zheming Li, Weiran He, Jialei Cui, Feng Ren, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. Mooncake: Trading more storage for less computation a KVCache-centrig architecture for serving LLM chatbot. In 23rd USENIX Conference on File and Storage Technologies (FAST 25), pages 155-170, Santa Clara, CA, February 2025. USENIX Association.
[42] Tareq Si Salem, Gabriele Castellano, Giovanni Neglia, Fabio Pianese, and Andrea Araldo. Toward inference delivery networks: Distributing machine learning with optimality guarantees. IEEE/ACM Transactions on Networking, 2023.
[43] Victor Sanh, L Debut, J Chaumond, and T Wolf. Distilbert, a distilled version of bert: Smaller, faster, cheaper and lighter. arxiv 2019. arXiv preprint arXiv:1910.01 2019.
[44] Ying Sheng, Shiyi Cao, Dacheng Li, Banghua Zhu, Zhuohan Li, Danyang Zhuo Joseph Gonzalez, and lon Stoica. Fairness in serving large language models. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pages 965-988, 2024.
13

---

[45] Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning. arXiv preprint arXiv:2010.037668, 2020.
[46] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomput. 568(C), February 2024.
[47] Malavikha Sudarshan, Sophie Shih, Estella Yee, Alina Yang, John Zou, Cathy Chen. Quan Zhou, Leon Chen, Chinmay Singhal, and George Shih. Agentic I1m workflows for generating patient-friendly medical reports, 2024.
[48] Biao Sun, Ziming Huang. Hanyu Zhao, Wencong Xiao, Xinyi Zhang, Yong Li, and Wei Lin. Llumnix: Dynamic scheduling for large language model serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pages 173-191, Santa Clara, CA, July 2024. USENIX Association.
[49] Yashar Talebirad and Amirhossein Nadiri. Multi-agent collaboration: Harnessing the power of intelligent Ilm agents. arXiv preprint (xXx::2300603314, 2023.
[50] Jiahao Wang, Jinbo Han, Xingda Wei, Sijie Shen, Dingyan Zhang, Chenguang Fang, Rong Chen, Wenyuan Yu, and Haibo Chen. Kveache cache in the wild: Characterizing and optimizing kvcache cache at large cloud provider. In 2025 USENIX Annual Technical Conference (USENIX ATC 25), pages 465 465-482,21 2025.
[51] Wei Wang, Dan Zhang, Tao Feng, Boyan Wang, and Jie Tang. Battleagentbench: A benchmark for evaluating cooperation and competition capabilities of language models in multi-agent systems. arXiv preprint 2rXi / 15971, 2024.
[52] Xiaohui Wang, Ying Xiong, Yang Wei, Mingxuan Wang, and Lei Li. Lightseq: A high performance inference library for transformers. arXiv preprint viXiv:2010.33881 2020.
[53] Yiding Wang. Kai Chen, Haisheng Tan, and Kun Guo. Tabi: An efficient multi-level inference system for large language models. In Proceedings of the Eighteenth European Conference on Computer Systems, pages 233 248, 2023.
[54] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoe Le, and Denny Zhou. Chain-of offthhuggttt prompting elicits reasoning large language models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2022. Curran Associates Inc.
[55] Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li. Erkang Zhu, Li Jiang, Xiaoyun Zhang. Shaokun Zhang, Jiale Liu, Ahmed Hassan Awadallah, Ryen W White, Doug Burger, and Chi Wang. Autogen: Enabling next-gen IIm applications via multi-agent conversation, 2023.
[56] Ruixuan Xiao, Wentao Ma, Ke Wang, Yuchuan Wu, Junbo Zhao, Haobo Wang, Fei Huang, and Yongbin Li. Flowbench: Revisiting and benchmarking worktlow-guided planning for llm-based agents. arXiv preprint arXiv:2406. 14884, 2024.
[57] An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li. Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint 1.Xi::2407.10671 2024.
[58] Jayy Yao, Hanchen Li, Yuhan Liu, Siddhant Ray, Yihua Cheng, Qizheng Zhang, Kuntai Du, Shan Lu, and Junchen Jiang. Cacheblend: Fast large language model serving for rag with cached knowledge fusion. In Proceedings of the Twentieth European Conference on Computer Systems, EuroSys '25, page 94-109, New York, NY, USA, 2025. Association for Computing Machinery.
[59] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:22 10.00629, 2022.
14

---

[60] Yining Ye, Xin Cong, Shizuo Tian, Jiannan Cao. Hao Wang, Yujia Qin, Yaxi Lu, Heyang Yu, Huadong Wang, Yankai Lin, al. Proagent: From robotic process automation to agentic process automation. arXiv preprint 1rXiv:2311.
[61] Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung- Gon Chun. Orca: A distributed serving system for (Transformer- B generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pages 521-538, 2022.
[62] Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao. Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. Sglang: Efficient execution of structured language model programs. In A. Globerson, L.. Mackey, Belgrave, A. Fan, U. Paquet, Tomezak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 62557-62583, Curran Associates, Inc., 2024.
[63] Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. Distserve: disaggregating prefill and decoding for candput-optimized large language model serving. In Proceedings of the 18th USENIX Conference on Operating Systems Design and Implementation, OS,I'24, USA, 2024. USENIX Association.
NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitations of our work in S
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
15

---

The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't It acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof's
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification:
We provide proofs in appendix A
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
We provide the source code of our paper in supplement materials.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
16

---

While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g.. large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification:
We provide reproducible code of our paper in supplement materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
public/guides, (osoli/cuuldas//ddeubbasasssssss
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https wtt.:/dddde/.ooeSeeiiideieeeeeee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide the training and testing details in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
17

---

The full details can be provided either with the code, in appendix, or as supplemental
material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
We use box plots and KDE plots to show statistical significance. Some error bars are not reported because it would be too computationally and economically expensive. We have already conducted extensive experiments under various settings that are sufficient to verify the effectiveness of our proposed method.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide sufficient information on the computer resources (type of compute workers, memory) 77.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer:
[Yes]
Justification: Our research follows the NeurIPS Code of Ethics.
18

---

Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We have discussed both potential positive societal impacts and negative societal impacts of the work performed in $
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative socictal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
12. Licenses for existing assets
19

---

Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
The creators or original owners of assets used in the paper are properly credited. In addition, the license and terms of use explicitly mentioned and properly respected.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswitthheooo thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's ss creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
20

---

15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
Declaration LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
21

---

Appendix
A Proof of Theories
A.1 In-place KV cache calibration
RoPE is relative positional encoding method and uses circular function do positional encoding. In formal, we have a rotating matrix relative to the token position
coo( sin (pfb)
sin (pf0)
cos (pob)
0
0
0
0
cos(p)1)
R(p)=
cos(po (/22011/2211)
where 0g is a dimension relative to the dimension. The RoPE performs Op (x, (xp)(x for the token Tp. We denote Kp Ip W.. In other words,
cos(pO,) * K2p0 sin(pfa) Kp,1
sin(p Kp.0 + cos(pbo) Kp,1 cos(p(1) * Kp.2 - sin(p01) Kp,
Kp
sin(p0 1) Kp.N-2 + cos(p0 * *K Kp,N-
When truncation happens, we still have Ry. Kp which does not change.
cos(p'fo) Kp - sin(p' Kp,1
sin(p'0)) K p,0 + cos(p' (0) p,,1 cos(p'()) Kp.2 (nn(p())) Kp.3
Ky
sin(p'O 11/2-1)) ( R + (00(p00//221) RPNN-1/
Here, we verify the correctness of Eq. [
K2.21 cos((p+ + 8,) K p.2s sin((p + 8p)0.) K p,2111
(cos(p()) co,(6, sin (p(l) sin ),,:)) Kp,21
(sin(p0;) co,(0)))))))) + cos( sin sin((,,00)) (8,0,) K 12.2++1
cos(5,,0)) (pf,) K Kp,2s sin(pe,) Kp.2i+1)
(in(,,,,))sin(( K p.2t + cos(po,) (p.2111)
cos(6,,0,) p,
cos( 00,(-8,,,)KK sin(-5,0,)
On the other hand, for K 22+1
Ky K2 sin((p 8,) Kp.21 cos((p+ + 8,) Rp2241
= (sin(p0.) cos cos(6,,0)) + cos(po.) sin (8,0:) * 2
(cos(p0.) cos (o,(0)))) sin(p0.) sin (bp.))) K p.141
K2,20 + cas(pe) R p22
=
(in(&,,.) Kp,2 sin(p0,) kn,2+1)
= cos(o,,,)) Kp (in
cos(-oo ( Kp,2141 sin(-6, K p.24
22

---

A.2 The Optimality of ECE Policy under one Setting
this section, we prove the optimality of SAS's ECE policy the case that each session cache is of the same size.
Suppose there exists another eviction policy A such that it achieves higher session cache hit rate than ECE some session access sequence.
Let FECE be the hit rate achieved by ECE on session access sequence S, and HA be the hit rate incurred by A. By assumption,
HA > HECE-
Consider the first time i which A and ECE make different decisions about which session evict from the cache.
Let the sessions in cache just before time be the same for both policies, and suppose the current access at time requires an eviction. ECE evicts session Sc, and evicts different session SCa. + SCo-
Since ECE is defined to evict the session that will not be used for the longest time in the future, sC, the optimal choice given perfect knowledge of future accesses.
There are two cases to consider:
If sca is used sooner than 8c, the future, then A's decision will cause cache miss before ECE does, decreasing HA.
2. If sCa is used later than sc, or never used again, then ECE would have chosen sca instead, contradicting the definition of ECE.
In either case, A cannot produce higher cache hit rate than ECE, which contradicts our assumption. Thus, the optimality SAS's ECE policy is proved in this case.
B Additional Details of Background
As the R-Judge observes, an agent may face invalid actions or preset rule violation prior to final success or reaching round limit, We present a detailed analysis of agent outcomes for 20 Alfworld agent cases in Table There is increasing trend for invalid action format in the later stages and smaller models.
Table 4: Outcome comparison between Llama-8B and Llama-70B.
Model
Success
Reach Round Limit
Invalid Actions
Rule Violation
Llama-8B
35%
40%
15%
10%
Llama-70B
65%
30%
5%
0%
C Additional Details of Training
The Chatbot-Arena dataset includes 33k pairs of human-labeled preference data of the following models.
fastchat-t5-31
1ouna-13b-v1.2
chatglm-6b
1pt4all-13b-snoozy
mpt-7b-chat
wizardlm- 13b
vicuma-7b
RWKV
stablelm-tumee--llpha-
guanaco-33b
dolly-v2-12b
palm-2
oasst-pythia-11
claude-instant-vl
alpaca-13b
koala-1 13b
claude-v1
llama-133
gpt-3.5-turfe
vicuna-15 13b
gpt-4
C.1 Q-Judge Training
To facilitate training, we group the models in the arena into three tiers with increasing model sizes and higher ranking. A question is classified solvable by the smaller model if the smaller one either
23

---

Table 5: Data Distribution of Q-Judge Training Set
Small
Medium
Large
Accuracy
GT Eval
53.85%
33.52%
12.63%
N/A
44.35%
51.55%
4.10%
52.45%
Table 6: Data Distribution of R-Judge Training Set 0 (Unsatisfactory) (Satisfactory)
Accuracy
GT Eval
49.59%
50.41%
N/A
43.10%
56.90%
82.76%
wins or ties against a larger model's response; otherwise, considered challenging enough to require a more powerful model.
We train the Q-Judge for 10 epochs on top of BERT [2 with a batch size of 16 and a warm-up step of 500, utilizing one A6000 GPU. We choose BERT the backbone since Q-Judge is less frequently called and the routing task requires high precision. The training takes 2.9 hours. Under the customized loss, the Q-Judge achieves 52% accuracy on the evaluation set. Please be noted that the definition of accuracy here is different from the one in $7.4.2. where we define a much looser restriction due to the difficulty of annotation.
The distribution of the Q-uude train set and the results on the evaluation set are shown in Table 5 GT shows the distribution of ground truth labels and Eval shows the distribution of Q-Judge generated labels on the evaluation set.
C.2 R-Judge Training
We train the R-Judge on top of DistilBERT 43 using another customization of the Chatbot-Arena dataset. We choose DistilBERT as the backbone since R-Judge is more frequently called, and its efficiency is extremely important. We label the winner's 's response as capable, while the loser's 's as incapable. For tie match, we only take the smaller model's response capable and drop the larger LLM's. We train the R-Judge for 10 epochs with a weight decay of le-2 and warm-up steps of 500. The training takes 17 minutes. RJJ.ss achieves an 82.8% agreement rate with selanede preferences on the evaluation set. The distribution of the R-Judge train set and the result on the evaluation set are shown in Table with GT representing ground truth.
D Additional Implementation Details of AGSERVE
We implement the AGSERVE system in approximately thousands lines of Python and CUDA code. AGSERVE provides an interface for automatic agent programming and supports model hierarchy customization with agile agent serving support.
LLM Serving. The SAS part of AGSERVE is based on the SoTA LLM serving system, vLLM 24 AGSERVE supports session cache manager and two-level load balancing the instances to accelerate inference speed. We extend the APIs to record the session ID and session cache reuse pattern information (for truncation and restoration) for each request. We also provide more APIs to support the dynamic model allocation and proactive session cache release. AGSERVE supports starting session-aware instances of preset models via Ray (35
A Algorithm in ECE Eviction Decision. The pseudo code of the algorithm used in eviction decision is shown in Alg. D as we describe in $4 In L.1 L15,, adopt knapsack-like algorithm give the optimal recomputation overhead with the number of actual released blocks and evicted sessions. In L18 to L19, the algorithm calculates the recomputing penalty for each session. In L20 L26, the algorithm walks through different is, which defines the actual number of sessions to evict. AGSERVE predicts ETA in the following manner. First, AGSERVE maintains the metadata of each session. When session starts, AGSERVE initializes the metadata by the agent type. Throughout the session, AGSERVE updates the metadata with the last several rounds' round duration, and last arrival time data. AGSERVE E will use the average round duration to calculate the ETA. The ETA grows if the next request has not arrived at the origina ETA time, and finally reaches infinite in case of unexpected session interruption. The ETA prediction will not be drastically affected by the unexpected behaviors, since it keeps track of multiple rounds of history data. robust to workload shift, since each session is treated independently.
24

---

Algorithm 1:
ECE Eviction Policy
Input:
Number of ongoing sessions n, number of running sessions r, number of waiting sessions wt session cache size (can be zero) 1[1 [[1....) ranked by ETA, maximum batch size K, prefilling per-block overhead P. decoding time consumption
Output: Available session nums k, sessions to evict E
Function Knapsack (i, W):
Initialize dp[0] + 0, dp(l..... W] + 00, S[ + +
Initialize asp < 0, aval + oo, as t []:
// The first i sessions cache shall remain in the cache and is not considered in eviction
for r 4 -- n
for j t downto W - [[x] do
// for the cases where released blocks exceeds the requiring W
val = dpij] + if val < aval then
8
aval + val;
S +
10
asp = + [[]];
// No need to update dp, since no better solution comes based on asp
11
for j + W - [|rr] downto 0 do
// for the cases where released blocks does not exceed the requiring W
12
val = dp.pl u(z] if val < dp.j j + [x]]] then
13
dpij + [x]]] t val:
15
$lß + [x]]]
[x):
return aval, asp,ass as;
k D, (n r), E I]: 18 rsp + 0, asp - 0;
// Calculate the recomputation penalty for each session 19 for 0- 0 do
[0j1 P. [[i] (m-i+1)
// Walk through all possible batch size.
for +- to min(w, 2 K -r)
rsp rsp + r [[]];
// Check if new session needs to be evicted, improve efficiency
if aspsrsp then
24
L f,, asp, Eş + Knapsack
val = D,+s (n 1--11 i): 25
26
valck then
27
1 k i, E;;
28
return k,
Each SAS runs its ECE algorithm independently. The theoretical algorithm complexity is OW) We optimize the large constant W (knapsack size) with Python dictionary our implementation. Due to the long context of each session, the number of session cache maintained in the context is limited. As result, even in the worst case, the latency per ECE decision remains under 0.1 milliseconds, which we find negligible compared to typical inference latency.
Model Hierarchy. AGSERVE adopts a three-layer hierarchical model structure. We choose the
(Llama-8B) and 70B (Llama-70B) versions of the widely adopted Llama-3 [14] as the first two
layers. These two models also outperform in multiple benchmarks X 2]. We adopt GPT-4o
[ at the
third layer external API, which is one of the best models in multiple agent benchmarks
29 51]
25

---

Table AGSERVE APIs
API Implementation Building
Description
set_sp(
Set the shared prefix for the implementation.
set_tsp()
Set the task specific template for the implementation.
set tools(
Set the tools and its descriptions for the implementation.
Task Executing
Designate the cascading models, ranked by the order from weak to strong.
set_models()
set set_tau()
Designate the quality threshold for RP(()
set (
Designate the response checking frequency for RP(-)
set_task()
Set the task and assemble the prompt.
Adjust the availability of tools, and add/remove tools at the present round.
adjust_tooll
inject()
Inject the observation.
Query the LLM with the existing prompt, and return the LLM- generated actions.
action()
req_ change)
Request general migration.
req_update()
Request a service-upgrade migration.
go_ so_back()
Restore the saved status.
We choose the three-layer hierarchy due to the following reasons. Firstly, the physical limits of our testhed constrain the number L.LMs we can deploy. Secondly, we ensured that the models each layer exhibit substantial performance gap in reasoning quality, which takes advantage of model diversity. The benefit of Llama also applics to other models adopting the Llama structure, including the Qwen family [57]. Notably, AGSERVE does not benefit from dictionary reuse, since the tokenization process only takes no more than 2 milliseconds even under the full context window, and is not comparable to the TTFT. As matter of fact, AGSERVE re re-tokenize the prompt for each request. The cascading structure AGSERVE flexible. Adding or removing layers is straightforward: users only need customize the Chatbot-Aree dataset and retrain the QMM. Thirdly and importantly both the Llama-8B and Llama-70B models are adjustable models, as stated in $6. and are widely adopted in agents. They from the same family and share the same tokens and dictionary. GPT-4o selected due to its outstanding performance in multiple benchmarks, demonstrating AGSERVE'S support for commercial model APIs besides SAS.
Fault Tolerance. AGSERVE is capable to handle the possible failures in distributed inference. If an instance fails, the client sends request to the scheduler for rerouting. If the scheduler fails, AGSERVE defaults to a regular LLM serving system with session cache enabled, establishing sessions between the agent and a random instance.
Agent Programming. AGSERVE provides a set of APIs like LangChain [25] for agent programming. Agent operators can build their own agent implementation with available tools and prompt templates The clients can also designate the used I.Mss, the quality threshold, etc. through the APIs. We list the APIs for programmers to build the agent implementations as well as those for clients to use AGSERVE in Table 7 The agent operates on the Session class to perform LLM query and quality maintenance. Each agent can pass the т, ip, P parameters and the quality threshold 0 to Sessin onn. For each query, the Session class will return quality indicator in addition to the response. For the restorable agents, the quality indicator may instruct them to save or restore the previous history.
E Additional Details of Evaluations
E.1 Implemented Agents
We implement the following four agent implementations demonstrate the performance of AGSERVE.
The AlfWorld (AW) 45 is an embodied household assistant navigating and interacting within simulated environments. AW is standard agent that includes multiple rounds of interaction to attempt and fail.
26

---

plays two-player game called AquaWar against random- -cted baseline. The client performs two types of actions: attack and guess. Prompt templates of the two types vary. We implement CG with two AGSERVE sessions using multi-agent collaboration. The two sessions perform model cascading independently.
The card game (CG) 229
The knowledge graph (KG) [29] queries knowledge graphs to
question. We start Freebase knowledge graph via Virtuoso [|] on CPU node with 256 GB memory. Queries
do alter KG, we implement KG restorable agent introduced in $5.2
The Mind2Web (M2W) [12 navigates and performs tasks across various domains of real-world websites, While it presents a complex challenge, an LLM usually completes the task within few interaction rounds.
E.2
Quality for Different Agents
We outline the computation of the quality metric in the section. se is set to 25 if the session ends successfully in status COMPLETED or
_RECHHD,; otherwise, it is set to
E.2.1 AW Agent Quality
For the AW agent, the quality calculation is follows:
rounds
QAw sc cc + 25
2
The base quality is set to 25, with an additional 50 points of ve if there is valid result. If the agent accomplishes the task, ve is set to 50, otherwise, While fewer interaction rounds mean fewer moves for the embodied agent, score of 25 is added with a penalty of 0.5 for each move.
E.2.2 CG Agent Quality
For the CG agent, the quality is calculated based on meta-performance metrics:
full_play play
harm on _n_enemy
Qce
x 25 +
x 50
test_ test_times
total, HP x test times
round num
+ 25 max
50 X test_ som_timm"" 1)
The quality of CG partly breaks the common requirement of quality factors due to its "game" character. It comprises three parts: the normal behavior part by the ratio of full play rounds (in how many test times the CG agent operates normally), the accomplishing part by the win round ratio, and a survival part replacing the efficiency part. In most cases, we find LLM loses the battle. In a losing game, longer survival should be given a bonus rather than quick death.
E.2.3 KG Agent Quality
For the KG agent, the quality is computed as:
rounds
QKc sc x1 25
2
A base quality of se is added, followed by the F1 score. The F1 reflects how well the answer given by the KG agent matches the ground truth. If the score is greater than zero, an additional score of 25 is applied with penalty based the number of actions, with reason similar the AW.
E.2.4 M2W Agent Quality
For the M2W agent, the quality is derived from multiple metrics:
Quaw + element_aat acc x 25 + step_sr x 25
action x 25
This calculation is based on three key metrics: element accuracy, step success rate, and action Fl score. Each metric contributes equally to the overall quality, with a maximum of 25 points each.
E.3
Cost of Open-Source Models
We use the retail price of cloud server rentals estimate the cost of open source models. our knowledge, the cost of open-source models could be significantly lower (30% to 60% lower) for
27

---

Table Cost of AGSERVE and baselines for Alfworld under different pricing plans.
AGSERVE
Llama-8B
Llama-70B
GPT-4o
RouteLLLM
Cascade
Original Cost
0.0177
0.0023
0.0376
0.1073
0.0337
0.1395
Cost(-30% Rent)
0.0137
0.0016
0.0263
0.1073
0.0324
0.1353
Cost(-609 Rent)
0.0097
0.0009
0.0151
0.1073
0.0312
0.1311
Quality
90
21
72
94
25
69
self-operated data centers long-term users with Azure's spot reserved plarf The lower cost could further exhibit the advantage of AGSERVE against the GPT-40-only approach.
E.3.1 The A6000 Testbed
We find the instance of the most similar computing capability offered by cloud service providers to our testbed is ecs. gn7i-8x..... 16xlarge on Alibaba Cloud with 8 A10 GPUs. The rent for this instance is approximately $10 per hour.
The Llama-8B 8B model requires only 1/4 of the node and can run 32 sessions concurrently with no
throughput loss. The price of Llama-8B per session is 10 + 3600 + 4 + 322 = $(2.17 x 10-5)/s. The Llama-70B model requires one node and can run 16 requests concurrently with no throughput loss. The price of Llama-70B per session is 10 3600 : 16 $(1.73 10 10-*)/s.
AGSERVE requires 5/4 of the node to perform both 8B and 70B models. Though with higher throughput, we calculate the price of
$1.73 10-4 = $(2.17 10 10-4)/s.
E.3.2 The A800 Testbed
We find the instance of the most similar computing capability offered by cloud service providers to our A800 testbed is ND96amsr A100 v4 offered by Azure. The rent for this instance is approximately $33 per hour, with tax included.
All three agent serving systems evaluated in 7.. utilize two 8x80GB A800 instances. We find out that the testbed is capable of handling about 64 sessions (requests) simultancously without computing resource preemption. We define the price of using the A800 testbed for each session as 33 + 3600 64 = $(2.66 10 10-4)/s.
E.4 AGSERVE Settings
We set the quality control threshold 0 of R.Js()) to 0.5. The reasoning quality check frequency is set to 4 for all agents except M2W. AGSERVE checks the LLM's reasoning quality for every response (v 1), as an M2W session usually finishes in 2 to 3 rounds.
E.5 Details of End-to-End Cascaded Serving
In 7.2, we show that AGSERVE shows superior performance under the retail rental price. In this section, we display the cost of AGSERVE and other baselines for operating the Alfworld agent under different pricing plans in Table AGSERVE'S 's rental price is still lower than that of Llama-70B which has worse quality than it. With a discount in the rental prices, AGSERVE further breaks the current cost-quality tradeoff curve with only 9.04% cost to achieve similar performance with GPT-4o.
E.6 Details of Multi-Agent Traces
Uniform, Gamma, Poisson, and Burst traces that we evaluated in $7.3 comprise 100 agents each. The uniform, Gamma, and Poisson traces share an average sending rate of 4 seconds per agent. Running traces is very expensive. It costs us more than 500 USD produce Fig just for the OpenAl API.
E.7
Details of Ablation Studies
F.7.1
SAS Efficiency
We set the max_meeee www.mmemmuiiiittttt.. included) and the max_ ceaxx_odee__leee to 4096 on one A6000 GPU. Under this setting, there are 745 blocks for KV cache, fitting 6 to 3 sessions depending on their round. This setting promotes eviction and truncation during agent execution demonstrate the efficiency of SAS's ability better. Nevertheless, this setting could be very common in real-world serving scenarios 23 where GPUs have limited memory, such as NVIDIA(R) RTX 4090 (24GB) and A10 (24GB) GPUs.
of SAS to 0.37 (equivalent to 17.8GB, 3GB of CUDA graph
See ttops://azzrrs www.ooootooooooommtmmmtttwww.... http:://......aks......ssssssssr
28

---

Table 9: QMM finishes evaluation job in real-time.
Q- QJuudee
R-Judge
AW
CG
KG
M2W
Overhead (s)
0.05
0.03
0.3
0.04
0.005-2
0.03
Liama-88 8B
Llarna-708
40000
Throughpur (takenon))
30000
20000
W
(a) Avg Prompt Throughput Over Time.
2000
800
Thrnughput (tolernon))
400
My
200
Time(Minutel
(b) Avg Generation Throughput Over Time
Figure 15: The prefilling and generation throughput of different models in AGSERVE. The thin red line at 4.67 min shows the launch two-A800 70B instance and a half-cut of a four-A800 8B instance. The bold red line 17.67 min shows the two-A800 70B instance replaced by an 8B instance.
E.7.2 QMM Efficiency
To minimize the bias, we only show the observation and LLM's response round by round to annotators, with no previous knowledge of the session's final result or the model size. We also utilize the voting mechanism and each label is agreed by least three annotators.
We compare the overhead of running Q-Judge, R-Judge, and the response time for four agents in Table 9 The results show that the R-Judge can finish in the time response generation most cases. The Quuudee only introduces negligible overhead in its infrequent calling. Please note that we take the time from the generation of request the generation of first token TTFT in Fig including the overhead for node routing, which is not counted in other experiments.
E.7.3 Details and Case Study of Dynamic Allocation
To create an imbalanced demand-supply ratio, we continuously launch agent samples every 5 seconds with 0 at a low value of 0.01, deferring sessions to the Llama-70B instances, mimicking the burst of difficult tasks the very beginning. After period, we raise mimic a change low-cost pursuit. We forbid AGSERVE from visiting OpenAl APIs to prevent the assistance of external machines.
In this section, we give a case study on the impact of dynamic allocation. The agent launcher keeps launching the agent at a given speed and shifts the quality maintenance threshold at some time point. Fig [3] the prompt and generation throughput of different models during the experiment. The prompt throughput shows the prefilling speed, while the generation throughput reflects the decoding speed, AGSERVE first moves 2 GPUs from the instance 70B instance due to the high volume of 70B instances. And it dynamically moves the 2 GPUs back when the demand for 8B models
29

---

Method
Cost (.01S)
Quality
Overhead(s/req)
Gross Cost(.01S)
Human 1.505
91.75
14.71
2.944
AGSERVE
1.768
89.57
1.768
Table 10: Evaluations of human-like judges in agent serving of Alfworld. The overhead column
shows the time it takes for human judge the satisfaction per request. Gross cost shows the cost if the human-judge overhead included the calculation.
Table 11: Comparison of centralized (C) and (istributed(D) scheduling by e2e latency (EL), quality (Q), and scheduling latency (SL) under different agent launch rates (LR). EL and SL are in seconds.
LR (slagent)
EL-C
Q-C
SL-C
EL-D
Q-D
SL-D
0.8
64.76
84.18
0.42
47.66
85,83
0.09
0.4
97.45
83.39
0.43
60.12
85.86
0.08
0.2
129.99
85.03
0.46
68,94
85.06
0.09
increases. Notably, AGSERVE keeps the functionality of both models throughout the time, since there are always requirements for both models. As we discussed in $7.4.3 we face 6% failures in FSI, FS2 and AGSERVE due to the L limited capability in long-context scenarios. In routine operation, AGSERVE will choose upgrade the model GPT-4o.
E.8 Performance of an Ideal Human-Like Judge
In fast-paced request pattern in agent serving, it is impossible for a human judge in real-time. Our experience shows that it takes 10 seconds or more for humans to read and understand the context before rating the response, missing the latency objective of many agents. Fig 13 depicts 75.8% accuracy in response quality judgment relative to human preference. The KV cache of different models not be reused. AGSERVE adopt periodical quality check and retry strategy stated in $5.2 to avoid excessive model migration, which may lead to extra overhead of KV cache re-computation. In this experiment, we want to show the performance of an ideal human- like judge on the AW agent. To do this, we let human expert determine whether to migrate or not for each response, and evaluate the cost and quality of each agent serving. We do not include the time for humans to judge in the cost and latency calculation. Table Toble shows the performance of the human judge, which only outperforms AGSeRve with margin benefits, showing that R-Judge strong enough for the three-layer session lifetime management. We acknowledge that it may not apply to cascades with more layers of models.
F AGSERVE on Large Scale
While AGSERVE employs centralized scheduler for distributed agent serving, it is designed to sup- port distributed scheduling, where each district has its own RS. We simulate AGSERVE'S performance on large scale with the following approaches:
1. The inference latency on instances is based on sleep operation, and we profile the inference duration by the GPU setting, concurrent requests, and model size.
2. The quality issues are triggered certain probability related to the size/capability of LLM.
3. We define a virtual round-to-go. One interaction round with a larger LLM has higher probability of decreasing the virtual round-to-go more than a smaller one. We simulate the behavior of a 60-node cluster with 8 GPUs per node. The result is shown in Table showing that the distributed scheduling version of AGSERVE reduces e2e latency while accelerating scheduling speed and improving fairness in the cluster.
30


Reviews:
[{"review_id": "97NMoEOQl4", "reviewer": "Reviewer_Z6NR", "summary": "The paper presents a session-aware system for LLM agent serving, addressing inefficiencies in KV cache management, model selection, and resource allocation. Key innovations include ETA-based KV cache eviction, in-place positional embedding calibration for middle-truncation reuse, quality-aware cascading via QMM, and dynamic GPU scheduling. Evaluated on AgentBench with agents (e.g., ALFWorld, Mind2Web) and models (Llama-3 8B/70B, GPT-4o), it achieves 83.5% cost reduction at GPT-4o quality, 1.8x quality at same cost, 2.84x cache hit rate, and up to 50% latency cut vs. baselines like vLLM and Llumnix++.", "questions": "- The authors mentioned the long-input scenarios in sec 7.2. Could the authors elaborate a bit more on how to handle the straggler settings with the four agents?\n- Could the authors provide more cascade performance with more levels/non-Llama models, and potentially extended to large-scale serving involving more than two nodes?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n- The paper is clearly presented and well written.\n- The paper proposes the session-aware kv cache strategy, the proposed eta eviction and calibration enhance reuse for multi-round agents, outperforming LRU with 2.84x hit rates. This could offer tailored advantages against existing prefix-focused systems, which optimize general inference but overlook agent-specific middle-truncation.\n- The paper proposes quality-aware cascading, the session-level upgrades via QMM (95.2% recall) minimize costs by preferring underkill, surpassing RouteLLM, and could achieve similar quality to GPT-4o with a 16.5% cost in AW.\n- The proposed system integrates cache, cascading, and scheduling for 65.6% multi-agent savings and provides throughout empirical evaluations on AgentBench and synthetic request traces from Poisson and Gamma distributions. Theoretical proofs has also been provided for in-place kv-cache calibration. \n\nWeakness:\n- The assumption of ece is uniform/burst, which may be vulnerable to the variability as noted in [1]. \n- In sec 7.4.2, the authors discuss the low overhead of QMM, but this may accumulate with long-input or long-context checks (different scenarios of prefilling/decoding bound), providing more details about the context-length, quality & efficiency tradeoff could make the claim stronger; \n- It is understandable that the study already provides throughout analyses on A6000 and recent AgentBench, but the scalability might be untested vs. heterogeneous LLMs and the fixed three-tier cascade lacks diversity; \n\n[1] Wang, Jiahao, et al. \"KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Provider.\" arXiv preprint arXiv:2506.02634 (2025).", "was_revised": true, "final_justification": "I appreciate the clarification and new experiments addressing other reviewers' comments. I have decided to keep my original ratings.", "rebuttal": "Thank you for your postive appraisal of our work!\n\n[Weakness 1] The assumption of ece is uniform/burst, which may be vulnerable to the variability as noted in [1].\n> Thanks for bringing [1] to our attention. ECE relies on the assumption that the estimated time of arrival (ETA) for the next request in a session is independently predictable. This assumption holds well in practice for agent workloads.\n>\n> Interestingly,  as a contemporaneous work, [1] investigates multi-turn request traces (particularly agent workloads as cited by their [55, 57]) and arrives at a similar conclusion. As shown in their Section 3.3,  _the probability of issuing the next turn request is predictable given a workload category and time interval_, which supports our design. We will explicitly discuss and cite [1] in Section 2.2 of our revision to acknowledge this connection.\n\n[Weakness 2] The low overhead of QMM may not work under accumulated long contexts.\n> Thanks for your question. The two judges’ context window is 512, limited by their backbone (i.e. BERT, DistilBERT). Q-Judge's input of $t$ and $p$ is usually long enough to saturate the window. As for R-Judge, we feed as much information as possible in $e$ to fill the window up, since longer context facilitates R-Judge to capture the LLM’s repetitive behavior if applicable. As a result, AgServe almost always utilize full context window of both judges in our evaluation. Despite that, we witness no significant overhead, since QMM overlaps with observation generation. We will clarify it in Section 7.4.2 of our revised manuscript.\n\n[Weakness 3, Question 2] Could the authors provide more cascade performance with more levels/non-Llama models, and potentially extended to large-scale serving involving more than two nodes?\n> Thanks for your question. We will address your concern in three parts.\n> 1. AgServe selects the three-tier cascade due to the capability gap of LLMs and the limited testbed size, justified in Appendix C (L639). We will add reference to it in the main body.\n> 2. Llama structure is adopted by many other LLMs, such as the Qwen family. Therefore, these models benefit the same from SAS's technology just like Llama models. Moreover, AgServe does not take advantage from tokenization reuse, as tokenization only takes negligible time of 1~2 milliseconds compared to the 2 second decoding duration for Llama-8B. Therefore, heterogenous models will not affect AgServe's performance.\n> 3. Since each SAS processes its sessions independently in AgServe, we expect little performance divergence for larger testbeds. We are working on finding a larger testbed, and we expect to evaluate on a larger scale in the revised version.\n\n[Question 1] The authors mentioned the long-input scenarios in sec 7.2. Could the authors elaborate a bit more on how to handle the straggler settings with the four agents?\n> Thanks for your question. We would like to explain why we claim that RouteLLM can not tackle the long-input scenarios. RouteLLM is tailored for short-input applications (e.g. MMLU[1] and GSM8K[2]), where average input is less than 100 tokens. However, in general, the input for agent serving exceeds thousand tokens due to the long few-shot examples and tool instructions. To tackle this problem, AgServe's QMM selectively picks the information that are necessary for quality assessment. This enables QMM to perform well in the long-input agent serving scenarios. We will clarify it in the revision.\n>\n> [1] D.Hendrycks, _et al_. Measuring massive multitask language understanding. ICLR 2020.\n>\n> [2] K.Cobbe, _et al_. Training verifiers to solve math word problems. arXiv:2110.14168, 2021."}, {"review_id": "f9aTBbXvpS", "reviewer": "Reviewer_WMmU", "summary": "This paper introduces AGSERVE, a novel system designed to overcome the cost-quality tradeoff in serving Large Language Model (LLM) agents. Unlike traditional LLM serving systems, AGSERVE is optimized for the unique characteristics of agentic workflows, such as predictable request patterns, increasing quality requirements over multi-round sessions, and unique prompt formatting including middle truncation. AGSERVE features a session-aware server for efficient KV cache management, a quality-aware client that employs real-time quality assessment for dynamic model cascading, and a dynamic resource scheduler to maximize GPU utilization. Experimental results demonstrate that AGSERVE achieves comparable response quality to GPT-4o at a significantly lower cost (16.5% in some cases) , delivers a 1.8x improvement in quality relative to the cost-quality tradeoff curve , and increases KV cache hit rates by up to 2.86x compared to existing methods.", "questions": "- What are the specific heuristics or machine learning models used to predict the Estimated Time of Arrival (ETA) for sessions, which is crucial for the ECE eviction policy, and how robust are these predictions to unexpected agent behaviors or workload shifts?\n\n- Beyond the qualitative \"satisfactory rate drops as round increases\" observation, can a more quantitative characterization of the increasing task difficulty for LLMs as context grows be provided, perhaps in terms of specific metrics or a more detailed analysis of failure modes at later stages of agent sessions?\n\n- Given that the paper \"disregard[s] the sessions that trigger rule violations\"  when evaluating the performance of the RS Dynamic Allocation, how frequently do such violations occur in practice, what are their root causes, and how does AGSERVE handle them to maintain overall system stability and performance?", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n- The paper identifies and tackles the unique challenges of serving LLM agents, distinguishing it from traditional LLM serving systems that are not optimized for agent workflows. This focus on session-awareness, predictable request patterns, and growing context is a key strength.\n\n- AGSERVE offers a holistic solution by integrating a session-aware server with innovative KV cache management (ETA-based eviction and in-place positional embedding calibration), a quality-aware client for session-aware model cascading, and a dynamic resource scheduler. These components work together to transcend the cost-quality tradeoff.\n\n- The paper provides strong empirical evidence on real testbeds, showing that AGSERVE achieves comparable quality to GPT-4o at a substantially lower cost (16.5%), delivers a 1.8x quality improvement relative to the tradeoff curve, and significantly boosts KV cache hit rates (2.84x) while reducing latency.\n\n## Weaknesses\n\n- The evaluation primarily focuses on four specific agent implementations (AlfWorld, card game, knowledge graph, and Mind2Web) . While these cover different interaction patterns, the generalizability of AGSERVE's benefits across a wider range of LLM agent applications is not fully explored.\n\n- The in-house deployment of AGSERVE primarily uses Llama-3 (8B and 70B versions) as the adjustable models . While GPT-4o is used as an external API , the system's performance and optimizations might be less effective or require significant re-tuning for other LLM architectures or model families that are not from the same family and do not share the same tokens and dictionary .\n\n- The cost estimation for open-source models is based on retail cloud server rentals, and the paper acknowledges that \"the cost of open-source models could be significantly lower (30% to 60% lower) for self-operated data centers or long-term users\" . This simplification might not fully capture the true cost-efficiency advantages or disadvantages in all deployment scenarios, potentially skewing the cost-quality comparison.\n\n- The R-Judge training relies on a customized Chatbot-Arena dataset where human experts labeled responses. The paper notes the difficulty of annotating prompt difficulty for Q-Judge , and it's unclear if the \"human-selected preferences\" for R-Judge fully capture the nuances of agent task success or are prone to subjective biases.\n\n- While the paper mentions related works, some aspects of AGSERVE, like KV cache management and model cascading, could benefit from a more in-depth comparison with the very latest developments in those specific areas, beyond just general LLM serving systems. For instance, the discussion on cache reuse mentions CacheBlend , but a detailed comparative analysis of their respective strengths for agent serving is limited.", "was_revised": true, "final_justification": "I've confirmed the author's rebuttal and will keep my original score.", "rebuttal": "Thanks for your detailed review, which helps us improve our quality of manuscript.\n\n[Weakness 1] The evaluation primarily focuses on four specific agent implementations . While these cover different interaction patterns, the generalizability of AGSERVE's benefits across a wider range of LLM agent applications is not fully explored.\n> Thanks for your advice. The four agents cover multiple interaction patterns, showing that AgServe improves agent serving efficiency with our proposed techniques based on agent serving behaviors. We’ll clarify AgServe’s generalization to agent serving scenarios in our revision, and expand agent implementation in the future.\n\n[Weakness 2] The system's performance and optimizations might be less effective or require significant re-tuning for other LLM architectures or model families that are not from the same family and do not share the same tokens and dictionary .\n> Thanks for your question. Generally speaking, LLMs with different dictionaries in the cascade will not affect AgServe’s performance.\n> 1. For generalizabilty, AgServe does not reuse tokenization result when performing service upgrade. Therefore, we expect little performacne divergence between heterogenous models and the current setup.\n> 2. Our profile on Llama-8B shows that tokenization takes only 1~2 milliseconds, while generation takes 2 seconds per round. Therefore, we believe the gain from dictionary reuse is limited. \n> \n> We‘ll clarify it in Appendix C.\n\n[Weakness 3] The simplification of VM rentals might not fully capture the true cost-efficiency advantages or disadvantages in all deployment scenarios, potentially skewing the cost-quality comparison.\n> Thanks for your question. AgServe keeps most lifetime of sessions on locally-deployed models, and only upgrades to GPT-4o when necessary. Therefore, lower rental price is beneficial for AgServe. The table below shows the performance of AgServe for Figure 8(a) if rental is 30% or 60% lower. We’ll add these statistics to Appendix D.\n\n||AgServe|Llama-8B|Llama-70B|GPT-4o|RouteLLM|Cascade|\n|-|-|-|-|-|-|-|\n|Original Cost|0.0177|0.0023|0.0376|0.1073|0.0337|0.1395|\n|Cost(-30\\% Rent)|0.0137|0.0016|0.0263|0.1073|0.0324|0.1353|\n|Cost(-60\\% Rent)|0.0097|0.0009|0.0151|0.1073|0.0312|0.1311|\n|Quality|90|21|72|94|25|69|\n\n[Weakness 4] It's unclear if the \"human-selected preferences\" for R-Judge fully capture the nuances of agent task success or are prone to subjective biases.\n> Thanks for your question. To minimize the bias, the annotators were shown the observation and LLM’s response round by round, with no previous knowledge of the session’s final result or the model size. We also utilize the voting mechanism and each label is agreed by three annotators at least. We’ll clarify it in our revision.\n\n[Weakness 5] Some aspects of AGSERVE, like KV cache management and model cascading, could benefit from a more in-depth comparison with the very latest developments in those specific areas, beyond just general LLM serving systems. For instance, the discussion on cache reuse mentions CacheBlend , but a detailed comparative analysis of their respective strengths for agent serving is limited.\n> Thanks for your advice. As we discussed in Appendix F, CacheBlend is orthogonal to AgServe. CacheBlend selectively updates the KV cache for pre-computed RAG contents, and is applicable to AgServe. We will discuss CacheBlend in the main text in the revised version, if the space permits.\n\n[Question 1] What are the specific heuristics or machine learning models used to predict the Estimated Time of Arrival (ETA) for sessions, which is crucial for the ECE eviction policy, and how robust are these predictions to unexpected agent behaviors or workload shifts?\n> Thanks for your insight. AgServe predicts ETA in the following manner. First, AgServe maintains the metadata of each session. When a session starts, AgServe initialize the metadata by the agent type. Throughout the session, AgServe updates the metadata with the last several rounds’ round duration, and last arrival time data. AgServe will use the average round duration to calculate the ETA. The ETA grows if the next request has not arrived at the original ETA time, and finally reaches infinite in case of unexpected session interruption. The ETA prediction will not be drastically affected by the unexpected behaviors, since it keeps track of multiple rounds of history data. It is robust to workload shift, since each session is treated independently.\n\n[Question 2] Beyond the satisfactory rate observation, can a more quantitative characterization of the increasing task difficulty for LLMs as context grows be provided, perhaps in terms of specific metrics or a more detailed analysis of failure modes at later stages of agent sessions?\n> Thanks for your question. Task difficulty is a subjective metric, which is hard to quantify. We discussed different types of quality issues in Section 5.2 (L139), which can be considered as different modes of failures. Taking Alfworld as an example, here is a detailed analysis of agent outcomes for 20 AW cases. There is an increasing trend for invalid action or format in the later stages and smaller models. We’ll add more discussions on failure modes in our revision of Section 2.2.\n\n\n||Success|Reach Round Limit|Invalid Actions|Rule Violation|\n|-|-|-|-|-|\n|Llama-8B|35%|40%|15%|10%|\n|Llama-70B|65%|30%|5%|0%|\n\n[Question 3] Given that the paper \"disregard[s] the sessions that trigger rule violations\" in Sec 7.4.3, how frequently do such violations occur in practice, what are their root causes, and how does AGSERVE handle them to maintain overall system stability and performance?\n> Thanks for your question. These violations mainly refer to the violation of preset agent-specific rules or action format. This will make agent exit prior to task completion or reaching round limit. Such violation takes place in 6% of sessions. This probability is similar for FS1, FS2 and AgServe. This is mainly due to the LLM’s limited capability in long-context scenarios. In its normal operation, AgServe will choose to upgrade the model to GPT-4o."}, {"review_id": "M9sgblMsel", "reviewer": "Reviewer_j2me", "summary": "The work proposes an agent serving system called AgServe, which jointly optimize KV cache reuse, model selection and resource allocation, reducing the agen serving cose while achieving higher quality results.\n\nThe author starts by arguing that the current serving systems are largely developed for serving generic LLMs and are not optimized for the agent workflows. The authors have identified the following agent serving characteristics:\n- predictable and high frequency interaction: the interactions with agents are light-weight (less tokens) and frequent\n- growing context and increasing difficulties: agents interact in multi-round fashion, making the context length grows rapidly\n- unique prompt formatting: agent has specific ways of formatting prompts\n\nThe main idea is to predict sessions to reduce inference latency and depending on the phase of the session, use different-sized models to reduce cost.\n\nThe major challenges to achieve this are:\nsession aware cache policy: current system’s reuse and eviction policies are not well-suited for agent serving\n- quality assessment: existing systems do not estimate the session lifetime and do not have the ability to dynamically assess model response quality and thus cannot choose the best model for each stage\n- resource allocation: the gpu usage of model switching is not optimized", "questions": "- I am curious what the Q-Judge / R-Judge are practice? Are they also LLMs? How can you ensure these instances fulfill the designated tasks? E.g., the Q-Judge should select the most cost-efficient model for the session by assessing the task’s difficulty, but how can you be sure it chose the most cost-efficient model? Do you need to train them or provide additional context? Similarly, the R-Judge should evaluate the reasoning quality, how do decide the consistency in evaluation and how do you set the standard / threshold?\n- The author mentioned the customized CUDA kernels in chapter 7, what exactly does the kernels do and why do you need them? There is no explanation of that.\n- The ECE eviction policy seems reasonable, but it comes with additionally computation overhead, because it needs to make prediction? LRU follows the simple but universal rule of least recently used to evict cache and since LRU has been adopted by the two most popular inference framework vLLM and SGLang, I believe it has shown robustness in large scale caching. Whether ECE is truely beneficial needs more large scaled validations", "limitations": "- Generally I feel like the delivery of this paper needs further improvement to convey their message more cleaner; currently they keeps on mentioning new concepts, without previous definition or explanation, making the reading difficult. For example, in the beginning of section 7, the author mentioned different agens (AW, CG, KG, M2W), but you have to search through the whole paper and find the definition in the Appendix, or in the ECE Eviction Policy section, it states the formula for calculating the eviction cost, and the P is also not defined. There are other examples like this.\n- The strictness of R-Judge theta seems to be an important hyperparamter, as it greatly controls the decision of R-Judge, but how it is obtained and quantified seems unclear (judging from the section 5.2 and the Appendix B) \n- Model allocation seems to be too simplified, the RS profiles the nunimum number of GPUs required, but in practise, increasing the number of GPUs will lead to faster inference, this trade-off should be considered when allocating memory", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n- clear motivation and detailed problem analysis: the authors start by claiming that there does not exist a serving system that is optimized for serving LLM agents, then\nanalyzing the problems of using the generic LLM’s serving systems, making its motivation clear\n\n\nWeaknesses\n\n- The idea of introducing a Q-Judge and R-Judge is interesting, but I am having doubts that the mentioned training of Q-Judge / R-Judge is sufficient to accomplish their designed roles, by only training on Chatbot-Arena for such little time, also the only training on this dataset seems limited, leaving doubt that whether the trained Q-Judge and R-Judge is able to handle all kinds of tasks, or it would require task specific training", "was_revised": true, "final_justification": "The authors have addressed most of my comments and promised to update the paper accordingly with more explanations. I feel like this paper can have a borderline accept, but still needs some polish in the writing and presentation to qualify for better score.", "rebuttal": "Thanks for your feedback, which helps us improve the quality of our manuscript.\n\n[Weakness 1, Question 1] （1）I am having doubts that the mentioned training of Q-Judge / R-Judge is sufficient to accomplish their designed roles, by only training on Chatbot-Arena for such little time, also the only training on this dataset seems limited, leaving doubt that whether the trained Q-Judge and R-Judge is able to handle all kinds of tasks, or it would require task specific training. （2） I am curious what the Q-Judge / R-Judge are practice? Are they also LLMs? How can you ensure these instances fulfill the designated tasks? E.g., the Q-Judge should select the most cost-efficient model for the session by assessing the task’s difficulty, but how can you be sure it chose the most cost-efficient model? Do you need to train them or provide additional context? Similarly, the R-Judge should evaluate the reasoning quality, how do decide the consistency in evaluation and how do you set the standard / threshold?\n\n> Thanks for your insightful question. We would like to address your concerns twofold. \n> 1. Q-Judge and R-Judge are lightweight classifiers built upon BERT and DistilBERT, respectively. Both judges are trained on customized Chatbot-Arena datasets. Q-Judge learns from RouteLLM [35], which has demonstrated robust performance across a wide range of tasks in routing scenarios without task-specific tuning. To minimize unnecessary use of powerful models (\"overkill\"), Q-Judge employs a novel cost-aware loss that encourages routing to cheaper models when appropriate. R-Judge focuses on evaluating reasoning quality by detecting whether the response addresses the original task, avoids redundant reasoning steps, and maintains logical coherence.\n> 2. We agree that generalization is an important concern. In Section 7.4.2, we present a micro-benchmark evaluating the two judges across four implemented agents. Q-Judge agrees with human preferences in over 70% of cases, and R-Judge achieves 90.5% recall on detecting low-quality reasoning. These results suggest that despite being trained on a single dataset, the judges generalize well in practice. We will clarify this further in the revision.\n\n[Question 2] The author mentioned the customized CUDA kernels in chapter 7, what exactly does the kernels do and why do you need them? There is no explanation of that.\n> Sorry for the confusion. The customized CUDA kernel is designed to batch tokens from multiple sessions and perform in-place calibration of positional embeddings, as described in Section 4. This operation allows us to better utilize GPU memory and reduce kernel launch overhead in high-throughput settings. We will add a detailed explanation of the kernel’s role and its benefits in the revision.\n\n[Question 3] The ECE eviction policy seems reasonable, but it comes with additionally computation overhead, because it needs to make prediction?  I believe LRU has shown robustness in large scale caching. Whether ECE is truly beneficial needs more large scaled validations.\n> Thanks for your question. We acknowledge the concern about potential overhead. In practice, the ECE policy incurs only about 0.1 milliseconds per eviction decision, which we find negligible compared to overall serving latency of 2 seconds. To achieve this, AgServe maintains lightweight metadata for each session (such as prior arrival times and interval) and leverages statistical estimation to predict the expected time of next arrival (ETA).\n> While LRU is very useful in general LLM serving and widely used by many frameworks, it may evict the session cache just before reuse in agent serving, as depicted in Figure 7. To tackle this problem, we introduce estimated-time-of-arrival to agent serving for the first time and proposes ECE.\n> ECE mainly focus on the KV cache of tokens that are not shared by other sessions (i.e. task-specific template and truncatable). The eviction decision of such content is handled entirely within each node. Therefore, the scale of nodes will not reduce ECE’s effectiveness.  We will include this clarification and the runtime overhead measurement in the revised manuscript.\n\n[Limitations 1] Currently they keeps on mentioning new concepts, without previous definition or explanation, making the reading difficult. For example, in the beginning of section 7, the author mentioned different agens (AW, CG, KG, M2W), but you have to search through the whole paper and find the definition in the Appendix, or in the ECE Eviction Policy section, it states the formula for calculating the eviction cost, and the P is also not defined. There are other examples like this.\n> Thank you for pointing this out. We will revise the manuscript to ensure all technical terms and abbreviations are clearly defined when first introduced. Specifically, we will add a forward reference to Appendix D.1 in Section 7 for the agent definitions (AW, CG, KG, M2W). Additionally, we will clarify that P in the eviction cost formula denotes the penalty of recomputation—i.e., the ratio between prefill and decode latency per token. A thorough pass will be conducted to ensure that all such symbols are properly introduced and explained.\n\n[Limitations 2] The strictness of R-Judge theta seems to be an important hyperparamter, as it greatly controls the decision of R-Judge, but how it is obtained and quantified seems unclear.\n> We appreciate your observation. The strictness parameter $\\theta$ in R-Judge controls the pass threshold for response quality. A value of 0 means no response will pass the quality check, while 1 accepts all responses. We train the Q-Judge for strictness of 0.5. Users can adjust this threshold based on their own quality requirements or workload characteristics. We will include an explanation of this hyperparameter and its practical implications in the revision.\n\n[Limitations 3] Model allocation seems to be too simplified, the RS profiles the nunimum number of GPUs required, but in practise, increasing the number of GPUs will lead to faster inference, this trade-off should be considered when allocating memory\n> Thanks for your insight. While RS profiles the minimum number of GPUs required to fit a model, it does not restrict allocation to the minimum. AgServe supports tensor parallelism, and RS can assign more GPUs than the minimum when available. For example, if one GPU suffices but six GPUs are idle, RS will spawn two SAS instances using 4 and 2 GPUs respectively, allowing faster inference and flexible resource adjustments in the meantime. We will clarify this design in the revision, and will support more types of parallelism (e.g. pipeline parallelism) in the future."}, {"review_id": "NUUxwlPO0F", "reviewer": "Reviewer_E9yH", "summary": "This paper presents AGSERVE, a session‐aware agent serving system designed to “transcend” the traditional cost–quality tradeoff in LLM‐based agent workflows. By combining (1) in‐place RoPE calibration and an ETA‐based cache eviction policy to boost KV‐cache reuse under middle‐truncation, (2) a Quality Maintenance Module (QMM) with a Q-Judge for initial model selection and an R-Judge for runtime quality monitoring and migrations, and (3) a dynamic GPU scheduler for demand‐driven resource allocation, AGSERVE claims up to 83.5% cost savings versus GPT-4o at equivalent quality, 1.8× quality gain at fixed cost, 2.86× cache‐hit improvement, and 1.2× end-to-end speedup on real multi-GPU testbeds across four agent benchmarks.", "questions": "1. Have you evaluated AGSERVE on any non-programming agent workloads to assess generality?\n\n2. Can you provide false-positive/false-negative curves for the R-Judge, and quantify migration overhead when misclassifications occur?\n\n3. What is the worst-case decision latency of the ECE algorithm as agent count and cache size scale?", "limitations": "Experiments are confined to four programming‐agent tasks, leaving AGSERVE’s effectiveness on dialogue, planning, or other agent workloads unverified.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n1. Tailors cache and eviction policies to multi-round agent contexts, yielding substantial cache reuse and latency gains.\n\n2. Integrates caching, quality judgers, and resource scheduling under a coherent API, simplifying adoption.\n\n3. Thorough ablations isolate the benefits of in-place calibration, ECE vs. LRU eviction, Q-Judge/R-Judge overheads, and dynamic allocation across two testbeds (A6000 and A8000 clusters).\n\n\nWeaknesses\n1. All experiments target four programming-agent tasks on AgentBench. It remains unclear whether AGSERVE’s session-aware optimizations generalize to non-coding agents (e.g., dialogue, planning).\n\n2. While Q-Judge and R-Judge training details and accuracy/distribution tables are provided, the paper omits any analysis of misclassification trade-offs (false-positive vs. false-negative rates) or calibration curves that would clarify migration costs under varying thresholds.\n\n3. The ECE eviction decision algorithm is described in pseudocode, but its runtime complexity and decision-latency overhead under tight memory budgets are neither analyzed nor experimentally measured.", "was_revised": true, "final_justification": "I assign a final score of 4 (Borderline Accept). The clarifications and new experiments provided during the rebuttal largely resolve my concerns. Please incorporate these experiments and clarifications into the next revision of the manuscript.", "rebuttal": "Thank you for your detailed review and feedback.\n\n[Weakness 1, Question 1, Limitations 1] Have you evaluated AGSERVE on any non-programming agent workloads to assess generality?\n> Thank you for your question. We implement four agents, AlfWrold (simulated household jobs / embodied agents), Card Game (card games/ game playing), Knowledge Graph(knowledge graph / Q&A dialogue) , Mind2Web (web browsing / planning) on top of AgServe, spanning across a wide range of agent application and request patterns. As shown in Figure 8, AgServe consistently outperforms baselines across all four domains by pushing the cost-quality frontier. We will clarify this generality and add a forward reference to Appendix D.1 at the beginning of Section 7 in the revision.\n\n[Weakness 2, Question 2] Can you provide false-positive/false-negative curves for the R-Judge, and quantify migration overhead when misclassifications occur?\n> Thanks for your question. Sure! Here is some data points on the False-Positive/False-Negative curve. We’ll add the curve to our revision. False positive means that the R-Judge mistakenly classifies a satisfactory result as unsatisfactory, while false negative menas that the R-Judge takes an unsatisfactory one in the ground truth as satisfactory. To reduce the early-migration cost of false positive, AgServe adopts the retry strategy and assesses the response again prior to migration, as described in Section 5.2.\n\n| Data Point | 0     | 1      | 2      | 3      | 4      | 5      |\n|---------------|-------|--------|--------|--------|--------|--------|\n| False Positive Rate| 0.00% | 28.09% | 40.06% | 41.22% | 47.93% | 57.55% |\n| False Negative Rate| 33.64%| 17.60% | 8.98% | 3.94%  | 0.91%  | 0.00%  |\n\n> We also conduct a study on the cost of early migration in Appendix D.7. An ideal human judge with no overhead and perfect accuracy can further reduce the cost by 14%, yet only improves the performance by 2.4%. However, human-as-a-judge introduce second-level latency in reality and raises the cost significantly.\n\n[Weakness 3, Question 3] What is the worst-case decision latency of the ECE algorithm as agent count and cache size scale?\n> Thanks for pointing this out. Each SAS runs its ECE algorithm independently. The theoretical algorithm complexity is O(k\\*n\\*W). We optimize the large constant W (knapsack size)  with a Python dictionary in our implementation. Due to the long context of each session, the number of session cache maintained in the context is limited. As a result, even in worst-case conditions, the observed latency per ECE decision remains under 0.1 milliseconds, which we find negligible compared to typical inference latency. We will clarify this in the final version and include this worst-case estimate."}]

Author rebuttal:

Meta-review: The paper presents an agentic and session-aware KV-cache management system. The system leads to better GPU utilization without compromising the accuracy and quality of responses. All the reviewers give 'borderline accept' ratings to the paper. Some of them increased their initial ratings after reading the authors' rebuttal and additional experimental results. The reviewers expressed some concerns about the writing and clarity of the paper, which I encourage the authors to address in the revised version of the paper.
Decision: Accept (poster)