ICaRus: Identical Cache Reuse for Efficient Multi-Model Inference
Authors: Sunghyeon Woo
Jaeeun Kil
Hoseung Kim
Minsub Kim
Joonghoon Kim
Ahreum Seo
Sungjae Lee
Minjung Jo
Jiwon Ryu
Baeseong park
Se Jung Kwon
Dongsoo Lee
Venue: ICLR 2026 Poster (2026)
Keywords: multi model inference
agentic ai
prefix caching
fine-tuning
Primary area: other topics in machine learning (i.e., none of the above)
TL;DR: We propose ICaRus, the first architecture that enables multi model to fully share KV caches across all layers, providing a principled solution to inefficiencies in conventional multi model serving.

Abstract: Multi model inference, where multiple task-specialized models collaborate to solve complex real-world problems, has recently emerged as a prominent paradigm, particularly in the development of agentic AI systems. However, in such scenarios, each model must maintain its own Key-Value (KV) cache for the identical prompt, leading to explosive memory consumption.
This explosive growth of KV caches forces LLM serving systems to evict previously stored caches, which in turn introduces significant recomputation overhead whenever the evicted caches are required again. 
Moreover, prefix caching is inherently infeasible across different models, forcing each model to recompute KV cache for the identical prompt, which leads to signficant overhead. To alleviate these issues, we propose Identical Cache Reuse (ICaRus), a novel architecture that allows multiple models to share identical KV caches across all layers. ICaRus is based on the key observation that a decoder-only Transformer can be conceptually decomposed into a logical encoder, which generates KV caches, and a logical decoder, which predicts output tokens from the KV caches. ICaRus fine-tunes only the logical decoder while freezing the logical encoder, enabling multiple models to share an identical KV cache. This eliminates cache memory explosion and unexpected evictions while also allowing cross-model reuse of KV caches for new input tokens, thereby removing redundant recomputation in multi model inference achieving both efficiency and scalability. Moreover, by incorporating lightweight adapters such as LoRA, ICaRus parallelizes KV cache generation and next-token prediction during decoding. ICaRus achieves comparable accuracy to task-specific fine-tuned model across a diverse set of tasks, while allowing multiple specialized models to fully share KV caches. ICaRus achieves up to $11.1\times$ lower P95 latency and $3.8\times$ higher throughput in multi agent scenarios with 8 different models, compared to prior multi model system.

Published as a conference paper at ICLR 2026
ICARUS: IDENTICAL CACHE REUSE FOR EFFICIENT MULTI MODEL INFERENCE
Sunghyeon Woo", Jaeeun Kil", Hoseung Kim, Minsub Kim, Joonghoon Kim,
Abreum Seo, Sungjae Lee, Minjung Jo, Jiwon Ryu, Baeseong Park, Se Jung Kwon,
Dongsoo Lee
NAVER Cloud
Sunghyyon.wooo. wool, jaeeun.kil| navercorp.com
ABSTRACT
Multi model inference, where multiple task-specialized models collaborate to solve complex real-world problems, has recently emerged a prominent paradigm, particularly in the development of agentic Al systems. However, such scenarios, each model must maintain its own Key-Value (KV) cache for the identical prompt, leading to substantial memory consumption. This explo- sive growth of KV caches forces LLM serving systems to evict previously stored caches, which in turn introduces significant recomputation overhead whenever the evicted caches are required again. Moreover, prefix caching is inherently infea- sible across different models, forcing each model to recompute KV cache for the identical prompt, which leads to significant overhead. To alleviate these issues, propose Identical Cache Reuse (ICaRus), novel architecture that allows multiple models to share identical KV caches across all layers. ICaRus is based the key observation that decoder-only Transformer can be conceptually de- composed into a logical encoder, which generates KV caches, and a logical de- coder, which predicts output tokens from the KV caches. ICaRus fine-tunes only the logical decoder while freezing the logical encoder, enabling multiple mod- els to share an identical KV cache. This eliminates cache memory explosion and unexpected evictions while also allowing cross-model reuse of KV caches for new input tokens, thereby removing redundant recomputation in multi model inference achieving both efficiency and scalability. Moreover, by incorporating lightweight adapters such as LoRA, ICaRus parallelizes KV cache generation and next-token prediction during decoding. ICaRus achieves comparable accuracy task-specific fine-tuned model across a diverse set of tasks, while allowing mul- tiple specialized models to fully share KV caches. ICaRus achieves up 11.1x lower P95 latency and 3.8x higher throughput in multi agent workflow with 8 different models, compared conventional multi model system.
INTRODUCTION
Large Language Models (LLMs) have shown strong performance across domains (Zhao et al. 2024 Dubey et al. 2024: Comanici et al. 20255 Yang et al. 2025);; however, single model struggles with complex tasks that demand multi step reasoning and Nomain-specinfe expertise (Tang et al. 2020 Yao et al. 2023 Sun al., 2024). Recently, the emerging paradigm of multi model inference addresses this limitation by orchestrating ausk-specialized models, achieving higher accuracy and problem-solving ability than a general-purpose model (Fu al.) 2023 Du et al. 2024 Shen al., 2024 [Subramaniam et al. 2025). However, this paradigm introduces severe challenges in managing the Key-Value (KV) cache: each model maintains its own cache even for identical prefixes, causing memory consumption to grow rapidly with the number of models. Once GPU memory is saturated by KV cache, serving systems (K et al., 2023] ZZheng et al. 2024) must evict caches, which triggers redundant recomputation and significantly degrades throughput. Furthermore, because KV caches are model-specific, prefix caching (Kwon et al. 2023 Zheng et al. 2024) cannot be applied
"Equal contribution

---

Published as a conference paper at ICLR 2026
Workflow
KV Cache
KV Cache
Single
Multi Model
Agent A Planner)
Prompe #f
Promot af
Model
Conventional
ICaRus (Ours)
Prefix
Time
Agent B (ex, Executor)
Prompt
caching
Training Method
Performance
Prompinng
Weak
Fine-tuning
Strong
(only logical Deesder)
Fine-tuning
Strong
Prefix
Recompute
KV load
caching
KV Sharing
Inberent
Unsupported
Supported
Agent (ex. Summarizar)
Prompt
o3
KV Memory Usage
Low
High
Low
# Prefill
Recompute
KV load
Recomputation
Low
High
Low
Conventional Approach
ICeRus
(a) KV Cache management strategies in agent workflow using multi model
(b) Comparison of ICaRus und conventional approaches
Figure 1:
Comparison of KV cache management strategies and effectiveness in multi model scenar- ios between conventional approaches and ICaRus.
across different models, which forces identical prompts to rebuild KV caches independently and thereby increases latency.
Previous KV cache optimization techniques, such as pruning (Zang et al. 2023 quantization (Hooper et al. 2024; Yang et al.) 2024), and inter-layer sharing QQiao et al. 2024). ). reduce cache size while minimizing accuracy degradation. Unlike traditional LRU-based prefi caching, KVFlow (Pan al. 2025) schedules KV cache eviction and prefetching based on anticipated agent work- flow, reducing recomputation overhead. However, these methods focus only on single model cache management, leaving unresolved the challenges of cache explosion and the lack of KV cache shar- ing of prefix in multi model settings. DroidSpeak (Liu et al., 2024b) addressed multi model KV cache management by sharing non-sensitive layer caches between base model and its fine-tuned variants, thereby reducing recomputation cost. However, this approach has inherent limitations, as caches from sensitive layers remain unshared and must still be recomputed.
To address these issues, we propose Identical Cache Reuse (ICaRus), a novel architecture that en- ables multiple models share and reuse the same KV cache across all layers. The core idea of ICaRus originates from conceptually decomposing a decoder-only Transformer into two parts: a logical encoder, which is responsible for generating KV cache, and a logical decoder, which pre- dicts the next token from the cache. We freeze the logical encoder of pretrained LLM (i.e. base model) and fine-tune only the logical decoder. Since all specialized models share the identical log- ical encoder, the KV cache generated for identical prompt is likewise identical, enabling direct sharing without redundant memory usage as shown in Fig. IT). This prevents GPU memory from rapidly saturating due to KV cache growth, avoiding costly recomputation caused by cache eviction. Moreover, shared KV caches enable prefix caching across models, eliminating redundant compu- tation for identical prompts and further improving efficiency ICaRus leverages the adapter architecture to generate the KV cache for the next step in parallel with the next-token computation during the decode phase. We evaluate ICaRus across diverse tasks including matherattcs, coding, and knowledge understanding on a wide range of model families and scales (LLaMA-3.1-88) Dwwn3-1.7B/8BB/442 The results demonstrate that ICaRus achieves accuracy comparable to task-specific fine-tuned models, even though ICaRus-tuned models are able to share KV caches across tasks. Furthermore, when integrated into the vI.LM serving system and evaluated in multi agent scenario, ICaRus delivers much as a 1 11.1 x reduction in percenteee (P95) latency and a 3.8x throughput gain compared to conventional multi model system.
depicted in Fig. [b). In addition,
In summary, the main contributions of this work are follows:
We propose ICaRus, the first architecture that enables multiple decoder-only Transformers to fully share KV caches, guaranteeing high generation quality in real serving scenarios by explicitly modeling the fully shared-KV setting already at training time.
We demonstrate that ICaRus achieves accuracy comparable to task-specific fine-tuning across diverse tasks (mathematics, coding, and knowledge understanding) and model ar- chitectures (I.LaM--3. 1-8B, 1. 17B/8B/14BB
2

---

Published as a conference paper at ICLR 2026
We confirm that ICaRus significantly improves efficiency in multi agent workflow, achiev- ing up to 11.1x reduction in P95 latency and 3.8x improvement in throughput compared to conventional multi model system.
BACKGROUND & MOTIVATION
Key-Value Cache in LLM Serving Systems. During autoregressive inference, decoder-only Transformers generate tokens sequentially, where each new token depends all previously gener- ated tokens. Computing self-attention naively for every step requires recomputation over the entire sequence, incurring per-token complexity of On²²) where
the sequence length. To avoid this quadratic overhead, modern LLM serving systems cache the key and value representations of previ- ously processed tokens (Vaswani al., 2017). By reusing these cached states, each new decoding step only attends to the most recent token, reducing the per-token attention complexity to O(n) and thereby significantly lowering computational cost. However, the size of KV caches grows linearly with both sequence length and model depth, imposing substantial memory pressure on GPU-based serving systems (Kwon et al. 2023 Zheng et al.] (024).. Consequently, memory-efficien cache management has emerged as a critical challenge for scalable LLM deployment.
Prefix Caching in LLM Serving Systems. Prefix caching is widely adopted optimization that reuses the KV cache corresponding to a fixed prefix across multiple queries sharing the same initial context (Kwon al. 2023 Zheng et al. 2024). This technique is particularly effective in scenarios
generation (RAG) (Lewis et al., 2020) and instruction-tuned applica- tions (Chung et al. 2024 Ouyang al. 2022), where prompts often contain long but invariant components like system prompts, task-specific templates, or retrieved documents. By reusing the cached veyuv states of these repeated prefixes, serving systems can avoid redundant computation during the prefill phase, effectively reducing the computational complexity from (n ) to ((m). where n. denotes the sequence length and denotes the variable suffix length with m < n. thereby improving both throughput and latency. Moreover, prefix caching is highly beneficial in multi-turn conversational settings, where a large dialogue history is preserved across turns and only the most recent user utterance changes; by caching the KV states of the shared history and computing atten- tion only for newly appended tokens, serving systems can efficiently support interactive dialogues without recomputing the entire context at every turn (Kim et al. 2025).
such as netrieval-augment
Agentic AI Workflow and Multi Model Inference. Agentic AI and workflow- based reasoning have given rise to complex pipelines in which models are orchestrated to perform specialized roles. For instance, ReAct (Yao et al. (002) alternates between Thought - Aci - Observation, Reflexion (Shinn et al., 2023) incorporates self-evaluation loops, LATS (Zhou et al. 2024) explores reasoning through parallel branch expansion, and L.Moompiler (Kim et al. 2024) constructs a DAG to sched- ule overlapping tool and model calls. When executed within a single model, such workflows can leverage prefix caching to avoid redundant computation, thereby reducing effective memory usage, lowering P95 latency, and improving throughput (Kim et al. 2025). However, in multi model set- tings where task-specialized models collaborate within single pipeline, each model must maintain its own KV cache even for identical prefixes. Such KV cache duplication leads to memory usage that grows linearly with the number of active models; once GPU capacity is saturated, this growth inevitably triggers cache eviction, which in turn forces recomputation of evicted prefixes. More- over, since prefix caching typically operates only within individual models, identical prefixes must be recomputed separately across models, leading to redundant prefill computation that inflates both latency and energy consumption. These limitations underscore the need for new architectures that support cross-model KV sharing and prefill de-duplication in multi model inference.
DESIGN OF ICARUS
3.1 DECODER-ONLE TRANSFORMER AS LOGICAL ENCODER AND DECODER
We first present a mathematical formulation of the decoder-only Transformer, which predicts the next token conditioned on the current token context. Specifically, we abstract x,, k; and v; as the i-th token, its key representation, and its value representation, respectively, and denote the decoder-
3

---

Published as a conference paper at ICLR 2026
only Transformer by F. In this case, the next-token generation from the current token context in a decoder-only Transformer can be expressed as 24+1 (11, .22..... ..). To generate the next token 1, the model requires two types of information: the current token æ; and the accumulated key-value pairs. We denote the key set and value set to step i as 11 = {ki, .2,.... k;,, V1 = {11, "2,,",,- ..... ..). More concretely, in the attention operation, the query derived from x; is generated anew at each step, whereas the keys and values are continuously appended to the cache and reused across subsequent decoding steps. other words, the query does not persist beyond its step, but the KV pairs accumulate and form the long- terg-termmn memory. This dependency can be expressed as
2++1
= F(2,, K1g, V1:s).
(1)
Bq indicates that decoder-only Transformer predicts the next token conditioned the current token T2 and the KV cache constructed up to this point. More generally, the generation process can be decomposed into two conceptual stages: (I) constructing the key set K₃ and the value set V from the input sequence 21:4- {I1, ...,,;,,, J. J. and (2) decoding the next token 2++1 based on the current token z; together with the accumulated sets (K,, V.). Formally, this can be expressed as
K Vi = E(x1s),
Ti+1 = Dzs,, K13, V1:a).
(3)
where we introduce a logical encoder, denoted by E, that transforms the input sequence into its key and value representations, thereby constructing the KV cache, and a logical decoder, denoted by D, that consumes the current token and the KV set to generate the next token. Importantly, decoder- only Transformer can be interpreted as the special case where the parameters of the logical encoder and logical decoder are identical. More detailed concept of logical encoder-decoder architecture is depicted Appendix C
3.2 ICARUS: IDENTICAL CACHE REUSE ACROSS LLMS
As described in Section 3.1 decoder-only model can be decomposed into a logical encoder, which generates key-value pairs from given token, and a logical decoder, which predicts the next token using the current token and the accumulated KV cache, shown in Eqs. 2 From this perspective, task-specific fine-tuning can be viewed as jointly training both the logical encoder and the logical decoder to specialize
given task. While such task-tuned models achieve strong task-specific capabilities, each maintains its own logical encoder thereby preventing KV cache sharing even when prompts are identical across models.
Building on this insight, we propose the ICaRus architecture which fine-tunes only the logical de- coder of decoder-only Transformer as below.
Kisi. Vi Erase (21x);
(4)
T1+1 Drack (Ii, Kins Via),
(5)
MelaMechQA-40k
poo--iinttuttCode--Be
0.8
Conventional Fine Tuning
Conventional Fine lunennuning
1.0
0.8
0.6
Loss
0.6
0.5
0.4
www.
50
100
150
200
250
300
0
100
200
300
400
500
600
Training Steps
Training Steps
Figure 2: Training loss curves conventional fine-tuning and ICaRus, both applied with LoRA on LLaMA-3.1-88. trained on the MetaMathQA-400 and Soad-nnsttut-Code---5 dataset.
4

---

Published as a conference paper at ICLR 2026
Here, Ebase is the frozen logical encoder inherited from the base model, and Dtask is the logical decoder fine-tuned for the target task. Specifically, the logical encoder and the logical decoder initialized with the parameters of the base model, pretrained decoder-only Transformer (i.e. Fhase = Fbase = Daasa). The task-specific logical decoder Drask in Eq. is then trained, starting from the base decoder Dbases to predict the next token x1+1 under two objectives: (1) specializing in the target task, and (2) leveraging the KV cache generated by the frozen logical encoder in Eq. As result, multiple task-specific logical decoders (e.g., mathh Dooding Dreasoning) can share a single logical encoder, which is identical to the base model (i.e., Emath = Ecoding = Ereasoning Ebase), thereby enabling all models to reuse the identical KV cache generated by the shared encoder, as illustrated in Fig. [
During training. the input data are duplicated and provided to both the logical encoder and the logical decoder. The logical encoder generates the corresponding key-value representations, while the log- ical decoder computes attention over these representations with final output used compute the training loss for gradient updates. The logical encoder is kept frozen during training to ensure cache sharing across tasks. This training procedure, which explicitly accounts for KV cache sharing, helps ensure robustness when KV caches shared inference time in real serving scenarios, especially compared with approaches that attempt to share KV caches across models trained independently without considering KV cache sharing.
Figure 2 shows the training loss of LLaMA-3.1-88 on MetaMathQA-40k (Yu et al. 2023) and Evol-Instruct-8Ok (Roshdieh) 2023). The ICaRus curves almost perfectly overlap with those of conventional task-specific fine-tuning, indicating that restricting learning to the logical decoder does not hinder optimization and is sufficient for task-specific adaptation even when the logical encoder is shared across models. In other words, freezing the logical encoder forces all task-specialized models to reuse a common sequence representation and express their differences only through the decoder, which can be interpreted as a form of implicit regularization.
The core idea of ICaRus is to factorize a decoder-only Transformer into a logical encoder and a logical decoder, and to train only the logical decoder so that KV caches can be shared across different models. Consequently, ICaRus is largely agnostic to how the logical decoder adapted: in principle, the logical decoder can be trained using full-paramete fine-tuning, LoRA (Hu et al. (022). or
related variants (Liu et al., 2024a Diang et al. 2024; Woo et al.) 2025). Among these adaptation methods, we adopt LoRA to train the logical decoder because LoRA offers high training efficiency, which enables rapid deployment of new agents in multi-agent systems, while achieving performance comparable full-parameter fine-tuning (Schulman & Lab 2025) and making it straightforward to optimize the decoding phase in ICaRus for inference efficiency. In the following section, we describe how integrate LoRA into ICaRus and how this design further optimizes the overall inference cost.
3.3 OPTIMIZING ICARUS FOR MULTI MODEL INFERENCE
In Section 3.2 we introduced the concept and training methodology of ICaRus. In this section, we explain how ICaRus operates in multi model inference scenarios and discuss its key optimization strategies. During the prefill phase, ICaRus uses only the logical encoder, which encodes the input prompt into a KV cache and produces the next token. In the subsequent decode phase, ICaRus duplicates the current token (I;;) and performs two operations: (I) encoding æ; into key-value pair (ki, vy) through the logical encoder, and (2) predicting the task-specific output token (#:41) through the logical decoder by using the duplicated together with the accumulated KV cache (fki,...,k:)) { Ui..... 2t)), as Eq. 5 Consequently, regardless of the task, the KV cache is always generated by the logical encoder, and other role-specific decoders can directly reuse this shared KV cache without any need to recompute or further update it. The details can be found in Appendix C
Sequential execution of the logical encoder and decoder may incur up to 2x latency overhead com- pared to a single model execution, since both weights and KV caches are accessed twice. To mitigate the problem, we insert and fine-tune only lightweight adapters within the logical decoder instead of fully fine-tuning the decoder. Consequently, the logical encoder and logical decoder share most pa- rameters except for the adapters, enabling the shared parameters to be loaded only once and allowing the computations of the two modules to be executed in parallel as depicted in Fig.
5

---

Published as a conference paper at ICLR 2026
preftt outpur
fine-tuned pulput using base model RY coche!
using bese model
sky
Ibeat used
:1 Iayer
not used
:adlapter
not used
Attention
Rg
Attention
matmul
GQA
XN
GQA
XN
Computed kV
Looded
uscd
[CaRus
flies
The
dipllicate
(a) Profill Phase
(b) Decode Phase
Figure 3: Overview the ICaRus architecture. The base model, pretrained decoder-only Trans- former, serves as the logical encoder, while the adapter-tunets model (consisting of the base model and tunable adapter) serves as the logical decoder. The blue and orange lines indicate computations performed by the base model and the adapter-tunce model, respectively. The purple square denotes that the same base model generates the KV cache during both the prefill and decoding phases. In ICaRus architecture, KV caches can be reused regardless of the task, since the KV cache is always generated by the same base model (i.e., the logical encoder).
Table 1: Complexity comparisons between single model and multi model scenarios.
Memory
Latency
Scenario
Method
Decode (per token)
Total
Prefill
Memory Access
Compute
Single Model
O(M + L.)
O(ML. + Li)
O(M + L.)
O(M + L.)
BaseLine ICaRus
O(M + NL.)
O(N(ML, + 1Z)) O(M + Li)
O(M + Le) O(M + L.)
O(M + L.) O(2M + +2L)
Multi Model
OMM + Lt)
In addition, because both models attend to the identical KV cache generated by the base model, we optimize attention computation by concatenating the query representations of the logical encoder and decoder along the head dimension (Fig. 3). This enables parallel attention computation without redundant KV cache reads. Consequently, although the decoding phase of ICaRus appears to dou- ble the computational workload by running both the logical encoder and decoder, the system adds only negligible latency overhead. This is because parallel execution generates memory traffic (base parameters, KV caches, and lightweight adapter weights) that is almost the same as that of a single model. The detailed algorithm can be found in Appendix B
To validate the effectiveness of ICaRus, we further analyze the time and space complexity of multi model system built with the conventional approach (baseline) and with ICaRus, using N adapters in multi agent scenarios. Table I summarizes the results. We denote the input prompt length as L, the number of interaction turns per adapter as t, and the number of output tokens per turn as La, with the total sequence length L,, = L + lT,,, The base model size is represented by M. In the baseline, each model independently allocates KV memory and recomputes prefill for the same prompt, yielding memory O(M + N Li) and prefill complexity (M + L))). In contrast, ICaRus shares a single KV cache across models, reducing both to single model order, with space O(M+L,) L,) and prefill O(M L.). The advantage grows with longer sequences from inter-model communication and with larger agent counts N.
During decoding, the baseline requires O(M + L,) memory access and computation per token be- cause each adapter-tuned model reads the model weights and its own KV cache. ICaRus computes both the logical encoder and decoder (O(2M + 2T.)) but parallelizes most of the computation so

---

Published as a conference paper at ICLR 2026
Table Comparison of conventional methods and ICaRus on diverse datasets. Base Model denotes the pretrained decoder-only Transformer without fine-tuning. Multi Model consists of three inde- pendently fine tuned models: one on MetaMathQAA-00K one on Evol-Instruct-Code and one on Oasstl. ICaRus uses the same three specializations, but trains only task-specific logical decoders on a shared logical encoder, enabling KV cache sharing across models.
KV Sharing
Math
Coding
Knowledge
Model
Method
GSMSK
GSM+
HEval
HEval+
GPQA
Base Model
25.9
18.0
36.6
29.9
16.7
LLaMA3.1-8B
Multi Model
69.7
48.5
48.2
41.5
27.3
ICaRus (Ours)
67.9
45.8
48.2
43.9
28.8
Base Model
11.8
12.5
68.3
61.6
24.2
Qwen3-8B -BBBase
Multi Model
X
85,4
66.1
81.7
75.6
34.3
ICaRus (Ours)
0
87.3
67.5
86.6
79.9
33.8
that the model and KV cache are read only once. restoring O(M + L.). In context, many-turn settings where decoding is memory- -ound,, memory access dominates; accord- ingly, ICaRus achieves decoding latency comparable the baseline.
long-
EVALUATION
4.1
EXPERIMENTAL SETUP
We evaluate ICaRus from two perspectives: (1) accuracy and (2) performance in multi model infer- construct multi model systems follows. Starting from -LaMA-3.1-4 (Dubey et al., 2024) and Swen3-1.7B/8BB. BBBBBa/ (Yang et al.) 2025), we build three speciicc models per base model by fine-tuning MetaMathQA-40k for mathematies (Yu et al. 2023 Evol- Instruct-Code-O00 for coding (Roshdieh) (023). and OASSTI for instruction tuning Köpf et al.] 2023) using either conventional fine-tuning or ICaRus. These systems are then evaluated on bench- marks aligned with each task: GSM8K (Cobbe et al. 202T) and GSM-Plus (Li et al. 2024 for mathematics, HumanEval (Chen et al.
ence. In section 4.22
2021) and HumanEval+ (Liu al., 2023) for coding, and GPQA-Diamons (Rein et (0024 for knowledge understanding, using im-eval-hamees, (Biderman jet al. 2024 and EvaiPlus (Liu et al.
2023) to measure zero-shot accuracy. For comparison, both
the conventional fine-tuning and ICaRus use LoRA (Hu et al. 2022) the adaptation method.
we measure latency and throughput multi agent setting, using representative agentic patterns such as ReAct (Yao et al. (023) and Reflexion (Shinn et al.) 2023) on the HotPoiQA dataset (Yang et al., 2018). We evaluate configurations with 2, 4, and agents. We further extend this evaluation to a multi-model, multi-turn request-routing setup: within a single workflow, successive requests from a multi-turn interaction are routed in a round-robin manner different models. In this setting, the baseline is conventional multi-LoRA system, whereas ICaRus replaces it with a cache-sharing multi agent system. To ensure a fair comparison, integrate both systems into the vL.LM serving framework and evaluate them under identical settings. More details can be found in the Appendix A
For multi model inference (Section 4.3)
4.2 ACCURACY EVALUATION
Accuracy on diverse task. We first train and evaluate ICaRus alongside conventional fine-tuning across mathematics, coding, and instruction-tuninn tasks using LLaMA-3.1-- 1-8BBBBBBB and Qwen3-8B, as reported in Table The results show that ICaRus achieves accuracy comparable to, or even surpass- ing, task-specific fine-tuning across all tasks. In particular, for the Qwen3-8B-Bas model, ICaRus outperforms prior task-tuned models by at least 1.4% on benchmark evaluations for both mathemat- ics and coding tasks. We expect that the superior accuracy of ICaRus stems from generalization effect: by fine-tuning only the logical decoder while keeping the logical encoder frozen, ICaRus reduces the risk of overfitting compared to full task-specific fine-tuning.

---

Published as a conference paper at ICLR 2026
Table Comparison of conventional fine-tuning and ICaRus across different model sizes (Qwen3- 1.7 1.7B/8B/14H 17/8//14BBB1 1BB-Base) trained on the MetaMathQAA40KK dataset.
Model
Qwen3-1.7B-Bal
Qwen3-8B-Bas
Qwen3-14B-Ban
Method
Baseline
ICaRus
Baseline
ICaRus
Baseline
ICaRus
GSM8K
73.2
74.0
85.4
87.3
85.6
88.8
GSM+
53.7
54.1
66.1
67.5
66.7
68.8
Scaling with model size. We also examine the scalability of ICaRus with respect model size by conducting experiments on Qwen3-1.7B/8B/14BBBBB
in Table 3 The results show that ICaRus consistently achieves higher accuracy compared to conventionally fine-tuned baseline, with im- provements exceeding 2% on Qwen3-14B-Baue demonstrating that our method remains compet- itive as model capacity increases. Additionally, we verify the robustness of ICaRus across tasks and its scalability to larger model sizes by evaluating Qwen3-32B on tool-calling tasks, described in Appendix D
Table 4: Comparison conventional methods and ICaRus in multi-model inference scenarios. Base Model denotes the LLaMA-3. 1-8B-Base model without fine-tuning, while Math, Coding, and IF denote models fine-tuned on MetaMathQAA40K, Evvol-nnstruct-Codde and OASST1, respectively. Multi Model and ICaRus both consist of these three task-specific models; in ICaRus, however, only the logical decoders are fine-tuned while the logical encoder is shared across models.
# Model
Method
KV Sharing
Math
Coding
Knowledge
Avg.
GSM8K
GSM-Plus
I[Eval
11Eval+
GPQA
Base Model
25.9
18.0
36.6
29,9
16.7
25.4
Math Model
69.7
48.5
42.7
36.6
20.7
43.6
Coding Model
22.8
17.5
48.2
41.5
21.7
30.3
IF Model
16.5
44.5
39.0
27.2
30.3
Multi Model
X
69.7
48.5
48.2
41.5
27.2
47.0
3
ICaRus (Ours)
O
67.9
45.8
48.2
43.9
28.8
46.9
Multi domain orchestration results. Table 4 compares ICaRus orchestration with diverse single and multi model configurations using LLaMA-3.1-8B Each task-tuned model is fine-tuned on single domain-specific dataset (MetaMathQA for mathematics, Eyvol-Insrruct-Code--000 for coding, and OASSTI for instruction-tuning The results show that while single task-specific fine-tuned model achieves high accuracy on its target task, the model suffers from significant performance degradation on other tasks. In contrast, a multi model system composed of multiple task-specific fine-tuned models achieves consistently high accuracy across all tasks. Our ICaRus also attains accuracy comparable such multi model system, while additionally benefiting from KV cache sharing across agents, which enables orchestration at substantially lower computational cost.
4.3 PERFORMANCE MULTI MODEL INFERENCE
P95 latency and throughput across QPS. ICaRus consistently outperforms baseline multi model system across all load levels in both latency and throughput, as evaluated on LLaMA-3.1 I- 8B under the ReAct pattern (Fig-F 4p. We measure performance as the number of queries per second (QPS) increases; latency is reported at the 95th percentile (P95).
A key advantage of ICaRus is its ability to reuse identical prefix caches across models, avoiding the redundant recomputation required baseline system where each model reconstructs its own cache. For example, at QPS 0.3 with 4 models, ICaRus reduces P95 latency by 5.1x compared to the baseline, and this benefit becomes more pronounced as the number models increases.
As the QPS increases, the cumulative KV cache size of baseline system soon exceeds GPU memory capacity, triggering eviction previously stored KV caches and their subsequent recomputation. Consequently, throughput first plateaus and then declines, with the degradation occurring earlier as
8

---

Published as a conference paper at ICLR 2026
10
Baseline (N
Throughpur Trequesrsi
Basetine (Nkm4)
Biseific (N+d;
Beseline (N+6)
ICaRus (N=2)
KSauus (N-2)
895 Litenncy
ICaRus (Ne4)
KtaRus (N
ICaRus (N-8)
0.1
QPS (queriesis)
QPS (queries/s)
(a) P95 latency across QPS
(b) Throughput across QPS
Figure 4: P95 latency and throughput of ICaRus compared with multiple task aesc-specific agents fine- tuned from the LLaMA-3.1-80 base model under the ReAct pattern. Here, N denotes the number of LoRA modules, which are integrated into multi model system built using either the conventional approach or ICaRus.
10
60
LLMMAS.1 MMMAS.1-00
P96 Latency
LLaMA3 1-88
Through (reals)
7.5
40
40
5.0
20
20
2.5
0.0
100
0.4
0.6
0.8
0.2
0.4
0.6
0.8
Qwan3-14B-BR Base
100
Owen3- 148 Base
Max Throughput (req)s)
P96 p96 Latency
50
50
0.1
0.2
0.3
0.4
0.1
0.2
0.3
0.4
8
2
8
QPS
QP5
Num Agents
Num Agents
IaaRus (N-2)
ICaRus BaseLine (N=4)
ICaRus (N-8)
Beseline (N=2)
BeseLine (N=#8)
BeseLine
Figure 5: Comparison of P95 latency and maximum throughput across QPS for LLaMA3.1-8B and Qwen-3-14B Base under ReAct and Reflexion patterns.
the number of models increases (e.g., at 0.6 QPS for two models and 0.3 QPS for four models; Fig. [b). In contrast, ICaRus avoids redundant cache growth through cross-model KV sharing, allowing throughput to continue increasing even as baseline system plateaus and declines.
Consequently, when comparing maximum achievable throughput, ICaRus outperforms the baseline by 1.4x, 2.3x, and 3.8x with 2, 4, and models, respectively. At the QPS where baseline system reaches its peak throughput, ICaRus also achieves substantially lower P95 latency-3.8x, 5.1 and 11.1x for 2, 4, and 8 models, respectively. Furthermore. we confirm that ICaRus continues to achieve lower latency and higher throughput than the baseline even in scenarios where evicted KV cache entries are managed by swapping rather than recomputation, as detailed in Appendix
Performance under diverse workflows or models. We further evaluate ICaRus system across different models (LLaMA-3.1-885 and Myven3-14BBaasee and different agent patterns (ReAct and Reflexion). Specifically, we measure p95 latency over varying QPS and the maximum throughput achieved the optimal QPS setting, as summarized in Fig. 5
baseline system and
ICaRus prevents KV cache explosion and enables cross-model prefix caching, thereby achieving lower P95 latency and higher throughput in multi agent workflow. These gains persist even for larger models like Qwen3 14BB, where ICaRus achieves up to 7.4x lower latency and 3.6x higher throughput compared to the baseline. Additionally, preserved even under more realistic agentic patterns, where agents are invoked in a random order and the workload is skewed across agents, demonstrated in Appendix F
verify that the advantages of ICaRus

---

Published as a conference paper at ICLR 2026
5 RELATED WORK
Multi model Inference Leveraging multiple models has been widely explored as a way improve performance over a single model. Routing methods either select the most appropriate model or use multiple models cascade (Chen et al. 2024; Shnitzer et al., 2024), while ensemble approaches combine the outputs of multiple models, either at the token level Yu et al. 2024 Huang et al. 2024) or at the reasoning step level (Park ct al. (025). Multi model approaches have also been applied in multi agent systems, where interactions among agents have been shown to enhance performance across diverse tasks (Fu et al., 2023) Sun et al., 2024, Du et al. (024). In these systems, each agent used either base model fine-tuned variants obtained with methods such as LoRA instruction tuning (Mineiro 2024 Liu et al. 2025b).
KV Cache Optimization KV cache stores the keys and values of previous tokens avoid redun- dant recomputation during autoregressive generation and traditionally used on a per-request basis Vaswani et al., 2017). Prefix caching techniques extend the lifetime of the KV cache beyond single request, enabling multiple turns or related requests to share the same cache (Gao et al. 2024 Gim et al. 2024). However, prefix caching alone cannot address the challenge of deploying multiple models, as KV caches cannot be shared across different models even for identical prompts, and each model generates a distinct KV cache. DroidSpeak (Liu al.) (025b) addresses this issue by reusing the KV cache of shared foundational model for non-sensitive layers, while selectively recomputing only the sensitive layers in each agent model. This approach requires identifying sensitive layers that must be recomputed by the agent model, thereby affecting subsequent layers. On different axis, KVFlow (Pan et al. (025) manages KV caches by evicting and prefetching based on predetermined agentic workflows instead of an LRU policy, but it remains single model approach with agents defined by prompts.
6 CONCLUSION
In this work, we presented ICaRus, a KV cache-sharing architecture for multi model inference. ICaRus addresses the memory inefficiency of conventional system by enabling cross-model KV cache reuse, while maintaining accuracy through fine-tuning. Experiments across mathematics, coding, and instruction-following tasks confirm that ICaRus delivers accuracy on par with task- specific fine-tuned models, yet achieves significantly lower latency and higher throughput in multi agent workflow. Taken together, these results establish ICaRus as principled approach for scalable and efficient multi model inference. Looking ahead, we expect ICaRus to extend to large-scale models, heterogeneous agent systems, and real-world deployment scenarios where scalability and efficiency are increasingly critical.
REPRODUCIBILITY STATEMENT
We formulated the concept of the logical encoder and decoder in detail, which forms the foundation of the ICaRus algorithm, in Section 3.1 Furthermore, we provided rigorous mathematical formu- lation of ICaRus, along with its training procedure and convergence of the loss curve, Section The inference process of ICaRus and the corresponding optimization strategies are described in Section [.3] with pseudocode provided in Appendix [B] Finally, the detailed experimental setup for both training and inference is presented in Section 4.1 and Appendix A
3.2
REFERENCES
Stella Biderman, Hailey Schoelkopf, Lintang Sutawika, Leo Gao, Jonathan Tow, Baber Abbasi, Al- ham Fikri Aji, Pawan Sasanka Ammanamanchi, Sidney Black, Jordan Clive, Anthony DiPofi, Julen Etxaniz, Benjamin Fattori, Jessica Zosa Forde, Charles Foster, Jeffrey Hsu, Mimansa Jaiswal, Wilson Y. Lee, Haonan Li, Charles Lovering, Niklas Muennighoff, Ellie Pavlick, Ja- son Phang, Aviya Skowron, Samson Tan. Xiangru Tang, Kevin A. Wang, Genta Indra Winata, François Yvon, and Andy Zou. Lessons from the trenches on reproducible evaluation of language models. arXiv preprint arXiv:2405. 14782, 2024.
10

---

Published as a conference paper at ICLR 2026
Lingjiao Chen, Matei Zaharia, and James Zou. FrugalGPT: How to use large language models while reducing cost and improving performance. Transactions on Machine Learning Research, 2024.
ISSN 2835-8856.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained code. arXiv preprint arXiv:2107.03377 2021.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani. Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pel- lat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Y. Zhao, Yanping Huang, Andrew M. Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. Scaling instruction finettund language models. J. Mach. Learn. Res., 25:70:1-70:55: 2024.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10.4468,,,, 2021.
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit S. Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, Luke Marris, Sam Petulla, Colin Gaffney, Asaf Aharoni, Nathan Lintz, Tiago Cardal Pais, Henrik Jacobsson, Idan Szpektor, Nan- Jiang Jiang, Krishna Haridasan, Ahmed Omran, Nikunj Saunshi, Dara Bahri, Gaurav Mishra, Eric Chu, Toby Boyd, Brad Hekman, Aaron Parisi, Chaoyi Zhang, Kornraphop Kawintiranon, Tania Bedrax-Weiss, Oliver Wang, Ya Xu, Ollie Purkiss, Uri Mendlovie, Iaii Deutel, Nam Nguyen, Adam Langley, Flip Korn, Lucia Rossazza, Alexandre Ramé, Sagar Waghmare, Helen Miller, Nathan Byrd, Ashrith Sheshan, Raia Hadsell Sangnie Bhardwaj, Pawel Janus, Tero Rissa, Dan Horgan, Sharon Silver, Ayzaan Wahid, Sergey Brin, Yves Raimond, Klemen Kloboves, Cindy Wang, Nitesh Bharadwaj Gundavarapu, Ilia Shumailov, Bo Wang. Mantas Pajarskas, Joe Hey- ward, Martin Nikoltchev, Maciej Kula, Hao Zhou, Zachary Garrett, Sushant Kafle, Sercan Arik, Ankita Goel, Mingyao Yang, Jiho Park, Koji Kojima, Parsa Mahmoudieh, Koray Kavukcuoglu, Grace Chen, Doug Fritz, Anton Bulyenov, Sudeshna Roy, Dimitris Paparas, Hadar Shemtov, Bo- Juen Chen, Robin Strudel, David Reitter, Aurko Roy, Andrey Vlasov, Changwan Ryu, Chas Le- ichner, Haichuan Yang, Zelda Mariet, Denis Vnukov, Tim Sohn, Amy Stuart, Wei Liang, Minmin Chen, Praynaa Rawlani, Christy Koh, JD Co-Reyes, Guangda Lai, Praseem Banzal, Dimitrios Vytiniotis, Jieru Mei, and Mu Cai. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. CoRR, abs/2507.06261, 2025. doi:
Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. Improving fac- tuality and reasoning in language models through multiagent debate. In Forty-first International Conference Machine Learning, 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, and et al. The llama 3 herd of models. CORR, abs/2407.21783. 2024. doi: 10.48850/ARX112247.22222
Yao Fu, Hao Peng, Tushar Khot, and Mirella Lapata. Improving language model negotiation with self-play and in-context learning from ai feedback, 2023.
Bin Gao, Zhuomin He, Puru Sharma, Qingxuan Kang, Djordje Jevdjic, Junbo Deng, Xingkun Yang, Zhou Yu, and Pengfei Zuo. (fft--Effiieett large language model serving for multi-turn con- In 2024 USENIX Annual Technical Conference (USENIX
versations with { ATC 24), pp. 111-126. 2024.
In Gim, Guojun Chen. Seung-scob Lee, Nikhil Sarda, Anurag Khandelwal, and Lin Zhong. Prompt cache: Modular attention reuse for low-latency inference. In P. Gibbons, G. Pekhimenko, and C. De Sa (eds.), Proceedings of Machine Learning and Systems, volume pp. 325-338, 2024.
Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh. Michael W. Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami. Kvquant: Towards 10 million context length LLM inference with KV cache quantization. In Advances in Neural Information Processing Systems 38, 2024.

---

Published as a conference paper at ICLR 2026
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In The Tenth Interna- tional Conference on Learning Representations. OpenReview.net, 2022.
Yichong Huang, Xiaocheng Feng, Baohang Li, Yang Xiang, Hui Wang, Ting Liu, and Bing Qin. Ensemble learning for heterogeneous large language models with deep parallel collaboration. In The eighty- Anmual Conference on Neural Information Processing Systems, 2024.
Ting Jiang, Shaohan Huang, Shengyue Luo, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang, Deqing Wang, and Fuzhen Zhuang. Mora: High-rank updating for parameter-efficieet fine-tuning. CoRR, abs/2405, 125/24 2024. doi: 10.48550/ARXIV2405.. 12130.
Jiin Kim, Byeongjun Shin, Jinha Chung, and Minsoo Rhu. The cost of dynamic reasoning: Demystifying Al agents and test-time scaling from an Al infrastructure perspective. CoRR, abs/2506.04301 2025. doi: 10.48500ARXXV..556604523
Sehoon Kim, Suhong Moon, Ryan Tabrizi, Nicholas Lee, Michael W. Mahoney, Kurt Keutzer, and Amir Gholami. An LLM compiler for parallel function calling, In Forty-first International Con- ference on Machine Learning. OpenReview.net, 2024.
Andreas Köpf, Yannic Kilcher, Dimitri Von Rütte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Due Nguyen, Oliver Stanley, Richárd Nagyfi, al. Openassistant conversattios-ddmoraatiiii large language model alignment. Advances in neural information processing systems, 36:47669-47681 2023.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention, In Proceedings of the 29th Symposium on Operating Systems Prin- ciples, pp. 61 1-66..6. ACM, 2023, doi: 10. 00.1145/3600006361333
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retreeval-augmentee generation for knowledge-intensive NLP tasks. In Advances in Neural Information Processing Systems 33, 2020.
Qintong Li, Leyang Cui, Xueliang Zhao, Lingpeng Kong, and Wei Bi. Gsm-plus: A comprehensive benchmark for evaluating the robustness of lims as mathematical problem solvers. arXiv preprint wrXi::2402.19255 2024.
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chat- gpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems, 36:21558 2222522211572 2023.
Shih- Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu- Yu-Chiang Frank Wang, Kwang- Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposse low-rank adaptation. In Forty-first International Conference on Machine Learning. OpenReview.net, 2024a.
Weiwen Liu, Xu Huang, Xingshan Zeng, xinlong hao, Shuai Yu, Dexun Li, Shuai Wang, Weinan Gan, Zhengying Liu, Yuanqing Yu, Zezhong WANG, Yuxian Wang. Wu Ning, Yutai Hou, Bin Wang, Chuhan Wu, Wang Xinzhi, Yong Liu, Yasheng Wang, Duyu Tang, Dandan Tu, Lifeng Shang, Xin Jiang, Ruiming Tang, Defu Lian, Qun Liu, and Enhong Chen. ToolACE: Winning the points of LLM function calling. In The Thirteenth International Conference on Learning Representations, 2025a.
Yuhan Liu, Esha Choukse, Shan Lu, Junchen Jiang, and Madan Musuvathi. Droidspeak: Enhancing cross-llm communication. CoRR, 0ix/22111.22800 2024b. doi: 10.48550/ARXIV2411.000
Yuhan Liu, Yuyang Huang, Jiayi Yao, Shaoting Feng, Zhuohan Gu, Kuntai Du, Hanchen Li, Yihua Cheng, Junchen Jiang. Shan Lu, Madan Musuvathi, and Esha Choukse. Droidspeak: Kv cache sharing for cross-llm communication and multi-llm serving, 2025b.
Paul Mineiro. Online joint fine-tuning of multi-agent flows, 2024.
12

---

Published as a conference paper at ICLR 2026
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems 35, 2022.
Zaifeng Pan, Ajjkumar Patel, Zhengding Hu, Yipeng Shen, Yue Guan, Wan-Lu Li, Lianhui Qin, Yida Wang, and Yufei Ding. Kvflow: Efficient prefix caching for accelerating Ilm-based multi- agent workflows. CoRR, abs/2507.0740M 2025. doi: 10.48550/ARXIV.5500.000
Sungjin Park, Xiao Liu, Yeyun Gong, and Edward Choi. Ensembling large language models with process reward- eeward-guided tree search for better complex reasoning. Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Proceedings of the 2025 Conference of the Nations of the Americas Chap- ter of the Association for Computational Linguistics: Human Language Technologies ( (Volume I: Long Papers), pp. 10256-10277, Albuquerque, New Mexico, April 2025. Association for Com- putational Linguistics. ISBN 979-8-891 119999166-9999 189-6
doi: N..18653/v1/202555 http5/////2223.maaloooo....5..5 ma
Aurick Qiao, Zhewei Yao, Samyam Rajbhandari, and Yuxiong He. Swiftkv: Fast prefill-optimizz inference with wwwp:wlldge--reeeeviii model transformation. CoRR, abs/2410.03960. 2024. doi: 10.45500ARX1.22100009966
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang. Julien Di- rani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a bench- mark. In First Conference on Language Modeling, 2024.
Nick Roshdieh. Exol-instruut-code-BBk 2023, Hugging Face dataset.
John Schulman and Thinking Machines Lab. Lora without regret, 2025. Blog post.
Weizhou Shen, Chenliang Li, Hongzhan Chen, Ming Yan, Xiaojun Quan, Hehong Chen, Ji Zhang, and Fei Huang. Small lims are weak tool learners: A multi-llm agent. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 16658 16888. Associa- tion for Computational Linguistics, 2024. doi: 10. 1/22://11/2224.EMMLPPPPPPP
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflex- ion: language agents with verbal reinforcement learning. In Advances Neural Information Processing Systems 36, 2023.
Tal Shnitzer, Anthony Ou, Mírian Silva, Kate Soule, Yuekai Sun, Justin Solomon, Neil Thomp- son, and Mikhail Yurochkin. Large language model routing with benchmark datasets. In First Conference on Language Modeling. 2024.
Vighnesh Subramaniam, Yilun Du, Joshua B. Tenenbaum, Antonio Torralba, Shuang Li, and Igor Mordatch. Multiagent finetuning: Self improvement with diverse reasoning chains. In The Thir- teenth International Conference on Learning Representations. OpenReview.net, 2025.
Qiushi Sun, Zhangyue Yin, Xiang Li, Zhiyong Wu, Xipeng Qiu, and Lingpeng Kong. Corex: Push- ing the boundaries of complex reasoning through multi-model collaboration. In First Conference on Language Modeling, 2024.
Hongyan Tang, Junning Liu, Ming Zhao, and Xudong Gong. Progressive layered extraction (PLE): A novel multi-task learning (MTL) model for personalized recommendations. In RecSys 2020: Fourleenth ACM Conference on Recommender Systems, pp. 269-278. ACM, 2020. doi: 10. 145/ 3383313.3412236,
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances Neural Infor- mation Processing Systems 30, pp. 5998-6008, 2017.
Sunghyeon Woo, Sol Namkung, Sunwoo Lee, Inho Jeong, Beomseok Kim, and Dongsuk Jeon. Paca: Partial connection adaptation for efficient fine-tuning. In The Thirteenth International Conference Learning Representations. OpenReview.net, 2025.
13

---

Published as a conference paper at ICLR 2026
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang. Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin. Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yugiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025.
June Yong Yang. Byeongwook Kim, Jeongin Bac, Beomseok Kwon, Gunho Park, Eunho Yang, Se Jung Kwon, and Dongsoo Lee. No token left behind: Reliable KV cache compression via importance-aware aware mixed precision quantization. CoRR, abs/2402. 18096, 2024. doi: 10.48550/ ARXIV.2402.1809.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2369 2380, Brussels, Belgium, 2018. Association for Computational Linguistics. doi: 10.18653/v1/DIs 1259.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations. OpenReview.net, 2023.
Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James Kwok, Zhen- guo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309. 12284, 2023.
Yao-Ching Yu, Chun Chih Kuo, Ye Ziqi, Chang Yucheng, and Yueh-Se Li. Breaking the ceiling of the LLM community by treating token generation as classification for ensembling. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Com- putational Linguisties: EMNLP 2024, pp. 1826-1839, Miami, Florida, USA, November 2024. Association for Computational Linguisties. doi: 18.18633/v 11865//w/2024..... 1e2p:22..ininng-eemmete
Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark W. Barrett, Zhangyang Wang, and Beidi Chen. H20: heavy-hitter oracle for efficient generative inference of large language models. In Advances in Neural Information Processing Systems 36. 2023.
Justin Zhao, Timothy Wang, Wael Abid, Geoffrey Angus, Arnav Garg, Jeffery Kinnison, Alex Sher- stinsky, Piero Molino, Travis Addair, and Devvret Rishi. Lora land: 310 fine-tuned Ilms that rival gpt-4, A technical report. CoRR, abs/2405.00732, 2024. doi: 1044550/AXXI..445.00777
Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, lon Stoica, Joseph E. Gonzalez, Clark W. Barrett, and Ying Sheng. Sglang: Efficient execution of structured language model programs. In Advances in Neural Information Processing Systems 38, 2024.
Andy Zhou, Kai Yan, Michal
Haohan Wang, and Yu-Xiong Wang. Lan guage agent tree search unifies reasoning. acting, and planning language models. In Forty-first International Conference on Machine Learning. OpenReview.net, 2024.
14

---

Published as a conference paper at ICLR 2026
APPENDICES
A EXPERIMENTAL SETUP
A.1 TRAINING SETUP
All experiments were conducted on single node with 8XNVIDIA A100 GPUs (80GB each). Each GPU processed a micro-batch of size I, and we applied gradient accumulation over 16 steps, result- ing in an effective batch size of I 128 examples across all devices. This corresponds to approximately 13lk tokens per optimization step when the maximum sequence length was 1024, and 262k tokens when it was 2048.
We trained on three datasets: MetaMathQA (40k sampled examples), Evol-Instruct (80k full set), and OASSTI (10k sampled examples). The maximum sequence length set to 2048 for Evol- Instruct and 1024 for the others. The number of training epochs was for MetaMathQA and Evol- Instruct, and 3 for OASSTI.
Optimization was performed using the AdamW optimizer with default hyperparameters (3) =0.9, 82=0.999) and weight decay of 0.01. We used cosine learning rate decay schedule with warmup ratio of 0.03, and performed a grid search over learning rates (1 x 10-1, 2 x 10-1, x 10-1). No additional regularization techniques (e.g. dropout or gradient clipping) were applied.
For all experiments, we applied low-rank adaptation (LoRA) with rank of 128 and an a of 256.
A.2
MULTI MODEL INFERENCE SETUP
A.2.1 AGENT WORKFLOW SELECTION AND DESIGN
We designed our experimental setup to evaluate the scalability and performance characteristics of multi model Al agent systems under realistic workload conditions. For this study, we selected two representative agentic patterns that reflect common production use cases and exhibit distinct reasoning behaviors:
ReAct Yao et al. 2023): This framework synergizes chain-of-thought reasoning with external tool use through an iterative process where agents generate reasoning traces and task-specific ac- tions in interleaved manner. In the ReAct paradigm, agents alternate between internal reason- ing (thoughts) and external actions (tool calls), with each iteration consisting of a thought-action observation cycle. This pattern is particularly effective for tasks requiring dynamic interaction with external knowledge bases and APIs
Reflexion (Shinn et al.) (023): This framework reinforces language agents through linguistic feed- back, maintaining reflective text in an episodic memory buffer to improve decision-making across multiple trials. Unlike ReAct, Reflexion adds self-evaluation capabilities where agents generate ver- bal reinforcement cues to assist in inerovmtpoveent,, storing these experiences in long-term memory for rapid adaptation. This approach enables agents to learn from past mistakes without requiring model fine-tuning, achieving superior performance on complex reasoning tasks.
A.2.2 MULTI MODEL. ARCHITECTURE WITH LORA ADAPTERS
To simulate realistic multi-tenant agent deployments, we implemented multi model inference setup where each agent instance operates with its own Low-Rank Adaptation (LoRA) adapter. This con- figuration mirrors production scenarios where different agents may require specialized model be- haviors domain-specific fine-tuning, Specifically, we matched the number of concurrent agents to the number LoRA adapters, ensuring that each agent maintains its own parameter space.
In evaluation, multiple task-specific LoRA adapters share the same base model on single GPU. Under this setup. both the baseline multi-LoRA system and ICaRus already leverage the standard prefix/K mechanisms of the serving stack: requests routed to the same LoRA module reuse the existing KV cache for identical prefixes whenever possible, thereby sharing KV-cache memory and avoiding redundant prefill recomputation within each model.
15

---

Published as a conference paper at ICLR 2026
A.2.3 WORKLOAD CHARACTERIZATION
For workload modeling, used the HotPolQA dataset (Yang et al., 2018) the underlying question-answerie benchmark for both ReAct and Reflexion workflows, following the setup of Kim et al. )225). Input/output distributions and tool-calling patterns were based on empirical mea- surements from Kim et al. (025), which provides comprehensive statistics on real-world agent workflow characteristics. These patterns informed our synthetic workload generation, ensuring our experiments reflect actual deployment scenarios.
A.2.4 EXPERIMENTAL PARAMETERS
We conducted systematic scaling experiments with the following configuration:
Agent Scaling: We evaluated system behavior with and 8 concurrent agents to understand how resource contention and memory pressure evolve with increasing agent density.
Request Rate (QPS):
For Qwen3-14B-Ban Tested 0.1, 0.2, 0.3, and 0.4 QPS
For Llama--.1-8B: Tested at 0.2, 0.4, 0.6, and 0.8 QPS
The different QPS ranges reflect the computational differences between model sizes, with the smaller 8B model capable of sustaining higher request rates.
differrnnces
Throughput Measurement: We measured actual system throughput at the 0.8 QPS configuration to empirically determine system saturation points under peak load conditions.
Batch Size and Latency Dynamics: To understand latency behavior under constrained conditions, fixed the total request count at 128 while varying QPS. This experimental design differs from unbounded request streams where continuously arriving requests would cause monotonically in- creasing batch sizes and consequently unbounded growth in 95th percentile latency. Under our fixed-request protocol, we observed that 95th percentile latency initially increases with QPS but eventually saturates at plateau, indicating the system reaches steady-state where requests are being processed within the available compute budget.
This saturation behavior provides critical insights into:
The maximum sustainable batch size for each agent configuration
The point which additional request rate increases no longer impact tail latency
The effective capacity limits of multi agent systems under resource constraints
A.2.5 RATIONALE AND IMPLICATIONS
Our experimental design captures several critical aspects of production multi agent systems:
1. Resource Isolation: By assigning separate LoRA adapters to each agent, we model sce- narios where agents require distinct specializations (e.g., different domains, languages, or task-specific fine-tuning).
2. Memory Pressure: The multiplicative effect of agent count on KV cache requirements reflects real-world memory bottlenecks in multi-tenant deployments.
3. Workflow Diversity: The combination of ReAct's tool-calling patterns and Reflexion's 'ss self-improvement cycles represents a broad spectrum of agent behavioral patterns, from reactive tool use to iterative refinement.
4. Scaling Characteristics: Our range of agent counts (2-8) and QPS values provides in- sights into both vertical scaling (request rate) and horizontal scaling (agent parallelism) dimensions.
This setup enables to quantify the trade-offs between agent autonomy, system throughput, and resource utilization in modern agent deployments, providing actionable insights for practitioners deploying multi agent systems scale.
16

---

Published as a conference paper at ICLR 2026
B PSEUDO ALGORITHM
B.1 PREFILL PHASE IN ICARUS
Algorithm 1: Prefill Phase (Standard Linear Only)
Input: Prompt tokens P E V
Output: First token Mprreu X1 t Embed(P) RNxd
V. V.KV_CACH[[1.. L]
for i to L do
Qi < Lin ("; K; t- Lincar(X,, Wj), V32 + Lincar(X,; W;)
Qi, Ki E NNdx, Vi E Ngd
** generate KV cache (w. the Logical Encoder)
KV_CACHE[i] + (K, Vi)
A, AttentionnQ K,, Vi) E Rxd,
X +- FFNAAtteeniionOuupuu(AA (A,)) E Nxd
Prrefill + Sample(LMHeex((X441111)
// Prefill Result
17

---

Published as a conference paper at ICLR 2026
B.2 DECODE PHASE IN ICARUS
Algorithm 2: ICaRus Linear
Input: X E 2xxxx
// batch=2, seqlen T, hidden size d
X[O]: Input for Logical Encoder (Base model)
X[I]: Input for Logical Decoder (Base model Adaptive model)
Output: Y 2xxxx
/* Parallel execution for Base Model and Adaptive Model
#/
Xump f- Linear(X)
X Meempl [1 f- X Xeemp [] + alaptiveLinear(Xeem)
Xremp
Algorithm 3: Decode Phase (w. ICaRus Linear)
Input: Prreiu E V. KV_ KV_CACHE(1... L] CACHE::: Prompt KV cache from Logical Encoder (Base Model)
Output: Generated tokens Y = (1N+1, -N (N+T)
(where N the prompt length, T is the number of generated tokens)
Input Jnput_Toker Token < 3/prefill
for f 1... do
X1 < Emneed(nppt__Tonn Token) E Nxd
/* Stack hidden states for ICaRus Execution
*/
Xpair +- stack baack_batch((1, X1)
// shape: [2,1,d]
for i to do
/* KV cache from base model for sharing Ketep +- LLinerrXX W Vatep <- Linear (inear(X;; W (Kgache V,eache) CVCHA(c
K; - concat_scqueett Keachh Katep)
10
V; < concal_sequencee cacche ystop)
11
KV []_CACHE[i] (K,, Vi)
13
Q pair < ICaausLinear( X pair W Ag)
// shape: 12,1, d_k]
14
/* Enable attention parallelism via GQA
*/
Q concat_nnmmern nuwnead [0], aaair
(1]]
// shape: [1,2HH,d_k)
16
A + GQA(Q,, K,, Vi)
// shape:
[1,24H,d_V
17
A ppair + transpose and reshape(Ai)
shape:
1,,1,
18
Zpair + ICaaRusLinear(AA pair W A:,A))
// shape: [2,
/* FFN: up act - down (W.IIaRusLinear)
*/
19
ppair < FFN(2.air)
// shape: [2,1,d]
/* use only Adaptive Result
*/
new_loken +- Sanple(LMMeddd(PTii)
Y t concat(Y, new new_kok Input Token +- new_token
22
18

---

Published as a conference paper at ICLR 2026
C
LOGICAL ENCODER-DECODEER CONCEPT AND INFERENCE WORKELOW
Xi+1
K1:t=11 11--1
Logical encoder
Kid-1i Viu-1
Logical encoder
K2 V1
Kid Vna
Logical decoder
x,,
X2,
X
X
X
< Prefill Phase
< Decode Phase>
Figure 6: Inference workflow of the logical encoder-decoder.
In this section, we provide a more detailed explanation of the logical encoder-decoder concept. Inference in decoder-only Transformer can be viewed as consisting of two phases: a prefill phase and decode phase.
Prefill: generate the KV cache for the input prompt.
Decode: (1) generate the KV cache for the current token, and (2) predict the next token.
Motivated by this behavior, we conceptually decompose the model into logical encoder and logical decoder. The logical encoder denotes the part of the computation that is solely responsible for producing the KV cache, whereas the logical decoder denotes the part that predicts the next token during decoding and does not produce any new KV entries: treats the KV cache as pre- computed sequence representation and only issues queries against generate tokens. Under this decomposition, inference can be reinterpreted as follows:
Prefill: the logical encoder generates the KV cache for the input prompt.
Decode: (1) the logical encoder generates the KV cache for the current token, and (2) the logical decoder predicts the next token.
ICaRus fine-tunes only the logical decoder and freeze logical encoder. Specifically, the task- specialized decoders consume the shared KV cache from the common logical encoder for attention computation, shown in Fig. [6] enabling heterogeneous, task-specialized decoders to operate on single shared representation without any approximation or recomputation. In other words, ICaRus models can reuse KV cache entries produced not only in the prefill phase but also in the decode phase without any updates or reconstruction, because all KV entries are always generated by the same logical encoder.
D ROBUSTNESS OF ICARUS ON TOOL-CALLING TASKS WITH LARGER MODELS
To demonstrate the scalability and robustness of ICaRus, We conducted experiments with Qwen3- 32B on the ToolAce dataset (Liu et al. 2025a) for tool calling related task, and evaluated the resulting models on the BFCL benchmark shown below.
shown Fig. 7 the loss curve of ICaRus converges smoothly and is comparable to that of the baseline, which is consistent with the behavior observed in Fig. of the manuscript for math and coding tasks with 8B models. This indicates that our training procedure remains stable even when scaling larger models and to different task domain.
Moreover, as reported in Table even with a larger 32B model and the tool calling task, ICaRus achieves comparable accuracy than baseline that does not share the KV cache. This suggests that our method not only trainable and stable, but also robust and effective, both in terms of model scale and task type.
19

---

Published as a conference paper at ICLR 2026
ToolACE
1.2
Conventional Fine-Tuning
ICaRus
1.0
0.8
Loss
0.6
0.4
0.2
MMM
100 Training Steps
250
Figure 7: Training loss curves of conventional fine-tuning and ICaRus, both applied with LoRA Qwen-3-32B, trained on the ToolAce dataset.
Table 5: Comparison of conventional fine-tuning and ICaRus when training Qwen3-32B the ToolAce dataset.
BFCL Non-live (AST)
Model
Method
Simple Python
Simple Java
Simple JavaScript
Baseline
96.5
62.0
74.0
Qwen3-32B
ICaRus (Ours)
94.5
63.0
76.0
E ICARUS UNDER SWAP-BASED KV CACHE MANAGEMENT
We conducted experiments with swap enabled (4GB swap space) using an earlier version of vLLM that supports this feature. The experimental results reported below.
Sigure&shows that ICaRus continues to provide lower P95 latency and higher throughput even when the mulfi-model system swap for KV cache management. In particular, with 8 LoRA modules, ICaRus achieves to 12.1x lower P95 latency and 3.8x higher throughput than the baseline. This because ICaRus reduces the KV cache footprint itself, so that even higher QPS the GPU does not saturate and expensive swap operations are rarely triggered in the first place.
In summary, we emphasize that recompute/swap strategies and ICaRus address orthogonal as- pects of the problem. Concretely, recompute swap determine how to manage KV cache once GPU memory becomes full (e.g., whether evict and reload from host storage or to recompute), whereas ICaRus fundamentally reduces KV pressure by enabling cross-model KV sharing across lask-specialized models. By avoiding redundant KV construction across models, ICaRus effectively
Deselin 21-21
Throughput (requestsis
10
Beadine (hwd)
newdre (o-D)
(Calus 20
Katas food)
P95 Latency
25
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
QP5
QPS
(a) P95 latency across QPS
(b) Throughput across QPS
Figure 8: P95 latency and throughput of ICaRus compared with multiple task-specific agents fine- tuned from the LLaMA-3.1-88 base model under the ReAct workflow with swap-based KV cache management. Here, N denotes the number of LoRA modules, which are integrated into multi model system built using either the conventional approach ICaRus.
20

---

Published as a conference paper at ICLR 2026
delays mitigates the point at which the KV cache saturates GPU memory, thereby improving performance regardless whether the underlying system chooses recompute or swap as its eviction policy. In principle, ICaRus could also be combined with wwap-based KV management.
F PERFORMANCE UNDER RANDOM AND SKEWED AGENTIC PATTERN IN REAL-WORLD SCENARIOS
We evaluate the scenario which the controller invokes agents at random with a skewed workload under ReAct workflow, that typical turn only subset of agents is active, better reflecting such real-world scenarios. Specifically, unlike the round-robin invocation pattern in Section 4.3 construct skewed workload in which one agent is invoked with probability 50% on each turn, while the remaining agents share the rest of the probability mass and are invoked in random order rather than a fixed sequence. The experiments are conducted on the vLLM vO architecture and the results are reported below.
125
Boseline (W-21 Boolis (bw4)
Eaealize
Throughput (req/s)
100
Kalko (1-21
8
ICalu (1-21
icamus (hwe)
p95 Latency
ICaPLe a1
75
6
50
25
2
0
0.1
0.2
0.3
0.4
0.5
0.6
QPS
QPS
(a) P95 latency across QPS
(b) Throughput across QPS
Figure 9: P95 latency and throughput of ICaRus compared with multiple task-specific agents fine- tuned from the LLaMA-3. 1-8B base model under the ReAct workflow where the agent invocation pattern is random and skewed. Here, N denotes the number of LoRA modules, which are integrated into multi model system built using either the conventional approach or ICaRus.
Fig. shows that ICaRus maintains low P95 latency and high throughput under dynamic and skewed agentic patterns. For example, with models at 0.4 QPS, ICaRus achieves 15x lower P95 latency and 1.2x higher throughput than the baseline, demonstrating that the core advantage of ICaRus, en- abling per-model prefix caching on top of cross-model KV sharing, is preserved even under skewed and random agent invocation patterns. Furthermore, in the baseline, throughput quickly saturates beyond certain QPS because rapid growth of the KV cache triggers frequent evictions and recom- putations. In contrast, ICaRus allows multiple models to share single KV cache pool, keeping entries within the available GPU memory budget without evietion so that throughput continues to increase with QPS without saturation. As result, in the 8-model setting, ICaRus achieves up to 3.5x higher throughput than the baseline under skewed and dynamic agent invocation patterns.
21


Reviews:
[{"review_id": "oNluIPWreP", "reviewer": "Reviewer_7YdR", "summary": "The core idea of the work is to attempt to re-use the KV for the prefill phase, compared to the traditional LoRA based approach.The work instead finetunes a decoder style model to handle the sub tasks, using a light weight adapter.", "questions": "1. was swap space turned on for the experiments? Can all the KV cache be evicted/loaded back instead of recomputation?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. I think this is an interesting twist on traditional LoRA by choosing to change just the decoding side. \n2. provides a 11x decent speedup over the existing gpu cache system", "weaknesses": "1. I feel like the optimization provided by Icarus, while very impressive, feels a bit too incremental on existing LoRA systems.\n\n2. From Appendix A2.2, the experiments tested the case where each agent has it's own LoRA adapter. Due to arrival patterns and batching, It is unclear the amount of LoRA used per GPU might not be true in practice. \n\n3. The main alternatives the paper lists are either recompute or save. It is possible that swap is a valid strategy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "G0bkZFjFPd", "reviewer": "Reviewer_1HaB", "summary": "* The paper proposes a novel method to reuse KV Caches across different models to reduce redundant prefill operations. The method achieves up to 11.1× lower P95 latency and 3.8× higher throughput in agentic workflow across 8 different models.", "questions": "* In the round-robin evaluation setup, how many different models are used? I missed this detail and would encourage the authors to clarify.\n* In Figure 2, please clarify the insight as the training loss curves look almost identical and I am missing the takeaway.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The method decouples the decoder-only model into a logical encoder and a token predictor, and freezes the encoder for fine-tuning. This makes the fine-tuning faster and more memory efficient. All LLMs also share the same logical encoder.\n* Instead of fine-tuning the entire decoder, lightweight adapters are used for fine-tuning for more efficiency to reduce the overhead of multiple KV accesses.", "weaknesses": "* The paper mentions DroidSpeak as related work, which also performs sharing of the KV Cache across multiple LLMs, but points out that DroidSpeak requires some re-computation of sensitive layers. It would be useful to see a more direct comparison with DroidSpeak as a baseline and compare the cost of fine-tuning adapters in ICaRus vs selective re-computation in DroidSpeak.\n* The evaluation only considers a round-robin routing scheme, which deliberately increases the number of models being used in a session. It would be useful to see the evaluation under prefix-aware or KV-aware routing. Do the gains still hold in terms of the latency and throughput if reuse across models is not required?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "czv7OKtjLr", "reviewer": "Reviewer_eTzX", "summary": "ICaRus introduces “Identical Cache Reuse” to let multiple task-specialized LLMs share the same KV cache for identical prompts, avoiding redundant memory use and recomputation in multi-model inference. It splits a decoder-only Transformer into a frozen logical encoder (that builds KV caches) and a fine-tuned logical decoder (that predicts tokens), so all models reuse one shared cache. Using lightweight adapters (e.g., LoRA), ICaRus achieves similar accuracy to task-specific fine-tuning but cuts P95 latency by up to 11× and boosts throughput up to 3.8× in multi-agent workflows.", "questions": "1. How does the work deal with cases when agent finishes prefill and does some decoding? How to reuse the decode token for that part? Or how does system perform if that part is not used? Especially in long decoding scenarios like coding.\n\n2. How does the accuracy compare with not sharing at all? A simpler workflow without multi-model inference. Not sure if the dataset needs multi-model inference.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. Innovative idea that changes how multi-model share KV cache.\n2. Demonstrates good potential system improvement over previous methods by the new design.\n3. Promising path to efficient multi-model inference.", "weaknesses": "1. Unclear accuracy comparison to workflows without any sharing or naive baselines. How does a base model perform if I do not use the new workflow?\n2. Training seems to be more complex. How robust is this training method? It will be more convincing to do more experiments on bigger models and more datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Pag5O3Kk8z", "reviewer": "Reviewer_QNML", "summary": "This paper introduces ICaRus, a novel architecture designed to optimize multi-model inference by enabling full Key-Value (KV) cache sharing across decoder-only Transformer models. The core innovation lies in conceptually decomposing a Transformer into a logical encoder (which generates KV caches) and a logical decoder (which predicts output tokens). By freezing the encoder and fine-tuning only the decoder using lightweight adapters (e.g., LoRA), ICaRus allows multiple specialized models to reuse identical KV caches, significantly reducing memory usage and recomputation overhead. The authors demonstrate that ICaRus achieves comparable accuracy to task-specific fine-tuned models while delivering up to 11.1× lower P95 latency and 3.8× higher throughput in multi-agent workflows.", "questions": "1. In Figure 3, does the decode phase update the **shared** KV Cache? If yes, does another role-specific decoder directly reuse the updated KV or it always requires the logical encoder (Figure 3(a)) to recompute the KV cache?\n\n2. Is ICaRus coupled with LoRA finetuning? Or it is compatible with other finetuning approaches (e.g., full parameter finetuning or prefix tuning)? Which adaptations are needed for other finetuning approaches?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The logical encoder-decoder decomposition is an interesting and insightful abstraction that enables cross-model KV cache reuse, a previously unsolved challenge.  \n* ICaRus shows substantial improvements in latency and throughput, especially in multi-agent workflows like ReAct and Reflexion, while surprisingly matches or even exceeds the performance of fully fine-tuned models in several benchmarks.", "weaknesses": "* Limited scope of sharing. The proposed approach assumes problems are solved with multiple task-specialized models that **have different weights** and **are finetuned from the same base model**, such an assumption limits its applications. Is such a setting common? To my best knowledge, multi-agent systems typically use the same model for all roles, with only the system prompts customized for each.\n\n* Synthetic evaluation setup. The experimental setting for multi-model inference (Lines 318–320) relies on a synthesized workload that routes multi-turn requests in a round-robin fashion to different models. While this setup enables controlled benchmarking, it may not accurately reflect the dynamic and heterogeneous nature of real-world multi-agent systems, where agent interactions are often asynchronous, task-dependent, and influenced by external stimuli. This raises concerns about the generalizability of the reported latency and throughput improvements.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)