Cartridges: Lightweight and general-purpose long context representations via self-study
Authors: Sabri Eyuboglu
Ryan Saul Ehrlich
Simran Arora
Neel Guha
Dylan Zinsley
Emily Ruoyu Liu
Atri Rudra
James Zou
Azalia Mirhoseini
Christopher Re
Venue: ICLR 2026 Poster (2026)
Keywords: test-time training
self-distillation
Primary area: foundation or frontier models, including LLMs
TL;DR: We show how to use self-distillation to reduce long context memory consumption.

Abstract: Large language models are often used to answer queries grounded in large text corpora (e.g. codebases, legal documents, or chat histories) by placing the entire corpus in the context window and leveraging in-context learning (ICL). Although current models support contexts of 100K-10M tokens, this setup is costly to serve because the memory consumption of the KV cache scales with input length. We explore an alternative: training a smaller KV cache offline on each corpus. At inference time, we load this trained KV cache, which we call a Cartridge, and decode a response. Critically, the cost of training a Cartridge can be amortized across all the queries referencing the same corpus. However, we find that the naive approach of training the Cartridge with next-token prediction on the corpus is not competitive with ICL. Instead, we propose self-study, a training recipe in which we generate synthetic conversations about the corpus and train the Cartridge with a context-distillation objective. We find that Cartridges trained with self-study replicate the functionality of ICL, while being significantly cheaper to serve. On challenging long-context benchmarks, Cartridges trained with self-study match ICL performance while using 38.6x less memory and enabling 26.4x higher throughput. Self-study also extends the model's effective context length (e.g. from 128k to 484k tokens on MTOB) and surprisingly, leads to Cartridges that can be composed at inference time without retraining.

CARTRIDGES: LIGHTWEIGHT AND GENERAL-PURPOSS LONG CONTEXT REPRESENTATIONS VIA SELF-STUDY
Sabri Eyuboglu' 1+
Ryan Ehrlich
Simran Arora
Neel Guha' Christopher Ré
Dylan Zinsley²
Emily Liu'
Atri Rudra² 3
James Zou
Azalia Mirhoseini"
Stanford University Caltech
University at Buffalo
*Equal contribution
ABSTRACT
Large language models are often used to answer queries grounded in large text corpora (e.g. codebases, legal documents, or chat histories) by placing the entire corpus in the context window and leveraging in-context learning (ICL). Although current models support contexts of (00K-10M tokens, this setup is costly to because the memory consumption of the KV cache scales with input length. We explore alternative: training a smaller KV cache offline each corpus. At inference time, we load this trained K VVccache, which we call CARTRIDGE, and decode response. Critically, the cost of training CARTRIDGE can be amortized across all the queries referencing the same corpus. However, we find that the naive approach of training the CARTRIDGE with next-token prediction on the corpus is not competitive with ICL. Instead, we propose SELF-STUDY, a training recipe in which we generate synthetic conversations about the corpus and train the CARTRIDGE with a context-distillation objective. We find that CARTRIDGES trained with SELFF STUDY replicate the functionality of ICI., while being significantly cheaper to serve. On challenging long- mongccotext benchmarks, CARTRIDGES trained with SELF- STUDY match ICL performance while using 38.6 6x less memory on average and enabling 26.4x higher throughput. SELF-STUDY also extends the model's effective context length (e.g. from 128k to 484k tokens on MTOB) and surprisingly, leads to CARTRIDGES that can be composed inference time without retraining.
INTRODUCTION
Large language model (LLM) users often place large text corpora into the context window. For instance, a user or organization may use LLMs to understand a codebase (Nam et al.) (024). financial document (Islam et al.] 2023), legal texts (Guha al. 2023). textbook (Ouellette et al 2025).
or personal files (Arora & Ré 2022). LLMs excel here due to in-context learning (ICL), enabling accurate responses diverse queries (e.g., questions, summarization, reasoning) (Dong et al. 2022)
Despite its flexibility, this usage paradigm is costly to serve. ICL requires maintaining a KV cache that grows linearly with the input length. For example, LLaMA 70B needs 84 GB of memory (at 16-bit precision) to answer a single question over I 128k-token context (Dubey et al. 2024). This severely limits user throughput: on a single H100 GPU, LLaMA 8B's peak throughput (tokens/s) drops by 77x when increasing the context from 1k to 120k tokens (Figure 2
Prior work has thus explored ways to reduce KV cache memory usage. For instance, prompt compression methods reduce the number of tokens stored in the cache via summarization, or self- information filtering (Diang et al. 2023b Li 2023) Chuang et al. [024). while KV cache compression techniques directly compress the stored key-value pairs (Ge et al. 2023a Zhang al. 2023b Tang ct al., 20244) Oren et al.) 2024). Unfortunately, there are memory-qualln
tradcoffs associated with these methods: in experiments challenging long-context tasks, find that performance degrades rapidly when applying these methods with compression ratios greater than 2x (see Figure B
Motivated by the observation that the cost of preparing KV cache be amortized across many queries that reference the same corpus, we explore a complementary approach based on offline training. Given specific text corpus (e.g. patient's medical record) we freeze the LLM and train

---

Problem Setting
In-context learning
represaenec picdused standard prefit Sapports cxetee High GPU wenery comaurtti
Cartridges
thatis irsiied with nal-dtddy
Document Corpus
Queries
Gestre
Self-study
Cartridge
Les menery
Geawdd senti
Preffl
KEBI
k[4]
A|AI
38.4
LUM Cartridge
Users send many messages groundled in single largo corpus of toxt.
NATI AM
Phe 254 Mer
PITS OAA
Figure Producing CARTRIDGES via self-study. For a given document corpus, we train a CARTRIDGE by distilling the corpus into parameterized KV cache through a process we call SELF-STUDY. At inference time, this CARTRIDGE can be loaded into an LLM, which can then be used to answer diverse queries about the corpus while requiring substantially less memory.
smaller KV cache offline by backpropagating loss into the key and value vectors in a process equivalent to prefix tuning (Li & Liang 2021 Lester et al. (021). We call the trained KV cache representing the corpus "CARTRIDGE." At inference time, we load the trained CARTRIDGE, append the user's messages, and decode. Because users repeatedly reference the same corpora (e.g. SEC filings, codebase, personal files), each CARTRIDGE can be trained once offline and reused. This approach also integrates cleanly with existing inference servers, which are already designed to manage per-user KV caches (Kwon et al., 2023; Zheng et al.] 2024; Juravsky et al. 2025 Ye et al. 2025).
The central challenge of this work lies in training CARTRIDGES that exhibit the generality of ICL. Due to ICL, a standard KV cache is a remarkably general-purposee albeit large, representation of a corpus: single cache can support diverse interactions from answering factual questions to writing poems (Dong et al.] 2022). In contrast, naively training a CARTRIDGE with next-token prediction on the raw corpus yields compact but restricted representations of the corpus. With next-token prediction, we show we can memorize the corpus perfectly using a CARTRIDGE with 107x less memory than the standard K KV-cache. However, the CARTRIDGE not a general-purpose representation - it can only regurgitate the corpus, not answer diverse queries (Figure 2). The challenge is to reduce memory consumption while maintaining generality.
To address this challenge and produce general-purpose and compact CARTRIDGES, we propose an automated method called SELF-STUDY. SELF-STUDY has two steps:
1. Synthetic data generation (Section We generate synthetic training data by prompting the model to quiz itself about the corpus content. Training on the resulting conversations lets us avoid training on the same exact text multiple times and improves generality (see Figure 2). To support corpora that exceed the effective context length of the model, we chunk the corpus when generating synthetic conversations. We also curate set of seed prompts that bias the synthetic conversations towards global reasoning, improving structural awareness (see Figure
Context distillation (Section 4.2): We train on the synthetic conversations using a context- distillation objective (Bhargava et al. 2024] [Sell et al., 2022), which aligns the CARTRIDGE- augmented model's next-token distributions with the distributions of the model with the corpus in context. We find that the context distillation substantially improves the quality of the CARTRIDGES compared to next-token-predictionn (see Figure center).
In summary, given a large corpus of text, our goal to train small virtual KV cache, termed CARTRIDGE, that when used by the model, mimics the conversational behavior of the model with the entire corpus in context. To do this, we generate synthetic conversations and train the CARTRIDGE on them with a context distillation objective
recipe we call SELF-STUDY.
Evaluations. We evaluate CARTRIDGES trained with SELF-STUDE on set of challenging bench- marks that pair a single large text corpus (100k-484k tokens) with diverse set of queries (Islam) et al. 2023; Adams et al. 2024 Tanzer et al. 2023). We make three claims. First, SELF-STUDY expands the qemoiyy frontier- averaged across the benchmarks, CARTRIDGES produced with SELF-STUDY match ICL generality and quality while consuming 38.6x less memory, enabling a 26.4x increase in peak throughput (tokens/s) when serving many users with different corpora. These memory reductions and speedups represent an order of magnitude improvement over sfate-of-the- art cache compression baselines (e.g. DuoAttention (Xiao et al. 2024b)). Second, CARTRIDGES enables context length extrapolation. On the MTOB benchmark (Tanzer et al. 2023), where models must translate from Kalamang, low-resource language, into English, we use SEL.F-TUUDY with LLAMM--
2

---

8B construct small CARTRIDGE from a 484k token textbook. This CARTRIDGE outperforms ICL over the first 130, 000 tokens of the textbook by 11.0 chrF points and matches the ICL performance over curated subset of the textbook. Third, SELF-STUDY also yields CARTRIDGES that can be composed without joint optimization: when we concatenate two CARTRIDGES the model can answer queries requiring knowledge from both (see Figure
Additionally, we ablate the design decisions in SELF-STUDY and CARTRIDGES (Section 5.3 and Appendix A). Notably, we compare CARTRIDGES parameterized as a KV cache (Li & Liang 2021) with CA ATRIEEEES parameterized as a LoRA (Hu 2022) and find that KV cache parameterization performs better on both in-domain and oft-of-domain tasks.
In this work. we demonstrate how we can reduce memory consumption during language model serving by scaling offline training compute. We hope this new axis of scaling will enable new applications that are currently bottlenecked by KV cache memory consumption, like coding agents with full cel-iropys context or long-term memory in chatbots.
PRELIMINARIES
We begin by discussing related work (Section 2.1 formalizing our problem (Section 2.2p, and providing background on language models and KV caches (Section 2.3
2.1 RELATED WORK
See Appendix Boor more comprehensive discussion of prior work.
Parameter Efficient Fine-Tuning and Knowledge Injection In order to adapt language model to specific task or domain, practitioners commonly train small number of parameters (usually a low-rank adapter). which augment or modify the original model (Hu et al. 2022 Li & Liang 2021 Lester al. 202T, Meng et al., 2024 Zaken et al.) 202T). In our work, we build upon less popular technique, prefix-tuning (Li & Liang, 2021 Lester et al. (021), where we optimize internal activations for a set of "virtual" " tokens preceding the input. Recent works on knowledge injection apply LoRA (or variants (Mao et al. 2025)) to store text corpus in a small number of
parameters (Zhang et al. 2023a Xiao et al. [023] Kujanpää et al. 2024| Mao et al.) 2025 Kuratov work, these papers do not focus on memory reductions or throughput improvements enabled by knowledge injection and do identify the
al. 2025 Su al. 2025| Caccia al. 2025). In contrast
importance of the prefix-tuning parameterization.
Prompt and KV-cache compression Many works have proposed techniques to reduce the size of the KV cache. One set of approaches focuses making the prompt smaller explicit methods alter
the prompt text through summarization and filtering (Jiang et al. [023b Li 2023 Chuang et al.
2024, Zhang al.
2024b Pan al. 2024), while implicit methods compress prompt representations
into a set of "soft" tokens (Chevalier et al. [2233 Yen 2024, Ge et al., 2023b Mu et al. 2023 Qin
al. 2023 Lester al. (021). Another of approaches exploits observations about the structure
of the KV cache Yu et a 2024 Chang et al. 2024, Kim et al. 2024) to drop (Ge et al. 2023a
Zhang et al. 2023b) lang et al.) 2024] Oren et al. [0244] Li et al. 2024b) or merge tokens (Wang
al. 2024
Zhang et al. 2024d
Wan et al. 2024).
Architectural changes A large body of work has studied architectural changes to the original multi- head attention operation Vasswni et al. (2017) with the aim of reducing the memory footprint of the KV cache or replacing it with memory object of constant size (inter aliaZaheer et al. [al.](2)))) (2020) Shazeer (2019) Liu et al. (2024a): Gu & Dao (2023); BBehrouz et al. (2024). In Appendix E provide a theoretical analysis comparing CARTRIDGES with linear attention, one such architecture with constant memory footprint. Unlike SELF-STUDY and the compression approaches discussed above, which can be readily applied to any pre-trained Transformer, these architectural changes typically require retraining the model from scratch or using complex architecture conversion techniques (Zhang
et al. 2024a).
2.2
PROBLEM SETUP
We assume a setting in which users issue a stream of diverse queries about a common corpus of text. We denote the corpus C and the query Q {q), G2; -2...... ; )- 9m
For example, C may
3

---

Specifically, we allocate KV cache correspond to the 2022 Form 10-K filing for AMD, which is almost 100k tokens. Analyst might as diverse queries with respect to this filing, including: (1) recalling factual information, (2) performing mathematical reasoning, (3) even generating creative responses (e.g., poem). Other illustrative examples of C. include legal filings, code repositories, chat histories, and medical records.
Let R {ri, :............................... Fm) denote the responses the LLM produces for the queries. We have two objectives. First, we wish to maximize the quality of responses R under some quality metric (e-g- accuracy). Second, we wish minimize the LLM's memory footprint while it is answering questions with respect to the document. This is because larger memory footprints decrease throughput and necessitate more hardware serve the same number of users (Figure Right).
2.3 LANGUAGE MODELS AND KV CACHES
Recall that an LLM accepts as input a sequence of N tokens X V drawn from discrete vocabulary V C Z of tokens, each represented by unique integer. The output, which we denote F(-(x), corresponds to a categorical distribution over vocab V conditioned on the prefix x E V". Inside the language model, each token xr[i] in is embedded into a d-dimensional space, yielding matrix u E IRax The matrix u is passed through a stack of I. model layers, which each mix the matrix along the and d dimensions, with layer l outputting y' E Raxdd The final y is mapped to the logits over V with a linear projection.
Most modern language models use the self-attention operator (Vaswani al. 2017). Given input E [Rexd for sequence length n and embedding dimension d, computes the output y' E Raxd exp(ali] k(t)/ vdvvls]
where weight matrices W, W 1 exp(qls) k(t/vdd) over projections q. k, v uW q: uW ks uW.
via the softmax y [i] Σj=
and W. for each layer learned during training.
We generate text from JF one token a time by sampling from F(. x) and appending the sampled token to Critically, the attention operator is causal: every output y(i] is conditioned on prior tokens. This means we can store the keys and values for the prior tokens in a KV cache (k[ý), v(jl which grows in i. Thus, generation proceeds in two phases: (1) prefill, where cache for the initial prompt x and (2) decode, where we generate the response token by token and append to the cache. The KV cache effectively serves representation of the corpus C.
compute the KV
THE CARTRIDGE PARADIGM
In this section, we describe the CARTRIDGE paradigm, in which we generate representations of the corpus C' offline with training, instead of constructing them on-the-fly with prefill.
3.1 FORMALIZING CARTRIDGES
Our goal is to train CARTRIDGE for given corpus A CARTRIDGE a small set of parameters Z E IR* (i.e. an adapter (Li & Liang 2021 Hu et al. 2022) that augments an LLM F and causes behave as if it had Č in context window. Formally, augmented with Z given query q. For all E Q. we want to ensure that samples ~ F2 (-1g) are as good or better than the ICL sample Tg ~ F(-IC @ q), according to some query-specific scoring function. Because Q might span diverse range of question types (e.g. mathematical reasoning, factual recall comprehension, summarization, and more), it is essential that Fz can generalize across different q Q. This is non-trivee non-trivial because Q is unknown when Z is being learned offline.
Fz(-1g) denote the distribution of F
3.2 PARAMETERIZING CARTRIDGES
We parameterize Z using prefix-tuning & Liang 2021). composed of trainable key and value vectors Zμ, Z, E Foxd The size of the full E RLxpxdx2 is controlled by the hyperparameter p.
In ICL, the KV cache for Fe(g) (where is of length ne and Q is of length no) would contain ne + ni key-value pairs, with the first ne corresponding to C. and the last na corresponding Q:
4

---

Generalization to diverse queries
Quality-momory tradeoff
Poak throughput vs. cache size
113x
There comples the nut of paurge..
2.25
Pook throughput (tokemsts)
104
Data struchuing tasks
46x
Symthesis tmika
2.00
12x
1x
A8dS) IDC.
1.75
100
Creative tasks
sp. Wiite coen AMDD pelimuuiio
1.50
Mettermttical reasoning
1:25
202
121x
38
Disjoitt reasoning
1.00
0.8
11x
1K
Factual
ng. "Wu Auri
of trrgge"
Cache Size (GB)
Weetortattin oc wilred the
Quary
Seff-Study
Cache Size (GB)
(eptperpccst))
Neco tokan prodict
Pull
Figure 2: CARTRIDGES trained with SELF-STUDY balance the generality and memory con- sumption tradeoff. (Left) We evaluate on different slices from the GENCONVO dataset. CAR- TRIDGES trained with next-token prediction performs well on memorization queries, which resemble its training distribution, but cannot generalize to other queries like the other methods. (Center) The z-axis measures the size of the KV cache in GB for the different methods. The y-axis shows log-perplexity the GENCONVO dataset averaged over the query types. (Right) Peak throughput (tokens/s) measured for different cache sizes for LLAMA-3B and LLAMA-8B with SGLang (Zheng al. 2024) on an IxH100 (See Appendix(A)
ICL KV Cache
CARTRIDGE KV Cache
(k1, v1),... (Kner Vnc) ) (kre+1, Vnc+1)...
(2): 21), (Z) zz), (kne+1, Vno+1)
KV pairs for C
KV pains for
Trainable KV pairs in
KV pairs for
To train a CARTRIDGE, we substitute the key-value pairs corresponding to C with Z, and directly optimize them by back-propagating the loss into the key and value vectors. We freeze all model parameters, only training the keys and values in 2. We discuss the choice of loss in Section 4.2
Initialization Prior work finds that optimizing randomly initialized cache Z is unstable and leads to degraded performance (Li & Liang 2021). Instead, these works initialize the trainable cache with a smaller dimensionality and then re-project contrast, we find that proper initialization of 2 allows us to directly optimize the full cache without reparametrization. Specifically, we initialize the corpus C. Alternativel,, we could use a summary of the corpus or filter tokens using off-the-shelf prompt compression strategies (Xiao et al., (024b). Section 5.. we show that our initializations lead to stable training and faster convergence than the random initialization.
the original dimension with an MLP. In
the KV cache corresponding to the first p tokens of
Why this parameterization? We note that the parameter-efficient fine-tuning literature provides other ways to augment LLM with a set of additional parameters, in particular low-rank adaptation (LoRA) (Li & Liang 2021 Hu et al., 2022; Lester et al., [02I). In Section [.3] we perform a comprehensive comparison of CARTRIDGES parameterized with prefix-tuning and LoRA.
3.3 SERVING CARTRIDGES
A CARTRIDGE can be served efficiently with minimal changes to existing LLM inference servers (Zheng et al., 2024 Kwon et al.) 2023 Juravsky et al.) 2025). Because a CARTRIDGE be loaded directly into the KV cache managers of existing inference servers. LLM inference servers are heavily optimized for managing distinct KV-caches for multiple users (Ye) et al. 2025), meaning CARTRIDGES can be served high throughput using existing inference servers. Decoding tokens with a CARTRIDGE identical to serving request with a prefix of length p (the hyperparameter denoting the number of tokens in the CARTRIDGE). This contrasts with other methods like LoRA, which require custom infrastructure to serve efficiently to multiple users (Chen al. 2024a). See Figure for the relationship between prefix length and throughput.
is a KV cache,
5

---

4 SELF-STUDY: A SELF-SUPERVISED METHOD FOR TRAINING CARTRIDGES
In this section, we describe SELF-STUDY, a simple approach for training CARTRIDGE Z on any corpus of text. The design of SELF-STUDY is motivated by the observation that CARTRIDGES trained with a simpler recipe fail to generalize to diverse user queries.
Motivating observation The naive method for constructing a CARTRIDGE would be to fine-tune the parameters of Z with the next token prediction objective on the corpus text directly. We show results experimenting with this approach in Figure 2 where we evaluate on a dataset derived from FinanceBench (Islam et al. 2023). which we refer to GENCONVO (see Appendix D for details). GENCONVO contains multiple types of questions (e.g. synthesis, reasoning). We find that the naïve next-token prediction approach can memorize with near perfect perplexity (Figure left), while consuming 107x less memory than ICL Figure@]center However, generalization to other slices is poor, as shown Figure We seek training objective that allows the responses from model that uses the CARTRIDGE to generalize to a diverse set of user queries, resembling ICL.
Motivated by these observations, describe synthetic data generation recipe in Section 441 Jnnd context- context-distillation objective in Section As we show in Figure CARTRIDGES trained with this approach can generate responses to many types queries that match the quality of queries generated with ICL. See Figure for a visualization of the CARTRIDGE approach.
4.1 SELF-SUPERVISED SYNTHETIC DATA TO AVOID OVERFITTING
To improve CARTRIDGE generality, we propose generating a synthetic training dataset D
Overall synthetic data pipeline Our overall pipeline puts information from the corpus C in context and prompts the model to have a conversation with itself about the corpus to generate the synthetic query-response pairs as shown in Algorithm [
We represent the concatenation with æ / y.
Algorithm 1 SELF-STUDY: Data Generation
Input: C: Corpus, F Model
Output: {aj, bi...., ag. b Convo
1: è + ch)
(1) Get a subcorpus of C that fits in the context window
2: s +
(2) Get a prompt to seed the first message from A
3: for = to k do
(3) Sample conversation with & back and forths
4: a; ~ F(. F(-
c a @ .... .....eb. b
D (3.1) Sample A's message with č and s in context
5:
F: cear ... + b a.)
(3.2) Sample B's message with č in context
6: end for
7: return (ai, b bi...... ak. b
The conversation is generated by iteratively sampling generations from two LLM participants A and (which are the same model). We maintain two different conversation histories: A's starts with a user message containing seed prompt (e.g. "Please start conversation by asking question about the document above. ") followed by alternating assistant and user messages from A and B, respectively. B's conversation history does not include the seed prompt and contains the same messages as A's but with the roles of A and B swapped. Both have the subcorpus in the system prompt. To build training dataset, sample Marin independent conversations and concatenate the messages from A and B into a single sequence of tokens:
D = (x() a b(d)
4 a2
by
a
(1) 4 b
where each x is concatentation of the messages. Note that all of the datasets on which evaluate in the main paper involve a single-turn, So, we set i 1, generating a synthetic conversation with one user message and one assistant message.
Note that the chunk and _eet__sed_____ee__ee_eeeee prompt functions expose two different ways to control the data distribution of the synthetic data. We find that these two design decisions are critical for training high quality CARTRIDGES with SELF-STUDY.
Chunking We use short subcorpora ē (between 512 and 4096) tokens to let the LLM focus on different parts of the corpus when generating data. This is motivated by observations in prior work (Liu

---

H-O- Carsidges
ICL L ICL INU bosk book)
Truncated
@ Sanmary
C Das Attention
Multi-key NIAH
Health
MTOB (KE)
QASPER
Accureyy
000-000
800
8
6
0.50
3P Co)
1
10.50
612
Mernary conmumption (Cacheatee
Merrory conmupptio
|Cache
Cacha
ICL
ICL
0.33
0.49
644
0.55
22
30
330
5EC
3003 S- compute
200
4000
199 190
2001
1000
156
2000
Soll stady compute of coimarggaaees
Sudd computs
Sell-ihuy compare
of skyel
Figure CARTRIDGES can match ICL quality with lower memory costs by scaling SELF-STUDY compute. (Top) We measure response quality (ı-axis) against KV cache memory consumption (I-axis) for different methods, at different KV cache sizes. The dashed line marks the quality of standard ICL. (Bottom) We measure response quality (y-axis) against scale of self-study compute (I-axis). The dashed line marks the quality of ICL and prompt compression baselines.
al. 2024c| Narayan et al. 2025). Furthermore, chunking also allows us train CARTRIDGES on corpora longer than the model's 's context window.
Seed prompts Instead of using, just one seed prompt, we curate a list of five different seed prompt types: structuring, summarization, question, use cases, and creative. The full list of seed prompts used in experiments is provided in Appendix [ Critically, in all our experiments the seed prompts are generic: they do not mention anything related to the specifics of the corpora we evaluated (e.g. mention of translation for MTOB medical terms for LongHealth). We use the same set of seed prompts across all of the experiments. In Section [.3] we ablate the use of diverse seed prompts and find that it improves performance over single generic seed prompt by up to 4.8 accuracy points (43.6 + 48.4 on LONGHEALTI).
4.2 SELF-STUDY CONTEX
OBJECTIVE
Given a fine-tuning dataset D Dimin we adapt standard techniques from the model distillation litera- ture (Kim & Rush 2016 Snell et al. 2022 Kujanpää et al. 2024). We let F(-(x) denote the next token distribution given some input text x. Our teacher is the model with the subcorpus, č, in context F(-(E) and our student is the same model adapted with a trainable cache Fz We use a classic distillation objective (Hinton et al.) 2013) that minimizes the KL-divergence between the teacher and student next-token distributions over a sequence of tokens x and the corresponding subcorpus used to generate them è.
arg min Z
Σ
DKL. F((/0x x|:
F2(+1x(:
(2)
(x.e)
In Appendix A ablate the use of the context-distilatton objective and show that improves accuracy when controlling for the amount of synthetic data (e.g. 3.7 accuracy points on LONGHEALTH).
5 RESULTS
We describe experiments evaluating the effectiveness of CARTRIDGES trained with SELF-STUDY in various long-context scenarios. Our results support the following claims. First, CARTRIDGES trained with SELF-STUDY
match outperform ICL while maintaining generality and reducing serving costs [Section5.I] Second, SELF-StUDE is effective on corpora longer than the context window of the LLM (Section 5.2). Third, the parameterization ablations to assess the relative benefits of
7

---

Cartridge Purametoriaation
Cartridge
Sulf-study Objective
Sulfotudy Seed Prompt
4.44
2046
0.5
3.08
Clap
Da 0.0
40 Corteo-didtation
seed proept
XLL
2 Bankaay mociers
Mestickknn preckdian
propppt
Figure 4: Ablating CARTRIDGE and SELF-STUDY design choices. Here we include ablations for parameterization, initialization, objective, and seed prompts the MTOB or LONGHEALTH datasets (see Appendix ( full ablation experiments on all datasets).
different aspects of SELF-STUDY and CARTRIDGES (Section 5.3) Fourth, when we concatenate two different CARTRIDGES without any joint training, the model can respond to queries requiring information from both CARTRIDGES (Section 5.4)
Datasets We study datasets consisting of diverse (g, r) pairs about a single long document. Across datasets, C ranges between 100k and 484k tokens. Our datasets are drawn from popular long-context benchmarks, with some used as-released and others modified to meet this structure. These include: Multi-key Heyyst- (NIAH) (Hsieh et al., 2024), LONGHEALTH (Adams et al., 2024), MTOB (Tanzrr et al. 2.23), and QASPER (Dasigi et al. [201]) We evaluate LLM response quality using accuracy for NIAH and LONGHEALTH. log perplexity for QASPER, and character n- n-gram f-score (chrF) for MTOB (Tanzer et al. 2023 Popović, 2015). Because each dataset effectively consists of a "single" document, we train a single CARTRIDGE per dataset and evaluate on the queries response pairs (g,r). Appendix D provides further details.
5.1 EXPANDING THE QUALITY-MEOO FRONTIER BY SCALING SELF-STUDY COMPUTE
We assess how CARTRIDGES produced with against baselines on the NIAH, LONGHEALTH and QASPER datasets. For all three datasets, C fits within the model context window (128k tokens). We compare to traditional ICI., two prompt compression baselines (prompt truncation and prompt summarization using GPT-4o (OpenAl 2024 ). and the sate-of-the-aan KV cache compression baseline (Jiang et al. 2023a Xiao et al. (024b)). Please see Appendix A. I for comparisons with other cache compression baselines. We evaluate memory use in terms of KV cache size: the size of the KV cache for the ICL model and prompt compression methods, the size of the CARTRIDGE, and the size of the compressed KV cache for KV cache compression methods like DuoAttention.
fare in quality and memory consumption
The top of Figure 3 presents our main results on LLAMA 3. Compared with ICL, CARTRIDGES offers substantial memory savings at comparable performance: up 13.8x smaller for LONGHEAITH, up to 97.0x for QASPER, and up to 648.3x for NIAH. As Figure 2 (right) shows, these memory reductions translate peak throughput (tokens/s) increases of 11.5x and 76.6x for LONGHEALTH and QASPER, respectively. In contrast, all of the cache compression baseline methods fail to match ICL quality even at modest compression ratios of 2 4x. See Appendix A.2 for results with the QWEN3 family of models, where we observe even larger compression ratios: CARTRIDGES 106.4x smaller outperform full ICL KV caches by 3.8 accuracy points on LONGHEALTH.
These substantial compression ratios are not free lunch. As we show in the bottom of Figure 3 achieving ICL quality at large compression ratios requires spending between two to four orders of magnitude more compute (FLOPs) than we would running prefill with standard ICL. The value of SELF-STUDY, is that it gives practitioners the option to trade off increased offline compute for reduced online memory consumption, which is advantageous in settings where users care about tine-to-fisst first-tokken and latency, users issue many queries over the same corpus, or when we have access to cheap offline compute resources (e.g. night when user load is low (Jaiswal al. 2025 Goel et al. 2025)). Notably, on NIAH, LONGHEALTH, and QASPER, we observe that when we scale compute, performance improves steadily and eventually exceeds ICI. quality.
8

---

Cartridge Composition
Multi- doc Answering
avalted AMD
Papu 12
study
Pepai Cartridge
Tranceeed GI.
39.8 GB
stataments, respectively?
PepurCo and AMD stated eachiOkk"
One Cartridge [B
such the consolidared Aesnnial statementt of AMD was
some
19k
PapuiCo AMD
Self-etudy
AMD Cartridge
Composed Cartridge GB
performed by Brnst& & Young LuP
Unbleaee fin compentedd Red fIAk bhe compectorn
sudit contaliiated linancla statemment Dwistte lauche Lt!
the energy drikk marked AME
LLM Cariiegs
Mathod [Cache Size)
1.75 2,0 ligloeriictt))
competiior
Figure 5: CARTRIDGE Composition. (Left) Illustration of CARTRIDGE composition, where two independently trained CARTRIDGES (one for a Pepsi 10-K and one for an AMD 10-K) are concatenated without any additional training. (Middle) We evaluate compasition on a dataset of multi-document questions requiring information two different PJ 100k token documents with LLAMA-3B (see Appendix D). The x-axis shows log-perplexity (lower is better) on gold-standard answers. We compare CARTRIDGE composition with an (a) ICL baseline where we truncate the document to fit in the 128k token context length and (b) an CARTRIDGE baseline where only include the CARTRIDGE for one of the documents. (Right) Examples of responses to multi-document questions using composed cartridges.
5.2 EXTENDING THE EFFECTIVE CONTEXT WINDOW WITH SELF-STUDY
We evaluate whether SELF-STUDY allows us to accurately process corpora that exceed the context window length. To study this, we consider the MTOB dataset, and LLAMA-8B. which has context window of 128k tokens. MTOB provides two different long documents: a full 484k token latex textbook and shorter 60k token version, which was manually-curated by the dataset authors to exclude content not relevant to the translation task. Even though the 484k textbook is 356k tokens longer than LLAMA-8B context window length, we can produce a CARTRIDGE for the full textbook using the chunking strategy of SELF-STUDY. Figure 3 (middle plot) shows the performance of CARTRIDGES of various sizes trained with SELF-STUDY.
As point of comparison, we provide the results for KV cache baseline methods on the smaller 60k token textbook, and also include ICL on a truncated version of the long textbook. Like above, we observe that CARTRIDGE can match the performance of ICL on the hand-curated 60k token version, while requiring substantially less memory and only having access to the 484k token version, which exceeds the context window of LLAMA-8B CARTRIDGES also outperform competitive baselines at every KV cache size, by up to 11. chrF points.
5.3 ABLATING SELF-STUDY DESIGN CHOICES
We perform ablations to study different aspects of SELF-STUDY and CARTRIDGE parameterization, with full results in Appendix and key findings highlighted in Figure
First, we ablate the parameterization and initialization of CARTRIDGES. We find that the prefix-tuning parameterization substantially outperforms LoRA: on MTOB with CARTRIDGES z 0.6 GB, prefix- tuning achieves 4.5 ChRF points higher performance. More importantly, prefix-tuning maintains generalization to unrelated queries (MMI.U accuracy drops only from 54.7 to 54.3 as CARTRIDGE size increases from 0.15 GB to 0.96 GB), while LoRA suffers severe degradation (from 54.7 to 45.3 accuracy). Initializing the CARTRIDGE with the KV cache of thefirst tokens of the corpus achieves 55.3% accuracy on LONGHEALTII compared to only 29.9% with random vectors. Interestingly, simply initializing with the KV cache of a different corpus closes most of the gap, achieving 51.3% accuracy. See Figure 6 and Figure for complete results on other datasets.
Next, we ablate SELF-STUDE design choices. We find that context- objective significantly outperforms standard next-token prediction, improving ChRF by 8.6 points on MTOB (24.9 + 33.5) with similar gains LONGHEALTH and QASPER. Further, show that using diverse set of five generic seed prompts (provided verbatim in Appendix [... improves performance over a single prompt ("Please generate a single chat message to begin a conversation about the information in the corpus. Ask a question about the corpus or make a request. "): +7.9 ChRF points on MTOB (24.1 + 32.0) and +4.8 accuracy points on LONGHEALTH (43.6 - 48.4).

---

5.4 COMPOSING CARTRIDGES
We evaluate if independently trained CARTRIDGES can be composed (i.e. concatenated along the sequence dimension) in order to serve queries about two different corpora (see Figure 5). We train CARTRIDGES across sizes {512, 1024, 2048, 4096) and long 10-K documents from AMD, Pepsi, AMEX, and Boeing (Islam et al. ). For each pair of CARTRIDGES pairwise (6 pairs per cache size), evaluate using dataset of malti-document questions, i.c. requiring information from both 10-Ks. Surprisingly, we find composition not only leads to coherent LLM generations off- the-shelf without any re-training (Figure 5), but also substantially outperforms the use of a single CARTRIDGE (i.e. for only AMD) or ICL (which struggles due to context length limits) (Figure on the multi-document questions.
6 DISCUSSION AND CONCLUSION
We propose CARTRIDGES as an alternative to ICL for settings where many different user messages reference the same large corpus of text.
There are several limitations this work. First, this work does strive to reduce the SELF-STUDE training cost and there is ample room for future optimizations that would make SELF-STUDY training procedure less costly (e.g. shared-prefix attention kernels (Ye et al. 2025) or improved synthetic data mixtures (Chen et al. (024b)). Second, in our work, CARTRIDGES matches ICL quality on the LongHealth benchmark, which tests long-distana dependencies, and on MTOB, which cumulative. However, there remains headroom these benchmarks and other domains with long-distance dependencies (e.g. code repositories). Future work should explore improvements to self-study that would enable it to better handle cumulative corpora and long-tenn dependencies. Third, in this work, we share the surprising result that when we concatenate two different CARTRIDGES without any joint training, the model can respond to queries requiring information from both CARTRIDGES. However, we stop short of the stronger claim that CARTRIDGES are as effective when composed as they are when used in isolation. Future work should explore how more effectively compose CARTRIDGES.
This work demonstrates that it is possible to trade off' increased offline compute for reduced KV cache memory consumption. Looking forward, this could pave the way to new context-aware AI applications that are currently bottlenecked by memory consumption, from medical assistants that know a patient's full medical history LLW--ooered IDEs that understand entire codebases.
REFERENCES
Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell Hewett, Mojan Javaheripi, Piero Kauffmann, et al. Phi-4 technical report. arXiv preprint (tt::2412.089000 2024.
Lisa Adams, Felix Busch, Tianyu Han, Jean- can--Bptiste Excoffier, Matthieu Ortala, Alexander Löser, Hugo JWL Aerts, Jakob Nikolas Kather, Daniel Truhn, and Keno Bressem. Longhealth: A question answering benchmark with long clinical documents. arXiv preprint arXiv:2401. 14490, 2024.
Joshua Ainslie, James Lee- Thorp, Michiel De Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Gqa: Training generalized multi nulti-query transformer models from multi-head checkpoints. arXiv preprint arXiv:2305.
2023.
Anthropic. The Claude 3 Model Family: Opus, Sonnet, Haiku. arXiv preprint, 2024. URL https://
www- www-cdn..o anth com wwt.////eeeeeee de 9b0 htt.eccccccc/ccbaac33333000000 618867627
Model Model_Cardd cc_.ard__llaa www.eaaeee__...
Simran Arora and Christopher Ré, Can foundation models help us achieve perfect secrecy? arXiv preprint arXiv:2205. 2022.
Simran Arora, Sabri Eyuboglu, Aman Timalsina, Isys Johnson, Michael Poli, James Zou, Atri Rudra, and Christopher Ré. Zoology: Measuring and improving recall in efficient language models, 2023.
htI.https: httpp::///axiiii org/ abs////s//2222 04927
10

---

Simran Arora, Sabri Eyuboglu, Michael Zhang, Aman Timalsina, Silas Alberti, Dylan Zinsley, James Zou, Atri Rudra, and Christopher Ré. Simple linear attention language models balance the recall -hooul-trooghpptt tradeoff. arXiv preprint arXiv:2402.1866 2024.
Maximilian Beck, Korbinian Pöppel, Markus Spanring, Andreas Auer, Oleksandra Prudnikova, Michael Kopp, Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. xlstm: Extended long short-term memory. arXiv preprint arXiv:2405.04517, 2024.
Ali Behrouz, Peilin Zhong, and Vahab Mirrokni. Titans: Learning to memorize at test time. arXiv preprint arXiv:2501.00663 2024.
Ali Behrouz, Zeman Li, Praneeth Kacham, Majid Daliri, Yuan Deng, Peilin Zhong, Meisam Raza- viyayn, and Vahab Mirrokni. Atlas: Learning to optimally memorize the context at test time. arXiv preprint arXiv:2505.237.55 2025a.
Ali Behrouz, Meisam Razaviyayn, Peilin Zhong, and Vahab Mirrokni. It's all connected: A journey through test-time memorization, attentional bias, retention, and online optimization. arXiv preprint arXiv:2504. 13/73, 2025b.
Iz. Beltagy, Matthew Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint a5X5v:2004.05150, 2020.
Aman Bhargava, Cameron Witkowski, Alexander Detkov, and Matt Thomson. Prompt baking. arXiv preprint arXiv:2409.13697, 2024.
Lucas Caccia, Alan Ansell, Edoardo Ponti, Ivan Vulic, and Alessandro Sordoni. Training plug-n-play knowledge modules with deep context distillation. arXiv preprint arXiv:2503.08727: 2025.
Chi-Chih Chang, Wei-Cheng Lin, Chien- Yu Lin, Chong- Yan Chen, Yu-Fang Hu, Pei-Shuo Wang, Ning-Chi Huang, Luis Ceze, Mohamed S Abdelfattah, and Kai-Chian Wu. Palu: Compressing kv-cache with low-rank projection. arXiv preprint 24X.v:2407.21111 2024.
Vivek Chari, Guanghui Qin, and Benjamin Van Durme. Kv-distill: Nearly lossless learnable context compression for llms. arXiv preprint 0rXiv:2503.10333 2025.
Lequn Chen, Zihao Ye, Yongji Wu, Danyang Zhuo, Luis Ceze, and Arvind Krishnamurthy. Punica: Multi-tenant lora serving. Proceedings of Machine Learning and Systems, 6:1-13, 2024a.
Mayee F Chen, Michael Y Hu, Nicholas Lourie, Kyunghyun Cho, and Christopher Ré. Aioli: A unified optimization framework for language model data mixing. arXiv preprint arXiv:2411.05735, 2024b.
Alexis Chevalier, Alexander Wettig, Anirudh Ajith, and Danqi Chen. Adapting language models to compress contexts. arXiv preprint arXiv:2305. 14788, 2023.
Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv: 10509, 2019.
Yu-Neng Chuang, Tianwei Xing, Chia- Yuan Chang, Zirui Liu, Xun Chen, and Xia Hu. Learning to compress prompt in natural language formats. arXiv preprint arXiv:2402. 2024.
Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A Smith, and Matt Gardner. A dataset of information--seekng questions and answers anchored in research papers. arXiv preprint arXiv:2105.03011 2021.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Tianyu Liu, et al. A survey in-context learning. arXiv preprint arXiv:2301.00234 2022.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Llama Herd of Models. arXiv preprint arXiv:2407.21783 2024. URI.https: peatttss://axive http://bs/24400 21783

---

Saumya Gandhi, Ritu Gala, Vijay Viswanathan, Tongshuang Wu, and Graham Neubig. Better synthetic data by retrieving and transforming existing datasets. arXiv preprint arXiv:244.11366, 2024.
Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. Model tells you what discard: Adaptive kv cache compression for Ilms. arXiv preprint arXiv:2310.01801, 2023a.
Tao Ge, Jing Hu, Lei Wang, Xun Wang, Si-Qing Chen, and Furu Wei. In-context autoencoder for context compression in a large language model. arXiv preprint arXiv:2307.06945, 2023b.
Kanishk Goel, Jayashree Mohan, Nipun Kwatra, Ravi Shreyas Anupindi, and Ramachandran Ramjee. Niyama: Breaking the silos of Ilm inference serving. arXiv preprint arXiv:2503.22562. 2025.
Yunhao Gou, Zhili Liu, Kai Chen, Lanqing Hong, Hang Xu, Aoxue Li. Yan Yeung, James Kwok, and Yu Zhang. Mixture of cluster-conditional lora experts for vision-language instruction tuning. arviv preprint arXiv:2312.12379, 2023.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces, arXiv preprint arXiv:2312.00755 2023.
Neel Guha, Julian Nyarko, Daniel Ho, Christopher Ré, Adam Chilton, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel Rockmore, Diego Zambrano, et al. Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models. Advances in Neural Information Processing Systems, 36:44123 44279, 2023.
Dan Hendrycks. Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.088 03300, 2020.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint ar 1rxvv:1533.02555 2015.
Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. Ruler: What's the real context size of your long-context language models? arXiv preprint arXiv:2404.06654 2024.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin. Lorahub: Efficient cross-task generalization via dynamic lora composition. arXiv preprint arXiv:2307.13266 2023.
Pranab Islam, Anand Kannappan, Douwe Kiela, Rebecca Qian, Nino Scherrer, and Bertie Vidgen. Fi- nancebench: A new benchmark for financial question answering. arXiv preprint a11142 2023.
Shashwat Jaiswal, Kunal Jain, Yogesh Simmhan, Anjaly Parayil, Ankur Mallick, Rujia Wang, Re- nee St Amant, Chetan Bansal, Victor Rühle, Anoop Kulkarni, et al. Serving models, fast and slow: optimizing heterogeneous Ilm inferencing workloads scale. arXiv preprint arXiv:2502.1 14617, 2025.
Dulhan Jayalath, James Bradley Wendt, Nicholas Monath, Sandeep Tata, and Beliz Gunel. Long- range tasks using short-context 1lms: Incremental reasoning with structured memories. arXiv preprint arXiv:2412. 18914, 2024.
Simon Jegou, Maximilian Jeblick, and David Austin. kvpress, November 2024. URL https
Fengqing Jiang. Identifying and mitigating vulnerabilities in Imm- -lm-interrated applications. Master's thesis, University of Washington, 2024.
12

---

Huiqiang Jiang, Qianhui Wu, Chin Yew Lin, Yuqing Yang. and Lili Qiu. LLMLingua: Com- pressing prompts for accelerated inference of large language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference Empirical Meth- ods in Natural Language Processing, pp. 133881 Singapore, December 2023a. Associ- ation for Computational Linguistics. doi: 10. 10.18653/v1 1/tp://w/2222eeee 222000.ennppmmain.82... / https
//aclanchology- wwtp:/w..g//....../o emn mop-mmain.. 825/
Huiqiang Jiang. Qianhui Wu, Chin- Lin, Yuqing Yang, and Lili Qiu. Limlingua: Compressing prompts for accelerated inference of large language models. arXiv preprint arXiv:2310.05736 2023b.
Jordan Juravsky, Ayush Chakravarthy, Ryan Ehrlich, Sabri Eyuboglu, Bradley Brown, Joseph Shetaye, Christopher Ré, and Azalia Mirhoseini. Tokasaurus: An Ilm inference engine for high- throughput workloads. tokasaurus/ 2025.
stanforddedu/bleeee
Junhyuck Kim, Jongho Park, Jaewoong Cho, and Dimitris Papailiopoulos. Lexico: Extreme kv cache compression via sparse coding over universal dictionaries. arXiv preprint arXiv:2412.08890 2024.
Yoon Kim and Alexander Rush. Sequence-level knowledge distillation. Proceedings of the 2016 conference empirical methods in natural language processing, pp. 1317-1327, 2016.
Kalle Kujanpää, Harri Valpola, and Alexander Ilin. Knowledge injection via prompt distillation. arXiv preprint 2xXiv:2412.14664 2024.
Yuri Kuratov, Mikhail Arkhipov, Aydar Bulatov, and Mikhail Burtsev. Cramming 1568 tokens into a single vector and back again: Exploring the limits of embedding space capacity. arXiv preprint arXiv:2502. 13063, 2025.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pp. 611-626, 2023.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.00 00499999...... 2021.
Dengchun Li, Yingzi Ma, Naizheng Wang, Zhengmao Ye, Zhiyuan Cheng, Yinghao Tang, Yan Zhang, Lei Duan, Jie Zuo, Cal Yang, et al. Mixlora: Enhancing large language models fine-tuning with lora-based mixture of experts. arXiv preprint 15X5v:2404.15159, 2024a.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 4582-4597, Online, August 2021. Association for Computational Linguistics. doi: 10. 1011188661/22211 100 URL httptpps//aaaaannooooo...ooosrre 2021. acl. acl- 153/
Yucheng Li. Unlocking context constraints of 1lm:: Enhancing context efficiency of Ilms with self-information based content filtering. arXiv preprint arXiv:2304. 12102, 2023.
Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. Snapkv: Llm knows what you are looking for before generation. Advances in Neural Information Processing Systems, 37:22947-22977 2024b.
Jessy Lin, Vincent-Pierre Berges, Xilun Chen, Wen-Tau Yih, Gargi Ghosh, and Barlas Oguz. Learning facts at scale with active reading. arXiv preprint arXiv:2508.0940 094944 2025.
Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chenggi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. Deepseck-v2: strong, economical, and efficient mixture-of- experts language model. arXiv preprint 2405 2024a.
13

---

Akide Liu, Jing Liu, Zizheng Pan, Yefei He, Gholamreza Haffari, and Bohan Zhuang. Minicache: Kv cache compression depth dimension for large language models. Advances in Neural Information Processing Systems, 37, 2024b.
Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157-173, 2024c.
Yansheng Mao, Yufei Xu, Jiaqi Li, Fanxu Meng, Haotong Yang. Zilong Zheng, Xiyuan Wang, and Muhan Zhang. Lift: Improving long context understanding of large language models through long input fine- tun-ttnnig. arXiv preprint arXiv:2502. 146444 2025.
Fanxu Meng, Zhaohui Wang, and Muhan Zhang. Pissa: Principal singular values and singular vectors adaptation of large language models. Advances Neural Information Processing Systems, 37: 121038-121072, 2024.
Jesse Mu, Xiang Li, and Noah Goodman. Learning to compress prompts with gist tokens. Advances in Neural Information Processing Systems, 36: 36:19327-19352, 2023.
Daye Nam, Andrew Macvean, Vincent Hellendoorn, Bogdan Vasilescu, and Brad Myers. Using an IIm help with code understanding. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, pp. 1-13, 2024.
Avanika Narayan, Dan Biderman, Sabri Eyuboglu, Avner May, Scott Linderman, James Zou, and Christopher Re. Minions: Cost-efficient collaboration between on-device and cloud language models. arXiv preprint arXiv:2502.1 /5664, 2025.
Nihal V Nayak, Yiyang Nan, Avi Trost, and Stephen H Bach. Learning generate instruction tuning datasets for zero-shot task adaptation. arXiv preprint 2rtp:mmmmxxx22002288334000,000 2024.
OpenAl. Gpt-4o system card, 2024. URL http://pp.///aavv.ooo/////////..
Matanel Oren, Michael Hassid, Nir Yarden, Yossi Adi, and Roy Schwartz. Transformers are multi- state mns. arXiv preprint arXiv:2401.00 1044 2024.
Lisa Larrimore Ouellette, Amy Motomura, Jason Reinecke, and Jonathan Masur. Can ai hold office hours? Available at SSRN 5 5/66938, 2025.
Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G Patil, lon Stoica, and Joseph E Gonzalez. Memgpt: Towards Ilms as operating systems. arXiv preprint arXiv:2310.08560, 2023.
Zhuoshi Pan, Qianhui Wu, Huiqiang Jiang, Menglin Xia, Xufang Luo, Jue Zhang, Qingwei Lin, Victor Rühle, Yuqing Yang, Chin-Yew Lin, et al. Llmlingua-2: Data distillation for efficient and faithful task-agnostic prompt compression. arXiv preprint 1xXi:2403.12966 2024.
Maja Popović. chrf:
character a-gram f-score for automatic mt evaluation. In Proceedings of the lenth workshop on statistical machine translation, pp. 392-395, 2015.
Guanghui Qin, Corby Rosset, Ethan C Chau, Nikhil Rao, and Benjamin Van Durme. Dodo: Dynamic contextual compression for decoder-only Ims. arXiv preprint arXiv:2310.02409, 2023.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606. 05250, 2016.
Haris Riaz, Sourav Bhahesh, Vinayak Arannil, Miguel Ballesteros, and Graham Horwood. Metasynth: Meta-prommping--rriver agentic scaffolds for diverse synthetic data generation. arXiv preprint 1.Xiv:2504.12363. 2025.
Luka Ribar, Ivan Chelombiey, Luke Hudlass-Galley Charlie Blake, Carlo Luschi, and Douglas Orr. Sparq attention: Bandwidth-efffcient Ilm inference. arXiv preprint arXiv:23/2.04988 2023.
Melisa Russak, Umar Jamil, Christopher Bryant, Kiran Kamble, Axel Magnuson, Mateusz Russak, and Waseem AlShikh. Writing in the margins: Better inference pattern for long context retrieval. arXiv preprint ar 2008 /4906, 2024.
14

---

Utkarsh Saxena, Gobinda Saha, Sakshi Choudhary, and Kaushik Roy. Eigen attention: Attention in low-rank space for kv cache compression. arXiv preprint 4rX1v:2408.05666, 2024.
Noam Shazeer. Fast transformer decoding: One write-head is all you need. arXiv preprint arXiv:1911.02150, 2019.
Charlie Snell, Dan Klein, and Ruiqi Zhong. Learning by distilling context. arXiv preprint arXiv:2209. 1519, 2022.
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling IIm test-time compute optimally can more effective than scaling model parameters. arXiv preprint arXiv:2408.03314 2024.
Weihang Su, Yichen Tang, Qingyao Ai, Junxi Yan, Changyue Wang, Hongning Wang, Ziyi Ye, Yujia Zhou, and Yiqun Liu. Parametric retrieval augmented generation. arXiv preprint ar arXiv:2501.15915, 2025.
Yu Sun, Xinhao Li, Karan Dalal, Jiarui Xu, Arjun Vikram, Genghan Zhang, Yann Dubois, Xinlei Chen, Xiaolong Wang, Sanmi Koyejo, al. Learning to (learn test time): Rnns with expressive hidden states. arXiv preprint arXiv:2407.04622 2024.
Sijun Tan, Xiuyu Li, Shishir Patil, Ziyang Wu, Tianjun Zhang, Kurt Keutzer, Joseph E Gonzalez, and Raluca Ada Popa. Lloco: Learning long contexts offline. arXiv preprint (xXiv:240400799, 2024.
Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han. Quest: Query-aware sparsity for efficient long-context Ilm inference. arXiv preprint arXiv:2406.10774, 2024.
Garrett Tanzer, Mirac Suzgun, Eline Visser, Dan Jurafsky, and Luke Melas-Kyriazi. A benchmark for learning translate new language from one grammar book. arXiv preprint arXiv:339.16657, 2023.
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at practical size. arXiv preprint arxiv:2408.000 18, 2024.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017
Zhongwei Wan, Xinjian WU, Yu Zhang, Yi Xin, Chaofan Tao, Zhihong Zhu, Xin Wang, Siqi Luo, Jing Xiong, and Mi Zhang. D2o: Dynamic discriminative operations for efficient generative inference of large language models. arXiv preprint arXiv:2406. 2024.
Zheng Wang. Boxiao Jin, Zhongzhi Yu, and Minjia Zhang. Model tells you where to merge: Adaptive kv cache merging for Ilms on long-context tasks. arXiv preprint 2:Xiv:2407.08444. 2024.
Xun Wu, Shaohan Huang, and Furu Wei. Mixture lora experts. arXiv preprint arXiv:2404.13668 2024.
Chaojun Xiao, Zhengyan Zhang, Xu Han, Chi-Min Chan, Yankai Lin, Zhiyuan Liu, Xiangyang Li, Zhonghua Li, Zhao Cao, and Maosong Sun. Plug-and-play document modules for pre-trained models. arXiv preprint arXiv:2305.1 1766, 2023.
Chaojun Xiao, Zhengyan Zhang. Chenyang Song, Dazhi Jiang, Feng Yao, Xu Han, Xiaozhi Wang, Shuo Wang, Yufei Huang, Guanyu Lin, et al. Configurable foundation models: Building Ilms from modular perspective. arXiv preprint arXiv:2409 02877, 2024a.
Guangxuan Xiao, Jiaming Tang, Jingwei Zuo, Junxian Guo, Shang Yang, Haotian Tang, Yao Fu, and Song Han. Duoattention: Efficient long-context 1lm inference with retrieval and streaming heads. arXiv preprint arXiv:2410.108/ 2024b.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming lan- guage models with attention sinks, 2024c. URL htt.httts: wwtps/wwwwwww..bb.se/eebboesse /
7453
15

---

Prateek Yadav, Colin Raffel, Mohammed Muqeeth, Lucas Caccia, Haokun Liu, Tianlong Chen, Mohit Bansal, Leshem Choshen, and Alessandro Sordoni. A survey on model moerging: Recycling and routing among specialized experts for collaborative learning. arXiv preprint arXiv:2408.070557, 2024.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint 151 2024a.
Songlin Yang. Bailin Wang, Yu Zhang, Yikang Shen, and Yoon Kim. Parallelizing linear transformers with the delta rule over sequence length. arXiv preprint 2411:24066006844 2024b.
Songlin Yang. Bailin Wang, Yu Zhang, Yikang Shen, and Yoon Kim. Parallelizing linear transformers with the delta rule over sequence length, 2025. URL https://arxiv.zgg/ /206
06484
Zihao Ye, Lequn Chen, Ruihang Lai, Wuwei Lin, Yineng Zhang, Stephanie Wang, Tianqi Chen, Baris Kasikei, Vinod Grover, Arvind Krishnamurthy, and Luis Ceze. Flashinfer: Efficient and customizable attention engine for Ilm inference serving. arXiv preprint 2rX1v:2501.01005, 2025.
URL http:tttss///axxi..ooo www.or////////oeeeeeo.ooeeeeeeee /2011. 01005
Howard Yen. Long-context language modeling with parallel context encoding. Master's thesis,
Princeton University, 2024.
Hao Yu, Zelan Yang. Shen Li, Yong Li, and Jianxin Wu. Effectively compress kv heads for Ilm. arXiv preprint arXiv:2406.07056 2024.
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang. Li Yang, al. Big bird: Transformers for longer sequences, Advances in neural information processing systems, 73:7727717 2020.
Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-efffcient fine-tuning for transformer-based masked language-models. arXiv preprint arXiv:2106.10119 2021.
Michael Zhang, Simran Arora, Rahul Chalamala, Alan Wu, Benjamin Spector, Aaryan Singhal, Krithik Ramesh, and Christopher Ré. Lolcats: On low-rank linearizing of large language models. arXiv preprint arXiv:2410. 1024, 2024a.
Qianchi Zhang. Hainan Zhang, Liang Pang, Hongwei Zheng, and Zhiming Zheng. Adacomp: Extractive context compression with adaptive predictor for retrieval aatreval-augmeented large language models. arXiv preprint arXiv:2409.011 095999 2024b.
Rongzhi Zhang, Kuang Wang, Liyuan Liu, Shuohang Wang, Hao Cheng, Chao Zhang, and Yelong Shen. Lorc: Low-rank compression for llms kv cache with a progressive compression strategy. arXiv preprint arXi:22410.03111, 2024c.
Yifan Zhang, Yifeng Liu, Huizhuo Yuan, Zhen Qin, Yang Yuan, Quanquan Gu, and Andrew Chi-Chih Yao. Tensor product attention is all you need. arXiv preprint arXiv:2501.066. 06645,, 2025.
Yuxin Zhang, Yuxuan Du, Gen Luo, Yunshan Zhong, Zhenyu Zhang, Shiwei Liu, and Rongrong Ji. Cam: Cache merging for memory-efficiett lims inference. In Forty-first International Conference Machine Learning, 2024d.
Zhengyan Zhang, Zhiyuan Zeng, Yankai Lin, Huadong Wang, Deming Ye, Chaojun Xiao, Xu Han, Zhiyuan Liu, Peng Li, Maosong Sun, al. language models. arXiv preprint arXiv:2305. /76911 2023a.
knowledge injection for pre-trained
Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, al. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems, 36:34661-34710 2023b.
16

---

Ziyu Zhao, Leilei Gan, Guoyin Wang. Wangchunshu Zhou, Hongxia Yang, Kun Kuang, and Fei Wu. Loraretriever: Input-aware lora retrieval and composition for mixed tasks in the wild. arXiv preprint arXhv:2402.009977 2024a.
Ziyu Zhao, Tao Shen, Didi Zhu, Zexi Li, Jing Su, Xuwu Wang, Kun Kuang, and Fei Wu. Merging loras like playing lego: Pushing the modularity of lora to extremes through rank-wise clustering. arXiv preprint arXiv:2409.161/ 2024b.
Lianmin Zheng. Liangsheng Yin, Zhiqiang Xie, Chuyue Livia Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, al. Sglang: Efficient execution of structured language model programs. Advances Neural Information Processing Systems, 37: 62557-62583, 2024.
Yuhao Zhou, Sirui Song, Boyang Liu, Zhiheng Xi, Senjie Jin, Xiaoran Fan, Zhihao Zhang, Wei Li, and Xuanjing Huang. Elitekv: Scalable kv cache compression via rope frequency selection and joint low-rank projection. arXiv preprint arXiv:2503.01586. 2025.
17

---

Note LLM usage We used LLMs for polishing improving the grammatical correctness of the writing in this paper. We also used LLMs to identify related work and write code.
A EXTENDED RESULTS
In this section, include additional cache compression baselines, report results on an additional model family, and ablate the main design choices of CARTRIDGES and SELF-STUDY.
A.1
COMPARISON WITH ADDITIONAL CACHE COMPRESSION BASELINES
МТОB
Longhealth
Method
ChRF
# cache tok.
Compression
Accuracy
# cache tok.
Compression
Full ICL
36.5
48k
1x
53.6 %
114k
1x
CARTRIDGE
30.5
256
188% x
47.7%
512
223x x
AdaKV
29.1
9.6k
5x
41.8%
23k
5x
KeyDiff
27.1
9.6k
5x
40%
23k
5x
TOVA
24.7
9.6k
5x
32.2%
23k
5X
SnapKV
29.7
9.6k
5x
33.1%
23k
5x
Table 1: Comparison of CARTRIDGES, ICL baseline, and additional cache compression baselines on MTOB and LongHealth.
In Figure we include comparisons with additional cache compression baselines very strong GPT- 4o based summary prompt compression method and Duo-attention (the strongest cache compression method in NVidia's 's KVPress library (Jegou et al., 2024)). Here, we include results for the next four best performing cache compression methods
A.2 EXPERIMENTS WITH THE QWEN3 FAMILY MODELS
МТОВ
Longhealth
Method
ChRF
# cache tok.
Compression
Accuracy
# cache tok.
Compression
Full ICL
25.8
48k
1x
51.2%
109k
1x
CARTRIDGE
32.43
4096
11.7x
56.0%
4096
26.6x
CARTRIDGE
33.27
2048
23.4x
55.5%
2048
53.2x
CARTRIDGE
32.3
1024
46.9x
54.0%
1024
106.4x
Table 2: Performance of QWEN3 4B CARTRIDGES on MTOB and Longhealth with various sizes p.
In Figure 3 report results for the Llama-3 family of models. To confirm that our results are not specific that one family models, we also report results for the Qwen3 family of models this section. With Llama on the LongHealth we were able to achieve equivalent quality to ICI. with 10x smaller caches, on average. With Qwen the compression ratio even larger: on longhealth, we outperform the full KV cache by 3.8 accuracy points while being 106.4x smaller. The results presented in Table 2
A.3 CARTRIDGE DESIGN CHOICES: PARAMETERIZATION AND INITIALIZATION
In experiments, parameterize the CARTRIDGE with simplified version of prefix-tuning and initialize with a truncated KV-cache (see Section 3.2). this section, we describe ablation experiments motivating these design choices. First, compare two different CARTRIDGE parameter- izations (Figure 6): simplified prefix-tume beffi--tuning (Li & Liang 202T) and low-rank adaptation (LoRA) HHu al. 2022 Then, demonstrate the importance of proper CARTRIDGE initialization (Figure g
18

---

LongHealth vs. MMLU
MMLU vs, Cartridge Size
MMLU vs. Self-study duration
0.600
2.58
0.54
0.375
Recuracy
aMMUU
0.56
0.02
Accurccy
0.530
0.323
0.000
0.48
0.48
0A15
0.46
0.66
0450
0:45
0.50
0.55
0.5
1000
Accurccy
Cartridge Size
Self-study duration
(Longricath))
(GE)
(4 of tramiing stepu)
QASPER MMLU
MMLU Cartridge Size
MMLU Self-study duration
0.575
0.570
0.60
0.500
Accuraty
(MAAUS
Accuratr
Accurety
0.55
0.500
0:500
0,75
CA775
0.400
0.45
6-15
6:32
loglserploxtt) (CASPER)
Cartridge Size
Self studyy duration
(G8)
training
MTOB vs. MMLU
MMLU Cartridge Size
MMLU vs. Self-study duration
0.06
0.68
0.64
Accuracy
ITAMILS
0.64
Accuracy
(TOMILL
Acouracy
0.64
0.62
0.62
0
0.82
0.50
0.17
0.32
ChRf
Cartridge Size (GB)
Self study duration yf naiiiig stepsl
(MTOR)
Cartrigg Pansmataeniation
Easelnes
C-0-
Prefix-Tuning
0-0+0 LoRA
with incrsssing tahura
with
with fill
empty
Figure 6: Comparing CARTRIDGE parameterizations. We train CARTRIDGES using SELF- STUDY on the corpora from LONGHEALTH (Top), QASPER (Middle), and MTOB (Bottom) using two different parameterizations: simplified prefix-tuning (as described in Section 3.2) and low- rank adaptation (LoRA) (Hu et al. (022). We experiment with different CARTRIDGE sizes and choose LoRA rank and prefix-tuning cache size to align on memory consumption. We evaluate the performance of the CARTRIDGES on questions from the target dataset (LONGHEALTH or QASPER) using the same protocol as in Figure and also on questions from MMLU (Hendrycks et al.) 2020) that are unrelated to the corpora. (Left) The z-axis shows accuracy on MMLU and the y-axis shows accuracy on the target dataset. Each point represents a different CARTRIDGE size. (Center) The r-axxs shows CARTRIDGE size GB, and the y-axis shows accuracy on MMLU, (Right) The a-axis shows self-study duration in training steps, and the y-axis shows accuracy MMLU. The shade of the points represents the size of the CARTRIDGE.
19

---

Consumes limited memory
Retains corpus information
Supports diverse prompts
Method
In-context lcarning
Prompt / KV cache compression
CARTRIDGE Nexx-token-prediicion
CARTRIDGE SELF-STUDY
Comparing KV caching strategies. CARTRIDGE improves memory efficiency, while retaining the quality in-context learning across a broad set of prompts. / indicates a strength and X indicates limitation.
Figure 7:
Parameterization We evaluate CARTRIDGES trained corpora from LONGHEALTH QASPER on both in-domain (i.e. questions from LONGHEALTH or QASPER) and out-of-domain (i.e. ques- tions from an unrelated benchmark, MMLU (Hendrycks et al. 2020)) queries.
We find that the prefix-tuning parameterization is more effective than a memory parameterization both in-domain and out-of-domain queries. This is illustrated in Figure Figure((Left) where we see that prefix-tuning occupies the top-right corner of the plot (high accuracy on both MMLU and the target dataset).
memory-matched matched LoRA
Notably, we find that as we increase the CARTRIDGE size with LoRA tuning, performance on out-of- domain queries (MMLU) drops significantly. At 1.06 GB (LoRA rank 1632), MMLU accuracy drops from 60.0% to 45.3%. This drop in performance is highly correlated with the size of the CARTRIDGE, suggesting that LoRA is not well-suited to large Cartridges, which we show in Figure B are important for recovering ICL performance. In contrast, with prefix-tuning the accuracy only drops to 54.3% 1.06 GB. This degradation mostly invariant the size of the CARTRIDGE (54.7% at 0.15 GB), demonstrating that out-of-domain performance is robust across CARTRIDGE sizes.
On in-domain queries, CARTRIDGE sizes, the best LONGHEALTH accuracy prefix-tuning achieves is 55.6% at 0.96 GB, while the best LoRA accuracy 27% at 0.26 GB. Interestingly, LoRA accuracy at the largest CARTRIDGE sizes is lower; 41.3% at 0.96. It is possible that this is due to the out-of-domain degradation LoRA we discussed above. Since queries in LONGHEALTH test set are quite different from the synthetic queries generated by SELF-STUDY (e.g. they are multiple choice and require some complicated reasoning traces), out-of-domain robustness may be also important for "in-domain" performance.
also outperforms LORA, but the gap smaller. Across all
isn't clear why prefix-tuning is so much more robust than LoRA to out-of-domain performance degradation. is surprising given the similarity between KV-cache and MLP- both are linear transformations separated by a non-linearity. It is possible that this is due to the difference in the activation function (SiLU Softmax). We leave a more detailed investigation into the root cause of this difference for future work.
The standard way initializing k token CARTRIDGE in our main paper is using the KV cache from the first A tokens of the source document. In Figure & we ablate different initialization source. We try two additional initalizations: random vectors and random tokens.
Initialization
For random vectors, we simply initialize the parameters of the CARTRIDGE from a component-wise standard normal distribution. For random tokens, we initialize the CARTRIDGE as the KV cache of the first & tokens of arbitrary text (specifically, the Wikipedia page for gradient). The important difference between the these two strategies is that for random tokens the initial CARTRIdGE is "valid" KV cache produced by the model, while for random vectors is not.
Freezing the attention sink A small yet important detail of training a CARTRIDGE is that we do not let the first token's 's key and value vectors to be trainable. As studied in (Xiao et al., 2024c), the first key vector, which corresponds to the beginning of sequence token and is thus the same for every sequence, acts
"attention sink". We observed that when training CARTRIDGE, allowing those key and value vectors to be trainable led to training instability (see Figure D. For example, on some runs the MMLU accuracy would dip below 30%.
20

---

0.5
XAA 8192
First tokens corpus with difierent size Cartridges
Accuracy
0.4
Random tokens
0.3
with different size Cartridges
2048
0192
0.2
Random vectors
different Cartridges
1000
2000
3000
4000
Self-study duration
of training steps)
Figure 8: Ablating CARTRIDGE initalization. We train CARTRIDGES using SELF-STUDY the corpora from LONGHEALTH with 3 different initialization strategies. The æ axis is the number of training steps and the axis is the accuracy LONGHEALTH. The blue lines when initializing the CARTRIDGE using the KV cache from the first is tokens of the document. The purple lines initializing the CARTRIDGE from the KV cache of unrelated text. The green lines is initializing the CARTRIDGE with random vectors. Initializing from the first k tokens leads to slightly stronger results than initializing from the KV cache of random text. This difference may be more prominent on other corpora where the first tokens are more relevant to solving the downstream task.
the results
MMLU
LongHealth
0.55
0.55
0.50
Accuracy
0.50
Accuracy
Frozen first token
0.45
Trained first
0.40
0.40
long Salfeelidy
0.35
0.30
1000
2000
3000
4000
1000
2000
3000
4000
Self-study duration (# of training steps)
Self study duration (# of trainig steps)
Figure 9: Freezing the attention sink. In both plots, the y-axis is accuracy and the x-axis training step. The green line which corresponds to run where we allow trainable first token. (Left) The y-axis MMLU accuracy. This plot exemplifies the training instability we observed when the key and value vectors were trainable. The MMLU score dips below 30% before recovering, (Left) The y-axis is accuracy on questions from LONGHEALTII.
A.4 SELF-STUDY DESIGN CHOICES: DATA-GENERATION AND ORJECTIVE
In SELF-STUDY training we use seeded data-generation process and a context-distillation training objective (see Section 4). In this section, ablate these design choices, comparing against the performance of SELF-STUDY with simpler data-generation and objectives.
Data Generation In Section 4.1 we describe how we use five different seed prompt types when generating data with Algorithm These prompt types, structuring, sunmarization, question, use cases, and creative, are described in more detail in Appendix C.T
In this section, compare the performance of STUDY with these five prompt types against SELF-STUDY with a single prompt: "Please generate a single chat message to begin conversation about the information in the corpus. Ask question about the corpus make request."
Across three datasets, we find that using the five different prompt types during SELF-STUDY leads to higher quality CARTRIDGES (see Figure TT). On MTOB with CARTRIDGES of size 1024 tokens,
21

---

see 7.9 point ChRF improvement (24.1 - 32.0). On LONGHEALTH, the improvement is 5.5 accuracy points (45.8 + 51.3).
Interestingly, QASPER, we no benefit from using the five different prompt types. It is possible this is because the queries in the QASPER dataset are mostly factual questions that do not require complex reasoning like LONGHEALTH and MTOB do.
LongHealth
MTOB (KE)
QASPER
32.5
0.50
222
30.0
Arturay
0.45
27.5
(Perplosyy)
2.0
2 25.0
1.8
22.5
1.6
1.4
0.30
17.0
60
90
00
1000
2000
3000
4000
500
1000
1500
2000
0
500
1000
1900
2000
Self-study duration
Solistudy Seed Promeet
Self study duration oraltiinnng stops!
of lowiting steps)
5 seed prompts
seed prompt
Figure 10: Diverse seed prompts improve quality. We generate synthetic data according to Algorithm T and ablate the choice of seed prompts sampled on Line 2. We consider two approaches: using a single, broad seed prompt (Green) or randomly sampling prompts (Blue). We train CARTRIDGES using self-study with these two strategies on LONGHEALTH, MTÓB and QASPER corpora. In all plots, the æ axis is the number of training steps, and the y axis is either accuracy (for LONGHEALTH and MTOB) or perplexity on ground truth answer (for QASPER). We an CARTRIDGE size of 1024 tokens.
of five different types of seed
Training Objective In Section we describe the context-distillatinn objective we use (Snell et al., 2022 Kim & Rush, 2016 Bhargava et al.) (2024). This approach requires that we collect top output probabilities from the in-context model's output distribution during data generation. A simpler alternative would be to just use a next-token prediction objective with cross-entropy loss.
In our comparison, we find that this simpler objective underperforms the context-distillation objective (see Figure TI. Most notably, on MTOB with 2048 token CARTRIDGES, context-distillation outperforms next-token prediction by 8.3 ChRF points (24.9 - 33.2). On LongHealth, the gap 3.7 accuracy points (47.6 + 51.3).
LongHealth
MTOB (KE)
QASPER
0.50
Acourery
log (Porloxxo)
0.40
8
CHRF
0.35
20
0.30
1003
2000
3000
4000
5000
5000
1530
2000
2500
500
1000
1500
2000
Self-study duration
012
2000
Self study duration
trwining stepp))
Context-dieelletion with different sires
2 Next-token prediction
tratiing stops!
with different siree
Figure 11: Context-distilaatio objective improves training efficiency. We train CARTRIDGES using SEUDY on the corpora from LONGHEALTH (Left), MTOB (Center) and QASPER (Right) using two loss functions: a next token prediction loss (green) and distillation loss (blue). We evaluate the performance of the CARTRIDGES on questions from the target dataset (LONGHEALTH. MTOB or QASPER) using the same protocol as in Figure 3 In all plots, the æ axis is the number of training steps, and the y axis is either accuracy (for LONGHEALTI and MTOB) or perplexity on ground truth answer (for QASPER). The shade of the points represents the size of the CARTRIDGE. Using a distillation loss achieves higher accuracy (or lower perplexity for QASPER) across datasets and CARTRIDGE sizes.
22

---

As shown in Figure quality seems to be consistently improving with more SELF-STUDE compute. is possible, therefore, that by spending more during SELF-STUDY with the next-token prediction objective, could close the gap. However, for fixed amount of SELF-STUDY compute, context- distillation is considerably more effective.
These results demonstrate how context-distillation plays important role in efficiently recovering ICL performance with SELF-STUDY.
A.5 RESULTS ON SYNTHETIC RULER TASKS
In this section, we include results on synthetic tasks from the RULER framework (Hsieh et al. 2024). We focus on synthesizing RULER tasks that reflect the intended use case for Cartridges: settings in which many queries reference one context. The three tasks evaluate are: Multi-Key Key Multi culti--uery Single-value NIAH, Multi-Key Multi-Query Single-value NIAH, and Variable Tracking. Note: that for all of our RULER evaluations, did not change the generic seed prompts from the in the evaluations from the original paper. We ran all of the experiments below with Llama 3.2 3B at 94K context length.
used
Multi-Key Multi-Quer Single-value NIAH. On multi-key, single-value NIAH, Cartridges achieves near perfect accuracy (> 99%), while compressing the context up 92x. Further, we performed qualitative evaluation of the Cartridges trained on the NIAH corpus (at 92x compres- sion). The model with the Cartridge can summarize the background "haystack" context random assortment of essays from the investor Paul Graham):
User: In a sentence, can you summarize what you learned about?
Assistant: The text discusses various topics including the challenges of designing language that programmers will love, ... and the value of learning from others, including hackers and experts in the field.
We also checked that the model capable of extracting the needle (abundan--young-937554)) when prompted in non-standard ways:
User: What would the output of the following code be: printmmagic_numbeeyy "aauuberrs"abuddnotooonnn""""
Assistant: The output of the code print(magic_numbber maanersss"abudddntttttts (abundant-young"" would be 9375247.
Multi-Key Multi-Query Multi-value NIAH. The configuration of NIAH, in which the model has to output multiple values per key, is more challenging than the single-value variant above. The full KV-cache achieves only 28.8% 88.8% exact match accuracy, which Cartridges can match with 23x compression.
Note that the original Ruler paper evaluates only with recall (instead of exact match). found that the model often had high recall but low precision, report exact match to reflect this.
Variable tracking. On variable tracking, Cartridges outperforms the full KV cache (26.2% vs. 22.8% accuracy) while being 45.9x smaller.
A.6 THROUGHPUT MEASUREMENT DETAILS
We provide details for the throughput measurements in Figure 2 We use the state-of-the-art SGI Lang
inference system, with default parameters (Zheng al. 2024). We measure throughput on single H100 GPU.
We first determine the largest batch size b that fits in GPU memory, given cache of size k tokens. We then randomly initialize CARTRIDGES of size l and pre-load the CARTRIDGES into GPU memory. We finally measure the time taken to decode 128 tokens per sequence. The CARTRIDGES and decoded tokens are appended to KV-cache during generation. We report the average iterations after using 3 warm-up iterations.
23

---

B EXTENDED RELATED WORK
In this section, we provide a more in-depth discussion of the place our work occupies in the broader literature. The structure below mirrors the structure our paper: first we discuss work related to the parameterization and initialization of CARTRIDGES (Appendix B.Ip. then cover work that inspired the design of SELF-STUDY (Appendix B.2), and finally we describe other approaches aimed at reducing the size of the KV-cache, many of which compare against in experiments (Appendix B.3
B. PRIOR WORK RELATED TO THE PARAMETERIZATION OF CARTRIDGES
Below we discuss prior work from the parameter-efficient fine-tuning literature that inform the way we parameterize CARTRIDGES in our work.
1.1.1 CLAAMETER-EFFICIEES FINE-TUNING (PEFT)
In order to adapt large language models (LLMs) to particular domains or tasks in a more compute and nemory-efficienn manner, several parameter-efficiet fine-tuning (PEFT) methods have been developed. Some of the most widely used PEFT methods include Low-Rank Adaptation (LoRA) (Hu
al. 2022). prefix-tuning (Li & Liang. 202T) and prompt-tuning (Lester et al., 202T).
Leveraging prior observations that fine-tuned language models exhibit an intrinsic low rank structure, Hu al. propose LoRA, which freezes model parameters and injects trainable rank decomposition matrices between each transformer layer. LoRA exhibits on-par or better fine-tuning quality while reducing the number of trainable parameters by 10,000 times and the GPU memory requirement by times (Hu et al. 2022).
Li et al. and Lester et al. both take a different approach to lightweight fine-tuning, proposing tunable "prefixes" and "soft prompts" respectively to prepend to queries in order to steer the model to desired outputs. Li et al. proposes prefix-tuning which learns a continuous representation for the activation of the prefix each transformer layer. These learned activations are then prepended to activations obtained by passing the input prompt through the frozen transformer. In contrast, Lester et al. proposes prompt-tuning. which optimizes the discrete token level and prepends series of learnable tokens to the input prompt. Both methods show strong performance while greatly reducing the number of learnable parameters and improving compute and memory efficiency for language model adaptation.
Principal Singular values and Singular vectors Adaptation (PiSSA) (Meng et al. 2024 another more recent PEFT method that attempts to ameliorate the slow convergence problems of LoRA. PiSSA initializes the LoRA rank decomposition matrices with the principal components of the original matrix, and exhibits faster convergence and enhanced performance compared to LoRA on several tasks, including GSMSK and MATH.
Several of these methods, especially LoRA, have been adapted specifically for distilling knowledge provided context into the parameters of a language model. Some those methods are described in the sections below, and this work is extension of prefix-tuning for long-context tasks.
1.1 BARAMETER-EFFICIENT ADAPTER COMPOSITION AND MERGING
A number of works have explored the idea of composing multiple different parameter-efficient adapters (e.g. LoRAs) by summing them together, concatenating them, or using a dynamic mixture of experts (Zhao et al. 2024b Huang et al. 2023 Xiao et al.) 2024a] /hhao et al., 2024a Yadav et al.) 2024. Wu et al. 2024 Gou et al. 2023 Li et al.) 2024a). For example, Huang et al. propose LoraHub, framework for dynamically weighting and composing multiple language model adapters (Huang et al. 2023). Given a set of LoRA modules for different upstream tasks and new unseen task with in in-context examples, LoraHub dynamically weights the LoRAs and composes new LoRA module for the task. Similarly, Zhao et al. propose a method for dynamically retrieving the most relevant language model LoRAs As for given task (Zhao et al. 2024a).
24

---

B.1.3 PARAMETRIC KNOWLEDGE INJECTION
Several recent works have explored methods for integrating external knowledge directly into model parameters, known as parametric knowledge injection (Kujanpää et al., 2024 Mao et al. 2025 Su
et al. 2025 Caccia et al. 2025 Kuratov al. (025). To the best of our knowledge, these studies are the closest scope to ours. Like ours, these works address the problem of parametrie knowledge injection: how to store large text corpora within parameters a language model. Some use simple synthetic data generation pipelines context-distilkee lation objectives. Unlike our work, these studies do not highlight the memory reduction and throughput advantages of parametric knowledge injection techniques. We highlight other differences below.
One parametric knowledge injection method, recently proposed by Kujanpaa et al., is prompt distillation, in which teacher model with access to privileged knowledge generates question-answer pairs. These pairs are then used to train a LoRA adapter for a student model (identical to the teacher model, but without access privileged information) using a distillation objective (i.e. mimicking the teacher's s full token distribution) (Kujanpää et al. 2024). This closely resembles our context- distillation objective, which we also found works better than next-token prediction. However, unlike our work, Kujanpaa et al. only train LoRA adapters of single size (rank 1024) and don't 't assess memory reductions with respect to full in-context learning. Indeed, they do not evaluate against long-context ICL baselines at all, focusing instead on a comparison with RAG. Furthermore, they evaluate on a relatively simple long-context setting - a concatenation of SQUAD passages (Raipurkar et al. 2016) which does not exhibit long range dependencies require reasoning the way MTOB and LLOGGHHAATT do.
Similarly, Mao al. propose Long Input Fine-tuning (LIFT), which fine-tunes language model using a typical next-token prediction objective on overlapping segments of the corpus, as well as instruction tuning question answer pairs generated from the corpus. Unlike al. find that synthetic Q/A pairs "offer minimal benefit and can even degrade performance due to overfitting" (Mao al. 2025) The difference in our findings is perhaps due to the fact that they only generate ten synthetic examples, whereas we generate tens of thousands. Furthermore, they use weaker ICL baseline (Llama 3 8B) that only has 8k tokens of context. Any contexts longer than 8k tokens are truncated before being fed to the ICL baseline.
work, Mao
Concurrent work on deep context distillation performs knowledge injection with synthetic data and a context distillation objective (Caccia et al.] (002). In this work, the authors only report performance with LoRA adapters and do not explore a prefix-tuning parameterization. In further contrast to our work, their focus is not memory reductions or throughput improvements. They only report performance with a single adapter size (rank 16 LoRA adapters), and they do not report throughput nature of the method.
improvements. Instead, the paper highlights the "plug-and
Finally, Su et al. proposes Parametric Retrieval Augmented Generation (Parametric RAG), in which each document has a corresponding LoRA adapter, trained on augmented dataset consisting of the document, rewritten versions of the document, and question-an pairs generated from the document. At inference time, retriever is used to determine relevants documents, and the corresponding LoRA adapters are merged (Su et al. 2025). This method demonstrates significant gains over RAG variety of tasks, including WikiMultihopQA.
B.2 PRIOR WORK RELATED TO SELF-STUDY
B.2.1 SELF DISTILLATION AND CONTEXT DISTILLATION
Seff-distillation is another method used to internalize the performance gains provided by information in context (e.g. scratchpads, informative instructions) into the model parameters. In "Learning by Distilling Context", the authors distill model with instructions and scratchpads in context into parameters by conditioning the model on "[instructions] + [task-input]" to predict "[scratch- parr + [final answer]"; then fine-tuning the same model predict its own "[final answer]" conditioned on the "[task-input]", without seeing the "[instructions]" or using the "[scratch-pad]" (Snell et al.
2024).
25

---

B.2.2 SYNTHETIC DATA GENERATION
Due to the ubiquitous need for high quality data for fine-tuning (e.g. for use with the methods described above). a large body of work has focused on generating high quality synthetic data (Nayak] et al. 2024) (Abdin et al. 2024) (Gandhi et al., 2024) (Riaz et al. 2025). For example, Bonito is model that is fine-tuned to generate synthetic data (Nayak et al. 2024), and MetaSynth is method proposed by Riaz et al. that uses a language model to orchestrate several expert LLMs for domain- specific synthetic data generation (Riaz al. 2025) The training process for Phi-4, 14 billion parameter language model, also incorporates significant amounts of synthetically generated data (Abdin al. 2024). Incorporating synthetic data, in conjunction with new post-training techniques, allows Phi-4 to surpass its teacher model on STEM QA tasks, as well as perform well for its size on reasoning benchmarks. These works demonstrate the potential for synthetic data generation methods to augment the capabilities of language models.
Contemporaneous work by Lin al. proposes synthetic data generation recipe called Active Reading, which closely resembles self-study (Lin et al. 2025).
B.3 REDUCING THE SIZE OF THE KV CACHE
In this section, we discuss existing approaches for reducing the size of the KV cache.
First, in Appendix B.3.3 we describe works that propose architectural changes to the multi-head attention operation, which reduce the memory footprint of the KV cache. Next, in Appendix [B.3.]] we discuss prompl compression methods, which reduce the size of the KV cache by converting a long sequence of input embeddings into shorter one. They be split into hard-token methods, which output discrete tokens from the vocabulary, and soft-token methods, which output new token embeddings not from the vocabulary. Finally, in Appendix [B..22 we describe KV cache compression methods. These methods directly modify the key and value matrices in the KV cache. Compared with prompt compression methods, these are more expressive because they can produce a KV cache that no sequence of input embeddings could have produced.
The methodology proposed in our work relies on cache-tuning, which could be viewed as a form of KV cache compression.
B.3.1 PROMPT COMPRESSION
Hard-token prompt compression Some works aim to reduce the size of KV cache by converting a
longer text into shorter text (Jiang et al. 2023b Li 2023 Chuang et al. [024]] Zhang et al.) 2024bb] Pan et al. 2024). These methods are typically referred to as hurd-token prompt compression methods because the resulting KV cache comes from discrete tokens from the vocabulary. Compared with soft-token prompt methods, these methods work well with black-box API models.
These methods can be broadly classified into two categories: filtering and summarization based methods. Filtering methods cut text from the original prompt using heuristics such as self-inforee reffinfoormatin. For example, LLMLLinuua and Selective-Conte dropping redundant tokens) before passing it to the main model (Jiang et al.] 20235] Li] (023). Summarization methods paraphrase long prompt into a smaller number of tokens (Chuang et al. 2024).
smaller LLM to filter long prompt (e.g.
Soft-token prompt compression with adapted LLMs In one line of work, researchers train a model (typically an adapted LLM) to compress a long prompt into a smaller number of soft
tokens (Chevalier et al., 2023:
Yen 2024 Ge et al. 2023b Mu et al. 2023 Qin et al. 2023
For example, Autocompressors and In-context Autoencoders (ICAE) are LLMS that are fine-tuned to output embeddings which can be used in soft-token prompts (Chevalier et al., 2023 Ge et al.) 2023b). Autocompressors are trained with full-parameter fine-tuning and leverage a recursive strategy to generate the soft prompts, whereas ICAEs are trained with LoRA and use single forward pass to generate the soft prompts. A recent method, LLoCO, train enable the decoder better leverage AutoCompressor embeddings (Tan et al.) 2024). This differs from CARTRIDGES in that the LLoCO LoRA adapters are trained for a doman (e.g. academic papers, news), not specific document. A number of other works also propose using auxiliary model
LoRA adapters that
26

---

to produce soft-tokens from long prompt (Ge al. 2023b: Qin et al. 2023 Gisting is another method that differs from those above in that it uses the same tokens as it uses to generate the response (Mu et al. 2023).
LLM to compress the prompt into soft
Soft-token prompt compression via gradient-descent Soft tokens also be produced by optimizing input token embeddings with gradient descent. This idea, called prompi tuning, was first proposed for the purpose of conditioning a frozen langauge model to perform specific tasks (Lester et al. 2 As such, it is an important part of the parameter-efficien fine-tuning literature and is discussed more detail in Appendix(B.... Since then, Li al. has extended prefix tuning techniques to long-contex, settings, proposing new method called prefix propagation, which conditions prefixes on previous hidden states to achieve superior performance on long-document tasks compared to prefix tuning (Li et al. 2024a).
B.3.2 KV CACIIE COMPRESSION
Hard-token KV cache compression Motivated by the observation that, in some settings, a small number of keys dominate the attention scores of subsequent queries, several works have proposed KV cache eviction policies wherein keys and values are dynamically dropped during generation (Ge jt al. 2023a Zhang et al. 2023b; Tang et al. 2024 Oren et al. 2024). For example, H20 drops keys and values from generated tokens based on a running sum of historical attention scores (Zhang et al., 2023b). Similarly, Snapk V drops keys and values from prompt tokens based on window of queries from the end of the prompt (Li et al. (2024b).
major limitation of eviction methods is that once a key is evicted, it cannot recovered. Instead of evicting keys permanently, another line of work focuses selectively loading keys from KV cache to SMs. While these works do not reduce memory consumption of the KV cache. they can speed up inference by making better use of GPU memory bandwidth (Ribar et al. [0233 Tang al. (2224). For example, the Quest method estimates critical tokens at each decoding step and selectively loads them to SMs TTang et al. 2024).
Compared with the hard-token prompt compression methods, KV-cache compression methods allow fine-grained control at the level of an attention head. This means that token can be dropped from one attention head but not another.
Soft-token KV cache compression with merging In another line of work, instead of evicting tokens from the KV cache, researchers propose merging similar tokens (Wang et al. 2024 Zhang et al. [0224]] Wan et al., 2024 Liu et al.] 2024b). For example, Cache Merge (CaM) takes keys marked for eviction and merges them instead, using weighting scheme based on attention weights (Zhang et al.) (024d). Wang el al. builds on this work by clustering key states into "merge sets" based on cosine similarity, and merging states within a "merge set" with a Gaussian kernel weighting scheme, which upweights states more similar to pivotal state chosen the token with the largest total attention score (Wang et al. 2024p. Wan et al. expands on both these works with Dynamic Discriminative Operations (D20). which performs optimizations at both the layer and token levels. D20 adjusts the KV cache budget for each layer based on its attention density and uses an exponential moving average mechanism dynamically determine when previously discarded token is similar enough to retained tokens to be merged back in (Wan et al. 2024). All of these works demonstrate promising results, offering similar or better performance on several tasks compared to full cache with a 50% or more reduction in cache size. However, there is still room for further improvement, as these methods still fail match full cache performance in several tasks, and even 50% reduction in cache size may still be prohibitively expensive for very large models or very long contexts. Additionally, these works do not evaluate the effectiveness of these methods in long-context settings.
Soft-token KV cache compression with low-rank projection A number of works leverage the observation that the KV cache exhibits low-rank structure to develop compression methods (Yu)
et al. 2024 Chang et al. 2024| Zhang et al. 2024c; Zhou et al. 2025 Saxena et al., 2024).. Similar to compression methods based on merging, compression methods based on low-rank adaptation achieve performances similar to or exceeding full caches on several tasks at 50% compression, while experiencing performance degradation upon further compression.
27

---

Soft-token KV cache compression with adapted LLMs Above discussed how some works adapt an LLM to output a shorter sequence of soft tokens given a long context. Similarly, one could adapt an LLM to output smaller KV cache given long context. While less explored than the analagous prompt compression approach, there is at least one published method that falls into this category. In KV-distill, the authors add LoRA adapters an L.M''s query projections and train them to to produce queries which aggregate information from prior tokens (Chari et al. 2025). The adapter
applied selectively to some tokens and only these tokens are kept in the KV cache. The idea is that these selected tokens can act as sinks to collect information from prior tokens. The adapter is trained with a distillation objective between a compressed and uncompressed KV cache. However, unlike our work, K V-distill does not use any training at test time.
Soft-token KV cache compression with gradient-descent The idea of treating the keys and value matrices in a KV cache as weights and training them with gradient descent was first discussed in the prefix-tuning paper (Li & Liang/ 202T]. In this work, the method was not applied to long-contexts, but rather as parameter-efficient fine-tuning method that can be applied to training datasets with input-output pairs, so we discuss it in more detail in [B..IT Since then, we are not aware of works that have applied this technique to handle long-contexts.
B.3.3 ARCHITECTURAL CHANGES
A number of works have proposed architectural changes to the original multi-head attention (MHA) operation (Vaswani et al.) 7017) that reduce the memory footprint of the KV cache. Because they fundamentally alter the architecture, these methods are not immediately compatible with pre-trained models using the standard MHA operation.
The earliest works in this direction developed fixed sparsity patterns in the attention map (Beltagy al. 2020, Child al., 2019 Zaheer et al., 2020). For example, many works use a sliding window sparsity pattern wherein each token attends to a fixed window of tokens around it. These approaches reduce the size of the KV cache because they require only keeping around a fixed number of tokens in the KV cache. More recently, some large language models have adopted sliding window sparsity in subset of layers/heads (Team et al. 2024).
While the methods above reduce the size of the cache by introducing sparsity at the token- an- other class of methods changes the structure of the attention heads. Multi-query attention (MQA), the earliest of such modifications, uses multiple query heads but only a single key and value head (Shazeer) 2019). While MQA dramatically reduces the size of the KV cache, it can lead to a significant drop in the expressive power of the model. Grouped-query attention (GQA) is middle ground between MQA and MHA that allows a group of query heads to attend to a single key and value head (Ainslie) al.
2023). Many frontier models GQA, including the Llama 3 architecture, which we in our experiments (Dubey et al., 2024: Jiang] 2024 Yang et al., (024a).. More recently, a number of other architectural modifications have been proposed including including Multi-head Latent Attention (Liu
ct al. 2024a) and Tensor Product Attention (Zhang et al. 2025).
In another line of work, researchers observe that without the softmax operation in the attention mechanism (i.e. linearizing the attention operator). the KV cache can be faithfully represented by the fixed size matrix K" TTV (Arora et al. 2024). This allows us to represent the KV cache with a single matrix whose size is independent of the context length.
Indeed, a large body of work has focused on developing architectures with fixed-size memory consumption (i.e. models that do away with the KV cache). Notable examples include state-space models (uu & Dao (202)), RNNs (Beck et al. 2024), and other linear attention variants (Arora et al., 2024
Yang et al. 2024b).
Prior work shows that there are tradeoff's between the memory consumption of an architecture and the ability of model to perform recall-intensio tasks, when controlling for compute (i.e. FLOPs) (Arora ct al. 2024). In this context, our work shows that by increasing compute (i.e. FLOPs), we can reduce the memory consumption of model without sacrificing performance. In Appendix E we provide prelinary theoretical analysis relating SELF-STUDY with recurrent architectures. However, future work should explore the relationship between CARTRIDGES and recurrent models in more depth.
Most related to our work are recent architectures (e.g. Titans (Behrouz et al,, 2024). TTT (Sun) et al. 2024 that use constant-sized memory object (like in linear attention) but apply gradient
28

---

descent-like memory updates (Sun et al. 2024; Yang al. 2025 Behrouz al. 2025a 2024
2025b).
Like our work, these architectures are motivated by the observation that gradient descent is very effective at compressing text into constant space and demonstrate the promise of using gradient descent at test time for long-context tasks. In contrast with our work, these architectures need to be trained from scratch, they have not heen validated large scale models, and do not match the quality of attention on recall-intensive tasks (Arora et al. 2024 Behrouz et al. (025a)..
B.3.4 ORCHESTRATION FOR LONG-CONTEXT
In this section, we describe strategies for managing long-context by orchestrating calls to LLMs. For instance, the approach by (Russak al. 2024) involves summarizing chunks of the context and then combining the summaries. Similarly, PRISM (Jayalath et al. 2024) treats the context as a sequence of chunks, capturing key information in structured data format. MemGPT (Packer et al. 2023) introduces virtual memory paging system, drawing inspiration from operating systems, As context length reaches the limit of available memory, the system strategically determines which information to retain.
B.3.5 SYNTHETIC DATA GENERATION
A large body of work has focused generating synthetic training data (Nayak et al. 2024 Abdin eet al., 2024 Gandhi et al. 2024 Riaz et al. 2025). For example, Bonito is a model that is fine-tune tuned to generate synthetic data (Nayak et al. 2024), and MetaSynth is method proposed by Riaz al. that uses language model to orchestrate several expert LLMs for domain-specific synthetic data generation (Riaz al. 2025). The training process for Phi-4, 14 billion parameter language model, also incorporates significant amounts of synthetically generated data (Abdin et al. 2024).
C EXTENDED METHOD DESCRIPTION
In this section, we detail the seed prompts and chunking strategy we used to train CARTRIDGES with SELF-STUDY.
C.I SELF-STUDY SEED PROMPTS
As discussed in Algorithm seed the synthetic conversation generation with prompt that elicits conversations about different aspects of the document. For each conversation, we randomly sample one of the following functions and create seed prompt by calling it:
29

---

Structuring Seed Prompt Generator
structuring_seed prompt (*akwargs);
DATA, FATA_FORMATS
1I8ON""
"YAMG", "TOML", "INI", "XMT."
"plain Lext",
data_format
zandom, choice (DATA_PORMATS)
EXAMPLES
you structure information [[subsection)) of [[document] related to [scmething specifie]]
f"in follewing format: rdata_formst]]
"Be include precise information like any dates, times, names, and numerical values.
19
21
example random.chot (EXAMPLES)
23
24
returr
r"P ("Please gererate aingle message instruct TM
Information in formai..
*Output only
chat message itself absolutely nothing else.
27
"Make sure it is claar what saction and document you are asking about. r"The message
follow the louing template, rilling in dataila the corpus: \n'n'' (exampla)**
29
31
Summarization Seed Prompt Generator
summarisation__ced promp : prompts
"Pl *Please generata single chat message instructing LTM
part of the corpus.
sure the instruction is very explicit about section of the corpus that want zo summarize.
"Irclude datails (ids, names, titles, dates, eto.) are asking about. I,
clear what
"Please generale single chat message instructing an LIM to summarize section.
10
"Nake sure the instruction in explicit about the section that should he summarized and the documens from."
11
prompt choice (prorpta)
14
recure prompt
16
30

---

Question Seed Promot Generator
questioon_aeed hewliioo__ee__eeeeeeee prompt teakwargs)
prompts -
"Generale question for LLM that will Lest lis knowledge of the information in the corpus above. "in your question etc.) that make it clear what you are asking about. "Output only single questien. NOT include any other explenation other than the question." 1r
include dstails (ida, names, titles, dates,
"Generate a message for an ULB that will test its knowledge of the information in corpus abave." "
10
"Be sure to include detalla (Ida, namea, titles, datea, etc.) in the question so that book setting)-
answezed without access corpus (i.e, closed
"Dutput only singla questien. explanation other than the question."
include any other text
12
1,
14
"You are helping to quia user about the information in the corpus. the subsection
corpus abeve. "ge sure to include details (ida, zames, titles, dates, etc.) in the question make clear what you are asking about.
"Please generate question
16
"Answer with the questien, not include any other text."
18
20
prompt random, choice (prompla)
21
prempt
22
Use Case Seed Prompt Generator
use lre__aae__sed__eee (**kwargs)
prompt
"You are working to :zain language model on the information in the following corpus.
"Your primary goal to to chink about practical, real-world tasks or applizatiens semeone
schieve using knowledge consained within this
corpus.
*Consider how vaer might want to apply this information, not Just recall ft.
"After considering potentiel use cases, your task will be to generate a sample question that raflects
thase downstream applications.
*Tii quest question rutttnnnttttttttaaaaaaaa corpus, might ask trying to accemplish their specific geal.
ahould be aomething usar, who has sccess to
*Output only single quastion, no NOT inglude any other text or explanation other than the qoestion."
returr: prompt
Creative Seed Prompt Generator
def creative weeeiiiee_eeee_eeeeee prompi (vxkwargs)
prompt
"You are having ereativa conversation inspired the information in the corpua.
*Please generata question for your corverastion partner start off the discuaston. only with questien, include any other Lext."
Fecurr cindom.ctoiae (prompt)
31

---

C.2 SELF- STUUY CHUNKING
For the SELF-STUDY data generation process, extract uniformly random token-level chunks from the imput corpus C. A corresponding textual description is generally prepended to each chunk to contextualize it when generating the seed prompt. This approach helps the model focus different parts of the corpus and generate diverse synthetic examples. The specific chunking parameters and descriptions tailored each dataset:
LoNGHEALTH: Chunks are sampled with a minimum size of 512 tokens and a maximum size of 4096 tokens. The accompanying description is: 'Below a section of patient's medical record. is part of larger corpus of medical records for Noptienss different patients."
AMD/FhnanceBench: Fixed-size size chunks of 8192 tokens are utilized. No specific descriptive text is prepended to these chunks.
МТОВ: Chunks are sampled with minimum size of 512 tokens and maximum size of 4096 tokens. The description used is: "The following is excerpt from grammar book about the Kalamang language."
QASPER: Following our general methodology, chunks are sampled with minimum size of 512 tokens and a maximum size 4096 tokens. A generic description used to contextualize the chunk as excerpt from research paper, in line with the nature of the Qasper dataset.
D DATASETS
D.1 GENCONVO
To evaluate the ability of our approach to handle diverse queries over long documents, we generated the GENCONVO dataset. We created GENCONVO using the AMD 2022 10-K filing, document from the FinanceBench corpus (Islam et al. 2023). The primary purpose GENCON VO is to simulate a wide range of tasks user might ask model to perform given a long document, thereby testing the model's comprehension, reasoning, and ability to extract varied types information. The generation process relies Claude Sonnet 3.7 (Anthropic 2024) and is structured follows:
The entire source document (e.g., the AMD 2022 10-K, which is less than 200,000 tokens and fits within the model's context window) is provided Claude Sonnet 3.7.
Document Input:
2. Question Generation: A series of distinet prompt templates (detailed below), designed to elicit different reasoning traces (e.g., factual recall, synthesis, multi-hop reasoning), are used to generate questions. For the given document and each prompt template, we ask the model to generate 16 unique questions. This involves providing the model with the full document content alongside the specific question-generation prompt.
Answer Generation: Subsequently, for each generated question, Claude Sonnet 3.7 is prompted again with the original full document and the generated question to produce answer. This process ensures that the answers are grounded in the provided document.
We hope GeNCONvo provides challenging benchmark that moves beyond simple fact retrieval, assessing a model's capacity for deeper understanding and more complex information processing over long contexts. The following prompt templates were utilized for the question generation phase:
Factual Prompt Template
Please generate a question to test soneone's ability to renember factual details from the document. The anawer should be few tokens leng and be a factual detail from the statement, such as number, entity, date, title, This question should name.
common knowledge: instead, should be scmething that is only answerable via information in the document.
32

---

Knowledge Prompt Template
Please generate a question that requi res combining information mentioned both inside and outside the document.
This question should require using Eact from the cocument and also fact that you the document. For instance: What are the founding dates the companies that got acquired this year? This is a good question because the names of the acquired companies are mentioned in the document and the founding dates are not mentioned. What is the name of the CEO's spouse? This is good question because the mentioned in the document and the spouse's name is not mentioned. The answer should be a fact that is a few tokens long such as a number, entity, date, title, or name.
confident about, but is not mentioned in
of the CEO is
Disjoint Prompt Template
Please generate a multi-hop question that tests someone's s ability to use factual information mentioned in at least two very different sub-sections of the document.
This question shouldn't be standard question about this kind of document. Instead, it should ask about two parti particularly di sconnected ideas, like comparing information about the amount of owned space for the company headquarters with the amount of dollars of est est imattddd liability comparing the revenue number with the number of employees. This question should also test one's ability give away part of the answer in the question. Ensure that for one to get the correct answer to the question, they need to understand the decument.
retrieval: do not
The answer should be short: for example, number, entity, date, title, name.
Synthesize Prompt Template
Please generate a question that requires synthesizing and aggregating information in the document. For instance, you could ask somcone to summarizo page of the document, list all the key competitors meationed the document, or summarize the company's s business model.
Structure Prompt Template
Please generate question that requires understanding the structure of the document.
This question should be more about the structure of the document, rather than the precise statement details. For instance, you could ask someone to 1ist the titles of all the sections in the document, describe the document structure, report the total number of pages, ask which section amongst two sections comes first, or report the section with the largest number of tables.
Creative Prompt Template
Please generate questien about the document to test someone's s ability comprehend the content the document. This question specifically shculd be focused on their ability to generalize the information about. the document. to a strange question of sorts.
This question shouldn't be standard question about this kind of document, it should ask to do something abnormal and creative, like writing poem about a financial docunent.
33

---

Counting Prompt Template
Please generate a question that requires res counting how frequently
different events occur in the document.
This question should be about statistical properties of the document, rather than the statement details. For instance, you could someone to ceunt the number of times the word "million" is mentioned or count the length of the shortest section title. The answer should be a number.
Reasoning Prompt Template
Please generate question that requires mathematical reasoning over the values the document.
This question should require going beyond the facts directly mentioned in the statement, such as asking to compute the percentage increase in revenue between two years, find the largest expense category, or calculate difference in profit between two years. The answer should a number.
D.2 NEEDLE-IN-A AAAYSA-K (NIAH)
The seedle-i--aHaastaar task provides controlled evaluation model's ability to precisely retrieve and recall specific information from long documents.
adopt the challenging multi-key variant from RULER benchmark (Hsieh et al. 2024). which requires models to locate and extract multiple pieces of information scattered throughout long document. We choose this version of the task because it is more challenging than the standard single-key saale-in-the-haystadd task and because it reflects the setting where CARTRIDGES intended to be used: a single corpus of text against which many different queries are issued.
The task construction proceeds in three steps:
The document consists random passages drawn from essays about startups by investor Paul Graham, creating realistic and semantically coherent text that serves as distracting context.
Background Generation:
2. Needle Insertion:
Multiple synthetic "needles" (key-value pairs) are inserted at random positions throughout the document. Each needle contains unique identifier and an associated magic number. For example, the identifier "gorgeous-bath" is associated with the magic number "9290765".
Query Formation: LLLM prompts are produced that prompt the model to retrieve specific magic numbers given their corresponding identifiers, requiring precise information extraction from the long context. For example, the prompt "What is the magic number for saggeous-bath?? requires the model to retrieve the magic number "9290765" from the long context.
This setup tests whether CARTRIDGES can maintain the same level of retrieval accuracy ICL while using significantly compressed representations. The task is particularly challenging because the needles are syntactically similar but semantically distinct, requiring exact pattern matching rather than approximate retrieval.
Consider the excerpt below, which shows how needles are embedded within the natural text:
startup they to come to life, because finally they're re working the way people are meant to. Notes[11] When talk about humans being meant or designed to live a certain way, mean by evolution. [2] It's s not only the leaves who suffer. The constraint propagates up as well as down. So managers
In the first couple weeks of working their
constrained too; instead of just doing things, they have to act through subordinates. One of the special magic numbers for gorgeous-bath is: 9290765. [3] Do not finance your startup with credit cards. Financing a startup with debt is usually stupid move, and credit card debt stupidest of all. Credit card debt is bad idea, period. It is trap set by evil companies for the desperate and the foolish.
34

---

In this example, the model must identify that "gorgeous-batte bath" is associated with the magic number "9290765" when queried.
D.3 LONGHEALTH
LONGHEALTH is benchmark for evaluating large language models ability analyze and interpret long clinical texts (Adams et al. 2024). The benchmark consists 20 fictional clinical case reports (each containing between 5,090 and 6,754 word) and 400 multiple-choice questions based on them.
In our experiments, the context C consists of the reports for a panel of n patients. We use n = 10 patients, with full panel of approximately 100k tokens, which fits in the context length of the LLAMA models.
The questions are categorized into information extraction, negation, and sorting.
sorting question is included below:
Please answer the question below about the following patient: ID patient_03, Name: Mr. John Williams, Birthday: 1956-08-08 00:00:00, Diagnosis: Multiple Myeloma
<question>
Mr. Williams received nultiple radiologic examinations. In which order did she receive them?
</question>
<options>
CT whole Rody MR Spine Scan CT Spine Scan PSMA-PRI--" Scan CT Chest > CT Whole Body > Whole Body CT scan
Whole Body CT scan CT Spine Scan > CT Whole Body > MR Spine Scan > CT Chest PSMA-PET-CT Scan CT Whole Body.
Whole Body CT Whole Body CT Chest сT Chest PSMA-PET-CT Scan > MR Spine Scan > CT Spine Scan > whole Body CT scan Chest X-ray CT Chest > ст Spine Scan > CT Whole Body > Whole Body CT scan >
PSMA-PET-CT Scan > MR Spine Scan > C Whole Body
Whole Body
сT Spine Scan CT Whole Body MR Spine Scan CT Chest > CT Whole Body PSKA-PET-CT Scan </options>
You should first think step by step. Then give your final answer exactly it appears in the options. Your output should be in the follewing format:
<thinking> { [YOUR_]THOSSHT_POOCSSSS)
</thinking>
<answer> (YOUR ANSWER)
</answer>
An example of a negation question is included below:
Please answer the question below about the following patient: ID patient_0l, Name: Anna Sample, Birthday: 1970-01-01 00:00:00, Diagnosis: DLBCL <question>
Which of these examinations were never performed in Mrs. Sample?
</question>
<options>
Rone marrow aspiration
CSF aspiration
MRT of the head
Pulmonary function testing Cardiac stress testing
</options>
You should first think step by step. Then give your final answer exactly as it appears in the options. Your output should
35

---

be in the following format: <thinking> [[FOUR_THOOCCHIPRRCEE))) </thinking>
<answer>
{YOUR [ </answer>
D.4 MТОB
The Machine Translation from One Book (MTOB) benchmark tests large language model's ability to learn to translate between English and Kalamang, low-resource language with virtually no web presence (Tanzer et al., 2023). The core task is perform translation (Kalamang English, and English to Kalamang) by primarily relying on single comprehensive grammar book and small set of accompanying linguistic resources. In our work, we focus on translating from Kalamang to English.
The source documents provided by the MTOB benchmark are:
A grammar of Kalamang: A comprehensive grammar texthook, with the original source provided in WiEx format. This book details the phonology, morphology, and syntax of Kalamang.
Bilingual Word List (W): A list of Kalamang words with their part-of-specch tags and English descriptions.
Parallel Kalamang-English Corpus (S): A collection of 375 paired Kalamang Engilish sentences.
The MТOB authors preprocessed the grammar textbook from its original ETRX source into several plaintext splits for their baseline experiments. These include:
Gth (Medium-length chunk): A plaintext segment of approximately 50k tokens consisting of an overview chapter, a morpheme table from the grammar book, and the complete bilingual word list (W).
G (Long-length chunk): A larger plaintext segment of approximately 100k tokens, containing chapters from the grammar book that the MTOB authors deemed most important for the translation task.
Full Plaintext Textbook (G): The entire grammar book converted to plaintext.
The combination of the long-length chunk (G"), the parallel sentences (S), and the word list (W) exceeds the context window Llama 3 models. We use the medium-length chunk G and the parallel sentence list S as input for our ICL. baseline.
D.5 QASPER
QASPER is benchmark for evaluating the ability of large language models to answer questions about scientific papers (Dasigi et al.) (2021). To create a challenging multi-query long-context setting resembling the setup described in Section 2.2] we concatenate 16 papers all related to QA NLP models form out corpus In total, there are 78 questions about these 16 papers the dataset, which we use as the queries Q
Because the dataset only includes short answers and ground- spans containing evidence for each answer, we rewrite the answers in a longer, more conversational format using GPT-4.1 and use these as the targets when evaluating.
E THEORETICAL ANALYSIS: RELATIONSHIP BETWEEN ATTENTION, LINEAR ATTENTION, AND CARTRIDGES
When generate text with autoregressive Transformer, have to maintain a KV-cache that grows linearly with the length of the input and text. In Appendix we discussed a number of architectural modifications that either reduce the size of the K KV-ccche do away with it altogether.
36

---

In particular, when generating text with linear attention (e.g. (Arora al. 2024 only need to maintain a constant- object the KV-state matrix during generation.
Like the KV-state matrix in linear attention, CARTRIDGES consume constant amount of memory (i.e. their size a hyperparameter, which can be set independently of the input length). However, they differ from the K V-state in how they updated. In this work, CARTRIDGES are updated using SELF-STUDY- gradient descent on synthetically generated data. On the other hand, KV-states are updated using a linear attention update rule.
In this section, we will study the update rules for attention, linear attention, and gradient descent when applied to the multi-query associative recall (MQAR) problem (Arora al. 2023). a popular synthetic benchmark task used for studying the capabilities of long-contedd architectures. In particular, we consider variant of the standard MQAR problem where key-value pairs are repeated. First, we highlight some equivalences between the update rules of these approaches in the case where input keys are orthonormal. Then, in the more challenging case where input keys are in a Johnson-Linee Lindenn-nnneessauuss embedding, provide separation result showing that the gradient descent update rule is able to exactly solve an MQAR problem that linear attention cannot.
These theoretical results provide intuition for why constant-sized CARTRIDGES are able to match the performance of full VV-achees in long-context settings when linear-attention architectures have struggled to do so.
E.1 NOTATION
All vectors are assumed to be row vectors.
Parenthesized superscripts (e.g. k()) denote some temporal quality of an element. Subscripts denote different elements in set, is standard.
concise explanation for each variable:
d : model (and token) dimension.
number of unique kayuvalue pairs.
n number of queries.
N number of key-value pairs in stream.
E.2 MQAR
We define the Multiple Query Associative Recall (MQAR) problem. Definition 1. There is universe of keys:
K C Rixd
and values:
V cxdd
Definition 2. Arora er al. 2023) In the MQAR problem, the input is:
(1)))())))) (N(N), (()) where (t 000) E for 1 S
followed by a of queries
a..... where qi E K for siii n.
Then for each E [n]], output:
Vi- where i" max E [1, 93 dd if no such i exists.
37

---

E.3
REPETITIVE MQAR
Definition 3. -rpeettiiivee MQAR is special case where each (K(1), v(t) E S, where:
S = {(ki, vi). (kim; Vm }m
Additionally, k; unique.
is defined as the number of occurrences of (ki, vi) in the stream at Definition 4. To capture this, vi (0) timestep t.
E.3.1
ORTHONORMAL EMBEDDING
First, we will look at the MQAR problem in restricted case, when all keys orthonormal.
Definition 5. We call the set K to be orthonormal if for all k, k' E K:
(k. (k. = {1
i K'
otherwise.
E.3.2 JOHNNNN-LIDENSTRRUUS EMBEDDING
Next, will look the MQAR problem in restricted case, when all keys are in JL embedding. Definition 6. Let > we call the set K to be e-Jliffor all k. k'
K:
(k. k2) = c|
i
1
otherwise.
E.4 MODEL DEFINITIONS
Below, will describe three different model architectures. While they each exhibit different performance and capabilities they can be describe with a common framework for the MQAR problem.
1. State: is how the model store Key- Value pairs.
Update rule: how the model incorporates new Key-Value pairs into its state.
3. Query rule: how the model uses its state answer look up value query.
E.4.1 TRANSFORMER
The state is:
w (K(1), v(t)),
where,
K(O E Rexd v(t) E Rexd
Note that this consumes more memory as the context gets longer.
2. The update rule is:
k(++1) ((() + ((+1)) v((+1) v(1) ((+1)
3. On query E K, return:
q (K())² T V(())
These nules define the transformer setting for MQAR.
38

---

E.4.2 LINEAR ATTENTION
1. The state:
W E Raxd
The update rule defined as:
W w + ( (v(1+))).
((((t)))(((((11))))))
With the initial matrix being initialized zeros. Le. ( 0dxd
3. On query return:
qW
Lemma 1. Yang et al. 2025) Linear attention rule emerges if we were update using the loss function -k "'
important mention here that we are not using any kernels for linear attention. These rules define the linear attention setting for MQAR.
Lemma 2. Yang et al. p025 ( w - (k() T k(t) w( + (k(t)) T w(t) is the update
rule that emerges when we the gradient descent loss function: =||l<(0) W(() 10) Definition 7.
L -- 00
Proof. general, gradient descent has the update rule:
( ((() V/ wi..
(3)
Taking the gradient the loss function gives us:
Vw W[llIK (1: (1)) 00 [ = (K(I) T (k w( ((1))))))) 000)
(M()) 1000()))) (A(O)) w(1)
Using the above and choosing 17 1, we get for Equation
((++1) (1(1) - ((s(0)) A. 10 w(() (A(1)) T 2(t))
-w(1) - (B!1)) w((w)) + (B:00) o(1))
E,4.3 GRADIENT DESCENT
Gradient descent training on the cache. We look at the capability of this trained state on a certain input.
1. The state at time t is defined as:
( E Rdddd
2. The update rule which follows from Lemma 2
(++1) = (1 - (k(1)) k (t) - (R10) o00).
With the initial matrix being initialized zeros. Le. (()) ddxd
3. On query q. return:
(1())
39

---

E.4.4 ORTHONORMAL CASE
now see how the three models perform on the m repetitive MQAR when K orthonormal. Transformer
Lemma 3. On every input MQAR (even those for I-rep-MQAR) the state of Transformer needs ((Nd) parameters.
Intuitively, at each timestep, you will append d parameters to the state. At timestep the model will have td parameters.
Linear attention
Theorem 1. Linear attention can solve repetitive MQAR for any m 2 1 and orthonormal K, up (0) scaling (producing ri Uú when ( is queried with kę) and all keys being distinct with (dd) parameters.
Proof. We first prove that for any t 2 0:
W
(0 kp Usr-
(4)
Base Case: Initially, W7() 0dxd From this, indeed have:
W
(0) kg
Uy:
since for all i' [m]]
T2
0.
Inductive hypothesis: Assume that the state matrix at some arbitrary integer timestep is as claimed.
L.e.:
W
kp vg.
Inductive step: If (1()), e()) appears at timestep + the update rule will be:
((++1) = w + (I((+))) (1))) ((() + (k;)) T 0j
By the inductive hypothesis, we have that:
( = ( + (1,)
(1 kJ KJJur k)(
(++1) k uyc
=
==1
The final step follows from the fact that (t+1) = (1)) + when (1((+1), o(++1) (b((+1)),x(+11) = (k2, vy) and
r (t++)) = r(()) for all * j.
The proof of Equation (4) complete by induction.
40

---

Finally, it is the case that query k;;
m
k
w (0)
k; Σ
(1)
k vp
x=1
k; k. 0y'
( p ke by +1; (1) k 0s
p .0.0000 Byy + +r (0) 11
(0)
9/
as desired. In the above, the second last inequality follows from from Definition 5 and the fact that all k; are distinct.
(dd) parameters are needed as the matrix must have dimension d x d
Gradient Descent
Theorem 2. Gradient descent is able to exactly solve the repetitive MQAR (produce vs when (9 queries with k;) with (dd)) parameters.
Proof. Here can handle repetitions because update rule includes "peel" term. This means it removes the current value stored under a key before updating it with a new value.
We will show by induction that for all 2 0:
(
Σ
1 2000
ug vg.
Base Case: Initially, the cache matrix is set all zeros. From this, naturally follows that:
Σ
0. k Uys
f=1
since for all i!
(0) = 0. T2
Inductive hypothesis: Assume that at some arbitrary timestep t, have:
(
Σ
1 KJ v.
Inductive step: If (ke, vz) appears at timestep t + the update will be:
m
m
m
ter 120
k u
Σ
Σ
1 co
k
k ny
+ kjj ve
the second term reduces just peeling the term relating ke. if it exists, as all other inner products are 0,
172
Σ
1 1. k:T 255
(1,00, k2 vz)
kjj we
m
1 1,
koj vy
ko kT:0 oz
41

---

This replaces the value associated with & with the new value, while keeping everything else the same. This is the form that we want, as the only time we want to add key it is an new key.
Finally, it is the case that query k;;
k; W k;
Σ
ky D2
k;
V2
(0)
1 12
120
(e) 12
120
Again here matrix of dimension d can store d orthogonal vectors. Thus this requires, (dd)) parameters.
E.4.5 JL EMBEDDING
We now see how the models perform on the m repetitive MQAR when K 6-JI.
Transformer
Lemma 4. On every input to MQAR (even those for I-rep-MQAR) the state of Transformer needs ((Nd) parameters.
We note that when K is c-JL it is longer possible to get the exact answer from query rule k; W
Thus, we need to add a decoding step.
Definition 8. The output decoding step is Us- where:
arg max (uv, (!)). & 8E(m
Definition 9. For all i,j j [m], define:
C. = kk))
Linear Attention
Theorem 3. Linear attention (+ decoding as in Definition is unable to solve even the 2 repetitive MQAR and each v, being 1-hot encoding unless K is w (t) JLL
Proof. Due to the agreeance between different keys, when querying for key i, there is noise from other keys returned along with the correct answer. While we can tolerate some error, this error scales with the number of times the model has seen single key. Making it unfit for longer contexts, or contexts with many repeats.
First, note that the base case Equation from Theorem T still holds. In general, this holds for all K.
Specifically, on query k we have:
k (t) = ri (2) (ki, ki) + 11 (k1. 02) 1 + +r2 (0) 61,202.
Now, consider an input to 2 repetitive MQAR such that
(t) (t)
r2 E1,2.
Note that in this case:
(1)) (1). ki W(0) < (v2, kn (())) = (0)
61,2
42

---

and hence output 02 instead of 01- the embedding was w( 1 the number of repeats could not overeome the value.
Gradient Descent
Theorem 4. Gradient descent (+ decoding as in Definition 8 is able exactly solve repetitive MQAR with O(d)) parameters for J-JL K, as long as S mm and a < m-1
m+1.
Proof. We define:
to be the coefficient associated with k vy in W Specifically, let
(
k Uj
(5)
We will prove by induction that:
C (1) (k:, my has occurred A
(6)
where,
A h
(mn
1)
(7)
Base Case: Initially, the state is all zeros. From this, naturally follows that all of the c
zero. L.e Equation (6):
Ai, = 0.
Inductive hypothesis:
Assume that all for some timestep t and S i,j S m:
c
1(k.. has occurred + A450
where A09 satisfies Equation
43

---

Inductive Step: If at timestep t + 1 are given (ke,ve), vz), from Equation (5)
the update looks like:
m7 m
w(++1)
(+++1) 6.
ki
( J k2 kek
+ ki ve
ΣΣ
C k2 0z
+ k2 v0
Y=1ý=1
V=1f==1
change the associativity of the summations,
m
m
T Σ
Σ
k.
CeN Clo
kig By
+ ki 18
1=13/=1
here we separate the first term where 1' l anddd i' f
KTy
E
ko 0z
+ ke ve
"+tj==1
f'=1
here we separate the first term where i' = l and i' +
kTy
Σ
E
ko 03
Σ
ke wy
+ kot ve
vf(j'=1
y'zl
remove Ejj+
+
Σ
k Byy - Σ
ki Dj
Σ
Σ
ko
+ k2 1z
j'=l
3' =1=1
cancel terms,
m m
m
KTy
Σ
Σ
C.ty 2'gg'
k 2x
+ ki U2.
y=11
Note with this we can see that:
!
if
c((e+1)
c22
0. + 11=
if
Thus, if i + E, we have:
(t+1) =c(0
for + e. The inductive statement holds for these pairs. Now let's consider (1+1) If j then:
-1+)==++|!
-110 Σ
C +1
44

---

and note that by the triangle inequality and Definition 6
A
(1+1)
Se
Σ
by the inductive hypothesis,
Se Σ (1+
(mm - 1)e)aa)
am
((m 1) + Σ ((m 1)
H+
((m 1)
as desired.
Then for j # l, we have:
A ((+1)
(1+1)
Σ C.t
i'ze
The bounding of [ is similar to the 8= case.
With this we have completed the inductive proof on error terms.
the we set:
m²(m 1)
get the following bound:
A (8)
((m 1)c)
(8)
(m 1)c
(9)
(m 1)e
(10)
m - 1
Before the next steps, we must bound:
(o,, wلس1 S
(11)
For query with k, assuming we have seen k; before, get:
k; ( + Σ ( By
Now for the decoding step where for an arbitrary vy we get:
(v,, K. (()) = (w,, v.) + (0,, Σ A
j'
45

---

For the case where i it is the case that:
(0, k; W(()) = -1+ (U;, Σ (.))
j'
1
2
a.
This follows from Equation (0) and Equation (11.
For the case where i # j it is the case that:
(uz, k4 w (0, Uz) + (Uj, Σ A
j'
<a
m+1
This follows from Equation 100 and Equation (T].
As result, we will always pick the correct value when or m-1
46


Reviews:
[{"review_id": "1S0HLbBI14", "reviewer": "Reviewer_RZFK", "summary": "This work explores the challenge of efficiently serving large language models (LLMs) for tasks requiring access to extensive text corpora.\nTraditional approaches rely on placing entire corpora in the model's context window, which is memory-intensive due to the scaling of key-value (KV) cache with input length.\nThe authors propose an alternative: pre-training a smaller, corpus-specific KV cache called a Cartridge offline, which can be reused across multiple queries referencing that corpus.\nWhile naive next-token prediction training does not yield competitive results compared to standard in-context learning (ICL), the authors introduce a \"self-study\" method, generating synthetic conversations about the corpus and using context-distillation objectives, to train Cartridges effectively.\nThe experimental results show that self-study-trained Cartridges match ICL performance on long-context benchmarks while dramatically reducing memory usage and increasing throughput.\nAdditionally, this approach extends effective context length and allows for composability of Cartridges without retraining.", "questions": "See weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The self-study technique leverages synthetic data generation and distillation objectives to further improve context compression, enabling high-quality representation learning beyond naive next-token prediction.\n2. The experimental results show that Cartridge significantly reduces memory consumption and increases throughput by over 26x compared to traditional ICL methods. Despite resource savings, Cartridge achieves comparable performance as full-context ICL on challenging benchmarks.\n3. The writing clearly demonstrates the paper's motivation, implementation, and analysis.", "weaknesses": "1. Effectiveness relies heavily on generating high-quality synthetic conversations. There should have been some analysis on the data quality and cost of synthetic data.\n2. It would be better to compare with some naive methods that have a similar idea with Cartridge, e.g., existing prompt compression methods + SFT with your synthetic data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oMzyjcrxmp", "reviewer": "Reviewer_aS8F", "summary": "This paper proposed a new method called self-study, which learns to store the document knowledge in the learnable KV cache parameters using a distillation loss; the authors propose to generate data augmentations based on the document to avoid training on the same document multiple times. Experiments show that self-study matches the performance of ICL with additional spend on training compute. In addition, this method enables the model to handle documents beyond its context length.", "questions": "1. See Weaknesses. Could you provide comparison with the missing baselines?\n2. What are the applications that you think will need such techniques the most--Do you envision the popular applications will have different parameters for different documents? In what case, the information loss due to compression will impact the model performance?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper proposed a new method called self-study, which learns to store the document knowledge in the learnable KV cache parameters. The method amortize the inference compute to training compute, which is useful in many real-world applications.\n2. The authors carefully studied the effect of different initialization strategies, self-study compute, diversity of seed data augmentation prompts, etc., showing solid investigation on the factors that can impact the performance.\n3. The work also studied different parameterization methods for the self-study, such as prefix-tuning and LoRA. The experimental ablations are comprehensive.", "weaknesses": "1. The Figures have very small font and is very hard to read.\n2. The paper didn't compare with existing baselines on memory layer and active reading, nor discussing these works in the related work section, e.g., [1][2].\n3. There is missing one strong baseline that a summarizer is used to condense the document in the text space before feeding long document to context.\n\n[1] https://arxiv.org/abs/2412.09764\n[2] https://arxiv.org/abs/2508.09494", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GFHcW6njv8", "reviewer": "Reviewer_mY8p", "summary": "The work attacks the problem of high computational cost and reduced throughput of LLMs queried repeatedly to analyse a large (common) corpus of in-context documents.\n\nThe authors defined a cartridge - a sequence of pairs of pxd matrices that represent KV caches at every layer of the language model at hand. Here, p is the hyperparameter of the size of a cartridge, and d is the dimensionality of the model.\n\nThey suggest treating a cartridge as training parameters to represent information about a large common prefix. Given a common prefix, they suggest:\n1. Generate a training dataset. In a loop, do:\n\n1.a Retrieve a random chunk from the prefix\n\n1.b. Seed a generic seed prompt from template categories (structuring, summarization, question, use case, creative)\n\n1.c Generate a synthetic user query using the context of prefix chunk  + seed prompt\n\n1.d Generate a synthetic assistant answer using the context of prefix chunk + synthetic user query\n\n2. Construct a dataset of prefix chunk + synthetic user query + synthetic assistant answer\n3. By minimizing the KL divergence between next-token predicted distributions produced by the model conditioned with the prefix chunk and the model conditioned with the cartridge over the parameters in the cartridge, we recover the optimal cartridge\n\nEvaluations of LLaMa3-8B and Qwen3-4B on NIAH, LONGHEALTH, MTOB, and QASPER show very little quality degradation when using the cartridge instead of the long common prefix, despite ∼38.6× less memory and ∼26.4× higher throughput.", "questions": "The authors did not compare the QA performance of their cartridge system with the most obvious candidate, which is the RAG. Comparison with other prompt-compression methods (Lexico, Minicache, PALU, and KVPress) is also not provided. What would be the trade-off in accuracy and throughput?\n\nComposition mechanism: The paper briefly notes that multiple CARTRIDGES can be composed at inference time, but the mechanism is unclear. Are there limitations in compositional depth or order sensitivity?\n\n\nEvaluation breadth: The appendix lists several potential applications (e.g., summarization, retrieval, reasoning), yet the main results only cover QA and translation tasks. Were there any qualitative checks on these other use cases, even if not reported? If not, could you comment on any limitations that prevented such evaluations?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper makes a clear, practical contribution. The proposed method has a high potential. Using self-study with chunking, the approach handles corpora beyond the model’s window. Cartridges can be concatenated at inference time. They are easy to serve with the existing infrastructure.\n- The paper offers useful design ablations, including parameterization and initialization strategies.", "weaknesses": "The reproducibility of this paper is a potential weak point. Authors don’t provide code to reproduce their results, which may significantly diminish confidence and the potential impact.\n\nEmpirical results lack evaluations on real-world heterogeneous benchmarks, such as those with code or multimodal documents.\n\nOperational procedures of versioning and updating cartridges are not discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "i5hGlzTPk2", "reviewer": "Reviewer_AHXY", "summary": "This paper introduces Cartridges, a memory-efficient, high-throughput, and general-purpose solution for processing long contexts in LLMs, promising to replicate the functionality of in-context learning (ICL). The authors argue that existing LLMs rely on costly KV caches for long-context processing or ICL. While KV cache compression can reduce memory, its performance collapses at high compression ratios.  \n\nTo this end, they propose Cartridges, a lightweight, parametric cache that is trained offline to represent the text corpus. Direct optimization with the next-token prediction objective yields poor generalization, so they introduce self-study, training on synthetic conversational histories with a context-distillation objective.  \n\nOn tasks such as NIAH, LongHealth, and MTOB, Cartridges achieves strong accuracy while reducing memory by 38.6× on average and boosting throughput by 26.4×, significantly outperforming state-of-the-art cache-compression baselines.", "questions": "See Weaknesses.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Long context is a critical research area for current LLMs, especially when it comes to both the ability to enhance the LLMs' own capabilities by processing ultra-long context as well as their efficiency.\n2. Cartridge, as an architectural innovation, breaks away from the traditional KV-cache optimization paradigm, and its self-study training can also inspire future long-context training research.\n3. The proposed method delivers extreme efficiency gains (38.6× memory reduction and 26.4× throughput boost) on very long contexts (up to 484k tokens) while still maintaining superior performance.", "weaknesses": "1. I would appreciate a better organization of the related work, instead of scattering it across Section 2.1, Appendix B, and Appendix E. From my perspective, it is a paper relatively focused on long-context efficiency, so it should be more concentrated on the comparison between cache optimization and linear attention. The key theoretical insights that appear only in Appendix B and E should also be highlighted in the main text. Although linear attention does require pre-training from scratch and non-trivial model conversions, a direct discussion of their trade-offs, whether in approach difference or in experimental performance, is essential for clarifying your methods.\n2. The authors should more clearly distinguish Cartridges from prefix-tuning methods, for both learn a parametric cache, so a detailed explanation of their methodological distinctions and an experimental comparison are needed.\n3. The paper reports results only in the Llama Series. Because Cartridges is presented as a general paradigm, additional experiments on Mistral and Qwen would strengthen the claim.\n4. The authors claim that Cartridges can replicate the functionality of ICL; this should be backed better with evaluation on LongICLBench[1] or similar scalability tests in [2]. Likewise, their stated ability to support ultra-long contexts should be corroborated with results on RULER [3].\n\n\n[1] LongICLBench: Long-context LLMs Struggle with Long In-context Learning https://arxiv.org/abs/2404.02060\n\n[2] Many-Shot In-Context Learning https://arxiv.org/abs/2404.11018\n\n[3] RULER: What's the Real Context Size of Your Long-Context Language Models? https://arxiv.org/abs/2404.06654", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)