Autoencoding-Free Context Compression for LLMs via Contextual Semantic Anchors
Authors: Xin Liu
Runsong Zhao
Pengcheng Huang
Xinyu Liu
Junyi Xiao
Chunyang Xiao
Tong Xiao
Shengxiang Gao
Zhengtao Yu
JingBo Zhu
Venue: ICLR 2026 Poster (2026)
Keywords: Context Compression
Long Context LLMs
LLM Memory
Primary area: foundation or frontier models, including LLMs
TL;DR: SAC achieves efficient context compression without autoencoder training by using contextual semantic anchors and bidirectional attention.

Abstract: Context compression is an advanced technique that accelerates large language model (LLM) inference by converting long inputs into compact representations.
Existing methods primarily rely on autoencoding tasks to train special compression tokens to represent contextual semantics.
While autoencoding tasks enable compression tokens to acquire compression capabilities, we remark that such capabilities potentially conflict with actual downstream
task requirements, prevent the models from learning the features more beneficial for real-world usage.
Based on this observation, we propose Semantic-Anchor Compression (SAC), a novel method that shifts from autoencoding task based compression to an architecture that is equipped with this compression capability \textit{a priori}. Instead of training models to compress contexts through autoencoding tasks, SAC directly selects so-called anchor tokens from the original context and aggregates contextual information into their key-value (KV) representations. To ensure that anchors can effectively collect information, SAC introduces two key designs: (1) anchor embedding, a learnable embedding vector attached to the selected anchor tokens to mark compression carriers and (2) bidirectional attention modification, which enables anchor tokens to integrate information from the entire context. Experimental results show that SAC consistently outperforms existing context compression methods across different compression ratios and model sizes on question-answering and long-context summarization tasks. Our data, model and code have been released at \href{https://github.com/lx-Meteors/SAC}{https://github.com/lx-Meteors/SAC}.

Published as a conference paper at ICLR 2026
AUTOENCODING-FRER CONTEXT COMPRESSION FOR LLMS VIA CONTEXTUAL SEMANTIC ANCHORS
Xin Liu Runsong Zhao' Pengcheng Huang',
Xinyu Liu'
Junyi Xiao
Chunyang Xiao
Tong X Shengxiang Gao
Zhengtao Yu? Jingbo Zhu 1.2
1 School of Computer Science and Engineering, Northeastern University, Shenyang, China
2 NiuTrans Research, Shenyang, China
3 Kunming University of Science and Technology, Kunming, China
ABSTRACT
Context compression is an advanced technique that accelerates large language model (LLM) inference by converting long inputs into compact representations. Existing methods primarily rely autoencoding tasks to train special compres- sion tokens to represent contextual semantics. While autoencoding tasks enable compression tokens to acquire compression capabilities, we remark that such ca- pabilities potentially confiict with actual downstream task requirements. prevent the models from learning the features more beneficial for real-worth usage. Based on this observation, we propose Semantic-Ancho Compression (SAC), a novel method that shifts from autoencoding task based compression an architecture that is equipped with this compression capability a priori. Instead of training models to compress contexts through autoencoding tasks, SAC directly selects so-called anchor tokens from the original context and aggregates contextual in- formation into their key-value (KV) representations. To ensure that anchors can effectively collect information, SAC introduces two key designs: (1) anchor embed- ding, a learnable embedding vector attached to the selected anchor tokens to mark compression carriers and (2) bidirectional attention modification, which enables anchor tokens to integrate information from the entire context. Experimental results show that SAC consistently outperforms existing context compression methods across different compression ratios and model sizes on question-answeriig and long-context summarization tasks. Our data, model and code have been released at https://gituub.com//x MeteorssSA
INTRODUCTION
The expanding scope of large language models (LLMs) to tasks such as processing long docu-
ments (Liu et al. 2024b) Li et al. 2024
Duan et al. [2025] Zhao et al. 20266), maintaining multi-turn
dialogue coherence (Zhang et al. 2025 Yi et al. [202 Guan al. (025). and generating responses
grounded in extensive external knowledge (Lewis et al. 2020 Karpukhin et al. 2020; Huang et al.) 2025) necessitates the incorporation of vast contexts into the model input. However, directly process- ing such extremely long contexts is fraught with challenges, including prohibitive computational costs, significant inference latency, and performance degradation, largely caused by the "lost-in-the- theddd-eteemiddee"
phenomenon (Liu et al. 2024a).
To address these challenges. recent studies have proposed context compression (Chang al. 2024 Li et al. 2025al Lv et al. 2026. Tang et al. 2026a b). a technique that typically appends special tokens (i.e. compression tokens) the end of the context and leverages the LLM's causal attention mechanism to compress contextual information into a compact representation within these tokens. Once this compact representation obtained, the LLM can generate responses conditioned on it, rather than being conditioned on the entire original context. The reduction in context length leads to substantial decreases in both inference time and GPU memory consumption. While effective, these approaches face key limitation: the compression tokens are randomly initialized and lack inherent
"quual contribution. Email: [liuxinl, zhaors| | @mails neu. edu.edu.c I Corresponding author. Email: x actong@ma weu.eeeeee edu.ch

---

Published as a conference paper at ICLR 2026
the i-th token of the context compressed representation
the i-h token of the question
c
the i-h token of the answer
compression token
M
AE
instruction token for the AE task
LM
instruction token for the LMIQA task
Pretraining
Prefraining
Finetuning
Target LLM
x
Target LLM
Target LLM
?
4%
LLMILORA)
LLM(LoRA)
LLM(LoRA)
[a] Autoencodiing
(b) Language modeling task
(c) answering task
Figure 1:
Three tasks for training the context compressor introduced by ICAE and followed by numerous works. The training uses (a) Autoencoding task and (b) Language modeling task pretrain the encoder, then finetunes on (c) Question answering task.
semantic information. To address this limitation, prior works (Ge et al. 2024 Wang et al. 2024 Li et al. 2025b Zhao et al. 2025; Tang et al. 2025b) typically rely on extensive pretraining on autoencoding (AE) and language modeling (LM) tasks (as shown in Fig. D to endow compression tokens with the ability to carry contextual information. However, while the model learns to reconstruct contexts from compression tokens through the AE task, such reconstruction objective with actual objectives (e.g. QA from compressd tokens) that downstream requires (we analyse such suboptimal and costly pretraining stage raises critical research question: is it possible to design a compression architecture that inherently understands context without demanding AE phase?
misalign
misalignment subsection (.T). Thus, this reliance
To answer this question, our work introduces Semantie-Anchor Compression (SAC) (Figure 2 novel architecture for the context compression task. Instead of appending new special tokens and requiring extensive autoencoding pretraining to learn their representations, SAC directly selects representative tokens from the original context to act as "anchor tokens" for compression. By leveraging these semantically meaningful anchors from the input itself, SAC incorporates natural semantic priors that obviate the need for autoencoding pretraining. To mark their special role, these selected tokens augmented with dedicated "anchor embeddings", enabling the LLM to distinguish them from regular tokens. Furthermore, to enhance their compression capabilities, we modify the standard causal attention to bidirectional attention mechanism. This allows anchor tokens to access information from the entire context, rather than being restricted to only preceding tokens. These modifications collectively foster more effective context compression by providing anchor tokens with both distinct representations and comprehensive contextual awareness. To evaluate the effectiveness of SAC, conduct experiments question answering tasks and long-context summarization tasks, where it consistently outperforms strong baselines. Results also show that 1) our proposed method consistently improves over strong baselines across different compress ratios, 2) our proposed architecture achieves its best performance in a simpler training setting without autoencoding training, arguably because the anchor tokens already contain enough information about the original context. Our analysis reveals that SAC's compressed representations more closely resemble original context token KVs feature space, which arguably enables LLMs to better understand them during inference.
RELATED WORKS
2.1 COMPRESSION METHOD
Many methods focus on reducing prompt lengths to improve LLM inference speed. CC (Wingate) et al. 2022) utilizes contrastive learning compress specific natural language prompts into shorter and unique soft prompt tokens. However, cannot generalize to unseen prompts and requires retraining for prompts. GIST (Mu et al.) 2023) compresses original prompts into KV values through finetuming and can handle arbitrary unseen contexts. AutoCompressor (Chevalier et al. 2023) recursively combines compressed vectors with sub-prompts and aggregates all compressed vectors to
2

---

Published as a conference paper at ICLR 2026
C
the i-th taken of the context
compression token
M
compressed representation
M
LLM(LORA)
LLM(LoRA)
C2
C3
C4
C5
C6
C3
C4
C5
C6
(a) overview of previous methods
(b) the overview SAC
Figure 2: The difference between SAC and previous methods. While previous methods (a) compress contextual information into dedicated special tokens (referred to as compression tokens), SAC (b) compresses the context directly into the original contextual tokens themselves. Here, M can represent either the output from the final layer of the LLM or the key-value pairs across all transformer layers, which are later used as compressed representations for LLM inference.
construct the final representation, enabling compression of longer contexts. However, both GIST and AutoCompressor require finetuning the LLMs (referred to later as target LLMs) to perform inference, which may affect LLMs' original capabilities.
ICAE (Ge et al. 2024) formulates context compression training a general encoder that compresses contexts into compact representations understandable by target LLMs without finetuning target LLMs. To train the encoder, ICAE introduces autoencoding tasks and performs joint pretraining with language modeling tasks, followed by finetuning on downstream tasks. 500xCompressor (Li et al. 2025b) improves upon ICAE by replacing the compression carrier from the last layer output of compression tokens with KV values at each layer, achieving higher compression ratios. EPL (Zhao ct al. 2025) identifies that ICAE and 500xCompressor neglect the impact of positional encoding and proposes distributing compression token position IDs uniformly across the entire context rather than placing them the end. However, these methods still rely on autoencoding tasks to endow the compression tokens with the ability to carry contextual information.
Another category of prompt compression methods is based token selection, which selects rep- resentative tokens from contexts based on token importance. SelectiveContext (Li et al., (003).. LLMLingua (Jiang et 2023), and LMnggL (Jiang et al., (024) employ causal small language models to evaluate token importance based on information entropy. LLMLingua-22 (Pan) al. 2024a) distills token classifier compute the probability of each token to be preserved. PerceptionCompress (Tang et al. 2025al preserves critical information at the token level while removing tokens that distract the LLM's attention. These works demonstrate that LLMs can un- derstand original contexts using small number of representative tokens. However, they do not perform compressed tokens training which limits the usability the selected tokens by target L.M.s. Our proposed SAC can be seen as combination of token selection methods and compressed token training methods: it derives and train compressed representations that are based on tokens selected directly from the context and indeed is compatible with the token selection methods above.
2.2 BIDIRECTIONAL ATTENTION
Recent studies have shown that, removing the decoder's unidirectional causal constraint and intro- ducing bidirectional attention can effectively enhance the model's representational capacity (Wang et al. 2020). For instance, NV-Embed (Lee et al., (2025) replaces causal attention with bidirectional attention during contrastive training, achieving strong performance on general text embedding and dense vector retrieval tasks. LLM2Vec (BehnamGhhader et al.] 2024), by enabling bidirectional attention alongside masked next-token prediction, significantly improves the model's ability to cap- ture global semantics in text embedding tasks. These works indicate that bidirectional attention advantageous for acquiring global semantic information and robust contextual representations. However, its effectiveness in context compression tasks remains underexplored. Motivated by these findings, we incorporate bidirectional attention into the compressor to enhance contextual modeling during the compression phase.
3

---

Published as a conference paper at ICLR 2026
3 METHOD
3.1 TASK FORMULATION
Context compression is formally defined as follows: encoder & compresses a context C' (c1, (2... 991) into compact representation M with M E(C). Subsequently, target LLM leverages the compressed representation M in place of the original context C to perform various tasks, such question answering.
To train the encoder to effectively extract contextual information, ICAE introduces three objective functions. The autoencoding loss LAE ensures that the compressed representation M generated by & preserves all tokens in the context by learning to reconstruct the entire context C' regardless of the token relative importance, as shown in Figure 1]; mathematically, LaE log M)C The language modeling loss LLM encourages M to maintain predictive capability for future context c +I04), enabling proactive information planning, as shown in Figure LTp: mathematically, LLM log P(C" [M). During pretraining. LaE and LLM jointly optimized to obtain an initially effective encoder
C" = (9C++,,CC
Additionally, during finetuning, the downstream task loss LQa (taking QA exemplar down- stream task) enhances the ability of M to extract information that is potentially relevant for down- stream tasks. For example, for question answering (QA) tasks, the encoder learns through ZQA to identify and preserve information in the context that likely be queried, enabling accurate answer generation A = (a1, .., as shown in Figure mathematically. foa log P(A Q))
, (1,A1) when presented with subsequent questions Q = (4), 92, .)Q)).
3.2 SENHNOIC-ANCHOR COMPRESSOR
key distinction between our approach Semantic-Anchor Compression (SAC) and previous methods is that we derive compressed representations directly from selected context tokens, as shown in Figure This involves selecting a subset of tokens from context C' as anchor tokens C C. We believe that good selection strategy benefits SAC, Following EPL, our default strategy divides the entire context into |S| chunks and selects the middle token from each chunk. This setting helps maximize coverage of context C. As illustrated in Figure 33,, each selected token c; S is enhanced with the anchor embedding A. yielding an embedding sequence E = (e1, e2c1]=
E;- Emb(c)) -..Es EA
(1)
where Laes an indicator function that equals 1 when G E S and 0 otherwise. Following previous works, we employ a LLM with LoRA parameters LORR as the compressor: M = ECC) LLM(EE(LORA). Overall, using original tokens from the context avoids learning compression tokens from scratch and we empirically validate that such design improves learning efficiency.
We notice that because the encoder uses causal attention, the anchor tokens do not have visibility to the full sentence, limiting their representation power. Hence in SAC, we modify the LLM to replace causal attention with bidirectional attention (see Figure (p), to enhance the LLM's encoding capability. Notably, this bidirectional attention operates across all tokens, rather than being limited to the anchor tokens, allowing the model to capture richer contextual dependencies.
M can be either the output of anchor tokens from the LLM's final layer or the key-value pairs from each layer. Following C00ppeoprrs,orr we use key-value pairs as the compressed representation M. For the encoder, we adopt the same language model as the decoder and leverage the original context's KV cache, enabling the decoder to comprehend the compressed representation more readily without requiring additional semantic alignment During pretraining, we only use LLM and do not use LAE to train the compressor. Following previous work, we use LQA for finetuning.
'oor context compression, adopting different L.MSs for encoder and decoder will create semantic gaps hetween the and will impair performance, as noticed by (L.v al. 2026).
4

---

Published as a conference paper at ICLR 2026
Ci
the token of the context
anchor token
attention flow
e
anchor embedding
M
compressed representation
c
e2
33
44
e5
66
M
eΛ
eA
Emb(
C1
C2
C3
C4
C5
C6
21
22
e3
e4
e5
66
(a) semantic anchor embedding
(b) bidirectional attention
Key differentiators within SAC model architecture. (a) Representative tokens are trans- formed into anchor tokens through anchor embeddings. (b) The encoder in SAC adopts bidirectional attention, while the decoder operates with causal attention.
Figure 3:
4 EXPERIMENTS
4.1 EXPERIMENTAL SETTING
Dataset. For continued pretraining, we use the corpus SlimPajama-66 (Soboleva et al. 2023). During finetuning and evaluation, we employ the standard MRQA (Fisch et al.) 2019) question- answering dataset, which consolidates multiple QA tasks and standardizes them into a unified format. We evaluate SAC on both test sets, namely in-domain (ID) and out-of-domain (OOD), to comprehensively assess in-distribution fitting ability and cross-domain generalization performance.
Implementation Details. SAC utilizes Llama-3.2- IB (Grattafiori & Dubey 2024) both the encoder and target LLM. We equip the encoder with trainable LoRA (Hu et al.) 2022) adapters (rank 128, 256), while the target LLM parameters remain frozen. For each context, partition into sub-contexts 510 tokens each. The compressor compresses each sub-context into a sub-compressed representation, and subsequently concatenates these sub-compressed representations form the complete compressed representation. The number of anchor tokens |S| L/r] is determined by the compression ratio and the length of the sub-context L. We train all models in two stages: pretraining for 20,000 optimization steps followed by finetuning for an additional 20,000 steps, both conducted with batch size of 16. Complete hyperparameter configurations are provided in Appendix [A]
Baselines. We use the Llama-3.2-1B model trained on the MRQA (Fisch et al. 2019) dataset as an uncompressed baseline (denoted "Full-FT"). We compare our method against several context compression techniques: for hard compression, we choose LLM LLMLingua-2 (Pan et al.) 2024b) and evaluate its performance on the Full-FT model; for soft compression, we select ICAE (Ge al. 2024), 500xCompressor (Li et al. 2025b), Activation Beacon (Zhang al. 2024), DAST (Chen et al. 2025), and EPL (Zhao et al., 2025). To ensure fair comparison, all these soft compression baselines are trained using the same experimental settings and data our SAC method.
4.2 FINE-TUNING RESULTS
Tables I and 2 report the evaluation results of SAC on in-domain and out-of-domain MRQA datasets.
Overall Performance. For in domain results, SAC consistently outperforms all baseline models as shown in Table [ EPL, SAC shows a maximum improvement of 23.5% Fl 26.8% EM and a minimum improvement of 6.7% F1 / 8.2% EM for in-domain evaluations. A similar trend can be observed for oft-o--domain evaluations in Table P where the maximum improvement is 27.4% 4% 28.9% EM, with a minimum improvement of 6.9% F1 / 9.2%% EM compared to ICAE and EPI.
At a 15x compression ratio, compared with the weaker ICAE and the stronger
5

---

Published as a conference paper at ICLR 2026
Table For the fine-tuning results, we report in-domain performance on the MRQA datasets using ROUGE-1 1 F1 (Lin 2004) and exact match (EM) (Maalouly, 2022). The compression ratio of Lingua-
2 Pan al. 2024b set to 5x.
Methods
SQuAD
NewsQA
TriviaQA
SearchQA
HolpotQA
NQ
Average
EM
Full-FT
77.69
59.71
63.50
46.04
68 88.80
60.54
73.25
62.07
74.78
59.26
71.01
53.47
71.51
56.85
Lingua-2
32.93
19.57
26.78
13:20
9.67
8.12
45.40
31.80
36.
22.05
40.08
22.01
31.83
19.46
15x compression construint
500x
40.68
24.97
32.01
16.76
53.84
44.86
65.65
53.70
53.01
36.30
50.93
33.26
49.35
34.98
Beacon
37.53
14.31
31.10
7.72
48.85
4.47
39.06
22.67
45.01
27.35
40.29
15.82
40.31
15.39
DAST
36.33
22.65
31.55
16.29
56.92
48.14
68.07
56.51
54,02
37.93
52.10
35.31
49.83
36.14
EPL 44.58
27.91
33.34
16.69
56.16
47.09
66.36
54.13
54.88
38.38
53.80
35.71
51.52
36.65
SAC
47.43
30.25
36.55
18.07
61.13
52.19
68.97
56.76
58.83
41.86
56.79
38.88
54.95
39.67
Table 2: For the fine-tuning results, we report out-of-domain performance on the MRQA datasets
using ROUGE-1 F1 (Lin 2004) and exact match (EM) (Maalouly 2022). The compression ratio of
Lingua-2 Paan et al. 2024b) is set to 5x.
Methods
BioASQ
DROP
DouRC
RACE
RE
TQA
Average
F1
EM
F1
EM
F1
EM
F1
EM
F1
EM
F1
EM
F1
EM
FT
49.37
36.77
44.67
34.46
48.82
35.51
35.57
9.64
83.34
72.46
53.32
32.40
52.51
36.87
Lingua-2
27.76
19.48
27.28
18.83
27.07
18.32
17.54
4.15
39.30
20.59
28.42
15.83
27.90
16.20
15x compression construint
ICAE
35.51
24.47
30.39
21.96
13.78
9.06
1.5.21
3.71
55.24
40.33
34.75
21.56
30.81
20.18
500x
36.30
25.93
33.46
23.55
20.53
12.72
18.49
3.41
54.37
41.11
41.09
25.82
34.04
22,09
Beacon
36.07
7.65
34.39
14.90
33.78
14.99
26.68
3.71
53.28
23.91
37.27
12.97
36.91
13.02
DAST
36.57
27.06
31.90
22.42
21.56
14.26
16.31
3.26
48.54
35.92
36.69
22.75
31.93
20.95
EPL
40.52
28.52
32.16
22.29
25.70
16.39
20.97
4.01
59.75
46.34
25 42
36.74
SAC
41.31
28.66
36.72
27.48
28.94
18.99
23.35
4.90
61.04
47.90
44.21
28.21
39.26
26.02
EPL performs strongly overall thanks to its position layout designed for soft compression. Interestingly, SAC can be building upon EPL with enhanced compressed token representations introduced Section 3 The overall results show that the token representation enhancement effective SAC outperforms EPL only overall but also for each individual datasets in MRQA.
Comparison with EPL. As shown in Tables] and [
4.3 ABLATION STUDIES
In this subsection, we first show ablation studies two SAC key components, namely anchor embeddings and bidirectional attentions. Then we study the effect of using different strategies to choose anchor tokens. Finally, show empirically that our carefully designed SAC performs better without autoencoding training, property that has motivated the SAC design.
Table 3: Component ablation results. We report the average F1/EM performance of the model on in-domain (ID) and out-of-domaiie (OOD) tasks after removing the bidirectional attention (w/o mask) and the anchor embedding (w/o anchor). Full results on all tasks are provided in the Appendix B.3 including Table 16 and Table 17
In-domain
Out-of-domain
Methods
TriviaQA
HotpotQA
Averuge
BioASQ
Textbook
Average
F1
EM
F1
EM
F1
EM
F1
EM
F1
EM
F1
EM
SAC
65.06
55.93
67.41
50.28
66.24
53.11
44.66
31.45
52.24
32,93
48.45
32,19
SAC(w/o mask)
62.60
53.27
64.63
47.43
63.62
50.35
41.93
30.65
48.29
29.67
45.11
30.16
SAC(wi anchor)
63.90
54.81
65.25
48.31
64.58
51.56
43.70
31.78
51.59
32.20
47.65
31.99
2This is because the anchor tokens selected by SAC share the same position IDs as the compressed tokens in
EPI.

---

Published as a conference paper at ICLR 2026
Component Ablation. As shown in Table 3 our ablation study demonstrates the critical roles of the bidirectional attention and anchor embedding. Removing either component results in significant performance degradation in both in in-domain (ID) and out-of-domain (OOD) settings. The bidirectional attention mechanism enables anchor tokens to more effectively integrate information from the entire context, producing compressed representations that more heneficial for downstream tasks. Meanwhile, the anchor embedding provides explicit structural signals that guide the model to accurately identify and process these anchor tokens, thereby ensuring the effectiveness of information compression.
Token Selection. Table 4 shows the results using different token selection strategies for SAC. The results indicate that random se- lection (Random) significantly degrades performance arguably for two reasons: the selected to- kens lack semantic importance, and their random positions re- sult in poor global coverage of the context, which together hin- der effective representation. In contrast, information-base lection (Lingua-2) and our de- fault strategy achieve on par results, and both substantially outperform existing baselines in Tables 2 This demonstrates that the SAC architecture can effectively leverage and enhance any high- quality token selection strategy, rather than relying specific choice, highlighting the generality and robustness of the SAC framework.
Table 4: Token selection results. Different token selection strate- gies are compared, including Random selection, Lingua-2-based selection (Pan et al. 2024b), and our uniform selection (Zhao et al. 2025). Average FI/EM scores are reported across in-domain (ID) and out-of-domain (OOD) tasks.
ID-Avergee
OOD-Average
Methods
F1
EM
F1
EM
SAC SAC(Random)
63.63
46.95
47.72
32.30
59.54
43.58
45.35
30.44
se- SAC(Lingua-2)
63.26
46.54
47.43
32.13
Table Ablation study on the effects of autoencoding (AE) and language modeling (LM) objectives.
In-domain Average
Out-of-domain Average
Methods
5x
15x
51x
5x
15x
51x
F1
EM
F1
EM
F1
EM
F1
EM
F1
EM
F1
EM
SAC
63.63
46.95
54.95
39.67
46.37
33.08
47.72
32.30
39.26
26.02
32.24
21.44
500x(w/ LM only)
53.23
38.70
49.76
35.71
44.46
31.41
38.22
25.73
33.99
22.05
30.09
18.99
500k(w/ AE+LM)
55.26
34.98
43.19
30.26
25,40
34.04
22.09
SAC(W/ AE only)
56.55
40.34
49.93
35.33
43.95
30.64
42.08
27.98
35.50
23.29
28.77
18.32
SAC(W/ AE+LM)
62.04
45.80
51.73
36.67
44.69
31.37
47.26
32.25
37.23
23.96
31.01
19,90
AE Effect. Soft compression methods often use AE so that the compressed representations learn to reconstruct the original context. However, AE objective itself can be a limiting factor as the reconstruction target can potentially be misaligned with downstream tasks. SAC is designed with compressed representations directly derived from the contexts to eliminate the need for AE and we show our empirical investigations in Table The experimental results show that training with only the AE objective leads to substantial performance drop, and even when combined with the LM objective, the performance remains inferior to that of the full SAC model.
worth noting that while ICAE (Ge et al.) 2024) empirically shows that combining AE with LM tasks yields better results, the results do not generalize to other architectures such as 500xCompressor where Table 5 shows mixed picture for performance. Overall, the results in Table S raise questions regarding the necessity of autoencoding tasks and suggest that autoencoding may not be entirely essential for context compression methods.
4.4 SCALABILITY RESULTS
To verify the generalizability of SAC across model scales, we conduct additional experiments on the larger Llama-3.2 3.B-BB and Llama-3. 1-B models. We show EPL as our baseline which performs the As shown in Table SAC outperforms EPL
second best in main experiments shown in Table I and 2 across all model sizes and evaluation settings, consistently achieving higher F1 and EM scores. On the 3B model. SAC achieves an average Fl score of 50.48 and an EM score of 34.73, outperforming the EPL baseline (47. 17.46/ / 31.82). On the 8B model, SAC scores 52.31 (F1) and 35.93 (EM), again
7

---

Published as a conference paper at ICLR 2026
Table We compare the performance SAC and EPL across different model sizes [Lama-3322-3B and Llama-3..13 under fixed 15x compression ratio. Evaluations are conducted in both in-domain (ID) and out-of-domain (OOD) settings, and we report ROUGE-1 F1 and Exact Match (EM) scores.
In-domain
Out-of-domain
Methods
SearchQA
NQ
Average
DROP
RE
Average
FM
F1
EM
F1
EM
F1
FM
EM
FM
EPL(3B)
73.61
61.56
63.59
44,43
68.60
53.00
46.20
36.46
65.73
53.70
55.97
45.08
SAC(3B)
73.88
62.29
65,98
47.18
69.93
54.74
48,46
38.39
75.11
62.89
61.76
50.64
EPL(8B)
74.86
63.23
67.24
48.71
71.05
55.97
49.98
39.92
69.77
57.84
59.88
48.88
SAC(8B)
76.92
65.29
67.77
49.21
72.35
57.25
51.55
40.65
78.91
67.71
65.23
54.18
surpassing the EPL baseline (50.82 34.42). We remark that as the model goes larger, the gain does not seem to diminish. These consistent gains demonstrate that SAC effectively generalizes to larger model sizes and maintains its performance advantage. More detailed results on MRQA can be found in Tables [8 and 19
4.5 SENSITIVITY ANALYSIS FOR COMPRESSION RATIOS
We conduct a detailed evalua- tion of model performance un- der different compression ratios
Table 7: In-domain and out-of-domain performance under differ- ent compression ratios.
(5x, 15x, and 51x), using the same experimental setup as Ta- [ The corresponding results are presented in Tables
In-domain Average
Out-of-domain Average
Methods
F1
EM
F1
EM
Sx compression
As expected, the F1 and EM scores of all methods decrease as the compression ratio increases from 5x to 51x, since higher compression ratios result more information being discarded. At highest compression ratio of 51x, the performance of differ- ent compression methods varies across datasets (see more de- tailed results in Appendix B.2 Tables [1] and 15): some meth- ods perform well on certain datasets but underperform on oth- ers. Nonetheless, SAC consis-
ICAE
47.53
33.82
31.22
20.51
500x
55.26
40.14
38.46
25.40
EPL
62.90
46.33
46.95
31.30
SAC
63.63
46.95
47.72
32.30
1.5x compression
ICAE 500x
44.50
31.28
30.81
20.18
49.35
34,98
34.04
22.09
EPL
51.52
36.65
36.74
23.83
SAC
54.95
39.67
39.26
26.02
5 compression
ICAE
40.39
27.99
27.98
17.85
500x
43.19
30.26
30.43
20.09
EPI.
43.22
30.26
30.22
19.48
SAC
46.37
33.08
32.24
21.44
tently achieves the best average performance across all compression ratios, demonstrating strong robustness and stability.
4.6 LONG-CONTEX. SUMMARIZATION RESULTS
To further validate the robust- ness and usefulness of SAC, we conduct additional experi- ments on two long-context sum- marization tasks using the QM- Sum (Zhong et al. 2021 and GovReport (Huang et al. 2021) datasets. In these experiments, the models are configured with maximum input length of 32K tokens and are trained using
Table 8: We compare the performance of SAC and EPL on long- context summarization tasks, including QMSum (Zhong 202T) and GovReport (Huang et al.) 2021) reporting ROUGE- Fl scores.
Methods
QMSum
GovReport
Average
EPL SAC
14.82
20.40 22.03
17.61
14.95
18.49
8

---

Published as a conference paper at ICLR 2026
datasets with x5x compression ratio. Both SAC and EPL models trained and evaluated on the corresponding datasets.
The results in Table 8 show that SAC maintains strong performance on long-context summarization tasks, surpassing EPL across all datasets. Because EPL is trained with AE objective, the results suggest that removing the AE objective for SAC does not impair the model's capability to model long contexts. Moreover, SAC able to capture sufficient global information to support tasks that require comprehensive understanding of the entire context, highlighting its adaptability and generalization performance in long-contexi generation scenarios. More details are provided in Appendix B.5
5 ANALYSIS
5.1 AE ANALYSIS
We attribute the negative impact of the AE task on SAC primarily the following two related factors.
Intuitive Perspective. The AE objective requires the compressed representations to reconstruct the complete context, forcing the model to encode a large number of tokens within its limited capacity; some tokens might require significant model memories while not useful for downstream tasks, thereby weakening model's ability retain task-critical information.
AE-LM Gradient Perspective. In Figure g we visualize the gradient cosine similarity between the autoencoding and the language modeling losses during training. The figure shows that while the two gradients largely align at the very beginning, the cosine similarity quickly approaches zero, suggesting that the two tasks are largely orthogonal in parameter space. This orthogonality causes the optimization of autoencoding to interfere with language modeling updates, thereby hindering downstream task performance.
5.2 ATTENTION VISUALIZATION
To understand the unique behavior of compressed models, we analyze the attention patterns of the final layer at a 5x compression rate.
At lower 5x compression rate, shown in Figure 6 SAC attention map presents clear positive diagonal, indicating that its compressed tokens primarily attend to local tokens, adhering to the attention prior similar to EPL. In contrast, the attention map of 500xCompressor appears more diffused which arguably complicates learning and impairs its performance. As the compression rate increases (as shown in Figure 7 and g). EPL attention pattern starts also diffuse while SAC exhibits a focused attention pattern, with its anchor tokens attending to only few key original context tokens. Hypothetically, the focused attention pattern retains important token information handle downstream tasks and explains the superior performance we empirically observe.
5.3 REPRESENTATION ANALYSTS
Key Representation Analysis. In the key representation space (see Figure 4 up). the compression tokens (orange) of SAC and EPL are distributed relatively close the context tokens (blue), while the compression tokens of 500xCompressor are clearly separated from the context tokens. This discrepancy arises from the architectural design of each method. EPL modifies the positional IDs of its additional compression tokens to share the same rotational angle (RoPE) as the original context tokens, thereby reducing the distance between them; without such modifications. 500xCompressor compression tokens do not align with context token key representations. In SAC, the anchor tokens are directly embedded within the original context, in consequence their representations maintain close semantic ties with the context tokens.
Value Representation Analysis. In the value representation space (see Figure bottom). the anchor tokens of SAC are uniformly distributed across all regions with the value representations of the context tokens, without forming independent sub-clusters. This suggests that SAC's anchor embedding strategy allows for compressed value representations that more closely match the distribution of the
'hhe autoencoding performance will also be impacted by the LM loss for the same orthogonality reason. We show empirically the AB performance in Appendix Table 14

---

Published as a conference paper at ICLR 2026
SAC Key Representations
EPL Key Representations
500x Key Representations
SAC Value Reoresentations
EPL Value Reoresentations
500x Value Representations
120
Figure 4: The t-SNE visualization shows the key representations of the final layer KV values for SAC, 500xCompressor (Li et 2025b), and EPL (Zhao et 2025). respectively.
original value space. contrast, although EPL's compression tokens also overlap with the context tokens, they overlap less than SAC's: the compression tokens appear relatively sparse in the regions and show a slight clustering tendency at the boundaries. This indicates that EPL's value representations still exhibit degree of semantic shift relative the original value space, which is even more pronounced in the 500xCompressor.
6 CONCLUSION
In this paper, we propose novel, autoencodingg-fres context compression method, Semantic-Anchor Compression (SAC). Unlike traditional context compression approaches, SAC does not rely on training compression tokens to reconstruct the original input. Instead, it directly selects representative anchor tokens from the context and aggregates contextual information into their key-value (KV) representations via a bidirectional attention mechanism. This approach effectively compresses lengthy contexts while avoiding any impairment the language model's original language modeling capabilities, while traditional context compression can hinder performance by introducing additional compression tokens and the autoencoding tasks, our analysis shows. Experimental results show that SAC achieves high compression rates while significantly outperforming existing methods across multiple question answering and long-context summarization tasks, demonstrating strong cross-task generalization and an effective balance between compression efficiency and model performance.
7 REPRODUCIBILITY STATEMENT
We declare that the work presented in this paper is reproducible. Our data, model, and code have been released at hhttps:/github.comm www.co////ettte///.... This code be used reproduce the experimental results. The repository includes detailed instructions for environment setup, running experiments, data processing, and result evaluation.
ACKNOWLEDGEMENTS
This work was supported in part by the National Science Foundation of China (Nos. 62276056 and U24A20334), the Yunnan Fundamental Research Projects 0..222001BB070022 the Yunnan Science and Technology Major Project (No. 202502AD080014, the Fundamental Research Funds for the Central Universities (Nos. N25BSS054 and N25BSS094), and the Program of Introducing Talents of Discipline to Universities, Plan 111 16099..
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapados, and Siva Reddy. LIm2vec: Large language models are secretly powerful text encoders, 2024. URL
/tltps://arxi...o http:/vvorrg/b///24/// 05961
Kaiyan Chang, Songcheng Xu, Chenglong Wang. Yingfeng Luo, Xiaoqian Liu, Tong Xiao, and Jingbo Zhu. Efficient prompting methods for large language models: A survey, 2024. URL
https::/arxiv. wttpa/awwww.......aoa//oooooaa// 0107
Shaoshen Chen, Yangning Li, Zishan Xu, Yinghui Li, Xin Su, Zifei Shan, and Hai tao Zheng. Dast: Context-aware compression in Ilms via dynamic allocation of soft tokens, 2025. URL
fttts:://axxiw.orr//bbbss 11499 11493
Alexis Chevalier, Alexander Wettig, Anirudh Ajith, and Danqi Chen. Adapting language models to compress contexts. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 3829-3846, Singapore, http://1222222///p/p/ main.
December 2023. Association for Computational Linguistics. doi: 10. 232. URL https: /ww.//acaaaaa http://aaaaaatooo....ooo.o org/2023 emnlp- innain.. 232/
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 2378-2378, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 00.18653/1//119-12444 htthttps
Shaohua Duan, Xinze Li, Zhenghao Liu, Xiaoyuan Yi, Yukun Yan, Shuo Wang, Yu Gu, Ge Yu, and Maosong Sun. Chunks as arms: Multi-armed bandit- bandit-guided sampling for long-context Ilm preference optimization. arXiv preprint arXiv:2508. 13993, 2025.
Matthew Dunn, Levent Sagun, Mike Higgins, V. Ugur Güney, Volkan Cirik, and Kyunghyun Cho. Searchqa: A new q&a dataset augmented with context from a search engine. CoRR, abs/ abs/1704.05519 2017. URL httt::
org /btrg////////////// 1704. 0
Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Eunsol Choi, and Danqi Chen. MRQA 2019 shared task: Evaluating generalization in reading comprehension. In Proceedings of 2nd Machine Reading for Reading Comprehension (MRQA) Workshop at EMNLP, 2019.
Tao Ge, Jing Hu, Lei Wang, Xun Wang, Si-Qing Chen, and Furu Wei. In-context autoencoder for context compression in a large language model, 2024. URL 2307. 06945
Aaron Grattafiori and Abhimanyu Dubey. The Ilama 3 herd of models, 2024.
URL https
//arxiv.org/ aarrxiv.org/abs/2007.22
Shengyue Guan, Haoyi Xiong, Jindong Wang. Jiang Bian, Bin Zhu, and Jian guang Lou, Evaluating based agents for multi-turn conversations: A survey, 2025. URLhttps:
abs/ 200..224555. 22458
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. International
Conference Learning Representations, 2022. URL https: fttt:::://opereyeee ///p///ppeeviieee/////ee www.nee//eeee nwt.forum?
li=nZeeKkeeFyt>
Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, and Lu Wang. Efficient attentions for long document summarization, 2021. URL https://arxxii wwtp://ggg/ab/aa..e.e/ 02111111.11122.
Pengcheng Huang, Zhenghao Liu, Yukun Yan, Xiaoyuan Yi, Hao Chen, Zhiyuan Liu, Maosong Sun, Tong Xiao, Ge Yu. and Chenyan Xiong. Pip-kag: Mitigating knowledge conflicts knowledge- augmented generation via parametric pruning. arXiv preprint arXiv:2502. /5543, 2025.
11

---

Published as a conference paper at ICLR 2026
Huiqiang Jiang, Qianhui Wu, Chin- Yew Lin, Yuqing Yang. and Lili Qiu. LLMLingua: Com- pressing prompts for accelerated inference of large language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference Empirical Meth- ods in Natural Language Processing, pp- 13358-13376, Singapore, December 2023. Associ- ation for Computational Linguistics. doi: 10. 1ttp///////22.mmm////.
URL https
//aclanthology. wwwp:/w.gg./..... emn moo-main... 8257
Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin- Yew Lin, Yuqing Yang, and Lili Qiu. LongLLMLingua: Accelerating and enhancing LLMs in long context scenarios via prompt compression. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computationall Linguistics (Volume 1: Long Papers), pp. 1658-1677, Bangkok, Thailand, August 2024, Association for Computational Linguistics. doi: 10. 16653//1/2024.aclloorr
URL
U http::ttp:///ttt patttss//aclantteen wow.aannnhoooooooooooooooo ogy. pogy.org/2222 aaaalcllonn 917]
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Regina Barzilay and Min- Yen Kan (eds.), Proceedings of the 55th Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pp. 1601-1611, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10.18653/v1//17-11447 URL https? //aclanthology wwtpp//ggg//p///.111
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6769-6781, Online, November 2020. Association for Computational Linguistics. doi: 10. https: http::://acaaattho wwwthhhoooy....o org/ 2020 emnlp-ma 5507
URL
Aniruddha Kembhavi, Minjoon Seo, Dustin Schwenk, Jonghyun Choi, Ali Farhadi, and Hannaneh Hajishirzi. Are you smarter than a sixth grader? textbook question answering for multimodal machine comprehension. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wes Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoe Le, and Slav Petrov. Natural questions: a benchmark for question answering research. Transactions of the Association of Computational Linguistics, 2019.
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. RACE: Large-scale ReAding comprehension dataset from examinations. In Martha Palmer, Rebecca Hwa, and Sebastian Riedel (eds.). Proceedings of the 2017 Conference on Empirical Methods Natural Language Processing. pp. 785-794, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 1/653//11D17-102/ URL
Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. Nv-embed: Improved techniques for training Ilms as generalist embedding models, 2025. URL
org/
abs httpa//////2...22
Omer Levy, Minjoon Sco, Eunsol Choi. and Luke Zettlemoyer. Zero-shot relation extraction via reading comprehension. In Roger Levy and Lucia Specia (eds.). Proceedings of the 2 Conference on Computational Natural Language Learning (CoNLI. 2017), pp. 333-342,, Vancouver, Canada, August 2017. Association for Computational Linguistics. doi: 10. 10.18653/11K117110344 URL https:/ fttt::://accaatholoog. wwwpe/ooyy.ooo/11oo/ 1034/
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. generation for knowledge-intent intensive nlp tasks. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS '20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 978191 9781713829546.
12

---

Published as a conference paper at ICLR 2026
Jiaqi Li, Mengmeng Wang. Zilong Zheng, and Muhan Zhang. LooGLE: Can long-contin language models understand long contexts? In Lun- Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computa- tional Linguistics (Volume I: Long Papers), pp- 16304-16333
Bangkok, Thailand, August 2024. Association for Computational Linguisties. doi: 10. 10.18855/v11/ 1/2553////22222c-llloog 859. URL http::///woaaatooooooooror////// -conog......-.g....... 8597|
Yucheng Li, Bo Dong, Frank Guerin, and Chenghua Lin. Compressing context to enhance inference efficiency of large language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Pro- ceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 6342- 6353, Singapore, December 2023. Association for Computational Linguisties. doi: 10.18653/v1/ 1023.emll--miin.39 URL https:: http:///aaaaathhooooooooooooo wwtptthhoooo..oog//222222222 emn main. 391/
Zongqian Li, Yinhong Liu, Yixuan Su, and Nigel Collier. Prompt compression for large language models: A survey. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguis- fics: Human Language Technologies (Volume I: Long Papers), pp. 7182-7195. Albuquerque, New Mexico, April 2025a. Association for Computational Linguistics. ISBN 979-8-89176- 189-6. doi: 10.18653/v1/2023 http:////2222...aooorrrg.. naacc-long..66.66 URL https::/aclaathol..om 00020/2225500
naacl - 1or.-oon........ 3687
Zongqian Li, Yixuan Su, and Nigel Collier. 500xCompressor. Generalized prompt compression for large language models. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mo- hammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 25081 25091, Vienna, Austria, July 2025b. Association for Computational Linguistics. ISBN 979-8-89176-251-44 doi: 18.15633/v1/ 2025 4025.acl-long.. 1219. URL B httpps://acaatttolll.ttt
org acl Lon.::.....1--o..01102...
Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74-81, Barcelona, Spain, July 2004. Association for Computational Linguistics. URL
/t0//////444000 W04 -013/
Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Trunsactions of the Association for Computational Linguistics, 12: 12:157-173, 2024a. doi: 10.1 a 00638. URL
httpss//acLanthooogy.oo/////////
Xinyu Liu, Runsong Zhao, Pengcheng Huang, Chunyang Xiao, Bei Li, Jingang Wang, Tong Xiao, and Jingho Zhu. Forgetting curve: A reliable method for evaluating memorization capability for long-context models. arXiv preprint arXiv:2410.04727, 2024b.
Kangtao Lv, Jiwei Tang, Langming Liu, Haibin Chen, Weidong Zhang, Shilei Liu, Yongwei Wang, Yujin Yuan, Wenbo Su, and Bo Zheng. Data distribution matters: A data-centric perspective on context compression for large language model, 2026. URLpattts://axxxv.oo /bs/
2662.01777
Nicolas El Maalouly. Exact matching: Algorithms and related problems, 2022. URL https 13899
a www.i..rrg//bbl...
Jesse Mu, Xiang Lisa Li. and Noah Goodman. Learning to compress prompts with gist tokens. In seirth Conference on Neural Information Processing Systems, 2023. URL https? htw.rreewweeeeettfoooroooooo-oee wtw.rrmmid=-2txpP1..333
//openreview.o net./
Zhuoshi Pan, Qianhui Wu, Huiqiang Jiang, Menglin Xia, Xufang Luo, Jue Zhang, Qingwei Lin, Victor Rühle, Yuqing Yang. Chin-Yew Lin, H. Vicky Zhao, Lili Qiu, and Dongmei Zhang. LLMLingua-2: Data distillation for efficient and faithful task-agnostic prompt compression. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp. 963-981, Bangkok, Thailand, August 2024a. Association for Computational Linguistics. doi: 10. 18.11653/ wttp://vv//22244 20t.///2.2.2.2/n-//iaaea 2024. find nd ata.g-aacl.5777
URL https: / www.aachlnoooo..ooooooo
13

---

Published as a conference paper at ICLR 2026
Zhuoshi Pan, Qianhui Wu, Huiqiang Jiang, Menglin Xia, Xufang Luo, Jue Zhang, Qingwei Lin, Victor Rühle, Yuqing Yang. Chin- Yew Lin, H. Vicky Zhao, Lili Qiu, and Dongmei Zhang. LImlingua- 2: Data distillation for efficient and faithful task-agnostic prompt compression, 2024b. URL bittps://arxiin org/ abs 2403. 12968
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Jian Su, Kevin Duh, and Xavier Carreras (eds.), Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383-2392, Austin, Texas, November 2016. Association for Computational Linguisties. doi: 10.18653/v1/ D16- URL htt.ts/s//claatttoooooooooo////
Amrita Saha, Rahul Aralikatte, Mitesh M. Khapra, and Karthik Sankaranarayanan. DuoRC: Towards complex language understanding with paraphrased reading comprehension. In Iryna Gurevych and Yusuke Miyao (eds.), Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), pp. 1683-1693, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10. 11111866//111111111 URL httpp:: beattp::/aclaantho..e
org/P18-1156/
Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hestness, and Nolan Dey. SlimPajama: A 627B token cleaned and deduplicated version of RedPa- jama.
2023. BRL[https:/hhugiinggfee 627B
co/
Latasets/cerebras/S111
Jiwei Tang, Jin Xu, Tingwei Lu, Zhicheng Zhang, Yiming Zhao, Lin Hai, and Hai-Tao Zheng. Perception compressor: A training-free free prompt compression framework in long context scenarios. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Findings of the Association for Computational Linguistics: NAACI, 2025, pp. 4093-4108, Albuquerque, New Mexico, April 2025a. Association for Computational Linguistics. ISBN 979-8-89 891-98888877---9-9 195-77 doi: 10. 10.18653/v122055. 12116653////225..nnnnnn-..-.
229.
Jiwei Tang, Zhicheng Zhang, Shunlong Wu, Jingheng Ye, Lichen Bai, Zitai Wang, Tingwei Lu, Jiaqi Chen, Lin Hai, Hai-Tao Zheng, and Hong-Gee Kim. Gmsa: Enhancing context compression via group merging and layer semantic alignment, 2025b. URL http:tts///xxx..ooo////o///// 25C5 12215
Jiwei Tang. Shilei Liu, Zhicheng Zhang, Qingsong Lv, Runsong Zhao, Tingwei Lu, Langming Liu, Haibin Chen, Yujin Yuan, Hai-Tao Zheng, Wenbo Su, and Bo Zheng. Read as human: Compressing context via parallelizable close reading and skimming, 2026a. URL https: / //arxiivorrrr////.o
abs 10os/2602, 01840
Jiwei Tang, Shilei Liu, Zhicheng Zhang, Yujin Yuan, Libin Zheng, Wenbo Su, and Bo Zheng Comi: Coarse-to-fine context compression via marginal information gain, 2026b. URLhttps
Adam Trischler, Tong Wang. Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. NewsQA: A machine comprehension dataset. In Phil Blunsom, An- toine Bordes, Kyunghyun Cho, Shay Cohen, Chris Dyer, Edward Grefenstette, Karl Moritz Hermann, Laura Rimell, Jason Weston, and Scott Yih (eds.), Proceedings of the 2nd Work- shop on Representation Learning for NLP, pp. 191-200, Vancouver, Canada, August 2017. Association for Computational Linguistics. doi: 10.18653/-11117-2222.. URL https 1/aclantholon http://lgog.....///oo... orwpyy/og//111122221
George Tsatsaronis, Georgios Balikas, Prodromos Malakasiotis, Ioannis Partalas, Matthias Zschunke, Michael R. Alvers, Dirk Weissenborn, Anastasia Krithara, Sergios Petridis, Dimitris Poly- chronopoulos, Yannis Almirantis, John Pavlopoulos, Nicolas Baskiotis, Patrick Gallinari, Thierry Artières, Axel-Cyrille Ngonga Ngomo, Norman Heino, Eric Gaussier, Liliana Barrio-Alvers, Michael Schroeder, Ion Androutsopoulos, and Georgios Paliouras. An overview of the BIOASQ large-scale biomedical semantic indexing and question answering competition. BMC Bioin- formatics, 16(1))138, 2015. ISSN 147 1471-2105. doi: 10.1186/s1285..000 01550000 https://doe http://dooooggg///// .or. /10. 0t/.//..1.1116111 01107/812289-015-55555
URL
14

---

Published as a conference paper at ICLR 2026
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Superglue: A stickier benchmark for weneerl--urpos language understanding systems, 2020. URL BBLL:tttee//arxi.ooor//aa/////// 0
Xiangfeng Wang, Zaiyi Chen, Zheyong Xie, Tong Xu, Yongyi He, and Enhong Chen. In-context former: Lightning- compressing context for large language model, 2024. URL https:
//arxdv..rc/ab///2200000 13618
David Wingate, Mohammad Shoeybi, and Taylor Sorensen. Prompt compression and contrastive conditioning for controllability and toxicity reduction in language models. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Findings of the Asvociation for Computational Lin- guisties: EMNLP 2022, pp. 5621-5634, Abu Dhabi, United Arab Emirates, December 2022. 006655/11/22222 htgp:s/www2222..mnnnnnmmmmmmmmm findings-emmlp.42.. URL
Association for Computational Linguistics. doi: https://acl antho wtg.o/....gg...... f ndi ngs- 1p 412/
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answer- ing. In Ellen Riloff, David Chiang, Julia Hockenmaier, and un'iccii Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2369-2380, Brussels, Belgium, Sttober-November 2018. Association for Computational Linguisties. doi: N0.18633/1/DDIB 8-1599 URL htt.::tt::///tctt /ttp:////aaaaaahhaacooooo wtt.aantoooooooooooooooo....e... --91111119/
Zihao Yi. Jiarui Ouyang, Zhe Xu, Yuwen Liu, Tianhao Liao, Haohao Luo, and Ying Shen. A survey on recent advances in Ilm-based multi-turn dialogue systems, 2025. URL https:://axii:
org/abs/ 2402. 18013
Chen Zhang, Xinyi Dai, Yaxiong Wu, Qu Yang, Yasheng Wang, Ruiming Tang. and Yong Liu. A survey on multi-turn interaction capabilities of large language models, 2025. URL https
09959
Peitian Zhang, Zheng Liu, Shitao Xiao, Ninglu Shao, Qiwei Ye, and Zhicheng Dou. Long context compression with activation beacon, 2024.
Runsong Zhao, Xin Liu, Xinyu Liu, Pengeheng Huang, Chunyang Xiao, Tong Xiao, and Jingbo Zhu. Position ids matter: An enhanced position layout for efficient context compression in large language models, 2025. URL
2409. 14364
Runsong Zhao, Shilei Liu, Jiwei Tang, Langming Liu, Haibin Chen, Weidong Zhang, Yujin Yuan, Tong Xiao, Jingbo Zhu, Wenbo Su, and Bo Zheng. Comet: Collaborative memory transformer for efficient long context modeling, 2026.
01766
Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan Awadallah, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, and Dragomir Radev. Qmsum: A new benchmark for busey multi-domain meeting summarization, 2021. URL https: //tp:v/aaivvvvv//a///// org abs/
2104. 05938
15

---

Published as a conference paper at ICLR 2026
A EXPERIMENT DETAILS
We perform pretraining and fine-tuning using bfl6 precision on 8 NVIDIA RTX 3090 GPUs (24GB). For pretraining, we randomly sample data from the SlimPajama-68 dataset with a token length ranging from 510 to 2040. This data is then split into two halves: one for the auto-encoding (AE) task and the other for the language modeling (LM) task (the AE half is discarded for models without the AE objective). For downstream tasks, we process the MRQA dataset into (Context, Question, Answer) format for finetuning. Detailed hyperparameters can be found in Table 9
Table 9: Hyperparameters for training
Hyperparameter
Value
Optimizer
AdamW
Betas
(0.9, 0.95)
Weight decay
0.1
Learning rate
le-4 (pretrain)
Se-5 (finetuning)
Scheduler
Constant
Batch size
16
Warmup
300
Training steps
20k (pretrain)
20k (finctuning)
Clip norm
2.0
B DETAILED RESULTS
B.1 PRETRAINING RESULTS
As shown in Table 10, method, SAC, achieves the lowest perplexity (10.79) among all baseline models. This suggests that removing the autoencoding (AE) objective in SAC allows the model to better focus the language modeling task, thereby improving its predictive capability. Furthermore, since SAC avoids the additional computational overhead from independent compression tokens and the AE task, its training is approximately 31% % faster than ICAE and 26% faster than 500x and EPL.
Table 10: Pretraining comparison of SAC and existing context compression methods, results on LM perplexity and training time.
Methods
LM-PPL
Training Time(hours)
ICAE
12.35
3.85
500x
11.83
3.60
EPL.
10.88
3.60
SAC
10.79
2.66
B.2 FINE-TUNING RESULTS
Tables 11 and 15 report the evaluation results of SAC on in-domain and out-of-domain MRQA datasets, which we analyze from three perspectives: overall performance, effect of compression ratio, and domain generalization.
Overall Performance. SAC consistently outperforms all baselines across a variety of conditions, including compression ratios, and both in-domain and out-of-domain tests, as shown in Tables [] and
15 Averaging the results of the context compression methods across different compression ratios, SAC shows maximum improvement of 24.6% F1 / 28.6% EM and minimum improvement of 4.6% F1 / 5.7% EM in-domain evaluations. For out-oof-domann tests, the maximum improvement is 32.5% F1 / 36.2% EM, with minimum improvement of 4.6% F1 6.9% EM.
16

---

Published as a conference paper at ICLR 2026
Impact of Compression Ratio. We conduct detailed evaluation of model performance under different compression ratios (5x, 15x, and 51x), as shown in Tables Maaees[[aamm and [3] As expected, F1 and EM scores of all methods decrease with increasing compression ratio, from 5x to 51x, since higher compression ratios result in more information being discarded. At the highest compression rate of 51x, the performance of different compression methods is not consistent. While one method may perform well on certain datasets, it may underperform on others. Nonetheless, SAC consistently achieves the best average performance.
Generalization. We evaluate the generalization capability of SAC on out-of-domain datasets, as shown Table [5] Under all compression ratio constraints, SAC consistently achieves the highest average FI/EM scores among all methods. Specifically, at a 5x compression ratio, SAC attains average F1 and EM scores of 47.72 and 32.30, outperforming the second-best EPL. method by 0.77 and 1.0 points, respectively. At more challenging 15x compression ratio, SAC achieves average F1 and EM scores 39.26 and 26.02, surpassing EPI. by 2.52 and 2. points, with an EM improvement approaching 10%. Even at extreme 51x compression ratio, SAC maintains average Fl and EM scores of 32.24 and 21,44, still leading EPL by 2.02 and 1.96 points, respectively. These results indicate that the compressed representations learned by SAC exhibit strong cross-domain robustness.
Table 11: For the finetuning results, we report in-domain performance using ROUGE-1 (Lin
2004) and exact match (EM) (Maalouly 2002) scores the following datasets: SQuAD (Rajpurka
et al. 2016). NewsQA (Trischler et al. 2017). TriviaQA (Joshi et al.] (017). SearchQA (Dunn et al.
2017 HotpotQA Yang al. 2018), and NaturalQuestion (NQ) (Kwiatkosski et al. 2019).
Methods
SQUAD
NewsQA
TriviaQA
SearchQA
HotpotQA
NQ
Average
F1
EM
F1
EM
F1
EM
F1
EM
EM
F1
EM
F1
EM
Full-FT Lingua-2
77.69
59.71
63.50
46.04
68.80
60.54
73.25
62.07
74.78
59.26
71.01
53.47
71.51
56.85
32.93
19.57
26.78
13.20
9.67
8.12
45.40
31.80
36.10
22.05
40.08
22.01
31.83
19.46
compression
ICAE
36.20
22.12 12
28.06
13.77
54.63
45.5
65,
53.06
48.79
33.40
52.36
34.99
47.53
33.82
500x
51.62
33.63
39.70
22.63
$7.62
48.76
66.43
54.38
59.10
42.20
57.11
39.26
55.26
40.14
64.72
44.28
48.74
54.54
63.32
SAC
65.37
44.83
49.39
27.14
65.06
55.93
69.99
58.06
67.41
50.28
64.56
45.44
63.63
46.95
15x compressiin construint
ICAE
31.90
18.91
25
11.97
$1.78
42.94
64.81
52.89
45,22
30.32
48.01
30.67
44.50
31.28
500x
40.68
24.97
32.01
16.76
53.84
44.86
65.65
53,70
53.01
36.30
50.93
33.26
49.35
34.98
EPL
44.58
27.91
33.34
16.69
56.16
47.09
66.36
54.13
54.88
38.38
53.80
35.71
51.52
36.65
SAC
47.43
30.25
36.55
18.07
61.13
52.19
68.97
56.76
58.83
41.86
56.79
38.88
54.95
39.67
51x
compression construint
ICAB
26.17
14.58
22.448
9.69
47.62
39.23
64.31
52,80
38.91
24.78
42.87
26.86
40.39
27.99
46.07
EPI.
30.09
17.49
24.49
11.54
51.15
42.38
65.12
53.16
42.19
27.23
46.29
29.77
43.22
30.26
SAC
31.81
18.78
27.36
13.56
56.73
47.85
68.82
53.76
32.84
48.22
31.70
33.08
B.3 3 ABLATION RESULTS
In the main text, we discuss the significant performance gains of SAC over all baseline methods. To provide more detailed evidence, we present the full ablation study results here. As shown in Table [6] and Table [7] our conclusion holds not only terms of average performance but also consistently validated each individual dataset.
B.4 SCALABILITT RESULTS
We the same data and training settings described in the paper, with compression ratio of 15. The training is conducted on 8 A100 GPUs for Llama-3.2-3B and Llama-3.1-88
The results shown in Tables 18 and 19 Compared with the stronger baseline EPL, SAC achieved the best average performance in all cases, demonstrating its ability to effectively scale up to larger model sizes while maintaining its performance advantage.
17

---

Published as a conference paper at ICLR 2026
B.5 LONG-CONTEXT SUMMARIZATION RESULTS
To further investigate whether this degradation phenomenon is specific to QA or reflects a broader conflict between AE and downstream tasks, additionally evaluate SAC without AE and SAC with AE on other long-contex tasks. The evaluation metric used is ROUGE-FI.
For the summarization task, we train SAC and EPL on QMSum Zhong al. 2021 and GovRe- port (Huang et al. 2021) with a maximum input length of 32K tokens. The test results are shown in Table The results indicate that SAC achieves the highest average performance on long-document summarization tasks. Notably, the training data size of QMSum (1.26k) is much smaller than that of GovReport (17.5k), which may explain why SAC(ae+Im) performs slightly better on QMSum, while SAC performs better on the larger GovReport dataset. We hypothesize that redundant information increases long contexts and the information capacity of compressed tokens is limited, the AE objective of reconstructing all information may impose additional burden the model, thereby negatively affecting performance.
We further evaluate on long-context question answering benchmark with context length of 24K tokens, using the model checkpoints from Table which are trained with a 15x compression ratio on 2K-token contexts. The results are shown in Table 21 On both single-document and multi-document question answering tasks, SAC consistently outperforms the best baseline compression methods under 15x compression ratio. These results indicate that our method maintain its performance advantages even when applied to longer contexts.
B.6 FLOPS RESULTS
We want to clarify critical advantage SAC: while SAC does enable bidirectional attention for anchor tokens, does not require appending additional special tokens to the sequence like 500xCompressor. This means SAC operates on shorter sequences during inference, directly reducing computational overhead.
Let's quantify this advantage. Given a context with shape [b, s, h] where is batch size, is sequence length, h is hidden size, and / is the FFN intermediate size, we compare the theoretical FLOPs:
Table 12: Comparison of FLOPs between SAC and 500xCompressor for different modules.
Modules
SAC-FLOPs
500x-FLOPs
x <WQ / Wk/Wv)
3 . 2bsh
3 2b F)
QKT
2bs h
b(s + h)
AV
2bs h
b(s + h)2h
xWo
2bsh²
2b(s + k)h²
Xout W...
2bsh1
2b(s + kkhl
Xout W gate
2bshl
2b(s + +k)h
Xout W down
2bshT
2b(8 + k)hT
sum
bhs (h.(8hh.r) + 4s + 6T)
bh(s (8 + 2(s + k)) + 67]
Concrete comparison: With typical settings (b=1, s=510, /=:00/2088888- 1=8192) 1-248
At 5x compression(<=110): 500xCompressor requires 1.19x more FLOPPs than SAC
At 10x compression(k=51)) 500xCompressor requires 1.08 x more FLOPPs than SAC
To validate the theoretical analysis, we additionally measured empirical inference latency. Using single GPU with batch size 10 over 1000 sequences, we recorded the compression time for both methods. The results are reported Table
Table 13: Empirical compression time comparison (ms per batch) between SAC and 500xCompressor.
Methods
Compression Time
500x
257.43
SAC
243.87
18

---

Published as a conference paper at ICLR 2026
These results show that SAC not only reduces theoretical FLOPs but also achieves lower empirical compression latency. Despite the use bidirectional attention, SAC is more efficient than 500xCom- pressor, the computational cost of bidirectional attention is offset by operating the original, shorter sequence.
C VISUALIZATION ANALYSIS
C.I TRAINING CURVES ANALYSIS
Figure shows the training loss curves different compression ratios on the MRQA dataset. The training loss of our SAC model consistently converges better than other baseline methods across all compression ratios, which demonstrates that the compressed representations obtained from the SAC architecture are more beneficial for language modeling tasks. Notably, as the compression ratio increases appropriately, the difference convergence between SAC and the other baselines becomes more significant.
SFT Loss Curvas Sx Compression
SFT Loss Curvas 15x Compression
SFT Loss Curves at 51x Compression
40
50bxCompeess
30
15
Figure 5: Supervised finetuning loss curves. The figure illustrates the training loss trajectories of different models under three compression ratios: 5x, 15x, and 51x.
D THE USE LARGE LANGUAGE MODELS
We used large language model (LLM) general-purpose assist tool. The LLM'' primary role was assisting with writing and text editing, such as refining prose and correcting grammar and spelling ensure the paper's professionalism and fluency. We explicitly state that the LLM was involved in the core ideation or methodological design of this research. All core contributions of the paper, including the proposal of the methodology, the construction and execution of experiments, and the analysis of results, were performed independently by the authors.
Table 14: Analysis of autoencoding (AE) reconstruction and language modeling performance under different training objectives.
Methods
AE-PPL
AE-BLEU4 LM-PPL
ICAE
4.08
12.35
12.02
500xCompress
1.09
11.83
87.21
EPL
10.88
97.50
SAC(ae+Im)
1.03
11.01
98.19
SAC(only ae)
1.00
99.94
SAC
10.79
19

---

Published as a conference paper at ICLR 2026
500xCompressor
EPL
SAC
Figure 6: Attention maps of different models finetuned under 5x compression rate. From top bottom, the figure displays the final layer attention maps for the 500xCompressor, EPL, and SAC models, respectively. The x-axis represents the original context tokens, and the y-axis represents the compression tokens.
500xeompressor
EPL
SAC
Figure 7: Attention maps of different models finetuned under 15x compression rate. From top bottom, the figure displays the final layer attention maps for the 500xCompressor, EPL, and SAC models, respectively. The x-axis represents the original context tokens, and the y-axis represents the compression/anchor tokens.
20

---

Published as a conference paper at ICLR 2026
500xCompressor
EPL
SAC
Figure 8: Attention maps of different models finetuned under 51x compression rate. From top to bottom, the figure displays the final layer attention maps for the 500xCompressor, EPL, and SAC models, respectively. The x-axis represents the original context tokens, and the y-axis represents the compression tokens.
Live Gradient Cosine Similarity (AE vs. LM)
1.00
0.75
0.50
0.25
0.00
30
Raw Simllrity (Per Step)
Smoothed Similarity rWtronm 100) Conflict Threshold
-1.00
2500
5000
7500
10000
12500
15000
17500
20000
Training
Figure 9: Gradient Cosine Similarity between AutoEncoding (AE) Loss and Language Modeling
(LM) Loss.
21

---

Published as a conference paper at ICLR 2026
Table 15: For the finetuning results, we report out- of-domiin performance using ROUGE-1 1 FI and exact match (EM) scores on the following datasets: BioASQ TTatsaronis et al.) 2015), DROP Dua
et al. 2019), DuoRC (Saha et al. 2018), RACE (Lai et al. 2017), Relation Extraction (RE) Levy
al. 2017) and TextbookQA (TQA) (Kembhav et 2017).
Methods
BioASQ
DROP
DouRC
RACE
RE
TQA
Average
F1
F1
EM
Fuil-FT
49.37
36.77
34.46
48.82
35.57
83.34
46
32.4
52.51
36.87
Lingua
27.76
19.48
27.28
18.83
27.07
18.32
17.54
4.15
39.30
20.59
28.42
15.83
27.90
16.20
3x compression construid
ICAR
36.08
26.06
28.95
21.09
16.67
10.79
15.65
3.12
54.73
41.01
24
20. 96
31,22
20.51
500x
40.30
28.99
35.40
25.55
29. 29.43
19.32
21.57
4,90
65.43
50.88
38.62
22.75
38.46
25.40
EPL
46.05
32.58
39.94
28.94
39.10
27.12
30.99
6.08
76.07
62.31
49.54
30.74
46.95
31.30
SAC
44.66
39.48
77.87
32.93
47.72
32.30
15x compression constraiar
ICAE
35.51
24.47
30.39
21.96
13.78
9.06
1.5.21
3.71
55.24
40.33
34.75
21.56
30.81
20.18
500x
36.30
25.93
33.46
23.55
20.53
12.72
18.49
3,41
54.37
41.11
41.09
25. 82
34.04
22.09
EPL
40.52
28.52
32.16
22.29
25.70
16.39
20.97
59.75
46.34
25.
36.74
23.83
SAC
41.31
28.66
36.72
27.48
28.94
18.99
23.35
4.90
61.04
47.90
44.21
28.21
39.26
26.02
51x compression constroint
ICAE
33.82
23.67
27.94
19.29
11.14
14.89
47.02
34.02
33.08
19.83
27.98
17.85
500x
32.17
23.07
30.11
21.76
13.42
8.53
1.5.18
2.67
54.62
41.86
37.10
22.62
30.43
20.09
EPL
32.52
22.21
29.64
20.89
13.16
8.13
17.15
3.12
53.72
40.37
35.15
22. 16
30.22
19.48
SAC
36.95
21.85
14.26
15.87
48.19
41.05
26.21
32.24
Table 16: We conduct ablation studies for SAC under a 5x compression rate on the in in-domain dataset in three sets: component ablation, token selection, and the influence of the autoencoding (AE) task.
Methods
SQuAD
NewsQA
TrivinQA
SearchQA
HodpotQA
NQ
Average
EM
F1
EM
EM
FI
EM
RM
F1
EM
F1
EM
Component Ablution
SAC
65.37
44.83
49.39
27.14
65.06
$5
69.99
58.06
67.41
50.28
64.56
45.44
63.63
46.95
SAC(wro mask)
39.93
45.93
25.74
62.60
53.27
66.66
54.82
64.63
47.43
61.53
42.55
60.26
43.96
SAC(wio anchor)
61.59
41.72
46.52
25.45
63.90
$4.81
68.03
56.17
65.25
48.31
62.21
43.88
61.27
45.06
Token Selection
SAC SAC(Runddm) Rundom) SAC(Lingu--2)
58.16
39.00
42.94
22.98
62.37
53.17
69.
$7.36
63.86
46.70
60.76
42.26
59.54
43.5H
64.89
44.28 28
48.99
27.11
64.55
55.13
69.90
58.04
67.05
49.74
64.23
44.93
63.26
46.54
AE'' Effect
500xxw/ only)
300xtw/ AE+LM)
$1.62
33.63
39.70
22.63
57.62
48.76
66.43
54.38
59.10
42.20
57.11
39.26
55 55.26
40.
SAC(w only)
56.98
37.60
41.09
20.61
58.19
49.08
64.02
51.65
61.58
44.13
57.23
38.98
56.55
40.34
SAC(w/ AE+LM)
64.68
44.62
46.64
25.62
63.34
54.27
68.40
56.48
66.61
49.72
62.56
44.06
62.04
45.80
Table 17: We conduct ablation studies for SAC under a 5x compression rate on the out-of dataset in three sets: component ablation, token selection, and the influence of the autoencoding (AB) task.
out-of-domain domain
Methool
BioASQ
DROP
DouRC:
RACE
RE
TQA
Average
Component Ablation
SAC
44.66
31.45
41.55
30.87
39.48
26.92
30.53
6.23
77.87
40
52.24
32.93
47.72
32.30
SAC(wlo mask)
41.93
30.65
40.24
28.48
36.48
23.58
28.21
5,49
69.09
55.63
48.29
29.67
44.04
28.92
SAC(wlo anchor)
43.70
31.78
40.55
30.34
36.97
25.58
30.05
6.82
75.88
62.35
51.59
32.20
46.46
31.51
Token Selection
SAC(Random)
45.94
34.31
40.87
30.21
35. 35
24. 12
29.28
5.64
70.11
56.34
50.53
32.00
45.5
30.44
SAC(Lingua-2)
44.49
31.91
41.50
29.61
39.47
26.58
29.96
7.12
77.67
65.47
51.46
32.07
47.43
32.13
AE Effect
500xxww/ onlty)
17.59
300xiw/ AE+LM)
40.30
28.99
35.40
25.55
29.43
19.32
21.57
4.90
65.43
50.88
38.62
22.75
38. 38.46
25.40
SAC(w/ only)
40.85
29.39
35.32
25.28
31.55
21.32
25,86
4.90
72.29
$7.90
46.61
29.
42.08
27.98
SAC(W/ AB+LM)
44.84
32.31
41.47
31.14
39.29
27.58
30.11
6.23
77.12
64.42
50.74
31.87
47.26
32.26
22

---

Published as a conference paper at ICLR 2026
Table 18: Experimental Results of the 1B/3B/8B Models on the In-Domain QA Tasks.
Methods
SQuAD
NewsQA
TriviaQA
SearchQA
HotpotQA
NQ
Average
EM
EM
EPL(IB)
44.58
27.91
33.34
16.69
56.16
47.09
66.
54.13
54.88
38.38
53.80
35.71
51.52
36,65
SAC(IB)
47.43
30.25
36.55
18.07
61.13
52.19
68.97
56.76
58.83
41.86
56.79
38.88
54,95
39.67
63.59
SAC(3B)
63.11
43.51
49.23
28.80
69.65
60.85
73.88
62.29
68.87
51.40
65.98
47.18
65.12
49.01
(PL(8B)
65.79
51.18
72.40
48.71
SAC(8B)
64.92
44.80
49.77
29.20
72.37
63.70
76.92
65.29
70.80
54.25
67.77
49.21
67.09
51,08
Table 19: Experimental Results of the IB/3B/8B Models on the Out-of-Domain QA Tasks.
Methods
BioASQ
DROP
DouRC
RACE
RE
TQA
Average
F1
EM
F1
F1
EM
40.52
22.29
SAC(IB)
41.31
28.66
36.72
27.48
28.94
18.99
23.35
4.90
61.04
47.90
44.21
28.21
39.26
26.02
EPL(3B)
46.76
32.98
46.20
36.46
36.81
25.32
33.80
65.73
53.70
55.46
34.86
47,46
31.82
SAC13B)
47.96
34.51
48.46
38.39
42.21
29.85
33.61
7.72
75.11
62.89
55.53
35,00
50.48
34.73
EPL(SB)
52.52
37.57
49.98
39.92
40.09
27.24
35.22
7.27
69.77
57.84
57.36
36.66
50.82
34.42
SAC(8B)
51.55
40.65
35.52
78.91
57.87
35,93
Table 20: Model Performance on 32K Long-Conter Summarization Tasks.
Methods
QMSum
GovReport
Average
SAC
14.95
22.03
18.49
SAC(ac+Im)
15.32
20.15
17.74
EPL
14.82
20.40
17.61
Table 21: Model Performance on 24K Long-Contex. QA Tasks.
Methods
MultiFieldQA
NarrativeQA
Qusper
Sigle-Doc-Avg
2WikiMQA
MuSiQue
HotpotQA
Multi-Doe-Avg.
17.75
11.33
11.67
13,58
24.78
12.36
26.03
21.06
SAC(ac+Im)
17.38
10.49
9.05
12.31
22.47
7.76
23,88
18.04
ID Performance across Models and Compression Ratios
ICAE
62.5
500xCompress
EPL
60.0
55.0
Average
52.5
50.0
47.5
45.0
42.5
40.0
Compression Ratio
Figure 10: Efficiency and Performance Trade-off Curves on In-Domain (ID) Tasks.
23


Reviews:
[{"review_id": "dRvZ5nvfDv", "reviewer": "Reviewer_rbBK", "summary": "This paper proposes a context compression method that does not relying on autoencoding training. The proposed method, SAC, selects anchor tokens directly from the original tokens and aggregates contextual information into their key-value tokens using bidrectional attention. Experiments show that SAC outperforms existing context compression methods.", "questions": "1. Can you give a clear description of the method? For example, what are the anchor embeddings? How does anchor embeddings enable the compressor to identify critical tokens? The anchor tokens are just sampled evenly, so why are they more important? Why do you choose the output of anchor tokens from the LLM's final layers or the KV pairs from each layer? Any justifications? \n2. Can you demonstrate the effectiveness and performance improvement of the proposed method for long contexts? \n3. Can you provide evidence for the generalizability of the proposed model to larger models and other model families? \n4. What do you think might have caused the discrepancy in the observed AE effect?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Context compression is an important problem worthy of investigation. \n2. The proposed method is simple. It combines token selection, common in KV cache compression but probably not in context compression, and bidirectional attention. \n3. The proposed method outperfoms existing context compression methods.", "weaknesses": "1. The method is not clearly described and sometimes very confusing. \n2. The contexts in the experiments are in general short. It is of more interest to compress long contexts. \n3. The experiments use a single model, Llama-3.2-1B. It is unclear whether the proposed method generalizes well to larger models and other model families. \n4. The paper has an observation that challenges previous work, but does not provide any explanation, so it is hard to see if is a bug or something new.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wwY8sAIX3h", "reviewer": "Reviewer_NhSy", "summary": "This paper proposes Semantic-Anchor Compression, a novel framework designed to improve the efficiency and performance of Large Language Models on long-context tasks. The method introduces an autoencoding-free approach to context compression. The core novelty lies in three architectural designs: 1) It selects 'anchor tokens' directly from the original context to act as information carriers. 2) It introduces a trainable 'anchor embedding' to semantically distinguish these selected tokens. 3) It enables 'bidirectional attention' specifically for these anchor tokens, allowing them to effectively aggregate global context. This design allows the compression mechanism to be optimized directly for downstream tasks. Experimental results on a benchmark dataset demonstrate that the method achieves strong performance across various compression ratios and shows favorable generalization capabilities.", "questions": "1. Can you please provide a baseline experiment using the \"last token\" of each chunk as the anchor, while still applying your anchor embedding and bidirectional attention? I am very curious how this intuitive baseline compares to your \"middle token\" strategy.\n\n2. Do you have plans to validate SAC on other long-context tasks besides QA (e.g., summarization, multi-turn dialogue)? The current setup makes me highly concerned that the model is overfitting to QA while sacrificing general compression capabilities.\n\n3. Did you consider testing a SAC + AE setup (i.e., using your anchor architecture but retaining the AE loss)? What are your thoughts on the trade-off between QA performance and general-purpose robustness that this combination might offer?\n\n4. Could you provide a detailed analysis of the inference overhead? While SAC saves on AE training, what is the specific extra compute (FLOPs) or latency cost of enabling bidirectional attention for anchor tokens (either over the full sequence or within chunks) during inference, compared to standard KV caching or older compression methods?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces a novel and effective information compression method based on anchor tokens. This \"autoencoding-free\" paradigm successfully moves beyond traditional context-agnostic compression tokens.\n\n2. The proposed SAC method is simple, architecturally concise, and elegant in its design.\n\n3. The paper is exceptionally well-written and well-structured, clearly articulating the core problem, the proposed solution, and its advantages.", "weaknesses": "1. The paper's default strategy is to select the \"middle token\" of a chunk. However, the ablations (Table 4) are missing a critical and intuitive baseline: selecting the \"last token\" of each chunk as the anchor. In a causal model, the last token has naturally seen the entire chunk. This is a significant omission that prevents a full assessment of the \"middle token\" strategy and makes it difficult to judge if the performance gain comes from a truly better mechanism or just a better-positioned token.\n2. All experiments are conducted only on QA tasks. As the authors state, the AE task deviates from \"real-world usage,\" but using only QA to represent \"real-world usage\" is far from sufficient. SAC is a lossy compression (as it discards the AE objective), making it highly likely to have overfitted to the specific patterns of QA tasks. Its performance on other long-context tasks (e.g., summarization, story generation) is unknown.\n\n3. The paper's core argument is that AE-free is better. However, the authors never test the SAC + AE combination. That is, using SAC's anchor architecture (with bidirectional attention) to simultaneously perform the downstream task and the AE reconstruction task. It is plausible that SAC + AE would have slightly lower QA performance but be more robust and general-purpose (by retaining reconstruction). The authors claim AE is harmful but do not prove this experimentally (i.e., by comparing SAC vs. SAC + AE).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eoq0qdxKs5", "reviewer": "Reviewer_wACd", "summary": "This paper introduces **Semantic-Anchor Compression (SAC)**, a new training-based method for LLM context compression. Its generation pipeline follows the prevailing setup:\n\n- one language model acts as the compression model to produce compressed tokens from a given context, and\n- a (frozen) generation model consumes those compressed tokens to generate outputs for downstream tasks.\n\nPrior approaches----ICAE, 500xCompressor, and EPL ----first **pretrain with an autoencoding (AE) objective** to reconstruct the original context from compressed tokens, then fine-tune by passing the compressed tokens to a frozen generator, optimizing the LM objective and, finally, the downstream (e.g., QA) objective. The authors argue these methods require new, randomly initialized compression tokens*whose embeddings must be trained----on top of LoRA fine-tuning of the compression model----which \"lack inherent semantic information\" and thus demand extensive AE pretraining.  \n\nSAC addresses this by selecting anchor tokens directly from the original context, removing the need for AE pretraining and retaining only LM and task-level fine-tuning. Concretely, the context is partitioned into equal-length chunks, and the **center token** of each chunk is chosen as the **anchor** (following EPL). A fixed **anchor embedding** is then added to these tokens to highlight their special role. Finally, **bidirectional attention** is enabled **only in the compression model**, allowing each anchor to aggregate information from the **entire** context; the resulting **compressed KV states**----one per anchor----serve as the compact representation passed to the generation model. In experiments, the authors use Llama-3.2-1B for both compression and generation; during training, the compression model is fine-tuned with LoRA (r=128) while the generation model remains **frozen**.\n\nTo recap, SAC's contributions are twofold:\n1. **Eliminates AE pretraining**, relying only on **LM** and **downstream (QA) fine-tuning**.  \n2. **Replaces learned compression tokens** with **semantically grounded anchor tokens** taken **from the original context**.\n\nTwo simple architectural tweaks make this effective: (i) add a dedicated **anchor embedding** to mark anchors, and (ii) apply **bidirectional attention** (in the compression model) so anchors can gather information from both preceding and succeeding tokens.\n\nEmpirically on MRQA, SAC consistently outperforms strong AE-based baselines (500xCompressor, EPL) on both in-domain and out-of-domain datasets, with the gap widening at higher compression ratios (e.g., 15× and 51×). The method also shows better training efficiency due to removing AE pretraining.", "questions": "1. (Regarding Weakness 1) Have you experimented with or considered a *learned* selection mechanism? It seems plausible that a small, trained module could identify more semantically salient tokens to serve as anchors, potentially improving performance further, especially at very high compression ratios. In addition, attention/entropy-based KV selection (i.e., from the KV eviction literature) could also be worth a try.\n\n2. (Regarding Weakness 2) Can you add DAST[1] for comparison? Better if PCC [2] can also be included. \n\n3. (Regarding Weakness 3) How do you anticipate SAC would perform on non-extractive, non-QA tasks like long-document summarization? The AE objective, while mismatched for QA, does force a holistic reconstruction. Does the LM-only pretraining capture enough global information for tasks that require a broader, generative understanding of the entire context? \n\n3. (Regarding Weakness 4) Could you please clarify the precise implementation of the bidirectional attention? Is the attention mask modified only for the anchor token queries, or is the entire encoder pass non-causal for all tokens?\n\n4. It would be better if more compression ratios could be experimented with, producing a trade-off curve between performance and compression ratio to further validate the robustness of SAC.\n\n5. Can you clarify how you initialize the anchor embeddings?\n\n**Minor Issues (does not affect evaluation)**:\n\n1. At line 198, \"While using original tokens … limiting its representation power.\" should be 1 sentence instead of 2.\n2. At line 429, there should be a space within \"Analysis.In\".\n\n**Reference**:\n1. Shaoshen Chen, Yangning Li, Zishan Xu, Yongqin Zeng, Shunlong Wu, Xinshuo Hu, Zifei Shan, Xin Su, Jiwei Tang, Yinghui Li, and Hai-Tao Zheng. 2025. DAST: Context-Aware Compression in LLMs via Dynamic Allocation of Soft Tokens. In Findings of the Association for Computational Linguistics: ACL 2025, pages 20544–20552, Vienna, Austria. Association for Computational Linguistics.\n\n2. Liao. 2025. Pretraining Context Compressor for Large Language Models with Embedding-Based Memory. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 28715–28732, Vienna, Austria. Association for Computational Linguistics.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The critique of the AE's inefficiency is valid.\n\n2. The core idea of using in-context \"anchor tokens\" is reasonable. It simplifies the compression training process and, as the t-SNE analysis (Fig. 5) suggests, produces compressed representations that are more aligned with the original token space. \n\n3. SAC consistently outperforms baselines on MRQA, as well as during training (Appendix C trade-off curves). The fact that its performance advantage *increases* at higher compression ratios (15x, 51x) makes it more compelling. The experimental setups and compression ratios are consistent with prior works.\n\n4. Table 5 shows that adding an AE task to the SAC architecture (SAC w/ AE+LM) actually *hurts* performance compared to the LM-only (SAC) model. This provides powerful evidence for the \"AE mismatch\" theory that the authors argue for, which is quite different from what the literature used to believe.\n\n5. By removing the costly AE pretraining step, the method is inherently simpler and faster to train (as shown in Appendix B.1, Table 7).", "weaknesses": "1. The paper's default strategy (uniform chunking) is simple and effective. However, the ablation in Table 4 shows it performs similarly to a more complex method (Lingua-2). This suggests the selection strategy is important, but the paper stops short of exploring it deeply. \n\n2. The baselines compared in the paper seem to be a bit outdated. The paper is missing an important baseline, DAST [1], which is also chunked compression but with a dynamic token budget, achieving decent performance at a high compression ratio. \n\n3. The evaluation is performed exclusively on MRQA, a question-answering benchmark. This is a good choice, as QA is a key application. However, the paper's central critique is that the AE task is misaligned with *downstream tasks*. For a task like long-document summarization, an AE-style reconstruction objective might be *more* aligned than a QA objective. It is unclear if SAC's superiority would hold on tasks that require a more holistic understanding of the full context, rather than just extracting specific facts.\n\n4. The paper states it \"modifies the standard causal attention to a bidirectional attention mechanism.\" For clarity, it would be beneficial to explicitly state whether this bidirectional mask is applied *only* to the anchor tokens (i.e., their queries can see all keys/values) or if the entire compressor LLM (with LoRA) is run in a fully non-causal (BERT-style) mode during the compression pass.\n\n5. All experiments are conducted on a single model at a single parameter size. It does not guarantee that the performance is generalizable to other models or models with much larger scales.\n\n**Reference**:\nShaoshen Chen, Yangning Li, Zishan Xu, Yongqin Zeng, Shunlong Wu, Xinshuo Hu, Zifei Shan, Xin Su, Jiwei Tang, Yinghui Li, and Hai-Tao Zheng. 2025. DAST: Context-Aware Compression in LLMs via Dynamic Allocation of Soft Tokens. In Findings of the Association for Computational Linguistics: ACL 2025, pages 20544–20552, Vienna, Austria. Association for Computational Linguistics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9wHVGHYeIk", "reviewer": "Reviewer_4eQ5", "summary": "The paper proposes Semantic-Anchor Compression (SAC), which directly selects anchor tokens from the original context and aggregates contextual information into KV representations. It consists of two key designs---anchor embeddings and bidirectional attention to ensure the compression performance.", "questions": "see weakness", "limitations": "", "rating": 2, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The proposed solution is simple but effective. The author shows good results on many downstream tasks.\n2. The paper is generally well-written and easy to follow.", "weaknesses": "1. The intuition behind the proposed method is confusing. The authors argue that \"SAC is distinct to previous compression-based methods\" because \"previous methods compress contextual information into context-agnostic special tokens\". However, in a auto-regressive setting, the compressed token is actually context-dependent since it can see all tokens behind in attention computation.\n2. SAC add lora to Llama-3.2-1B with attention map modification, from causal to bidirection attention. This sounds quite confusing and seems not to be a normal routine. Why does not use encoder models directly?\n3. The idea of context compression has been well-studied in many previous works, such as [1]-[5]. These works impair the novelty of proposed method.\n\n[1] Yen, Howard. Long-context language modeling with parallel context encoding. MS thesis. Princeton University, 2024.\n\n[2] Li, Yuhong, et al. \"Snapkv: Llm knows what you are looking for before generation.\" Advances in Neural Information Processing Systems 37 (2024): 22947-22970.\n\n[3] Zhang, Zhenyu, et al. \"H2o: Heavy-hitter oracle for efficient generative inference of large language models.\" Advances in Neural Information Processing Systems 36 (2023): 34661-34710.\n\n[4] Han, Wei, et al. \"Two are better than one: Context window extension with multi-grained self-injection.\" arXiv preprint arXiv:2410.19318 (2024).\n\n[5] Zhang, Peitian, et al. \"Long context compression with activation beacon.\" arXiv preprint arXiv:2401.03462 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)