AdmTree: Compressing Lengthy Context with Adaptive Semantic Trees
Authors: Yangning Li
Shaoshen Chen
Yinghui Li
Yankai Chen
Hai-Tao Zheng
Hui Wang
Wenhao Jiang
Philip S. Yu
Venue: NeurIPS 2025 poster ()
Keywords: Language Model
Context Compression
Primary area: deep_learning

Abstract: The quadratic complexity of self-attention limits Large Language Models (LLMs) in processing long contexts, a capability vital for many advanced applications. Context compression aims to mitigate this computational barrier while preserving essential semantic information. However, existing methods often falter: explicit methods can sacrifice local detail, while implicit ones may exhibit positional biases, struggle with information degradation, or fail to capture long-range semantic dependencies. We introduce AdmTree, a novel framework for adaptive, hierarchical context compression designed with a core focus on maintaining high semantic fidelity while keep efficiency. AdmTree dynamically segments input based on information density, employing gist tokens to summarize variable-length segments as leaves in a semantic binary tree. This structure, combined with a lightweight aggregation mechanism and a frozen backbone LLM (minimizing new trainable parameters), enables efficient hierarchical abstraction of the context. By effectively preserving fine-grained details alongside global semantic coherence, mitigating position bias, and adapting dynamically to content, AdmTree comprehensively preserves the semantic information of lengthy context.

Adm Tree: Compressing Lengthy Context with Adaptive Semantic Trees
Yangning Li Shaoshen Chen'r, Yinghui Li Yankai Chen
Hai-Tao Zheng 121,, Hui Wang", Wenhao Jiang", Philipp S. Yu?
'Shenzhen International Graduate School, Tsinghua University "Peng Cheng Laboratory SUnieersity of Illinois Chicago Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ.
Abstract
The quadratic complexity of self-attention constrains Large Language Models (LLMs) in processing long contexts- capability essential for many advanced applications. Context compression aims alleviate this computational bottleneck while retaining critical semantic information. However, existing approaches often fall short: explicit methods may compromise local detail, whereas implicit methods can suffer from positional biases, information degradation, an inability to capture long-range semantic dependencies. We propose AdmTree, a novel framework for adaptive, hierarchical context compression with central focus preserving high semantic fidelity while maintaining efficiency. AdmTree dynamically segments input based information density, utilizing gist tokens to summarize variable- length segments as the leaves of a semantic binary tree. This structure, together with lightweight aggregation mechanism and frozen backbone LLM (thereby minimizing new trainable parameters), enables efficient hierarchical abstraction the context. By preserving fine-grained details alongside global semantic coherence, mitigating positional bias, and dynamically adapting to content, AdmTree robustly retains the semantic information of long contexts.
Introduction
Large Language Models (LLMs) 23 52
have demonstrated remarkable proficiency in processing and understanding long contexts, enabling advances eetrieval-ugmentii generation [9] 551 and agentic system [54), etc. However, handling long contexts remains computationally intensive due to the quadratic complexity of self-attention with input token length. This leads to high memory consumption and inference latency. Consequently, context compression has emerged as critical technique, aiming to reduce inpul token length while preserving maximal semantic integrify.
Despite promising results, most existing methods fail to simultaneously preserve information across multiple semantic dimensions, such as global versus local semantics, or information across positions. This inability in preserving different information dimensions may leads to poor general- ization across real-world tasks, which demand distinct types of semantic information. Specifically, existing methods can be divided into two main categories. Explicit methods [25] [56 57 directly shorten text by removing content deemed less essential for overall understanding. Although these methods effectively capture global meaning, they often disrupt local coherence due to excessive omission, leading to the loss of fine-grained details. In contrast, implicit methods [13] 20 33 41
encode long contexts into compact latent vectors (also called "gist tokens") in flat manner. These methods achieve higher compression ratios, but exhibit different compression efficiency to context at distinct positions. As evidenced by
23 40). implicit compression methods are prone to positional
"quual Contribution. 1, Corresponding Author, This work was primarily conducted GML under the
leadership Wenhao.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

68
22
Grack
SnapKV
"Suppy
(Curg)
50
Chred-book
Buacon
Admirec (Ours)
Accuracy (Hg)
20
Begiesing
Ead
Begioning
Middl
10
1st
5th
10tb
15th
20th
25th
30th
Beok Level
Level
Leve
Ground-trath Document
Granul-Granulritty Summarization. Implicit methods tend prioritize the preservation of global information.
Question Answering. Explicit methods exhibit varying compression effectiveness across content at different positions.
Figure 1: Pre-experiments demonstrate that existing methods struggle balance semantic information of different dimensions for different types of tasks.
bias, often overlooking information from the earlier or middle parts of the context. In other words, less salient information is easily overshadowed by more prominent content.
To mitigate semantic loss caused by position bias, some implicit methods also explored recursive compression [TT 21 58]. These methods progressively condense segmented contexts into serialized gist tokens. However, they often rely on fixed-size segments without considering variations in information density, leading to imbalanced compression loads across different input regions. More- over, such linear manner still causes semantic information to degrade progressively during recursive compression, making it difficult to capture long-range dependencies and maintain global coherence.
To overcome these limitations, draw inspiration from cognitive science, where hierarchical struc- tures are fundamental to human information processing (6 [2]). Hierarchical representations naturally balance semantic breadth (coverage across contexts) and depth (gramularity of detail). Moreover, it establishes bidirectional connections among leaf nodes, mitigating the information degradation by previous unidirectional compression. Motivated by this, we introduce the Adaptive Semantic Tree Compressor (Adm'Tree) a novel hierarchical context compression framework designed to preserve information integrity more effectively. Unlike methods relying on fixed chunking or flat representations, Adm Tree performs compression in a dynamic and structured manner: (i) first, long input context is dynamically segmented into variable-length units based on information density. (ii) then, gist tokens are interleaved among segments compress them, serving as the leaf nodes, and a semantic tree is then constructed bottom-up upon these leaves. (iii) finally, AdmTree generates responses conditioned on the semantic tree, which can be dynamically updated as new context arrives.
Extensive experiments are conducted to validate the effectiveness of AdmTree. In main experiments, Adm consistently achieves state-of of-the-art performance across five task types and more than ten datasets from LongBench [4) surpassing baseline methods by over 10% while maintaining high inference efficiency. In some QA tasks, Adm'Tree even outperforms the strongest baseline Beacon [58], by up to 20 points, demonstrating its remarkable ability to comprehensively preserve semantic information. Further analysis also confirms that Adm exhibits strong scalability and efficiency in dynamic context scenarios. Additionally, appendix results highlight the interpretability of AdmTree powered by attention over tree nodes, compared to previous black-box implicit compression methods.
Preliminaries
2.1 Task Formulation
Following the instruction tuning setting, define input context X (X Xdoc), where iinst task instruction (query) and doe input long document. The compressed context X is either text tokens semantic vectors, based which LLMs generate responses. The objective of context compression can be formulated as maximize information retention while minimizing redundancy:
max T(X; XR(X), X
in which I(X; Y) measures the mutual information between the compressed context X and ground- truth answer Y. ensuring that essential information is preserved. The second term R(X) is

---

Implieit Compresion
Input
GT
GI
Gi
GT
GF
Output
Store 1055, by los bon unter atstor Gongle
<0T>
<DT>
Implicit Compresion
Input Segment
GT
GT
Input Seguennt 2
GT
GT
Inpur Segment 3
G1
Output
Lickade the
polisa
(11) Aduptive rre--aased Impliici Compression Method (Oers)
Tree-basod Constructinn
5ug
Laar
(A.(A..)
Seyreet with Longe
PO17))
Imput Segmentt
Euput Segnentt
Input Segment
Output
882 Adoetie Leaf Node Conseraction
Figure 2: Comparison of compression frameworks. All methods generate responses conditioned on the representation in green. Unlike other methods compress in a linear manner, AdmTree dynamically balances context compression both horizontally and vertically through adaptive gist token allocation and tree hierarchy. Meanwhile, bidirectional aggregation mitigates information degradation.
regularization term that penalizes excessive redundancy, promoting shorter X. The parameter À controls the trade-off between information retention and compression.
To quantify the compression degree, the compression ratio is defined as the proportion of original input length to compressed context length, given by:
X
2 1, where is the token number.
X
2.2 Hard Tradc-offs Between Different Information Dimensions in Compression
To preliminarily illustrate the limitations of both explicit and implicit compression methods in balancing semantic information across multiple dimensions, we conduct experiments two probing tasks: (i) Multi-Granularity Summarization. Leveraging the BookSum dataset [31, this task requires the model to generate summaries at the book, chapter, and paragraph levels for given article. It evaluates the model's capacity to preserve both global and local semantic content. (ii) Multi-Documeer
Question Answering. For this task, the NaturalQuestion dataset [32] was employed. The task involves answering questions based on 30 input documents, only one of which contains the ground-truth answer. By placing the ground-truth document at different positions, verify the model capability to compress information across various locations.
As shown in Figure aa we observe that as the summarization granularity becomes finer (from book to paragraph level), the performance explicit method (LongL.MMingge) consistently declines, while no such trend is observed for implicit methods. This suggests that explicit compression methods tend to prioritize global information while neglecting finer details. Figure Tb] both Activation Beacon and SnapKV demonstrate better performance when the ground-truth document is placed at the end. This results in a performance curve characterized by an upward trajectory in its latter segment. The "lost in the middle" phenomenon 440 indicates that implicit methods are more prone to positional bias, preferentially retaining information closer to compression tokens. The above preliminary experiments demonstrate the existing compression methods struggle to retain semantic information in multiple dimensions completely, motivating us to propose a more balanced compression framework.
3 AdmTTree Framework
3.1 Overview
As shown in Figure 2 existing one-time recursive compression methods still operate in a linear or flat manner. They all suffer from information degradation due to position bias. In contrast, AdmTree is a novel framework for adaptive hierarchical context compression. Its dynamic tree structure not only balances semantic preservation terms of both breadth and depth, but also mitigates the information degradation caused by unidirectional compression through hierarchical aggregation. In general, it dynamically segments input context, assigns gist tokens to summarize each segment, and builds a tree structure to capture contextual information at multiple granularities. This hierarchical representation

---

is then utilized for compression, with only minimal parameters trained. AdmTree comprises three key stages: dynamic leaf gist token construction, semantic tree construction, tree-based compression.
3.2 Adaptive Leaf Gist Token Construction
This stage transforms an input context X into an alternating sequence of textual sub-segments $; and leaf gist tokens (GT),- g. The primary objective is to achieve adaptive segmentation based informa- tion density, governed by compression strategy, to prepare input for hierarchical summarization.
X Segmentation Budgeting
X'
[s (GT), $2, (GT) (GT) Ml
Each leaf gist token (GT); is designated summarize its preceding sub- sebbeeement Sk. and the attention scope is restricted to its associated sub-segment and preceding leaf gist tokens. The process involves initial uniform partitioning of X into segments X,, followed by adaptive budget reallocation (determining the number of sub-segments 1ú within each X;) using an information content score (Fq. 2) to define the final sub-segments 8₁ and the placement of their corresponding (GT)) A dedicated token, (GT), is incorporated into the LLM's vocabulary for this purpose.
Initial Segmentation Given an input context X [T [ #cl of length L, AdmTree first parti- tions it uniformly into initial segments Xj, each of length n:
Xi ( "mii))t+t......mm(m.mmm) mm,,((tx,
where is constrained not exceed the maximum sequence length supported by the LLM.
Adaptive Compression Budget Allocation Recognizing that information density varies across segments, initial uniform segments are adaptively further partitioned: segments with higher infor- mation density or complexity are divided into finer sub-segments with more gist tokens (implying a lower local compression ratio).
simple yet effective strategy is proposed for this budget reallocation. Given the overall compression ratio the initial budget of leaf gist tokens b; for each segment X; (of length n) is n/2r. We first quantify the informational value of each segment X; using an entropy-adjusted perplexity score:
(Xi) = (X;) exp (-X Entropy(X;,
where (ntropy()) measures its intrinsic information content, and À is a hyperparameter balancing PPL and entropy. Segments are then ranked based on this score, and the gist token budgets redistributed as follows:
n/T, if X; ranks in the top 25% by Score 61 n/2T, if X, ranks in the middle 25% by Score n/ /T, if X4 ranks in the bottom 50% by Score
(3)
This adaptive allocation strategy still maintains the overall compression ratio globally, while adapting locally to information density. Finally, Adm'Tree partitions each original segment X; into b/ finer sub-segments, appending one gist token (GT) after each sub-segment. Let 8j,j denote the j-th sub-segment of the original segment X,. The processed segment X becomes a sequence of sub-segments and their corresponding gist tokens:
x
[ (GT) 81,2, (GT) (.).............. S8,,, (GT))B)
where each sub-segment Sr has a length oi.j n/t/. These (GT), tokens form the leaf nodes of the semantic tree.
3.3 Semantic Tree Construction
Here, we detail the construction of current semantic tree Tex from the set of leaf gist token rep- resentations (hiGT), (TTT) The construction is scalable and incremental: when processing sub-segment 8;, the existing tree Tex (which summarizes preceding (..) (GT) is partially reused to form T&ı by integrating the new gist token's 's representation hIGT,, (obtained as described in Sec. 3.4). The general bottom-up aggregation process is:
Hienarchical Aggregation TM
h l(G m

---

While the tree structure can be flexible, we employ the binary tree. Meanwhile, to avoid introducing extra consumption, we do not use padding tokens keep tree balanced. Instead, any remaining unpaired gist token is deferred to the aggregation at next level.
Once the structure is determined, the aggregation function is defined Agg : P((Rd) Rd. where P denotes the power set and IR is the hidden state dimension. Starting from the leaf gist tokens, the aggregator iteratively combines hidden states h, for all children a of node u to form the parent node's 's hidden state hy:
h = Agg((((
C.
(5)
Here, C, denotes the set of children of node u. For efficiency, the aggregation function Agg(:) implemented by single-layer self-attention mechanism, followed by an averaging operation:
is
(C.) = Average (Self-Attnage (fhu u E (ag))))))
(6)
The trainable parameters size in Ongs is significantly smaller than the entire LLM, and are only built once during inference. Therefore, it does not affect the original inference complexity ((C[T)))) Furthermore, the self-attention in aggregation mechanism further allows information from subsequent segments influence preceding ones, mitigating the unidirectional limitations of causal LLMss
3.4 Tree-based Compression
This stage processes each sub-segment Sş and associated gist token (GT), by leveraging the semantic summary Tex of preceding content (i.e., tree built from gist tokens The primary goal is to generate contextualized representations for tokens in 8ş and for (GT) Once the representation is computed, the key (K (GT,)) ) and value (VGT)) representations of (GT), are also stored to accelerate the construction of subsequent semantic tree (as per Sec. 3.3
(GT) x-1>
LLM Encoding, (Hlg,, hoor) (K V(GT),))
(Sk, () Tek
Here, Ha, and hiGT), denote the output hidden states of text tokens in sy. and gist token (GT), respectively. Following prior work (21l 41), AdmTree leverages the original LLM to encode text token, while the (GT),, token and the nodes from Tea are processed independently with another trainable attention heads.
As illustrated in Figure distinctly encode gist tokens and regular text tokens, we introduce two separate attention branches before merging them for joint self attentotentoon Assuming h2 E RRd the hidden state of the j-th token at layer I. The query, key, and value vectors computed
d = $H==,,W
=
of
W
(7)
where placeholder E (gt, it} distinguishes gist tokens (gt) from text tokens (tt). Projection matrices for text tokens (e.g., W are derived from the original LLM, while those for gist tokens (e.g., Wy') are newly introduced and trained. After individually computing. the representations from the preceding tree context (T<x), the current text sub-segment (sk), and the current gist token ((GT) ) are concatenated back to form a unified sequence. The original order of each token type is maintained. The combined matrices are:
[Qu, Qul,
K [KTx Kns Kgl,
[VTeu> Vn, Vgd-
Then, the standard self-attention mechanism with causal mask M is then applied:
Hi+1 softmax (QKT + + V,
+MVV
VT
where de is the dimension of the key vectors. All tokens are encoded with relative positional embeddings for queries and keys. is actually the concatenation of Hs, and hIGT) Tree nodes from Tek & flattened in legh-to-right, bottom-to-top order to be integrated into the sequence.
3.5 Compression Learning
AdmTree learns to compress context into the semantic nodes of a tree T and then generate appropriate responses based on them. Similar to general LLMs, AdmTree is optimized via next-token prediction
5

---

task. The prediction for token 2j within the given sub-segment S₁ is conditioned on both the dynamically constructed semantic tree Tex (summarizing all preceding context) and the local intra- sub-segment context Fzjj. The objective is to minimize the negative log-likelihood:
Larain
min
Ex-D
Σ Σ log P (xz Tek; Tezi ag Og Oage)
(10)
PyLam..g gg.am........i fo..mb..ap Oreg
81EX*,EN,
where the outer sum iterates over all sub-segments S₁, derived from a long context input X. The trainable parameters consist of: Oglam. for the attention heads processing gist tokens; Ogt emb. for the token embedding of (GT); and Oage. for the aggregator parameters. The parameters of the backbone LLM, OLLM. remain frozen throughout the training.
4 Experiment Details
4.1 Experiment Setting
Baselines Following prior work 26 58 on compression, we introduce two main categories of baseline models for processing long contexts:
(i) Retrieval-based Methods. These methods employ retriever to obtain the most relevant segments based on their similarities to the given query (instruction). We utilize one sparse retrieval method
BM25 48 and two dense retrieval methods, Sentence-BERT 47 and OpenAl Embedding
(ii) Compression-based Methods. Unlike retrieval-based methods, these methods directly compress the input text into more concise representations. We select LongLLMLingua (26) representative of explicit compression methods. Auto AutoCompressor [I]] ICAE (2I). Activation Beacon [58], and SnapKV [38 are employed implicit compression methods. The former three methods recursively compresses long contexts into semantic vectors, enabling preliminary fine-grained compression.
We also fine-tuned the original LLM using the same training data to serve as the strong baseline, denoted as Original LLM-FT.
Evaluation Benchmark We evaluate AdmTree's effectiveness and efficiency on LongBench [4) LongBench is multitask benchmark assessing LLM comprehension of long contexts. The description for other used dataset can be found in Appendix.
2---n-rrr-t Instruct employed the backbone LLMs. We use LlaMA-2-77 2-7B-Chat because three out of the four compression baselines utilize
Backbone LLMs 11--A----B-Chae and Qwen-2-7B-
Implementation Details The training details for Adm'Tree can be found in Appendix. For the compression ratio in main experiments, given LLaMA-2's maximum sequence length of 4K, we apply x2 compression for 4K-8K contexts, x4 for 8K-16K, and x8 for 16K-32K. For Qwen-2, we uniformly set the compression ratio to x4. The results are averaged over three inference runs.
4.2 Main Experiments
We evaluate the AdmTree and other baseline on LongBench. The performance and latency are reported in Table [] from which we can observe that:
1. AdmTree significantly surpasses all other baselines. Specifically, on the Llama-2-7B and Qwen- 2-7B models, AdmTree outperforms the current state-of-the-art compression method Activation Beacon, by 10% and 5% in average scores, respectively. The performance gains are particularly striking in QA tasks, where AdmTree achieves improvements exceeding 327.5% in some tasks. Notably, Adm'Tree achieves these gains without incurring additional latency compared to other recursive compression methods. This performance gain is primarily attributed to Adm'Tree's 's adaptive leaf node allocation and hierarchical architecture, which effectively balances semantic information across different dimensions. Mcanwhile, the utilization of single-layer Transformer for the leaf node representation in AdmTree's 's semantic tree contributes to its computational efficiency. In contrast, AutoCompressor and ICAE exhibit poor performance, primarily due to their fixed number of gist tokens, which hinders their ability to adeptly handle the diverse long-context scenarios in LongBench.
We use text wtw---eeeddddnndddeeeaeee ada-002 OpenAl.

---

2. Adm AdmTree's advantages are more pronounced when compared with retrieval- methods. While retrieval methods enable processing lengthy context without any modification on original model, this comes at the expense of performance. These methods particularly struggle with complex, multi- hop question answering, since they may fail to retrieve all necessary relevant context. Consequently, there is marked performance decline on single and multi-document QA compared to Adm'Tree
Adm Tree is the only compression-bas based method that consistently outperforms the strong baseline of full fine- tuning on the original LLM (Original LLM-FT) across both backbone models. We posit that adm''ee's s ability to surpass Original LLM-FT stems from its capacity to retain more comprehensive information within its hierarchical structure. When presented with a task query, AdmTree efficiently "retrieval" and utilize the most relevant information while ef- fectively filtering out contextual noise. Other methods, such as Beacon, do not consistently achieve this, occasionally under- performing Original LLM-FT on specific tasks.
Table 1: Out of domain evaluation on LongBench [4]. Each task type includes at least two datasets. We report the micro-averaged performance across all datasets. i: the results cited from Bai et al. Zhang et al. 58
LongBench
Latency
Methods
SingleDoc
MutiDDc
Summ.
FewShox
Code
AYG
Luvetey
Speedup
L.ama-2-78
Original
57.7
Orginnal M-M-PT
$4.8
27.5
23.2
618
57.8
99.8
25.6
Retienl -usiee Methodl
OpenA
23.7
Compeezaion Metlodd
AuscCoureesser"
[1
12.9
16.4
23.8
39.4
23.5
12.8
ICABI [D
19.5
34.8
21.8
10.2
MLngLLMLnnpo
21.7
49.5
SuupXV7 8B
22.6
60.1
57.7
34.6
3.0x
Beacon
$4.9
27.5
25.0
61.4
57.8
40.1
8.0
9/3xx
Adnn (Our)
36.5
36.3
26.9
65.5
00.9
44.1
7.8
3.3x
Qwen-2-78
Driginal 1.1 M!
88.8
89.5
76.77
70.1
10.3
45.7
23.5
1.0%
Original LLM FT
41.0
40.6
26.8
68.5
66.1
47.4
23,5
Netioeaa Aword Methads BMES
28.8
36.1
23.8
34.0
52.0
34.1
5.9
4,0x
SIBRR
18.4
22,9
21.6
50.4
34.6
25,9
5.9
4.0x
OpenAl
300
19.1
23.8
22.9
51,5
27.9
5.9
4,0%
Congretaton bused Methods
Long MiLings!
24.7
20,3
26.3
55.0
50.1
7.8
3.0x
4. Among compression-bws based methods, explicit method (LongLLMLinggua) and im- plicit methods (represented by Activation Beacon) demonstrate varied efficacy across different task types. For instance, LongLLLMinngua achieves competitive performance with AdmTree on summarization tasks but significantly lags behind question answering tasks. This observation aligns with our assertion in the introduction that explicit compression methods may struggle with tasks demanding fine-grained details. Conversely, Adm'Tree exhibits consistently strong and balanced performance across all task types. As further evidenced by our analyses in Figure [ Adm'Tree maintains stable performance across summarization tasks of different granularities and QA tasks with different ground-trute information location. This consistent performance, with lower variance, demonstrating AdmTree's superior capability in preserving semantic integrity across multiple information dimensions, and also suggests the potential of integrating offuure--f-xxxerts (MoE) architectures further specialize compression across diverse task requirements.
SnapKV
33.7
31.6
26.2
67.1
00.3
45.0
7.8
30x
Beacos!
40.5
40.3
26.8
68.4
66.4
47.2
7.3
3.2%
AdmTree (Ours)
41.6
45.9
30.2
69.9
66.8
49.7
7.0
3.4%
4.3 Ablation Experiments
To quantify the contributions of each modules overall performance, we conducted ablation ex- periments covering different training stages, model structures, and tree construction strategies. As shown in Table 2 both pre-training and instruction fine-tuning enhance the compression learning. Pre-training exerts the larger impact, probably since it learned from the full-sentence contexts, whereas instruction tuning optimized LLMs only on task outputs. Removing the tree structure (while maintaining the compression ratio) resulted in severe performance drop to 28.5, demonstrating the effectiveness of our proposed tree architecture. When the adaptive leaf gist token allocation was replaced with uniform allocation, indicating that adaptive compression better balances semantic information across segments. Furthermore, removing the parameters from the self-attention layer in the tree aggregation process also led to a notable performance decrease. In general, single-layer self-attention strikes an effective trade-off between performance and efficiency.
Tree-node retrieval To further enhance inference efficiency, we experimented with directly re- trieving only the top 75% of tree nodes during inference, based on their attention scores relative to
7

---

Table Experiments in dialogue scenarios, where context arrives dynamically rather than requiring
one-time compression. ShareGPT dataset [12 is used. AdmTree demonstrates notable flexibility.
1 Turn (765 tokens)
TTur (3006 tokeus)
Turn (6491 tokens)
Method
PPL1
Lateacy
TFLOPs
PPL.
Latency
THLOPs
PPL1
Latency
TFLOPs
AutoCompessoors
8.32
0.76
8.41
6.09
1.38
38.18
7.51
1.82
96.63
ICAE
7.72
8.13
1.20
92.37
LongL..M..nggn .inggu
5.91
0.78
8.78
4.96
1.41
48.32
4.77
1.98
145.71
SnapKV
Beacon
4.27
0.37
8.63
3.08
0.72
34.54
2.98
0.94
75.41
Adm'Tree (Ours)
4.01
0.37
8.37
2.91
0.70
34,09
2.79
0.92
75.38
the last token. The resulting performance loss is marginal: the model still surpasses competitive
baselines such as SnapKV on single-document QA. The scalability of AdmTree thus enables further tiference-time optimizations under limited computational budgets.
Aree Conicall Lenght in 46
Cantrxi Lengla 1k
Connext Length
25
Table 2: Ablation experiments on LLaMA-based AdmTrree
ROUXIE-L
Ablation
Single-Doc
Full
36.5
wro Pre-training
26.6
woo Fine-tuning
29.3
wio Tree Strccuree
28.5
CompecssinnRalo
Comgrosios Ratio
Congressius Ratio
w/o Self-attention
29.6
w/o Adaptive Leaf Constr.
34,1
Figure 3: Model performance under varying context lengths and compression ratios.
Nodes Retrieval
5
Analysis Experiments
In this section, several characteristics of AdmTree are investigated, including flexibility, effects across varying context lengths and compression ratios, and compression effects on fine-grained information.
5.1 Compression Flexibility
We evaluate the flexibility of compression methods multi-turn dialogue scenarios, where demands efficient compression of dynamically updated conversational context (utterances). The ShareGPT dataset [1], comprising lengthy dialogues between humans and ChatGPT, was utilized for evaluation. We assessed methods over three consecutive dialogue turns, with the average context length pro- gressively increasing from 765 to 6,491 tokens. Perplexity (PPL) was employed to assess responses quality, while latency and TeraFLOPs (TFLOPs) were reported to measure compression efficiency.
As shown in Table 3 AdmTree consistently achieves the lowest PPL scores across all three dialogue turns. Notably, its PPL decreases as the dialogue progresses and context length increases, which indicates that AdmTree effectively compresses the dialogue history, preserving semantic integrity and conversational structure. Mcanwhile, one-time compression methods (e.g.. LongLLMLingur and ICAE) incur dramatic rise computational cost as the dialogue extends, since they need to the recompress entire context at each turn. While Adm'Tree and Activation Beacon mitigate this issue as recursive compression methods, which allow for the reuse of compressed representations from previous turns. This scalability is highly desirable for dynamic, real-world applications like online dialogue systems, where context is continuously updated, not merely processed a single pass.
5.2 Compression with Different Context Length and Compression Ratio
Figure presents the performance of various compression methods under different context lengths and compression ratios. Specifically, we use the Multi-Session Conversation (MSC) dataset, constructed
obtain dialogue samples with average lengths of 1K. 2K, and 4K tokens. Following MemGPT, ROUGE-L is employed to evaluate response quality. We can found that Adm'Tree consistently outperforms all baseline models across all context lengths and compression ratios by substantial margin. At a 1K context length, Adm'Tre maintains stable performance, even showing slight improvement as the compression ratio increases. For longer contexts, TrmT exhibits much smaller degradation in performance compared to other methods.
by MemGPT 44). We further process
8

---

L--7 with Adm Tre (Ours)
LIaMA-2-TB with Activation Bracon
618
641
1872
1100
710
Contest Length
Context Length
(a) ROUGE scores of responses generated by Troe show scores
(b) ROUGE scores of responses generated by Activation
Beacon.
Figure 4: Comparison of Needle-in-Ther The- StheeHaystcck [27 results with LLaMA-2-7B as backbone LLM. Notably, several baselines collapse under the combination of long context and high compression ratio, with ROUGE-L scores dropping below 10.
5.3 Fine-grained Information Compression
We evaluated the fine-graines information retention capabilities of AdmTree and the state-of-the-an Activation Beacon with the challenging Heedle-inthhe-Haystee dataset. In this task, specific piece of information ("needle") is embedded at a specific position within a lengthy document ("haystack"), and the model is prompted to retrieve this detail. As illustrated in Figure AdmTree consistently and precisely extracts the target information across documents of varying lengths and needle positions, demonstrating strong semantic preservation. Notably, the length of input contexts in this task substantially exceeds those seen during AdmTree's training, yet the model maintains stable performance.
6 Related Work
6.1 Compression Methods
As input contexts grow increasingly long, context compression has attracted significant attention. The goal is to reduce input token length while maintaining semantic integrity. The existing methods can be broadly categorized into two categories: (1) Explicit methods that directly shorten text by
removing less crucial content based on information theory [25] 26 37
45
[1] These methods range
from token-level pruning [25] 45), to sentence-level methods such as CPC [39 (2) Implicit methods
include encoding imputs into soft tokens using methods such as Gisting [1] and Gist-COCO (35),
or applying recursive-basee compression as AutoCompressor []] ICAE (20] and Activation
Beacon [58]] Specialized compression methods have also been proposed, including xRAG [10] for retrieval-augee auggentd generation and CCM (2 for online scenarios.
6.2 Retrieval Methods
Retrieval methods enable I.LMs to process long context by selecting the most relevant segments based on their similarity to a given query. These methods are generally categorized into two types: (1) Sparse Retrieval: use cern-frequency-basee represent documents as sparse vectors based on keyword occurrences. (2) Dense Retrieval: employ neural networks to encode queries and documents into continuous vector spaces, enabling semantic similarity matching. Common models include Sentence-BERY Contriever 24 and DPR 28
models, such as TF-IDF (46) and BM25 [48), which
7 Conclusion
In this paper, to address the limitations of existing context compression methods in preserving semantic fidelity, we propose AdmTree, a novel hierarchical compression framework that dynamically allocates gist tokens and constructs adaptive semantic trees. By integrating information-an awaae segmentation with lightweight tree-based aggregation, AdmTree comprehensively preserves semantic information across multiple dimensions. Experiments on LongBench show that AdmTree consistently surpasses sate-of-the-an baselines both performance and inference efficiency across diverse tasks. Further analysis also confirms that ArmTree exhibits strong scalability in dynamic context scenarios.

---

Acknowledgments
This research is supported by the National Natural Science Foundation of China (Grant No.622761.. 154), the Research Center for Computer Network (Shenzhen) Ministry of Education, the Natural Science Foundation of Guangdong Province (Grant No. 2024TQ08X729 and JCYJ2024081 12009013, Research Fund of Shenzhen City (Grant N0.1CY12021103241200100 and GJHZ20240218 13603006), and the Major Key Project of PCL for Experiments and Applications (PCL2023A09).
14), the Basic
References
[1] Sharegpt. 2023. URL httpp:://saargggt..co com/
[2] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303. 08774, 2023.
[3] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023.
[4] Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. LongBench: A bilingual, multitask benchmark for long context understanding. In Lun-Wei Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3119-3137, Bangkok, Thailand, August 2024. Association for Computational Linguistics.
[5] Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng. Xiaozhi Wang, Xin Lv, Shulin Cao, Jiazheng Xu, Lei Hou, Yuxiao Dong, et al. Longbench v2: Towards deeper understanding and reasoning realistic long-contex, multitasks. arXiv preprint ar arXi:22412.55204 2024.
[6] Matthew M Botvinick. Hierarchical models of behavior and prefrontal function. Trends in cognitive sciences, 12(5):201-20)) 2008
[7] Ruisi Cai, Yuandong Tian, Zhangyang Wang, and Beidi Chen. Lococo: Dropping in con- volutions for long context compression. In Forty-first International Conference on Machine Learning. 2024.
[8] Weilin Cai, Juyong Jiang, Fan Wang, Jing Tang, Sunghun Kim, and Jiayi Huang. A survey on mixture of experts in large language models. IEEE Transactions on Knowledge and Data Engineering, 2025.
[9] Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. Lon- gloRA: Efficient fine-tuning of long-context large language models. The Twelfth International Conference Learning Representations, 2024.
[10] Xin Cheng, Xun Wang, Xingxing Zhang, Tao Ge, Si-Qing Chen, Furu Wei, Huishuai Zhang, and Dongyan Zhao xrag: Extreme context compression for retrieval- -uggeeted generation with one token. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomezak, and Cheng Zhang, editors, Advances in Neural Information Processing Systems 38: Annual Conference Neural Informa- tion Processing Systems 2024, NeurlPS 2024, Vancouver, BC, Canada, December 10 files/ http://pper222222///2/ 15, 2024, 2024. URL http::/ nips wttpppperfillsee/////eeeeeeaeeee -ww.:00777--bbttraro Abstract- Conf erence html
[11] Alexis Chevalier, Alexander Wettig, Anirudh Ajith, and Danqi Chen. Adapting language models to compress contexts. In Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pages 3829-3846, 2023.
[12] Wei-Lin Chiang, Zhuohan Li, Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90% chatgpt quality. See http:://vicuna Imsys. org (accessed April 2023), 2(3); 2023.
10

---

[13] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jeny Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. CoRR, abs/21 14166, 2021. URL
http://arxii...or/a////110/00 14168
[14] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110. 14168, 2021.
[15] Chenlong Deng, Zhisong Zhang, Kelong Mao, Shuaiyi Li, Xinting Huang, Dong Yu, and Zhicheng Dou. silver bullet or a compromise for full attention? a comprehensive study of gist token-based context compression. arXiv e-prints, pages arXiv-2412, 2024.
[16] Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. complexity-basee prompting for multi-step reasoning. In The Eleventh International Conference Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https://openreviexx wwwt/eerrr/meee wowomoiddyyfiiiiioooi dfidddyyf1iccc1111
[17] Yao Fu, Rameswar Panda, Xinyao Niu, Xiang Yue, Hannanch Hajishirzi, Yoon Kim, and Hao Peng. Data engineering for scaling language models to 128k context. In Forty-first International Conference on Machine Learning, (CM4.2024 Vienna, Austria, July 21-27, 2024.
OpenReview.net, 2024. hww.l//pgg..eeepaaoooeeeeeee/e/e
[18] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang. Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The Pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.0000 2020.
[19] Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. Retrieval-augmennte generation for large language models: A survey, arXiv preprint arXiv:2312. 10997, 2023.
[20] Tao Ge, Hu Jing. Lei Wang, Xun Wang, Si-Qing Chen, and Furu Wei. In-context autoencoder for context compression in large language model. In The Twelfth International Conference on Learning Representations, 2023.
[21] Tao Ge, Hu Jing, Lei Wang, Xun Wang, Si-Qing Chen, and Furu Wei. In-context autoencoder for context compression in large language model. In The Twelfth International Conference on Learning Representations, 2024. URL
net/ffoumm juj 4ZuGJE 4ZuGJE
[22] Stephen C Hirtle and John Jonides. Evidence of hierarchies in cognitive maps. Memory & cognition, 13(3):208- 217, 1985.
[23] Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng. Haotian Wang, Qiang- long Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. A survey hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems, 2024.
[24] Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised dense information retrieval with contrastive learning. Transactions on Machine Learning Research.
[25] Huiqiang Jiang, Qianhui Wu, Chin- Yew Lin, Yuqing Yang, and Lili Qiu. LImlingua: Compress- ing prompts for accelerated inference of large language models. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023.
[26] Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin- Yei Lin, Yuqing Yang, and Lili Qiu. LMnnguML Accelerating and enhancing LLMs in long context scenarios via prompt compression. Lun-Wer Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 1658-1677, Bangkok, Thailand, August 2024. Association for Computational Linguistics.
11

---

[27] Greg Kamradt. Needle in haystack pressure testing llms. 2024.
https: com/
[28] Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen Tau Yih. Dense passage retrieval for open-domain question answering. In 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Association for Computational Linguisties (ACL), 2020.
pages
[29] Mang-Hyun oa Kim, Junyoung Yeom, Sangdoo Yun, and Hyun Oh Song. Compressed context memory for online language model interaction. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
URL htttts://pppereee/r//
[30] Sehoon Kim, Sheng Shen, David Thorsley, Amir Gholami, Woosuk Kwon, Joseph Hassoun, and Kurt Keutzer. Learned token pruning for transformers. In Aidong Zhang and Huzefa Rangwala, editors, KDD '22: The 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Washington, DC, USA. August 14 18, 2022, pages 784-79 794. ACM, 2022. doi: 10.1 45/3353467.35532666 URL.tttpe ptttpss:/doi... //t.i//doiioooo org/ /trg/10.. 10. 10e11144335555 0544655553677... 3539260
[31] Wojciech Kryściński, Nazneen Rajani, Divyansh Agarwal, Caiming Xiong, and Dragomir Radev, Booksum: collection of datasets for long-form narrative summarization. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages
2022.
[32] Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453-466, 2019.
[33] Kuang-Huei Lee, Xinyun Chen, Hiroki Furuta, John Canny, and Ian Fischer. A human-inspired reading agent with gist memory of very long contexts. In ICLR 2024 Workshop on Large Language Model (LLM) Agents, 2024.
[34] Dacheng Li, Rulin Shao, Anze Xie, Ying Sheng, Lianmin Zheng, Joseph E. Gonzalez, Ion Stoica, Xuezhe Ma, and Hao Zhang. How long can open-source 1lms truly promise on context length? 2023, URL
1000--006-9-----0 longhatt
[35] Xinze Li, Zhenghao Liu, Chenyan Xiong, Shi Yu, Yukun Yan, Shuo Wang, and Ge Yu. Say more with less: Understanding prompt learning behaviors through gist compression. arXiv preprint arXiv:2402. /6058, 2024.
[36] Yucheng Li. Unlocking context constraints of Ilms: Enhancing context efficiency of Ilms with self-information-bere based content filtering. CoRR, abs/2304. 2023. doi: 10.48550/ARXIV. URL https://doi. ftttp:://ddi.org/111/
org/1o 10. 18550/arXivy 2304. 12102
[37] Yucheng Li, Bo Dong, Frank Guerin, and Chenghua Lin. Compressing context to enhance inference efficiency of large language models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pages 6342-6353, Association for Computational Linguistics, 2023. doi: 18.13//3/V /22.:222..EP--MAIII...MM https://doi.. org/10. (18853/vv/220 wwn.p-main.nnmmmm
URL
main. 391
[38] Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. Snapkv: Llm knows what you are looking for before generation. arXiv preprint arXiv:2404. 14469, 2024.
[39] Barys Liskavets, Maxim Ushakoy, Shuvendu Roy, Mark Klibanov, Ali Etemad, and Shane K Luke. Prompt compression with context-aware sentence encoding for fast and improved Ilm inference. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 24595-24604, 2025.
[40] Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157-173, 2024.
12

---

[41] Jesse Mu, Xiang Li, and Noah Goodman. Learning to compress prompts with gist tokens. Advances Neural Information Processing Systems, 36, 2023.
[42] Erik Nijkamp, Tian Xie, Hiroaki Hayashi, Bo Pang, Congying Xia, Chen Xing, Jesse Vig. Semih Yavuz, Philippe Laban, Ben Krause, Senthil Purushwalkam, Tong Niu, Wojeiech Kryscinski, Lidiya Murakhovs' ka, Prafulla Kumar Choubey, Alexander R. Fabbri, Ye Liu, Rui Meng, Lifu Tu, Meghana Bhat, Chien-Sheng Wu, Silvio Savarese, Yingbo Zhou, Shafiq Joty, and Caiming Xiong. Xgen-7b technical report. CoRR, abs/2309.034500 2023. doi: 10.48550/RXXV.23300 03450. URL wtthh:ttts/aooo org/ 2..0..8550/aaa. arxiv 2309. 03450
[43] Charles Packer, Vivian Fang, Shishir G. Patil, Kevin Lin, Sarah Wooders, and Joseph E. Gonzalez. Memgpt: Towards llms as operating systems. CoRR, abs/2310.08566) 2023, doi: 1048550/ARXX.23100005555
https::////// 48550//4455/////// arrvvv 201.. 08560
[44] Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang. Shishir G Patil, lon Stoica, and Joseph E Gonzalez. Memgpt: Towards llms as operating systems. arXiv preprint arXiv:2310.08560, 2023.
[45] Zhuoshi Pan, Qianhui Wu, Huiqiang Jiang, Menglin Xia, Xufang Luo, Jue Zhang, Qingwei Lin, Victor Rühle, Yuqing Yang, Chin- Yew Lin, H. Vicky Zhao, Lili Qiu, and Dongmei Zhang. LImlingua-22 2: Data distillation for efficient and faithful task-agnostic prompt compression. In Lun-Wei Ku, Andre Martins. and Vivek Srikumar, editors, Findings of the Association for Computational Linguisties, ACI. 2024, Bangkok, Thailland and virtual meeting. August / 11-16, 2024, pages 963-981. Association for Computational Linguistics, 2024. doi: 10. 10.18653/11/202, FINDINGS-ACL.57. Jtthttps:://// org/ wtt../..../..../////// 00083/vv1/2224/ http://0..1../////222..oonoonnoo ac1.
[46] Shahzad Qaiser and Ramsha Ali. Text mining: use of tf-idf examine the relevance of words to documents, International journal of computer applications, 181(1):25-29, 2018.
[47] Nils Reimers and Iryna Gurevych. Sentence-BERN Sentence embeddings using Siamese BERT- networks. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) pages 3982-3992, Hong Kong, China, November 2019, Association for Computational Linguistics.
[48] Stephen Robertson, Hugo Zaragoza, al. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends@ in Information Retrieval, 3(4):333-389) 2009.
[49] ByteDance Seed, Yufeng Yuan, Yu Yue, Mingxuan Wang, Xiaochen Zuo, Jiaze Chen, Lin Yan, Wenyuan Xu, Chi Zhang, Xin Liu, et al. Seed-thinking-vl. 5: Advancing superb reasoning models with reinforcement learning. arXiv preprint arXiv:2504.13114 2025.
[50] Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou, and Jason Wei. Challenging BIG-bench tasks and whether chain-of-thoughtt can solve them. In Anna Rogers, Jordan Boyd- Graber, and Naoaki Okazaki, editors, Findings of the Association for Computational Linguisties: ACL 2023, pages 13003-13051, Toronto, Canada, July 2023. Association for Computational Linguistics.
[51] Jiwei Tang, Jin Xu, Tingwei Lu, Hai Lin, Yiming Zhao, and Hai-Tao Zheng. Perception compressora triiningffree prompt compression method in long context scenarios. CoRR, 19272, 2024. doi: 10.48550/ARXIV.24099005 19272. URL htttps:///oi.rrg///
18550/arXiv.. 2409. 19272
[52] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Amn Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13711, 2023.
[53] Maurice Weber, Daniel Fu, Quentin Anthony, Yonatan Oren, Shane Adams, Anton Alexandrov, Xiaozhong Lyu, Huu Nguyen, Xiaozhe Yao, Virginia Adams, et al. Redpajama: an open dataset for training large language models. arXiv preprint arXiv:2411.12: 2024.
13

---

[54] Zhiheng Xi, Wenxiang Chen, Xin Guo. Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang. Senjie Jin, Enyu Zhou, al. The rise and potential of large language model based agents: A survey. arXiv preprint 2wXiv2230..08844 2023.
[55] Peng Xu, Wei Ping, Xianchao Wu, Lawrence McAfee, Chen Zhu, Zihan Liu, Sandeep Sub- ramanian, Evelina Bakhturina, Mohammad Shoeybi, and Bryan Catanzaro. Retrieval meets long context large language models. In The Twelfth International Conference Learning Representations, 2023.
[56] Yang Xu, Yunlong Feng, Honglin Mu, Yutai Hou, Yitong Li, Xinghao Wang, Wanjun Zhong, Zhongyang Li, Dandan Tu, Qingfu Zhu, al. Concise and precise context compression for tool-using language models. In Findings of the Association for Computational Linguistics ACL 2024, pages 16430-16441, 2024.
[57] Chanwoong Yoon, Taewhoo Lee, Hyeon Hwang, Minbyul Jeong, and Jaewoo Kang. Compact: Compressing retrieved documents actively for question answering. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 21424-21439, 2024.
[58] Peitian Zhang, Zheng Liu, Shitao Xiao, Ninglu Shao, Qiwei Ye, and Zhicheng Dou. Long context compression with activation beacon. In The Thirteenth International Conference Learning Representations, 2025. URL
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We demonstrate the advantages of our AdmTree in both performance and efficiency the experiment section.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
In conclusion section, we point the limitations.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide implemention details in experiment section and appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [Yes]
Justification:
We use public dataset.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ccpppp.ccppulli//guiii wwwticccguie//eeeee CoodeSubeiiie ssi wswinnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide implemention details in experiment section and appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We report the average results over three inference runs.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
17

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes] We provide it in the Appendix
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: //t::://neuisseeeeeee httppubccc/thiccccueeeeeeeeeeeee
Answer: [Yes]
Justification: Our research fully adheres the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discussed in introduction.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
18

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: the paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: all creators and original owners of assets used our paper have been properly credited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwwpsspipetattteeeeee thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
19

---

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: we will release our code and model in Github with well-documented readme.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification: the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact core methodology, scientific rigorousness, originality of the research, declaration is not required.
20

---

Answer: [NA]
Justification: the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not non
involve LLMs as any important, original, non non-standard components.
Please refer to our LLM policy for what should or should not be described.
21

---

A Additional Related Work
A.1 Extending LLM Window Size
Unlike the above methods that improve LLM efficiency for long-text processing, some studies focus on expanding the context window of LLMs. A mainstream approach involves progressive pre-training strategies [17 49, which incrementally extend the maximum context length through phased training. Although these methods achieve strong performance, they incur high training costs and do not reduce inference overhead. In our experiments, we include fine-tuned LLMs trained on the same long-context datasets strong baselines (Table I).
B Experiment Setting
B.1 Method Details
Baselines Following prior work 26 58 on compression, we introduce two main categories of
baseline models for processing long contexts:
(i) Retrieval-based Methods. These methods employ retriever rank segmented long contexts based on their relevance to the given query (instruction). We utilize one sparse retrieval method BM25 [48) and dense retrieval methods, Sentence-BERT 47 and OpenAT Embeddings" After assessing the relevance of the input context to the query using these models, segments with lower relevance are discarded iteratively until the required compression ratio is achieved.
(ii) Compression-busee Methods. We select LongLLMLingua (26 as representative explicit compression methods. AutoCompressor [II. ICAE (2I, Activation Beacon (58), and SnapKV 33 are employed as implicit compression methods. The former three methods recursively com- presses long contexts into semantic vectors, enabling preliminary fine one-grained compression.
We also fine-tuned the original L.M using the same training data to serve as the baseline, denoted as Original LLM-FT.
Backbone LLMs -1aMA-2-7B--00 -B-Chat and Qwen-2-7B Instructt employed as the backbone LLMs. We use LlaMA-2-7B-Cha. because three out of the four compression baselines utilize Throughout the experiments, adopt LlaMA-2 the backbone model unless explicitly stated otherwise.
Implementation Details In training AdmTree, we use the same dataset as Zhang et al. ensure fairness. Specifically, pre-training is conducted 1B tokens sampled from RedPajama 53 During fine-tuning, we utilize LongAlpaca [9]. BookSum [31], and 16K synthetic samples generated by GPT-3.5 [8]. AdmTree is trained with batch size of 8 NVIDIA A800 GPUs, with learning rates set to 5e-5 for pre-training and le-5 for fine-tuning, respectively.
58
to
For the compression ratio in main experiments, given LLaMA-2's maximum sequence length of 4K, we apply x2 compression for 4K-8K contexts, x4 for 8K-16K, and x8 for 16K-32K. For Qwen-2, uniformly set the compression ratio to x4.
B.2 Dataset Details
Evaluation Benchmark In main experiments, we evaluate AdmTree's effectiveness and efficiency on LongBench [4] and Seaee-nn-The-Haystte (27). LongBench is multitask benchmark assessing L.LMs" comprehension of long contexts across various tasks such as single- and multi- document QA and summarization, with 32K maximum length. Needle-In-Thee Saale--n--Te--Hyystte Haystack assesses the model's s ability extract pertinent fact ("needle")) from lengthy context ("haystack")).
In other section, we further experiment on diverse benchmarks, including GSMSK [14] and BBH 50 for demonstration compression, MSC [44] and ShareGPT [12 for dialogue compression, Arxiv-March23 for summarization scenario, Topic Retrieval for retrieval scenario:
We use text-enbedding -dd--i002 OpenAl.
22

---

longbenchv2 This dataset, introduced by Bai et al. [5], spans six major task categories and features documents ranging from 8,000 words million words length. The benchmark presents challenging four-choice question format, with performance evaluated using accuracy as the primary metric.
GSM8K This dataset [3 is widely recognized benchmark for evaluating mathematical reasoning capabilities. It specifically assesses a model's proficiency performing arithmetic operations and constructing coherent, step-by-step solutions using natural language. In our implementation, we employ a 2-shot learning approach with the original prompt being a complex, multi-step chain-of- thought (CoT) [16] formulation.
BBH
This dataset [30] is designed for language and symbolic reasoning tasks, with a specific focus on evaluating chain-of- chain-of-thought (CoT) reasoning capabilities. In our experiments, we employ two distinct CoT prompts as the baseline for comparison.
ShareGPT This dialogue dataset 1 is sourced from ShareGPT.ccmm, platform where users publicly share their ChatGPT interactions across multiple languages and diverse conversational scenarios (e.g., casual chats, writing assistance, and more). For our evaluation, we adopt the curated subset provided by [58) as our test set.
Arxiv-March23 This dataset is constructed from the most recent academic papers published in the arXiv preprint repository during March 2023. For evaluation purposes, we utilize a curated subset of 500 samples collected our test set [36]. Input text length limited to 10,000 characters and employ DeepSeek to automatically generate summaries for each test instance, which serve as the reference answers
25
This evaluation dataset [34] systematically examines model adaptability to varying context lengths and compression requirements. The framework employs structured multi-turn dialog sequences between human users and chatbots
Topic Retrieval
MSC Derived from MemGPT 43), this multi-session benchmark comprises the MSC dataset containing extended human-human dialogues (avg. 2,000 tokens). We systematically transformed the original dataset through prompt engineering to create length variants (1K and 4K tokens) for comprehensive evaluation of context-length robustness.
C
Analysis Experiments on More Benchmarks and Scenarios
C.1 Compression on LongBench v2
To ensure consistency with settings in [58], the experiments in main body conducted the I LongBench benchmark. In Table we further report results on the more challenging LongBench
v2 [5], which requiring deeper understanding and reasoning over long contexts. On LongBench v2, AdmTree demonstrates consistent absolute improvements and even more pronounced relative gains. Specifically, it outperforms the state-of-the-ard Activation Beacon by +18.5% on LLaMA and +12.7% on Qwen. Moreover, on the newly introduced Long-dialogue History Understanding task, AdmTree brings notable performance boost, demonstrating superior ability to preserve dialogue structure and semantics during compression. Besides, compared to the performance on the original LongBench, Activation Beacon no longer consistently surpasses other baselines on some tasks (e.g. Multi-Document QA) in LongBench v2. This trend underscores the potential advantages of strecttrut compression in facilitating structured reasoning over long-context scenarios.
C.2 Compression on Reasoning Data
Following the experimental setup of Jiang et al. (25). GSMBK and BBH are employed to assess model performance reasoning and in-context learning scenarios. We introduce a strong baseline, Full-shot, whose prompt lengths on both datasets remain within the maximum context window of LLaMA-2-7B. As shown in Table ??, AdmTree consistently surpasses all baseline methods. Notably, even under the quarter-shot compression constraint, AdmTree outperforms the Full-shot baseline, achieving
23

---

Table 4: Experiment results (%) on LongBench [S]
We also report the micro-averaged performance
across all datasets.
LongBenchy2
Methods
SingleDoc
MultiDoc
ICL
History
Code
Data
AVG
LLama-2-7B
AutoCompressor
3.1
1.5
2.6
0.8
8.9
7.2
3.1
ICAE [21
4.2
2.7
2.1
3.5
9.6
8.5
4.1
LongLLMLingua 26
20.0
17.6
8.6
2.6
16.0
11.8
14.9
SnapKV 38
18.1
15.2
23.1
16.2
18.3
19.2
18.1
Beacon
[58
20.0
14.4
25,9
18.0
22.0
21.2
19.5
Adm Tree (Ours)
22.9
19.2
27.2
25.6
24.0
24.2
23.1
Qwen-2-7B
LongLLLMingg 26
22.3
19.6
12.3
8.5
13.2
20.1
18.1
SnapKV 38
29.5
27.2
28.6
29.1
15.3
27.9
27.9
Beacon [58]
34.3
18.4
29.6
30.8
16.0
33.3
28.3
AdmTreee (Ours)
34.9
25.6
33.3
38.5
22.0
33.3
31.9
Table Experiment results (Exact Match, EM) under different compression ratios on the GSM8K mathematical reasoning [14 and Big-bench Hard (BBH) datasets [50]
GSMSK
BBH
Method
1-sl, const.
half-st. const. (<<14
quarter-st. const.
1-sl. const.
half-st. const.
quarter-st. const.
(~5
(~~20
(~33 x)
(<55 x)
(<>7 x)
AutoComprssoo
22.7
21.5
21.2
13.71
11.23
ICAE
37.76
35,21
33.12
21.67
20.12
19.83
Snapky
44.56
42.95
43.81
38.61
36.83
36.01
LongLLMLiin
42.55
43.46
43.23
33.37
39.42
38.81
Beacon
45.74
44.36
44.13
43.95
44.25
42.44
Adm Tree (Ours)
48.12
47.93
46.85
45.06
53.53
Zern-shot
41.87
36.49
Full-shot
46.88
43.34
compression ratios exceeding 20x and 7x on GSM8K and BBH, respectively. These results indicate that AdmTree effectively retains the reasoning process encoded in the original prompts. Interestingly, on the BBH dataset, higher compression ratios yield performance improvements for AdmTree. We attribute this to the relatively short full-shot prompts in BBH (fewer than 500 tokens), where fewer but more compact compressed tokens may enhance compression quality and thereby improve overall performance.
C.3 Compression on Summarization Scenario
We evaluate the performance different compression models on the summarization task using an additional dataset, ArXiv-March23, As shown in Table we compare the models under 2x and 4x compression ratios. Across metrics, AdmTTree consistently achieves strong performance. Besides, its performance under the 4x compression ratio does not show significant degradation compared to 2x setting. contrast, Activation Beacon performs noticeably worse on the BLEU metric. This may be attributed to the nature of BLEU, which emphasizes n-gram continuity, unlike ROUGE, which focuses on longest common subsequences. The lower BLEU score potentially suggests that Activation Beacon may generate less coherent summaries, potentially introducing unwanted or disjointed tokens.
C.4 Compression on Retrieval Scenario
conduct additional experiments retrieval scenario using Topic Retrieval dataset as supplement to the Needle-in-a- Haystack experiment. Different from Needle-in-a-Hasstanl which varies the position the "needle" within fixed-length context, Topic Retrieval varies the context length by modifying the number of total topics and then querying information about the first topic.

---

Table Compression performance in summarization scenarios. The ArXiv-March23 dataset used for evaluation. Metrics include ROUGE-1, ROUGE-2, ROUGE-L. BERTSccore, and BLEU.
Methods
ROUGE-1
ROUGE-2
ROUGE-L
BS F1
BLEU
Original LLM
40.35
16.22
27.22
81.57
34.88
2x Compression Constraint
AutoCompressor
9.05
0.24
6.31
12.43
5.05
ICAE
12.87
1.03
8.42
15.25
15.77
LongLLMLingua
31.78
9.29
20.01
74.24
24.81
SnapKV
33.23
12.56
25.1
73.09
30.23
Beacon
35.42
15.47
25.91
77.69
16.05
AdmTree (Ours)
36.10
15.69
26.60
79.83
35.99
4x Compression Constraint
AutoCompressor
7.69
0.21
5.21
11.36
4.21
ICAE
10.56
0.89
7.65
13.21
13.52
LongLLMLingnn
29.3
7,43
18.2
69.04
26.36
SnapKV
31.25
8.92
20.53
70.21
23.21
Beacon
32.18
14.12
23.96
74.25
17.22
AdmTree (Ours)
35.51
15.06
25.84
79.97
34.82
Aubrroopresser
ICAE
LMLinnu
SnapkV
Beacen
Admirre (Ours)
#Topics-1, Context]-732
#Topics- 10, [Context
#Topics-15, [Contex]--9888
TTopics-30, (Context-18040
Accuncy
+15
Compression Ratio
Compussion Ratio
Compression Ratio
Compression Ratio
Figure 5: Retrieval accuracy comparison the Topic Retrieval dataset under different context lengths and compression ratios.
Accuracy is used to measured the retrieval effectiveness after compression. Figure presents the retrieval accuracy of different methods under varying compression ratios and context lengths. AdmTTree consistently achieves the best performance across all conditions. Notably, shorter context length (Context Length= 732), Adm'Tree maintains near-lossless compression even at 16 compression ratio. For longer contexts, AdmTTree also exhibits lower performance degradation compared to other methods. In contrast, AutoCompressor and LongLLMLingua fail to handle context lengths beyond 6000 tokens effectively.
D
Visualization Experiments
D.1 Attention Pattern Analysis
Figure presennss KL divergence analysis of the attention distributions for tasks requiring varied semantic information, comparing AdmTree and Activation Beacon. Specifically, we measure the attention scores from the sentence's final token to the preceding compressed (gist) tokens at the onset of generation. Tasks 0)-9 correspond to repetition tasks involving different semantic requirements, including the repetition of content from different positions (beginning, middle, end) and varying granularities (word, sentence, paragraph). AdmTree demonstrates substantially greater diversity attention distributions across tasks, indicating that its compressed tokens encode distinct and semantically differentiated information. This diversity likely contributes to its improved performance by enabling more task-specific representation retrieval. Conversely, recursive compression methods such Activation Beacon not only suffer from semantic degradation through iterative compression,
25

---

01
015
Attention
0.00
Attention to Target Segment
Attention Target Segment
(a) AduiTree: Higher KL divergence acroes tasks, indicating greater pressed tokens.
(b) Activation Beacon: Lower KL divergence across tasks, indicating more uniform attention com- pressed tokens.
in attention com-
Figure 6: The Kullback-Leibler (KL) divergence the attention distribution from the final token to the compressed tokens across tasks of varying semantic types.
Activation Beacon
1.2
Activation Beacon
2.2
AdmTree
Loss
1.0
1,8
0.2
0.6
0.8
1.0
0.0
0.2
0.6
1.9
Epoch
Epech
(a) Pre-training stage.
(b) Fine-tuning stage.
Figure 7: Loss curve during the different training stage of AdmTreee and Activation Beacon.
but also tend to homogenize the semantic content across compressed tokens, thereby limiting their generalizability across diverse tasks.
D.2 Training Trends Comparison
AdmTree was trained on the same dataset as Activation Beacon. Figure 7 illustrates the loss convergence trends during both the pre-training and fine-tuning stages for the two methods. As shown, AdmTree achieves better convergence in the multi-task fine-tuning sonin stage.
D.3
Case Study
We introduce the repetition task, which requires models to faithfully reproduce the original input text without omission or alteration. As shown Table this task provides an intuitive perspective through which we can observe how different methods preserve semantic information. Compared to the baseline model, our Adm achieves near-pette sear-cerfect reconstruction of the original text both cases, merely leveraging the gist tokens within its semantic tree structure. In contrast, Activation Beacon exhibits severe hallucinations. the first case, it fabricates content that does not exist in the original text based its internal knowledge; in the second case, it falls into uncontrolled repetition. These qualitative comparisons clearly demonstrate that Adm Tree more capable of preserving the original semantics in faithful and complete manner.
26

---

Table 7: Case studies on the repetition task. We sample several contexts from the GovReport subset of LongBench and evaluate whether models can accurately faithfully repeat the original content. We compare the outputs of our AdmTree and Activation Beacon. Content in red indicates hallucinated or redundant content that deviates from the original text.
Original Context
Activation Beacon
Adm'Tree (Ours)
Colombia, the oldest democracies the Westem Hemisphere and the third most on-
Colembia, one of the oldest democracies in the Western Hemisphere and the third most populous Latin American country. has endured multisided civil conflict for than five decades until President Juan Ma nuel Santos declared the conflict over in August 2017 at the end a U.N.- monitored disarmaaentt According to the National Center for Historical Memory 2013 report, presented to the Colombian govermeent as part of the peace process to end the lighting, some 220,000 Colom- hians died in the armed conflict through 2012, 81% them civilians. also provided statistics quantitying the scale of the conflict, which has on Colombian society: more than 23, 33,000
Colombia is founding member of the
populous Latin American country. dured a multisided civil conflict for more than five decudes until President Juan San- tos declared the comflict over in August 2017 at the end of U.N-monttored dis- armarennt Ascording to the National Cen- ter for Historical Memory 2013 report, pre- sented to the Colombian government as part of the peace process to end the fight- ing, some 220,000 Colombians died the armed conflict thruugh 2012, 81%t of them civilians. The report also provided statis- ties quantilying the scale of the conflict, which has taken huge toll Colombian society; more than 23. .00 selective assas- sinations between 1981 and 2012: inter- nal displacement of more than 5 million Colombians due to land seizure and vio- lence; 27,00 77,000 kidnappings between 1970 and 2010; and 11, 1,0000 deaths amputees from anti persoumel land mines laid primar- ily by Colombia's 's main insurgeut guerrilla group. the Revolutionary Arned Forces Colombia (FARC).
Pacilic Alliance, trade bloc that seeks to deepen trade intogration and investment with partner countries and reduce bar- riers to hilateral trade hetween the four countries. The Pacific Alliance aims to create a common market, allowing for the movement of businesses and people, and to serve as an export plattorm to the Asin- Pacific region. Colombia's leader- slip role in the Pacific Alliance and its ac- Organization for Economic Conperation and Development (OECD) May 2018, following review of its macroecomomic policies, are new devel- opeentts. The record expansion of Colom-
cession
report
buge toll seloctive assassinations between 2012 12, internal displacement of more than S million Colombians due to land scizure and violence: 27,000 kidhappiags between 1970 and 2010; and 11,000 deaths amputees from anti personmel land mines laid primur- ily by Colombia's 's main insurgeat guerrilla group, the Revolutionary Armed Forees of Colombia (FARC).
bia's coco crop and inereasing cocaine ex- ports to the United States, however, may significantly hinder elfort consoli- date peace in Colombia and could poten- tially increase coruuptio and extortion. A significatt portion of the Colombian public remains skeptical of the peace pro-
and
and the FARC is mol in Colombia's 's democracy. Other
The Navy's proposed FY2019 bad- gel requests lotal $2.347 million (i.e. about $2.3 billion) in pro- curement funding for the CVN.78 class shipbuikding program. Congress's decisions on the CVN 78 class program and other Navy shipbuilding programs
The Navy's to perform the test on the Navy's to perform the test on the Navy's ta perform the test the Navy's 'ss perform the Navy's 's to perform the Navy's 's to per- form the Navy's 's to perform the Navy's 's to
The Navy's propased FY2019 budget re- quests lotal $2.347 million (i.e.. about $2.3 billion) in procurement funding for the CVN-78 program. Congress's 's decisions on the CVN-78 program could substantially af- feet Navy capabilities and funding require-
perform the Navy's to perform the Navy's 's to perform the Navy's 's to perform the Navy's 's to perform
may have significant
substantially
ments and the shiphuilding industrial hase.
Navy's to performm the Navy's 's to perform the Navy's 's to per- form the Navy's 's to perform perform the Navy's to perform the Navy's 's perform Navy's 's perform Navy's 's to perform the Navy's as to perform the Navy's perform the Navy's to per- form the Navy's 's to perform the Navy's 's to perform the Navy's perform the Navy's
The Navy's 's's FY2020 hudgel submission also proposed tn not fund the mid-life nuclear refueling overhaul (called Refueling Com- plex Overhaul, or RCOH) For the aireraft carrier CVN-75 Harry S. Truman), and instead retire the ship around FY2024 and also denetivate one of the Navy's carrier air wings at about the same time. On April 30, 2019, however, the Administration an- nounced
effects on the Navy's capahilities and funding requirements and shiphuilding industrial base. The Navy's 's FY2020 budget submission also proposed not fund the mid-life nuclear refueling overhaul (called Refueling Complex Overhaul, RCOH), for the aircraft carrier CVN-75 (Harry S. Truman), and instead retire the ship around FY2024 and also deactivate one of the Navy's 's currier air wings about the same time. On April 30, 2019, however, the Administration announced that it was effectively withdrawing this proposal from the Navy's PY2020 budget submission. The Administation now supports funding the CVN-75 RCOH and keeping CVN-75 (and by implication associated wing) in service pusi FY2024. For additional this withdrawn budget proposal, see Appendix A.
Navy's 's to
to perform the Navy's 's to perform the Navy's perform the Navy's 's to perform the Navy's is to perform the Navy's 's to per- form the Navy's 's to perform the Navy's 's to perform the Navy's to perform the Navy's to perform the Navy's to the Navy's 's new aireraft carrier. USS Gerald is scheduled to be delivered to the Navy 2017. The Navy has been working to complete the ship's 's initial sea trials, which are scbeduled begin May 2017.
was ellectively withdrawing this proposal from the Navy's s FY2020 bud- get submission. The Administration now supports funding the CVN-75 RCOH and keeping CVN-75 (and by implication sociated air wing) in service past FY2024. For additional discussion of this withdrawn budgel propsaa, see Appendix A.
Ford,
discussion
27


Reviews:
[{"review_id": "rMW9M05FO7", "reviewer": "Reviewer_68Wb", "summary": "This paper deals with long contexts in LLM. Essentially, the proposed AdmTree incrementally builds a hierarchical summary as a binary tree of gists to leverage it for the prediction of the next word in a framework of causal language model. As described in Section 3.1, it dynamically segments input context, assigns gist tokens summarize them, and builds a tree structure to capture a long context. It consists of three stages, namely dynamic gist token constuction, tree construction, and tree-based compression for prediction while freezing original LLM.\nExtensive experiments show that the proposed AdmTree has much better accuracy for prediction, and the model is further analyzed from various viewpoints to show how the AdmTree actually works.", "questions": "- As described in Section 3.3, AdmTree utilizes self-attention from subsequent part of texts, mitigating the unidirectional limitation of causal LLM. This is conceptually good, but I wonder it might influence the causal structure as a probabilistic model, namely it could be somewhat cheating for predicting the next word. Is this true or not?\n- Semantic density of text segment is measured by Equation (2) using a combination of perplexity and entropy. However, perplexity and entropy are actually the same thing, just differing about some transformation. What is occurring here?\n- This is not a question, but it is a bit pity that the hierarchically organized gists are just flattened for  inherently flat attention mechanism of current LLM. If LLM itself utilizes this kind of hierarchical structure for prediction, depth-first flattening might be no longer necessary.", "limitations": "Nothing.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "This is a carefully built, and scientifically interesting model for intrinsic summarization for LLM objective. Hiearchical organization of gists is shown to be a crucial component for this model, and semantic heterogeneity of a text stream is also considered decently.\nTechnically speaking, I am not perfectly sure that the proposed model is completely causal (as described below), but this is ia minor problem and this paper is clearly worth being published.", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for the detailed and thoughtful review, as well as the recognition of our work. We address your comments and questions below:\n\n-----\n\n**Q1. On causal structure and use of self-attention from subsequent segments**\n\n> *“AdmTree utilizes self-attention from subsequent part of texts, mitigating the unidirectional limitation of causal LLM... it could be somewhat cheating for predicting the next word.”*\n\n**Response:**\n\n Thank you for pointing out this subtle but important concern. To clarify, AdmTree preserves the causal structure required for next-token prediction. The self-attention involved in tree aggregation (Section 3.3) operates **outside the autoregressive decoding path**. It only summarizes the **already-seen** gist tokens to build semantic tree. During next-token prediction in segment $s_k$, the model only attends to tokens from $s_k$ (autoregressively, via causal masking), and the semantic tree $T_{<k}$, constructed solely from previous sub-segments. Thus, no future tokens beyond position $j$ (the predicted token) are accessed or leaked into the prediction path.\n\n-----\n\n**Q2. On the use of both perplexity and entropy in Equation (2)**\n\n> *“Perplexity and entropy are actually the same thing, just differing about some transformation.”*\n\n**Response:**\n\nPerplexity and entropy would be mathematically equivalent only when entropy were computed as the cross-entropy loss on the target token. But in our framework, they serve different roles. For segment $X_i = [x_1, \\dots, x_n]$, PPL measures the model's sequence-level prediction difficulty, which is defined as: $\\text{PPL}(X_i) = \\exp\\left(-\\frac{1}{n} \\sum_{t=1}^{n} \\log P(x_t \\mid x_{<t})\\right)$\nThis reflects how “surprised” the model is by the segment. In contrast, Entropy in our design is the average token-level entropy, which is calculated as: $\\text{Entropy}(X_i) = \\frac{1}{n} \\sum_{t=1}^{n} \\left(-\\sum_{v \\in \\mathcal{V}} P(v \\mid x_{<t}) \\log P(v \\mid x_{<t})\\right)$\nwhere $\\mathcal{V}$ is the LLM vocabulary. This captures the intrinsic contextual complexity at each position. High PPL may arise from genuine surprising content or from low-value noise. The entropy term helps suppress segments that are surprising but semantically flat, avoiding over-allocation of gist tokens to noise. Empirically, our entropy-adjusted PPL shows better alignment with actual information density, especially in heterogeneous contexts. \n\nWe also conducted comparative experiments showing the superiority of our entropy‑adjusted PPL metric:\n\n| Metric                      | Single-Doc QA | Multi-Doc QA | Summarization | Few-Shot | Code |\n| :-------------------------- | :------------ | :----------- | :------------ | :------- | :--- |\n| TF-IDF                      | 35.5          | 35.8         | 25.9          | 64.0     | 59.4 |\n| PPL                         | 35.9          | 35.7         | 26.2          | 65.0     | 59.8 |\n| Entropy                     | 35.8          | 35.2         | 25.9          | 64.1     | 59.7 |\n| Attention Score             | 36.3          | 35.9         | 26.5          | 64.7     | 60.1 |\n| entropy-adjusted PPL (Ours) | 36.5          | 36.3         | 26.9          | 65.5     | 60.9 |\n\n----\n\n**Q3. On flattening the hierarchical structure before feeding into flat LLMs**\n\n**Response:**\n\n> *“It is a bit pity that the hierarchically organized gists are just flattened... If LLM itself utilizes hierarchical structure, flattening might be unnecessary.”*\n\nThis is an insightful comment and aligns with our future vision! We currently flatten the hierarchical gist tree to maintain compatibility with Transformer-based architectures, but this is a practical constraint, not a conceptual necessity.\n\nHowever, emerging architectures like **Hierarchical Reasoning Model** [1], which employs separate recurrent modules for planning and execution while using very few parameters, perform deep hierarchical reasoning without any flattening and still achieve state-of-the-art results. Similarly, recent research into **diffusion-based reasoning**, such as Diffusion-of-Thought (DoT) [2], allows reasoning steps to form non-linear, chain-of-thought-like trajectories during diffusion rather than a strict left‑to‑right generation. \n\nTogether, these models illustrate that hierarchical representations can be processed natively. In future work, AdmTree’s semantic tree could serve as a compression front-end feeding into these hierarchy-native backbones, thereby eliminating the need for flattening and enabling more natural hierarchical reasoning. We will add this forward-looking discussion in our conclusion section.\n\n*[1] Hierarchical Reasoning Model. https://github.com/sapientinc/HRM*\n\n*[2] Diffusion of thought: Chain-of-thought reasoning in diffusion language models. NIPS'24.* \n\n----\n\nThank you for your valuable insights and kind recognition. Your feedback has inspired us and made our work better!"}, {"review_id": "GYPSI5gO5R", "reviewer": "Reviewer_FQ4D", "summary": "AdmTree introduces an adaptive, hierarchical compression framework that first segments long input according to information-density, inserts learned “gist” tokens, and then builds a binary semantic tree whose nodes are aggregated with a single-layer self-attention module. Extensive experiments demonstrate the effectiveness of the proposed method.", "questions": "See Weaknesses.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper is well written and easy to understand.\n2. The figures are well plotted to help understand the workload of AdmTree.\n3. The proposed compression method looks interesting to me.\n\nWeaknesses:\n1. The adaptive token budget is decided solely by an entropy-weighted perplexity score. Have the authors profiled alternative signals—e.g., TF-IDF, mutual information, or a lightweight learned ranker—to confirm the choice is not dataset-specific? An ablation study is required to reveal whether performance gains stem from the tree itself or from the hand-picked heuristic.\n\n2. All internal nodes are compressed by a single self-attention layer followed by pooling. When the hierarchy reaches dozens of levels, can such a shallow module still propagate fine-grained cues?\n\n3. Results cover two 7B chat models on LongBench-style tasks. How does AdmTree fare on code or math benchmarks, or on other model sizes? More experiments on diverse model sizes and domains would demonstrate the robustness and generalization abilities.", "was_revised": true, "final_justification": "The rebuttal satisfactorily addresses my concerns. I look forward to seeing the additional experimental results incorporated into the final version and maintain my recommendation for acceptance.", "rebuttal": "We sincerely appreciate your recognition of our AdmTree's design and the inspiration your comments brought us. Below, we address the concerns you raised in detail:\n\n------\n\n**Q1. “The adaptive token budget is decided solely by an entropy-weighted perplexity score. Have the authors considered alternative signals like TF-IDF, mutual information, or a learned ranker?”**\n\n**Response:**\n\n In our early experiments, we explored several alternative scoring strategies, including attention-based scores and pure perplexity-based ranking, but found them less effective and stable. We have now included additional results comparing different scores:\n\n| Metric                      | Single-Doc QA | Multi-Doc QA | Summarization | Few-Shot | Code |\n| :-------------------------- | :------------ | :----------- | :------------ | :------- | :--- |\n| TF-IDF                      | 35.5          | 35.8         | 25.9          | 64.0     | 59.4 |\n| PPL                         | 35.9          | 35.7         | 26.2          | 65.0     | 59.8 |\n| Entropy                     | 35.8          | 35.2         | 25.9          | 64.1     | 59.7 |\n| Attention Score             | 36.3          | 35.9         | 26.5          | 64.7     | 60.1 |\n| entropy-adjusted PPL (Ours) | 36.5          | 36.3         | 26.9          | 65.5     | 60.9 |\n\nEmpirically, our entropy-adjusted PPL shows better alignment with actual information density, since it simultaneously accounts for the difficulty of the sentences and the contextual uncertainty.\n\nBesides, as you suggested, designing a lightweight learnable scorer is indeed a promising direction. For instance, such a scorer could jointly consider the input and the LLM’s own capacity, enabling it to dynamically assign scores that reflect how different LLMs truely perceive the information density of the same sentence.\n\n-----\n\n**Q2. “All internal nodes are compressed by a single self-attention layer followed by pooling. Can such a shallow module still propagate fine-grained cues in deep hierarchies?”**\n\n**Response:**\n\n The primary function of the aggregator is to propagate semantic information from child nodes in a bottom-up manner. Since it does not need to maintain its own semantic representation, we opted for a lightweight single-layer Transformer to ensure training efficiency.\n\nTo further investigate how aggregator depth affects performance at different tree depths, we compared 1-layer and 3-layer aggregators under varying compression ratios (i.e., different tree depths), using Qwen-2-7B as the base LLM. We selected QA and summarization tasks from LongBench, as they require semantic of different levels. LongBench allows inputs up to 32K tokens, so under 2× compression, tree depths can easily exceed 10 layers. The experimental results are as follows:\n\n|                     | 2× Compression Ratio |               | 8× Compression Ratio |               |\n| ------------------- | -------------------- | ------------- | -------------------- | ------------- |\n|                     | Single Doc QA        | Summarization | Single Doc QA        | Summarization |\n| 1-layer aggregators | 42.8                 | 30.5          | 40.8                 | 29.8          |\n| 3-layer aggregators | 43.2                 | 31.3          | 41.3                 | 30.9          |\n\nWe observe that deeper aggregators do indeed improve performance, especially in summarization tasks where higher-level semantics are more critical. This suggests that **the current aggregation design still has room for improvement**. Unlike our current shared lightweight aggregators, future work could explore **depth-aware or compression-aware aggregators**. For example, under high compression ratios, leaf nodes must carry more semantic load, whereas in low-compression ratios, deeper trees shift the burden upward, making it essential for aggregators to appropriately balance fine-grained and global semantics.\n\n-----\n\n**Q3. “Results cover two 7B chat models. How does AdmTree perform on code/math tasks or other model sizes?”**\n\n**Response:**\n\nThe LongBench benchmark already includes a Code Completion task. In addition, we provided results on LongBench v2 and GSM8K in Tables 4 and 5 of the appendix. LongBench v2 requires long-context reasoning, while GSM8K is a widely used benchmark for mathematical reasoning.\n\nTo further demonstrate generalization across model sizes, we extended our experiments to include larger LLMs such as Qwen‑2.5‑7B and Qwen‑2.5‑14B. The results are shown below:\n\n| Method                   | Single-Doc QA | Multi-Doc QA | Summarization | Few-Shot | Code |\n| :----------------------- | :------------ | :----------- | :------------ | :------- | :--- |\n| Qwen-2.5-7B              | 41.9          | 45.2         | 26.5          | 69.1     | 64.9 |\n| Qwen-2.5-7B-**FT**       | 42.7          | 46.1         | 26.7          | 67.6     | 66.3 |\n| Qwen-2.5-7B-**Beacon**   | 42.5          | 45.8         | 26.8          | 67.4     | 66.4 |\n| Qwen-2.5-7B-**AdmTree**  | 42.5          | 46.3         | 28.1          | 69.3     | 66.8 |\n| Qwen-2.5-14B             | 42.5          | 52.9         | 25.1          | 71.7     | 66.7 |\n| Qwen-2.5-14B-**FT**      | 43.9          | 50.5         | 27.1          | 68.8     | 67.1 |\n| Qwen-2.5-14B-**Beacon**  | 43.4          | 49.9         | 27.1          | 68.5     | 67.4 |\n| Qwen-2.5-14B-**AdmTree** | 43.5          | 50.8         | 27.5          | 69.7     | 68.1 |\n\nThese results confirm that AdmTree maintains strong performance on larger and more advanced models, consistently outperforming the state-of-the-art method Beacon and the strong baseline Original LLM-FT.\n\n----\n\nWe sincerely thank you again for your thoughtful suggestions and hope our clarifications address your concerns. We would greatly appreciate it if you would consider a higher score."}, {"review_id": "znxAsX8nVY", "reviewer": "Reviewer_eT7L", "summary": "This paper proposes AdmTree, an adaptive hierarchical compression framework for long-context processing in LLMs. By segmenting input based on information density and summarizing segments with gist tokens in a binary tree structure, AdmTree preserves both local details and global semantics. Extensive evaluations on synthetic and real-world tasks show that AdmTree outperforms existing methods in semantic retention, efficiency, and downstream task performance.", "questions": "1. What are the detailed configurations for the RAG experiments? Was the segment order preserved during retrieval?\n2. Why does the original LLM exhibit a speedup in Table 1 for LLaMA-2-7B?\n3. What compression ratio was used in the experiments presented in Figure 4?\n4. For the needle-in-a-haystack evaluations, what types of noise and needle content were used? Was the noise content repetitive?", "limitations": "The authors wrote in the checklists that the limitations are introduced in the conclusion section. However, the conclusion section does not include any limitations. If they were removed at the last minute, please provide them in the rebuttal to highlight any potential limitations.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n1. The paper is clearly written and easy to follow.\n2. The authors conduct a thorough ablation study, analyzing the contribution of each component that makes AdmTree effective.\n3. The proposed approach demonstrates strong empirical performance on both synthetic and real-world tasks.\n4. The paper introduces a novel hierarchical approach to manage gist tokens, which is both conceptually interesting and practically useful.\n\n**Weaknesses**\n1. The models used for evaluation (LLaMA-2 and Qwen-2) are relatively outdated compared to more recent LLMs.\n2. The RAG scores for QA tasks appear to be quite low, despite prior work [1][2] suggesting that RAG-based methods tend to perform well on QA benchmarks.\n3. Some aspects of the experimental setup require further clarification. Several examples are listed in the Questions section below.\n\n[1] Li et. al., Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach, EMNLP 2024 Industry Track\n\n[2] Yu et. al., In Defense of RAG in the Era of Long-Context Language Models, arxiv 2024", "was_revised": true, "final_justification": "The 'strength' section highlights the justification for recommending acceptance. The rebuttals provided further clarifications along with additional evaluations on more recent models, and all concerns are addressed.", "rebuttal": "Thank you for your thoughtful evaluation and detailed comments. We address each concern below:\n\n-----\n\n**Q1. “The models used for evaluation (LLaMA‑2 and Qwen‑2) are relatively outdated compared to more recent LLMs.”**\n\n **Response:**\n\n As explained in the paper, we use LLaMA‑2‑7B‑Chat because three out of the four compression baselines are based on it, ensuring fair comparisons.\nOur approach is model-agnostic and compatible with any Transformer-based LLM. To further validate its generality, we have extended our experiments to include more advanced and larger models, including Qwen‑2.5‑7B and Qwen‑2.5‑14B. The results are shown in the table below:\n\n| Method                   | Single-Doc QA | Multi-Doc QA | Summarization | Few-Shot | Code |\n| :----------------------- | :------------ | :----------- | :------------ | :------- | :--- |\n| Qwen-2.5-7B              | 41.9          | 45.2         | 26.5          | 69.1     | 64.9 |\n| Qwen-2.5-7B-**FT**       | 42.7          | 46.1         | 26.7          | 67.6     | 66.3 |\n| Qwen-2.5-7B-**Beacon**   | 42.5          | 45.8         | 26.8          | 67.4     | 66.4 |\n| Qwen-2.5-7B-**AdmTree**  | 42.5          | 46.3         | 28.1          | 69.3     | 66.8 |\n| Qwen-2.5-14B             | 42.5          | 52.9         | 25.1          | 71.7     | 66.7 |\n| Qwen-2.5-14B-**FT**      | 43.9          | 50.5         | 27.1          | 68.8     | 67.1 |\n| Qwen-2.5-14B-**Beacon**  | 43.4          | 49.9         | 27.1          | 68.5     | 67.4 |\n| Qwen-2.5-14B-**AdmTree** | 43.5          | 50.8         | 27.5          | 69.7     | 68.1 |\n\nThese results confirm that AdmTree continues to perform effectively on larger and more capable models, consistently outperforming the state-of-the-art method Beacon and the strong baseline Original LLM‑FT.\n\n------\n\n**Q2. “The RAG scores for QA tasks appear to be quite low.”**\n\n**Response:**\n\nThank you for highlighting the pertinent RAG literature. Our RAG setup follows the original LongBench protocol. We use a chunk size of 500 and retrieve the top‑k most query‑similar chunks to fill the model’s context window as fully as possible, without preserving the original chunk order.\n\nCompared to the paper you mentioned, the relatively low RAG scores in our work mainly stem from two factors. First, we use smaller open-source models, while the cited works rely on much larger and more capable closed-source LLMs such as Gemini‑1.5‑Pro and GPT‑4o. Second, we evaluate on LongBench, while those works use InfiniteBench. These two benchmarks differ substantially and cannot be directly compared. Besides, the original LongBench paper also observed cases where RAG negatively impacted performance.\n\nAs shown below, we further explored the effectiveness of the Order-Preserving (OP) RAG method from the paper [1] you referenced, testing it with various embedding models. \n\n|                          | Single-Doc QA | Multi-Doc QA | Summarization | Few-Shot | Code |\n| ------------------------ | ------------- | ------------ | ------------- | -------- | ---- |\n| **LLama-2-7B & vanilla RAG** |               |              |               |          |      |\n| BM25                     | 25.1          | 23.9         | 24.4          | 56.4     | 33.1 |\n| SBERT                    | 17.1          | 15.8         | 23.6          | 53.2     | 36.8 |\n| OpenAI                   | 28.3          | 16.4         | 16.9          | 23.7     | 50.3 |\n| **LLama-2-7B & OP RAG**      |               |              |               |          |      |\n| BM25                     | 22.6          | 21.2         | 15.5          | 55.2     | 30.2 |\n| SBERT                    | 20.3          | 18.9         | 21.2          | 53.6     | 29.1 |\n| SBERT                    | 32.1          | 19.2         | 15.3          | 21.8     | 48.3 |\n\nWhile OP‑RAG led to modest improvements in some cases, it still failed to outperform AdmTree. Additionally, we observed that RAG offers negative impact in summarization tasks, underscoring its limited generalization across task types. In contrast, AdmTree demonstrates stronger and more consistent generalization across a diverse set of scenarios.\n\n------\n\n**Q3. “What are the detailed configurations for the RAG experiments?”**\n\n**Response:**\n\nPlease refer to the response to Q2.\n\n------\n\n**Q4. “Why does the original LLM exhibit a speed-up in Table 1 for LLaMA‑2‑7B?”**\n\n**Response:**\n\nThe original LLaMA‑2‑7B model has a context window limit of 4096 tokens. Through fine-tuning (Original LLM‑FT), we extended the model’s window length to accommodate the full input from LongBench without truncation. As a result, the original model runs with shorter inputs and thus achieves significantly lower latency compared to the extended model, although its performance is naturally weaker.\n\n------\n\n**Q5. “What compression ratio was used in Figure 4?”**\n\n**Response:**\n\nIn Figure 4, for each context length, we selected the smallest compression ratio from the set {2, 4, 8, 16, 32} to ensure that (context_length / compression_ratio) < model_context_window. We will clarify this in Section 5.3 of the final version.\n\n------\n\n**Q6. “For the needle-in-a-haystack evaluations, what types of noise and needle content were used? Was the noise content repetitive?”**\n\n**Response:**\n\nWe followed the original Needle in a Haystack setup. The noise content comprises essays by Paul Graham, with no repetition. The needle is a synthetic question (to prevent real-world data leakage) about San Francisco.\n\n------\n\n**Q7. “Limitations were missing from conclusion.”**\n\n**Response:**\n\nThank you for catching this oversight. We will reintroduce the following limitations in the Conclusion section: (1) Our current experiments are focused on English long-text benchmarks, and the multilingual or cross-lingual capabilities of AdmTree remain unexplored. (2) AdmTree performs compression only on the input side. It does not currently address the compression of long outputs, which is increasingly important as LLMs produce longer intermediate reasoning chains.\n\n----\n\nWe sincerely thank you again for your constructive feedback. We hope that our responses have addressed your concerns and support a higher score!"}, {"review_id": "9mYEY5EhSM", "reviewer": "Reviewer_LLL7", "summary": "The paper proposes AdmTree, a tree-based context compression framework that enables large language models (LLMs) to handle long contexts efficiently. The key idea is to perform adaptive, hierarchical context compression by dynamically segmenting long input sequences using their information density and summarizing each segment using gist tokens. These tokens are organized as the leaf nodes of a binary semantic tree, and higher-level nodes are constructed through lightweight, bidirectional self-attention aggregation. This hierarchical structure makes AdmTree to preserve both local and global semantic information more effectively than flat or linear compression methods, while remaining efficient and scalable for long-context processing in LLMs. On the other hand, existing explicit or implicit methods often suffer from information loss, positional bias, or fixed-length chunking. Extensive experiments on LongBench and multi-turn dialogue tasks demonstrate that AdmTree achieves state-of-the-art performance across a wide range of tasks, outperforming prior methods by a significant margin while remaining scalable and effective under varying context lengths and compression ratios.", "questions": "1. The authors introduce separate projection matrices for gist tokens and regular text tokens in self-attention. Could the authors elaborate on the motivation behind this design choice? \n\n2. Have you considered combining AdmTree with KV cache compression (e.g., SnapKV) to further reduce memory/bandwidth usage during long-context inference, especially in autoregressive decoding scenarios? It seems that this integration could potentially offer complementary benefits, where AdmTree can reduce input length and KV compression further compresses hidden states in attention leading to a more comprehensive efficiency solution.\n\n3. In Equation 2, how the entropy is defined to a sequence of tokens X_i?\n\nTypos:\n- line 56, “then” -> “ten”\n- line 78, “the” -> “The”\n- line 208, “evlauate” -> “evaluate”\n- line 215, “method” -> “methods”", "limitations": "The proposed method requires the model training with standard next-token prediction loss, rather than a compression-specific objective (e.g., preserving downstream task performance or semantic similarity). A brief discussion of whether this limits compression quality, or whether auxiliary objectives were considered, would be helpful.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The paper proposes a novel approach of token compression that leverages a tree structure to effectively retain both local and global semantic information.\n\n2. AdmTree can dynamically compress token sequences based on local information density, allowing it to adaptively allocate compression effort and avoid both over- and under-compression.\n\n3. The method achieves strong empirical results, outperforming state-of-the-art compression and retrieval-based baselines on a variety of tasks, including question answering, summarization, and multi-document reasoning.\n\nWeaknesses:\n1. Although AdmTree keeps the backbone model frozen, it requires training additional parameters, i.e., aggregation function for building the semantic tree and the gist token attention mechanism. These additional parameters still introduce extra training complexity and computational overhead. This makes the method less plug-and-play than purely retrieval-based or hard-pruning approaches and may present a barrier for deployment in resource-constrained or latency-sensitive environments.\n\n2. AdmTree operates entirely on the input side, applying compression through token manipulation (gist token insertion and hierarchical structuring) without modifying the underlying model architecture. While this makes the approach model-agnostic and easy to integrate with any frozen LLM, the model itself is unaware of or unoptimized for the compression mechanism. As a result, there may be a mismatch between the compressed input structure and the model’s inductive biases, potentially limiting performance compared to approaches that jointly optimize compression and model behavior.", "was_revised": true, "final_justification": "The response have solved most of my concerns. In particular, combining with KV cache compression approach is promising and it would be great if this is discussed in the final draft.", "rebuttal": "We sincerely thank you for your thorough and insightful review. Below, we address your concerns point by point. Meanwhile, we would like to respectfully clarify some potential misunderstandings raised in the listed weaknesses:\n\n----\n\n**Q1. “Extra training complexity... makes the method less plug-and-play than purely retrieval-based or hard-pruning approaches.”**\n\n**Response:**\n\nWe think that **“train-free” and “plug-and-play” are not fully equivalent concepts**. Plug-and-play emphasizes the ability to integrate a module seamlessly into existing LLMs, without relying on their internal architecture or altering the original model. This is exactly what our AdmTree achieves, it can be adapted to any Transformer-based LLM without structural modifications.\n\nFurthermore, although AdmTree requires training, the cost is orders of magnitude smaller than training an LLM from scratch to handle long contexts. We believe this **offers a practical trade-off between efficiency and performance.**\n\nOn the other hand, while retrieval-based methods require no training, their performance is consistently inferior to our AdmTree. As suggested by Reviewer eT7L, we also explored more advanced RAG variants such as Order-Preserving (OP) RAG. As shown below, it still brings marginal improvements. Additionally, we observed that RAG can even have a negative impact on summarization and some other tasks, highlighting its limited generalization across task types. This issue has also been mentioned in recent GraphRAG [1], which aims to improve retrieval coverage for tasks that require global information.\n\n|                          | Single-Doc QA | Multi-Doc QA | Summarization | Few-Shot | Code |\n| ------------------------ | ------------- | ------------ | ------------- | -------- | ---- |\n| LLama-2-7B & vanilla RAG |               |              |               |          |      |\n| BM25                     | 25.1          | 23.9         | 24.4          | 56.4     | 33.1 |\n| SBERT                    | 17.1          | 15.8         | 23.6          | 53.2     | 36.8 |\n| OpenAI                   | 28.3          | 16.4         | 16.9          | 23.7     | 50.3 |\n| LLama-2-7B & OP RAG      |               |              |               |          |      |\n| BM25                     | 22.6          | 21.2         | 15.5          | 55.2     | 30.2 |\n| SBERT                    | 20.3          | 18.9         | 21.2          | 53.6     | 29.1 |\n| SBERT                    | 32.1          | 19.2         | 15.3          | 21.8     | 48.3 |\n\nAdditionally, training a powerful embedding model for RAG also incurs considerable cost, especially for nowadays LLM-based embedding models.\n\n*[1] From local to global: A graph rag approach to query-focused summarization. <https://arxiv.org/pdf/2404.16130?>*\n\n-----\n\n**Q2. “There may be a mismatch between the compressed input structure and the model’s inductive biases.”**\n\n**Response:**\n\nThank you for raising this concern. Our goal is to keep the original LLM’s semantic space intact, and instead align the semantic space of the newly introduced gist tokens to that of the frozen LLM. This ensures that AdmTree is model-agnostic, plug-and-play, and requires much less training data.\n\nIf the LLM’s parameters are activated and involved in training, it would require **an enormous scale of training data** to align the two semantic spaces while preserving the LLM’s original semantic understanding. This may contradict the fundamental goal of context compression and therefore is not a practical solution. Also, we previously tried training both the original LLM and the new parameters in preliminary experiments, but found that **it failed to converge.** \n\nIn fact, many representative prior works (e.g., Gisting, ICAE, Activation Beacon) similarly keeps the backbone frozen and trains only the newly introduced parameters. In the future, we will consider training strategies inspired by multimodal LLMs, using multi-stage training with more data and separating training of different modules at different stages. Thank you for your thoughtful concern.\n\n-----\n\n**Q3. “Why use separate projection matrices for gist tokens and regular text tokens in self-attention?”**\n\n**Response:**\n\nAs we mentioned in Q2, we want the AdmTree to gain compression ability without disrupting the LLM’s original semantic space. This design choice enhances plug-and-play applicability and significantly reduces alignment overhead.\n\n-----\n\n**Q4. “Have you considered combining AdmTree with KV cache compression (e.g., SnapKV)?”**\n\n**Response:**\n\nYes, we agree that combining AdmTree with KV cache compression methods is a promising step toward a more comprehensive and efficient solution.\n\nThis idea aligns well with the **tree-node retrieval strategy** proposed in Table 3 of our paper, which aims to reduce inference load. Based on your suggestion, we conducted an experiment by appending SnapKV (compression ratio 2×) to our method during decoding. The results are shown below:\n\n| Ablation                          | Single-Doc |\n| --------------------------------- | ---------- |\n| Full                              | 36.5       |\n| \\+ Tree Nodes Retrieval (Top 75%) | 32.8       |\n| \\+ Tree Nodes Retrieval (Top 50%) | 23.3       |\n| + SnapKV (compression ratio 2×)   | 20.5       |\n\nWhile some performance degradation is observed due to cumulative compression, the hybrid system still outperforms baselines such as AutoCompressor and ICAE from Table 1. We believe that building a unified framework that incorporates different types of compression is a valuable research direction, and we thank you for this inspiration.\n\n----\n\n**Q5. “In Equation 2, how is the entropy defined over a sequence of tokens X_i?”**\n\n**Response:**\n\nFor a sequence $X_i = [x_1, ..., x_n]$, the entropy is calculated as the average token-level entropy:\n\n$\\text{Entropy}(X_i) = \\frac{1}{n} \\sum_{t=1}^n \\left( -\\sum_{v \\in \\mathcal{V}} P(v \\mid x_{<t}) \\log P(v \\mid x_{<t}) \\right)$\n\nWe will clarify this in the revised version.\n\n-----\n\n**Q6. “The method uses standard next-token prediction loss rather than a compression-specific objective (e.g., preserving downstream task performance or semantic similarity).”**\n\n**Response:**\n\nWe employ the standard next-token prediction loss in both the pretraining and instruction-tuning stages. During instruction tuning, the task instruction (query) is not a target for prediction. Although the loss is not specifically designed for compression, it is still compression-aware, since the core goal is to train the attention branches of the gist tokens to compress the context semantic and align with the LLM’s original semantic space. **Notably, the instruction-tuning phase encourages the AdmTree to maintain the same output for the task, i.e., preserves downstream task performance.**\n\nWe fully agree with your suggestion on designing compression-specific objectives. For example, introducing a reconstruction or paraphrasing task, where the model learns to regenerate the original context based on gist tokens, could encourage more comprehensive semantic compression. We believe this is a promising direction and leave it for future exploration. Thank you again for your thoughtful suggestion.\n\n---\n\nTypos will also be corrected immediately. We hope our clarifications resolve your concerns and would greatly appreciate it if you would consider a higher score."}]

Author rebuttal:

Meta-review: ***Summary***

The authors introduce AdmTree, which is a mechanism for compressing the KV cache, and thus reducing the cost of attention in LLMs.  Building on prior published work, AdmTree first inserts *gist tokens* at equally spaced intervals in the input text.  The gist tokens attend to and summarize the input text.  Later tokens (in more distant segments), then attend to the gist tokens rather than attending directly over the entire sequence length.  

The key innovation of AdmTree is that instead of simply inserting gist tokens at regularly spaced intervals, it constructs a binary tree of gist tokens, where parents attend to children, and children attend to the input text.  The binary tree is then flattened to produce a KV-cache that other text segments can attend to.  In addition, AdmTree attempts to allocate gist tokens dynamically, by sorting segments according to a custom metric that includes average token entropy, so that more tokens are allocated to segments with greater information content.  

The authors compare against various baselines including RAG, and on various tasks and datasets, and show substantial improvements over SOTA.    

***Meta-review***

The reviews for this paper were mostly positive, and all in the range of "borderline accept": (4, 4, 4, 5).  The reviewers asked various clarifying questions during the discussion period, and the authors gave detailed and well thought-out responses.  The authors also ran additional experiments in response to the reviewer questions.  At the end of the discussion period, all of the reviewers stated that their concerns had been addressed.  

I have have gone over the paper myself, and I agree with the reviewers.  This paper lies somewhere between "borderline accept" and "accept".  To be honest, the basic idea is not very exciting in my opinion; it is an incremental improvement over prior work.  On the other hand, I can find no grounds to reject the paper either; the literature review is good, the experiments are solid and comprehensive, and my own questions were answered during the rebuttal period.

I am recommending "accept" mainly on the strength of the author rebuttal.  The authors worked hard to address all of the reviewer concerns, and the reviewers in turn stated that all of their concerns had been addressed.  Here's a summary of the discussion.  

**Reviewer 68Wb (rating 5)**:
* Voiced concern about whether the binary tree breaks causality.  (This was also a question that I had when reading the paper).  The authors gave a satisfactory explanation... causality is preserved.
* Asked about alternatives to entropy.  The authors provided a detailed table of alternatives that they tested.

**Reviewer FQ4D (rating 4)**: 
* Also asked about entropy, and received the same response as reviewer 68Wb.
* Wanted additional tests with bigger models on code and math benchmarks.  The authors ran additional experiments, as requested.

**Reviewer ET7L (rating 4)**:
* Wanted to know more about the RAG comparisons, which the authors answered.  The reviewer stated that their concerns were addressed.

**Reviewer LLL7 (rating 4)** writes:
* "While [the use of a frozen pre-trained LLM] makes the approach model-agnostic and easy to integrate with any LLM, the model itself is unaware of or unoptimized for the compression mechanism."  The authors gave a detailed discussion, and the reviewer was satisfied.
Decision: Accept (poster)