Title: AdmTree: Compressing Lengthy Context with Adaptive Semantic Trees
Venue: NeurIPS 2025
Accepted: yes
Keywords: Language Model

Context Compression
Abstract: The quadratic complexity of self-attention limits Large Language Models (LLMs) in processing long contexts, a capability vital for many advanced applications. Context compression aims to mitigate this computational barrier while preserving essential semantic information. However, existing methods often falter: explicit methods can sacrifice local detail, while implicit ones may exhibit positional biases, struggle with information degradation, or fail to capture long-range semantic dependencies. We introduce AdmTree, a novel framework for adaptive, hierarchical context compression designed with a core focus on maintaining high semantic fidelity while keep efficiency. AdmTree dynamically segments input based on information density, employing gist tokens to summarize variable-length segments as leaves in a semantic binary tree. This structure, combined with a lightweight aggregation mechanism and a frozen backbone LLM (minimizing new trainable parameters), enables efficient hierarchical abstraction of the context. By effectively preserving fine-grained details alongside global semantic coherence, mitigating position bias, and adapting dynamically to content, AdmTree comprehensively preserves the semantic information of lengthy context.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes AdmTree, a tree-based context compression framework that enables large language models (LLMs) to handle long contexts efficiently. The key idea is to perform adaptive, hierarchical context compression by dynamically segmenting long input sequences using their information density and summarizing each segment using gist tokens. These tokens are organized as the leaf nodes of a binary semantic tree, and higher-level nodes are constructed through lightweight, bidirectional self-attention aggregation. This hierarchical structure makes AdmTree to preserve both local and global semantic information more effectively than flat or linear compression methods, while remaining efficient and scalable for long-context processing in LLMs. On the other hand, existing explicit or implicit methods often suffer from information loss, positional bias, or fixed-length chunking. Extensive experiments on LongBench and multi-turn dialogue tasks demonstrate that AdmTree achieves state-of-the-art performance across a wide range of tasks, outperforming prior methods by a significant margin while remaining scalable and effective under varying context lengths and compression ratios.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper proposes a novel approach of token compression that leverages a tree structure to effectively retain both local and global semantic information.\n\n2. AdmTree can dynamically compress token sequences based on local information density, allowing it to adaptively allocate compression effort and avoid both over- and under-compression.\n\n3. The method achieves strong empirical results, outperforming state-of-the-art compression and retrieval-based baselines on a variety of tasks, including question answering, summarization, and multi-document reasoning.\n\nWeaknesses:\n1. Although AdmTree keeps the backbone model frozen, it requires training additional parameters, i.e., aggregation function for building the semantic tree and the gist token attention mechanism. These additional parameters still introduce extra training complexity and computational overhead. This makes the method less plug-and-play than purely retrieval-based or hard-pruning approaches and may present a barrier for deployment in resource-constrained or latency-sensitive environments.\n\n2. AdmTree operates entirely on the input side, applying compression through token manipulation (gist token insertion and hierarchical structuring) without modifying the underlying model architecture. While this makes the approach model-agnostic and easy to integrate with any frozen LLM, the model itself is unaware of or unoptimized for the compression mechanism. As a result, there may be a mismatch between the compressed input structure and the model’s inductive biases, potentially limiting performance compared to approaches that jointly optimize compression and model behavior.\n\nQUESTIONS:\n1. The authors introduce separate projection matrices for gist tokens and regular text tokens in self-attention. Could the authors elaborate on the motivation behind this design choice? \n\n2. Have you considered combining AdmTree with KV cache compression (e.g., SnapKV) to further reduce memory/bandwidth usage during long-context inference, especially in autoregressive decoding scenarios? It seems that this integration could potentially offer complementary benefits, where AdmTree can reduce input length and KV compression further compresses hidden states in attention leading to a more comprehensive efficiency solution.\n\n3. In Equation 2, how the entropy is defined to a sequence of tokens X_i?\n\nTypos:\n- line 56, “then” -> “ten”\n- line 78, “the” -> “The”\n- line 208, “evlauate” -> “evaluate”\n- line 215, “method” -> “methods”\n\nLIMITATIONS:\nThe proposed method requires the model training with standard next-token prediction loss, rather than a compression-specific objective (e.g., preserving downstream task performance or semantic similarity). A brief discussion of whether this limits compression quality, or whether auxiliary objectives were considered, would be helpful.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes AdmTree, an adaptive hierarchical compression framework for long-context processing in LLMs. By segmenting input based on information density and summarizing segments with gist tokens in a binary tree structure, AdmTree preserves both local details and global semantics. Extensive evaluations on synthetic and real-world tasks show that AdmTree outperforms existing methods in semantic retention, efficiency, and downstream task performance.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1. The paper is clearly written and easy to follow.\n2. The authors conduct a thorough ablation study, analyzing the contribution of each component that makes AdmTree effective.\n3. The proposed approach demonstrates strong empirical performance on both synthetic and real-world tasks.\n4. The paper introduces a novel hierarchical approach to manage gist tokens, which is both conceptually interesting and practically useful.\n\n**Weaknesses**\n1. The models used for evaluation (LLaMA-2 and Qwen-2) are relatively outdated compared to more recent LLMs.\n2. The RAG scores for QA tasks appear to be quite low, despite prior work [1][2] suggesting that RAG-based methods tend to perform well on QA benchmarks.\n3. Some aspects of the experimental setup require further clarification. Several examples are listed in the Questions section below.\n\n[1] Li et. al., Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach, EMNLP 2024 Industry Track\n\n[2] Yu et. al., In Defense of RAG in the Era of Long-Context Language Models, arxiv 2024\n\nQUESTIONS:\n1. What are the detailed configurations for the RAG experiments? Was the segment order preserved during retrieval?\n2. Why does the original LLM exhibit a speedup in Table 1 for LLaMA-2-7B?\n3. What compression ratio was used in the experiments presented in Figure 4?\n4. For the needle-in-a-haystack evaluations, what types of noise and needle content were used? Was the noise content repetitive?\n\nLIMITATIONS:\nThe authors wrote in the checklists that the limitations are introduced in the conclusion section. However, the conclusion section does not include any limitations. If they were removed at the last minute, please provide them in the rebuttal to highlight any potential limitations.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

comments: SUMMARY:
AdmTree introduces an adaptive, hierarchical compression framework that first segments long input according to information-density, inserts learned “gist” tokens, and then builds a binary semantic tree whose nodes are aggregated with a single-layer self-attention module. Extensive experiments demonstrate the effectiveness of the proposed method.

STRENGTHS AND WEAKNESSES:
Strengths:
1. The paper is well written and easy to understand.
2. The figures are well plotted to help understand the workload of AdmTree.
3. The proposed compression method looks interesting to me.

Weaknesses:
1. The adaptive token budget is decided solely by an entropy-weighted perplexity score. Have the authors profiled alternative signals—e.g., TF-IDF, mutual information, or a lightweight learned ranker—to confirm the choice is not dataset-specific? An ablation study is required to reveal whether performance gains stem from the tree itself or from the hand-picked heuristic.

2. All internal nodes are compressed by a single self-attention layer followed by pooling. When the hierarchy reaches dozens of levels, can such a shallow module still propagate fine-grained cues?

3. Results cover two 7B chat models on LongBench-style tasks. How does AdmTree fare on code or math benchmarks, or on other model sizes? More experiments on diverse model sizes and domains would demonstrate the robustness and generalization abilities.

QUESTIONS:
See Weaknesses.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper deals with long contexts in LLM. Essentially, the proposed AdmTree incrementally builds a hierarchical summary as a binary tree of gists to leverage it for the prediction of the next word in a framework of causal language model. As described in Section 3.1, it dynamically segments input context, assigns gist tokens summarize them, and builds a tree structure to capture a long context. It consists of three stages, namely dynamic gist token constuction, tree construction, and tree-based compression for prediction while freezing original LLM.\nExtensive experiments show that the proposed AdmTree has much better accuracy for prediction, and the model is further analyzed from various viewpoints to show how the AdmTree actually works.\n\nSTRENGTHS AND WEAKNESSES:\nThis is a carefully built, and scientifically interesting model for intrinsic summarization for LLM objective. Hiearchical organization of gists is shown to be a crucial component for this model, and semantic heterogeneity of a text stream is also considered decently.\nTechnically speaking, I am not perfectly sure that the proposed model is completely causal (as described below), but this is ia minor problem and this paper is clearly worth being published.\n\nQUESTIONS:\n- As described in Section 3.3, AdmTree utilizes self-attention from subsequent part of texts, mitigating the unidirectional limitation of causal LLM. This is conceptually good, but I wonder it might influence the causal structure as a probabilistic model, namely it could be somewhat cheating for predicting the next word. Is this true or not?\n- Semantic density of text segment is measured by Equation (2) using a combination of perplexity and entropy. However, perplexity and entropy are actually the same thing, just differing about some transformation. What is occurring here?\n- This is not a question, but it is a bit pity that the hierarchically organized gists are just flattened for  inherently flat attention mechanism of current LLM. If LLM itself utilizes this kind of hierarchical structure for prediction, depth-first flattening might be no longer necessary.\n\nLIMITATIONS:\nNothing.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-07"}