Title: Compress, Gather, and Recompute: REFORMing Long-Context Processing in Transformers
Venue: NeurIPS 2025
Accepted: yes
Keywords: Large Language Models

Long Context

Inference Efficiency
Abstract: As large language models increasingly gain popularity in real-world applications, processing extremely long contexts, often exceeding the model’s pre-trained context limits, has emerged as a critical challenge. While existing approaches to efficient long-context processing show promise, recurrent compression-based methods struggle with information preservation, whereas random access approaches require substantial memory resources. We introduce REFORM, a novel inference framework that efficiently handles long contexts through a two-phase approach. First, it incrementally processes input chunks while maintaining a compressed KV cache, constructs cross-layer context embeddings, and utilizes early exit strategy for improved efficiency. Second, it identifies and gathers essential tokens via similarity matching and selectively recomputes the KV cache. Compared to baselines, REFORM achieves over 50% and 27% performance gains on RULER and BABILong respectively at 1M context length. It also outperforms baselines on ∞-Bench, RepoEval, and MM-NIAH, demonstrating flexibility across diverse tasks and domains. Additionally, REFORM reduces inference time by 30% and peak memory usage by 5%, achieving both efficiency and superior performance.


Peer reviews:
comments: SUMMARY:
The paper studies architectural designs (without training) to improve model's memorization capability. Building on existing works, the paper proposes REFORM, which performs KV cache compression, extracting QKV states and recompute essential tokens on demand. The authors test their method on various benchmarks where the method shows significant improvement compared to not only existing methods but also competitive technology such as RAG. The ablation studies ablate on the state choice, compression choice as well as kernel size to select most similar tokens.

STRENGTHS AND WEAKNESSES:
Strength: The paper cites relevant works from both recurrent context processing and random access approaches which situate well the current's work contribution. The paper evaluates on many benchmarks with two open source LLMs which makes the contribution convincing. 

Weakness: Compare to state of the art compression techniques H2O, the paper seems to mainly contribute on better state choosing and pooling which makes the paper have relatively weak technical contributions.

QUESTIONS:
Compared to H2O, can the authors clarify what major contribution the current work brings please?

LIMITATIONS:
The paper mentioned transparently that directly using H2O as a limitation (although it is more like a performance bottleneck). I think the paper would benefit from a more deep and insightful limitation discussions.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-06-08

comments: SUMMARY:
This paper introduces REFORM, an inference framework to handle long contexts by combining ideas from two lines of long-context processing works -- those that process chunks of context to create "summary" KV caches, and those that compress the KV cache by only retaining relevant tokens. 
Experiments are on RULER and Babilong datasets, as well as infinity bench and repoEval.

STRENGTHS AND WEAKNESSES:
Strengths:
1. Interesting approach combining ideas from two parallel lines of work. 
2. The proposed approach shown improvement over the baselines reported in the paper. 

Weaknesses: 
1. Missing Baselines: the paper omits comparisons with strong baselines, such as SnapKV, DuoAttention, Minference, MoA, CEPE etc. Without comparisons with these baselines, it is hard to evaluate the strength of this approach. 

2. The paper does not include any discussion of complexity for the proposed approach and baselines. Furthermore, no implementation details are included. Is the method compatible with flash attention? REFORM uses H2O for token eviction, which I believe is incompatible. How does this affect latency. (related question: are the wall clock times in table 6b reported w/ or w/o flash attention?)

QUESTIONS:
1. The presentation of how attention heads are chosen is slightly confusing. 

2. What is the performance of vanilla attention? It is unclear what the drop in performance is to full attention. While I appreciate that this is probably not possible to compute for very long contexts, it would be useful to report this for the 64k context lengths.

 3. Comparison to missing baselines would be helpful to calibrate the strength of this method.

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-03

comments: SUMMARY:
The paper proposes REFORM, a two-phase approach for inferencing long-context LLMs. It first incrementally processes input chunks with a compressed kv cache with cross-layer context embedings and early exits strategy. It then gathers essential tokens and selectively recomputes KV cache. It improves results on several important benchmark such as RULER, inifinity-bench, RepoEval and MM-NIAH. For example, it achieves 52% and 34% performance gains on RULE and BABILong at 1M context length compared to baselines.

STRENGTHS AND WEAKNESSES:
Strength:
- The paper is well written and easy to follow: the motivation is clear, method is intuitive, and Figure 1 is illustrative of the proposed method.
- The problem is important, addressing the efficiency at extremely long (e.g. 1M) context length inference.
- The results are great, comprehensive (on multiple benchmarks). The ablation is throughout.

Weakness:
- Please consider adding larger models, e.g. 32B size.

QUESTIONS:
(1) Could the author adds larger models result (e.g. Qwen-32B)?
(2) Could the author discuss more on token eviction? Currently the reviewer understands it is from H2O. How would this affect the paper result?
(3) Can the author adds the similarity search result and MNR scores (Table 1 and Figure 2) on Llama8B as well, and discuss how different models could affect the method?
(4) In line 331, the author mentions using 129-token windows, and using 5-tokens would significantly degrades performance. Could the author adds more numbers on how this change with respect to the window size?

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nREFORM is a novel inference framework for efficiently handling extremely long contexts in large language models. It employs a two-phase approach: first, it processes input incrementally while compressing the KV cache, building cross-layer embeddings, and using early exits to enhance efficiency. Second, it selects essential tokens through similarity matching and selectively recomputes the KV cache. REFORM outperforms prior methods, achieving over 52% and 34% performance gains on RULER and BABILong at 1M context length. It also excels on Infini-Bench, RepoEval, and MM-NIAH benchmarks, while reducing inference time by 30% and peak memory usage by 5%, offering both speed and accuracy.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The compression and reconstruction method is pretty interesting. It shares the similar spirit of [1] for fixing linear attention forgetting problem. \n- The compress-gather-recompute pipeline is well-designed. It combines multiple technologies of H2O(compress), LayerSkip(early exit strategy), ShadowKV(cosine similarity), recomputation together to build a good sparse implementation. The combination is technically sound.\n- It evaluates on a wide range of models and the ablation experiments is well designed for understand the contribution of different components. \n- Performance improvement is significant and the implementation and setting is very practical. \n\nWeaknesses:\n- The evaluations are focused on retrieval tasks (including infinibench). It's understandable it will perform well since there's a query-based reconstruction phase. However, we suggest evaluation on more complicated tasks, such as GSM-Infinite[2], will make it more straightforward to see how the proposed method differs from RAG. (Now, this paper includes a comparison with RAG (8K budget), if only works well on retrieval tasks, considering that RAG does not even require a prefilling of LLM; therefore, it's not entirely convincing.)\n- Lack of discussion and comparisons of several important work in dynamic KV access, e.g. Quest[3], MagicPIG[4], and prefilling acceleration: LM-infinite. Especially a comparison of iteratively applying SnapKV (https://arxiv.org/abs/2404.14469) [Iteratively prefill chunks of tokens into LLM, each time we use SnapKV to select a subset of previous tokens]. These comparisons will enhance and clarify the contribution of the work.\n- The proposed method needs to reconstruct KV cache when a query comes. The reported re-computation budget is 8k-16k. Considering the settings in this paper, typically long-context and short output, this reconstruction overhead is significant. Thus, a more detailed efficiency analysis is necessary. For example, time for the first token, time per output token on different input lengths.\n- All Efficiency Analysis are generating 10 tokens conditioned on 256k/1M inputs, therefore most of time is spending on prefilling phase. However, with LLM development with reasoning models' outstanding performance, decoding phase would generate more tokens than 10 tokens. The performance of REFORM on decoding phase is unknown and unclear. A detailed time table with sperate prefill/decoding would help a lot.\n- The paper lacks theoretical analysis for why the selected attention heads perform well , and the compression-recomputation trade-offs. Can authors share the reasoning of selecting heads?\n\n\n[1] Simran Arora, Aman Timalsina, Aaryan Singhal, Benjamin Spector, Sabri Eyuboglu, Xinyi Zhao, Ashish Rao, Atri Rudra, Christopher Ré. Just read twice: closing the recall gap for recurrent language models. \n\n[2] Yang Zhou, Hongyi Liu, Zhuoming Chen, Yuandong Tian, Beidi Chen. GSM-Infinite: How Do Your LLMs Behave over Infinitely Increasing Context Length and Reasoning Complexity?\n\n[3] Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, Song Han. Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference.\n\n[4] Zhuoming Chen, Ranajoy Sadhukhan, Zihao Ye, Yang Zhou, Jianyu Zhang, Niklas Nolte, Yuandong Tian, Matthijs Douze, Leon Bottou, Zhihao Jia, Beidi Chen. MagicPIG: LSH Sampling for Efficient LLM Generation.\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-06"}