ReAttention: Training-Free Infinite Context with Finite Attention Scope

## Abstract

Abstract The long-context capability of the Large Language Models (LLM) has made significant breakthroughs, but the maximum supported context length remains a critical bottleneck limiting their practical applications. The constraint of context length in LLMs arises from the self-attention mechanism, which cannot effectively and efficiently capture the semantic relationships within infinitely long contexts via the limited pre-trained positional information and attention scope. In this work, we propose ReAttention , a training-free approach enabling LLM based on the self-attention mechanism to support an infinite context with a finite attention scope under sufficient memory resources. ReAttention performs the position-agnostic top- k 𝑘 k italic_k attention before the ordinary position-aware self-attention, freeing LLMs from the length extrapolation issue. We validate the performance of ReAttention on the LongBench, L-Eval, and InfiniteBench and demonstrate that it is on par with traditional methods. Furthermore, we also apply ReAttention on mainstream LLMs, including LLaMA3.1-8B and Mistral-v0.3-7B, enabling them to support context lengths of at least 1M and even expanding the context length of LLaMA3.2-3B-chat by 128 × \times × to 4M without any further training in Needle-In-A-Haystack tests. We also improve the efficiency of ReAttention with Triton and achieve an efficient extrapolation without additional overhead.

## 1 Introduction

Large Language Models (LLM) based on Transformer(Vaswani et al., ; OpenAI, ; Reid et al., ; Sun et al., ) has made great progress in the applications of Natural Language Processing (NLP). Particularly, in long context modeling, a considerable amount of research has been dedicated to extending the context length of LLMs(Chen et al., ; Peng et al., ; Liu et al., ; Xiong et al., ), increasing the maximum supported context length from the initial 2K(Touvron et al., ) to 2M(Ding et al., ) in open-source LLMs. However, the maximum context length still remains a bottleneck that limits the practical applications of LLMs(Press et al., ; Chen et al., ). To achieve infinite context length under sufficient memory for Transformer-based LLMs, the following three conditions must be satisfied:

- a
The position information in the inference phase should not be out-of-distribution (OOD) compared to the training phase;(Han et al., ; Liu et al., )
- b
The self-attention entropy in the inference phase should not increase with the length of the input;(Han et al., ; Xiao et al., )
- c
LLM should keep effectively aware of critical contextual information at each inference step.(Dong et al., ; Zhang et al., )

Early works in extrapolation focus on the first point, primarily by interpolating the position embedding(Chen et al., ; bloc97, ; ; Xiong et al., ) or limiting the relative position within the pre-training context length(Su, ; Jin et al., ; An et al., ). However, later studies have shown that even if positional information is not OOD, the attention entropy tends to increase logarithmically with the length of the self-attention window(Han et al., ). Specifically, the self-attention distribution becomes increasingly diffuse as inference length expands, making it difficult to effectively gather information from the context, which leads to unstable model outputs(Peng et al., ; Han et al., ). In response, methods like LM-Infinite(Han et al., ) and StreamingLLM(Xiao et al., ) have been proposed, which retain the starting and ending parts of the input, allowing LLMs to maintain stable outputs as input length increases. However, these approaches compromise the global awareness of contextual information, significantly affecting the performance on downstream tasks.

For human beings, although we have long-term memory, we only need a small amount of information to think and act in real-time. Similarly, while an LLM requires the full context to complete the inference process, it only needs limited contextual information at each inference step. This fact has inspired research like InfLLM(Xiao et al., ) and LongHeads(Lu et al., ) to perceive and extract critical information from the context before performing self-attention. However, LongHeads still face upper limits on positional information due to the use of NTK or PI for prefill(Lu et al., ), while InfLLM suffers from bias in representing contextual information, raised from the chunk-wise representation and ReRoPE-style position embedding when extracting key information.

In this work, we introduce ReAttention, a training-free method that extends LLMs with finite context lengths to process infinite contexts. ReAttention regards extracting critical contextual information as an additional attention process preceding traditional self-attention, akin to ”thinking before you act.” At each reasoning step, ReAttention selects the most critical finite segments from the KV cache without position information, concatenates them, applies positional embedding, and performs self-attention. By controlling the length of the selected KV cache segments, ReAttention can achieve infinite context length with a finite attention scope, while ensuring the position information and attention entropy are not out-of-distribution(Han et al., ). Furthermore, inspired by the optimization techniques used in self-attention, particularly FlashAttention (Dao et al., ; Dao, ), we use Triton (Tillet et al., ), a GPU programming language, to minimize read and write overheads in top-$k$ attention. With our custom Triton kernel, ReAttention avoids extra computational overhead and reduces memory usage for long contexts compared to traditional methods. Our contributions can be summarized as follows:

- •
We outline three fundamental conditions for infinite context extension in Transformer-based LLMs, position embedding not OOD, stable attention entropy, and effective awareness of critical contextual information. We also find that the third condition can be satisfied through the attention score without positional embedding.
- •
Based on this observation, we propose a training-free method, ReAttention, satisfying the aforementioned three conditions, thus extrapolating the context length of LLMs to infinity with a finite attention scope, and making LLMs free from the issue of length extrapolation.
- •
We validate that ReAttention matches the performance of self-attention in long-context scenarios with no computational overhead and less memory usage. Specifically, ReAttention extends the context length of leading LLMs, such as LLaMA3.1-8B (Dubey et al., ), to at least 1M tokens. For smaller models like LLaMA3.2-3B-chat (Meta, ), the context length can be increased by 128$\times$, reaching up to 4M tokens, without additional training.

## 2 Method

The overall structure of ReAttention is illustrated in Figure 1, which consists of a position-agnostic top-$k$ attention responsible for full-context cache selection and a traditional self-attention transform with position embedding. ReAttention achieves a training-free integration between the two components.

### 2.1 Full-Context Cache Selection

While LLMs require a complete long context to perform the entire inference process, only a limited context segment is needed at each inference step (Xiao et al., ; ; Lu et al., ). Considering the beginning and end of the input context correspond to globally important prompts and local information for inference (Xiao et al., ), ReAttention retains both the global and local segments of the KV cache for inclusion in the self-attention process.

$$ $\begin{gathered}\bm{\mathcal{K}}_{\text{cache}}=\left[\bm{\mathcal{K}}_{\text{ global}},\bm{\mathcal{K}}_{\text{middle}},\bm{\mathcal{K}}_{\text{local}} \right]\text{,\qquad}\bm{\mathcal{V}}_{\text{cache}}=\left[\bm{\mathcal{V}}_{ \text{global}},\bm{\mathcal{V}}_{\text{middle}},\bm{\mathcal{V}}_{\text{local} }\right]\text{.}\end{gathered}$ (1) $$

Figure: Figure 1: Overview of ReAttention.
Refer to caption: x1.png

Then, ReAttention uses the query vector of the current step to perform a top-$k$ selection on the middle part of the KV cache (Ribar et al., ), to identify the most important cache segments for the current step as shown in Figure 1.

$$ $\begin{gathered}\text{Indices}=\text{top-$k$}\left({\bm{q}_{t}\bm{\mathcal{K}} _{\text{middle}}^{T}}\right)\text{,}\\ \bm{\mathcal{K}}_{\text{select}}=\bm{\mathcal{K}}_{\text{middle}}[\text{ Indices}]\text{,}\[0.86108pt] \bm{\mathcal{V}}_{\text{select}}=\bm{\mathcal{V}}_{\text{middle}}[\text{ Indices}]\text{.}\end{gathered}$ (2) $$

ReAttention performs a full-context selection on the KV cache in each layer, allowing different layers to choose different KV caches for the calculation. Furthermore, since each attention layer has multiple attention heads and ReAttention adopts chunked streaming input during the prefilling stage, multiple query vectors may exist simultaneously. In this case, ReAttention votes based on the top-$k$ selections from different heads and query vectors to identify the top-$k^{\prime}$ KV caches. Additionally, to ensure semantic coherence, ReAttention not only retains the top-$k^{\prime}$ elements themselves but also extracts $m$ neighboring entries as a whole. Overlapping parts are deduplicated.

Unlike chunk-based selection in previous works (Lu et al., ; Xiao et al., ), ReAttention extracts critical contextual information with the dot-product between $\bm{q}_{t}$ and $\bm{\mathcal{K}}_{\text{cache}}$, rather than a chunk-wise representation of $\bm{\mathcal{K}}_{\text{cache}}$. This method offers strong adaptability and avoids the semantic fragmentation caused by fixed chunking (Luo et al., ). Moreover, since the intermediate result of $\bm{q}_{t}\bm{\mathcal{K}}_{\text{middle}}^{T}$ is too large for long-context scenarios, and would incur huge read and write overhead, we follow the approach of FlashAttention(Dao et al., ; Dao, ), fusing the whole process into one kernel using Triton(Tillet et al., ), as detailed in Section 3.4.

### 2.2 Training-Free Integration

ReAttention concatenates the selected KV cache segments between the global and local parts, applies positional embedding sequentially, and preserves the relative order while ignoring the absolute distance between the selected segments, as shown in Figure 1. Self-attention can then be applied to the concatenated KV cache.

$$ $\begin{gathered}\bm{\mathcal{K}}_{\text{cache'}}=\left[\bm{\mathcal{K}}_{\text {global}},\bm{\mathcal{K}}_{\text{select}},\bm{\mathcal{K}}_{\text{local}} \right]\text{,\qquad}\bm{\mathcal{V}}_{\text{cache'}}=\left[\bm{\mathcal{V}}_{ \text{global}},\bm{\mathcal{V}}_{\text{select}},\bm{V}_{\text{local}}\right] \text{,}\\ \tilde{\bm{q}}_{t},\tilde{\bm{\mathcal{K}}}_{\text{cache'}}=\text{PE}\left(\bm {q}_{t},\bm{\mathcal{K}}_{\text{cache'}}\right)\text{,\qquad}\bm{o}_{t}=\text{ SelfAttn}\left(\tilde{\bm{q}}_{t},\tilde{\bm{\mathcal{K}}}_{\text{cache'}},\bm {\mathcal{V}}_{\text{cache'}}\right)\text{.}\end{gathered}$ (3) $$

It’s important to note that, unlike the conventional implementation in Huggingface Transformers(Wolf et al., ), where the position embedding is applied before KV caching, in ReAttention, the position embedding is separated from the KV cache and performed after the KV cache selection. That is to say, the cached KV does not include the positional information. This design offers several advantages. On the one hand, as mentioned in Section 4, the attention score without positional embedding is more conducive to locating the key information in the context. On the other hand, since the concatenated cache length remains within the pre-training context length or the extrapolation upper bound, the position embedding will never be OOD (Han et al., ).

Furthermore, the unselected KV cache segments are unimportant in the current inference step, as their self-attention scores are minimal (Zhang et al., ). Therefore, this modification does not harm the self-attention output and can eliminates interference from irrelevant information(Zhang et al., ; Ge et al., ). Compared with InfLLM(Xiao et al., ), ReAttention’s position embedding and attention transform within the pre-training window do not introduce the untrained attention pattern, thereby preventing bias accumulation in the KV cache during the prefilling stage. Thus, ReAttention can disregard position information, achieve an unlimited attention context with a limited attention scope without any training, and remain compatible with the existing attention acceleration methods(Dao et al., ; Dao, ).

## 3 Experiment

### 3.1 Setup

We conduct experiments on LLaMA3-8B-8K(Meta, ), LLaMA3.1-8B-128K(Dubey et al., ), LLaMA3.1-70B-128K(Dubey et al., ), LLaMA3.2-3B-128K(Dubey et al., ), Mistral-v0.3-7B-32K(mistralai, ), InternLM2.5-7B-1M(InternLM, ), Qwen2-7B-128K(Yang et al., ), Qwen2-72B-128K(Yang et al., ), Qwen2-1B-32K(Yang et al., ). For all models, we set the length of $\bm{\mathcal{K}}_{\text{global}}$ to 32, the length of $\bm{\mathcal{K}}_{\text{local}}$ to 4096, and selected span size to 32. Moreover, we set $k=4,k^{\prime}=127$ in top-$k$ attention. Importantly, the attention scope in each step remains within the maximum attention window. For example, for the LLaMA3-8B-8K with ReAttention, the maximum attention scope size is 32 + 4096 + 127$\times$32, which exactly matches the maximum supported attention window of 8192. We use OpenCompass(Contributors, ) for validation. All experiments are performed with FP16 precision and accelerated with FlashAttention2(Dao, ).

### 3.2 Long-Context Benchmark Evaluation

Figure: (a) LongBench En. tasks
Refer to caption: x2.png

**Table 1: Results of existing mainstream LLMs, namely LLaMA Series(Meta, ; Dubey et al., ; Meta, ) on InfiniteBench(Zhang et al., ) in different context lengths. ”-” means LLM could not provide a stable output in a certain context length. ReAttention achieves superiority over InfLLM(Xiao et al., ) and full attention.**
|  | 32K | 64K | 128K | Avg. |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | MC | QA | Sum | MC | QA | Sum | MC | QA | Sum |  |
| LLaMA3-8B-Instruct-8K | - | - | - | - | - | - | - | - | - | - |
| + DynamicNTK | 50.66 | 7.44 | 21.76 | - | - | - | - | - | - | - |
| + InfLLM | 40.61 | 4.15 | 20.30 | 35.37 | 5.12 | 18.26 | 33.19 | 5.78 | 19.70 | 20.28 |
| + ReAttention | 46.29 | 5.33 | 21.76 | 44.54 | 5.25 | 20.15 | 49.78 | 5.98 | 20.36 | 24.38 |
| LLaMA3.1-8B-Instruct-128K | 37.12 | 18.85 | 1.60 | 31.88 | 23.32 | 1.39 | 21.83 | 25.43 | 1.40 | 18.09 |
| + InfLLM | 32.31 | 5.48 | 21.00 | 34.50 | 7.56 | 19.82 | 39.30 | 8.92 | 19.27 | 20.91 |
| + ReAttention | 36.68 | 12.71 | 19.68 | 35.37 | 12.14 | 18.84 | 40.61 | 12.63 | 18.14 | 22.98 |
| LLaMA3.2-3B-Instruct-128K | 14.41 | 15.65 | 1.38 | 11.79 | 18.16 | 1.27 | 13.10 | 17.63 | 1.30 | 10.52 |
| + InfLLM | 20.09 | 4.35 | 13.66 | 23.14 | 9.38 | 12.52 | 22.27 | 12.08 | 14.25 | 14.64 |
| + ReAttention | 19.21 | 11.80 | 16.75 | 17.47 | 11.82 | 15.06 | 20.09 | 11.91 | 14.13 | 15.36 |

We first evaluate all 9 LLMs on the commonly used long-context benchmark LongBench(Bai et al., ) and L-Eval(An et al., ), with a default context length of 32K and a middle truncation. For LLaMA3-8B-8K, which has a context length of less than 32K, we report its performance at its pre-trained context lengths of 8K and the results with Dynamic NTK(bloc97, ), uesd to extrapolate to the 32K context length. For the Dynamic NTK implementation, we use the default settings from the Huggingface Transformers(Wolf et al., ), setting the scaling factor to 4. Additionally, we compare the performance of all 9 LLMs with StreamingLLM(Xiao et al., ) as an ablation study, using the same global and local segment lengths as the ReAttention settings.

As shown in Figure 2 and detailed in Table 4, ReAttention outperforms StreamingLLM across all 9 models, indicating that the full-context selection acquires useful information for long-context inference. Furthermore, ReAttention performs on par with full attention and even surpasses it in some cases, such as LLaMA3.1-70B-128K(Meta, ) and Qwen2-1B-32K(Yang et al., ). This demonstrates that ReAttention can be applied to LLMs of various sizes and achieve performance close to full attention on downstream tasks(Jiang et al., ).

To further demonstrate the superiority and extrapolation capability of ReAttention, we validate our method on InfiniteBench(Zhang et al., ), a more challenging benchmark with a longer context length. We choose 3 commonly tested subtasks, En.MC, En.QA and En.Sum, evaluate models with varying context lengths, and compare ReAttention with DynamicNTK(bloc97, ) and InfLLM(Xiao et al., ) with the same selection config. The results(^1^11Due to the extensive evaluation for all 9 models, and considering that InfLLM has limited support for larger models such as LLaMA3.1-70B-chat and other series like InternLM2.5 and Qwen2, we only report results of the LLaMA series models at the 8B and 3B scales.) are shown in Table 1. Remarkably, ReAttention consistently outperforms full-attention and InfLLM at the 128K context length and in the average score. While DynamicNTK performs well at 32k, it encounters a clear upper bound on extrapolation, beyond which the model fails to produce stable outputs. Moreover, while InfLLM can extend context length indefinitely (Xiao et al., ), it still lags behind ReAttention in downstream tasks due to inaccurate extraction of critical information and differences in position embedding format compared to the pre-training phase.

Figure: (a) LLaMA3.1-8B-128K
Refer to caption: x8.png

Figure: (a) Multi NIAH of LLaMA3.1-8B-Instruct-128K
Refer to caption: x14.png

### 3.3 Needle-In-A-Haystack Evaluation

Building upon the long-context benchmark evaluation, we use models with strong retrieval capabilities within their training context lengths and conduct the Needle-In-A-Haystack (NIAH) evaluation (Contributors, ; ). We perform experiments on 8 A100 GPUs and extend the context lengths of LLMs with ReAttention to at least 1M tokens. As shown in Figure 3, LLMs with ReAttention maintain remarkably high retrieval accuracy across the entire range of context lengths they could support, regardless of their original attention windows. Importantly, we also extend the context of mainstream LLMs like LLaMA3-8B-8K(Meta, ) and Mistral-v0.3-7B-32K(mistralai, ) to at least 1M, providing the community with an effective solution for deploying long-context LLMs.

Building on this, we increase the difficulty of the evaluation by conducting experiments on multi-NIAH and single-NIAH in longer contexts, with results shown in Figure 4. Given the increased difficulty, we employ the corresponding instruct versions of the models. On the mainstream LLaMA3.1-8B-Instruct-128K(Dubey et al., ), we achieve a context length of 1M in the multi-NIAH task. For smaller models, such as LLaMA3.2-3B-chat-128K(Meta, ), we extend the context length to at least 2M. Notably, for Qwen2-1B-Instruct-32k(Yang et al., ), we extend the context length to 4M on 4 A100 GPUs, achieving a training-free context expansion of 128$\times$. To the best of our knowledge, it is the largest amplification of context length for LLMs achieved without additional training. This demonstrates that ReAttention can extend the context length while disregarding the position information, using a finite attention window to achieve an infinite attention context.

Figure: Figure 5: Overview of the kernel fusion in our customized top-$k$ attention kernel. The performance measurements reflect the execution time of the corresponding kernel functions, with the input length 8K for Llama3.1-8B inference tasks.
Refer to caption: x17.png

### 3.4 Efficiency Analysis

In the PyTorch framework, operators execute independently, requiring frequent I/O to the GPU memory, which introduces unnecessary overhead and latency. Inspired by FlashAttention(Dao et al., ; Dao, ), we develop a GPU kernel function for the top-$k$ attention described in Section 2.1 using Triton. As shown in Figure 5, our kernel fuses operators for attention score computation and top-$k$ calculation, enabling the entire process to run within the GPU cache. Thus GPU memory I/O is significantly reduced, improving both GPU memory usage and runtime. We added top-$k$ attention that reduces the time for the self-attention, and our kernel keeps top-$k$ attention’s overhead minimal, leaving overall latency unchanged.

We analyze our kernel’s efficiency in terms of GPU execution time and memory usage, showing that using Triton rather than PyTorch for the top-$k$ attention greatly enhances GPU performance while reducing GPU memory consumption. Additionally, we compare ReAttention with the standard implementation in the HuggingFace Transformers(Dai et al., ), measuring time to first token (TTFT) and memory cost in context lengths 32k to 256k. All experiments were conducted on a system with a 48-core CPU, 256GB RAM, and an A800-80GB GPU.

Figure: (a) GPU memory cost across different context lengths.
Refer to caption: x18.png

#### Triton Operator Efficiency

We tested our Triton operator using real inference inputs (e.g., $\mathcal{Q}$ and $\mathcal{K}$ matrices extracted from real-world inference tasks) to measure execution time and device memory usage. As shown in Figure 6(a), the standard PyTorch implementation exceeds 80GB of memory for sequences over 64k, while our Triton operator minimizes memory usage, limited to input and output matrices. It also achieves hundreds of times faster performance at 64k sequence lengths and scales efficiently as length increases (Figure 6(b)).

Figure: Figure 7: First token latency(FTL) and GPU memory consumption in the prefilling stage. FullAttn refers to the official HuggingFace Transformers implementation of Llama3.1-8B-Base, which runs out of memory after 192k.
Refer to caption: x20.png

**Table 2: Analysis of ReAttention hyper-parameters on LLaMA3-8B-8K and LongBench, by default chunk size 512, span size 32 as well as global size, local size 4096, and top-$k$ Selection.**
|  | En. | Zh. | Code | Avg. |
| --- | --- | --- | --- | --- |
| StreamingLLM | 29.48 | 13.22 | 70.27 | 32.49 |
| ReAttention | 33.42 | 18.74 | 70.38 | 35.03 |
| + chunk size = 1024 | 25.50 | 18.07 | 70.32 | 34.75 |
| + chunk size = 2048 | 25.39 | 18.07 | 70.30 | 34.72 |
| + span size = 8 | 13.02 | 8.74 | 58.54 | 22.96 |
| + span size = 16 | 14.13 | 9.43 | 60.92 | 24.29 |
| + span size = 64 | 25.27 | 17.98 | 70.83 | 34.69 |
| + span size = 128 | 25.23 | 16.97 | 70.54 | 34.57 |
| + top-$k$ = 1 | 25.04 | 16.57 | 70.75 | 34.45 |
| + top-$k$ = 8 | 25.40 | 17.51 | 70.64 | 34.70 |
| + local size = 2048 | 24.51 | 16.77 | 70.39 | 34.08 |
| + local size = 1024 | 22.31 | 13.64 | 69.48 | 32.23 |

#### Memory Usage in Prefilling Stage

Memory overhead limits the prefilling for longer sequences. Figure 7 shows that our method surpasses the standard implementation in HuggingFace Transformers as context length grows, continuing to work when the standard implementation runs out of memory. We also recorded first-token latency, which remains comparable to the standard implementation.

## 4 Discussion

### 4.1 Analysis on Hyper-parameter

We first discuss the selection of hyper-parameters. We evaluate the performance of the LLaMA3-8B-8K(Meta, ) on the LongBench benchmark(Bai et al., ), comparing hyper-parameters including chunk size, span size, local size, and top-$k$ values. Since the prefilling chunk is always contained within the local part in self-attention, the chunk size must be smaller than the local size. Additionally, to ensure fairness in our comparisons, we maintain a consistent maximum attention window size across different settings, specifically keeping the summation of global size, span size times $k’$, and local size equal to 8192. To achieve this, we set global size equal to span size, retaining the earliest portion as the first selection segment. As span size increased, we correspondingly reduced the $k’$ to limit the upper bound of cache selection. Additionally, when local size decreases, $k’$ is increased. The final results are shown in Table 2.

We find that as the chunk size decreases from 2048 to 512, the performance gradually improves. The prefilling stage becomes more precise in extracting critical information during each top-$k$ attention step. However, since prefilling takes more time as chunk size decreases, we select 512 as our default value without further reduction. Next, we examined the impact of different span sizes. Among these hyper-parameters, span size has the most significant effect on downstream performance. When span size is relatively small, fragmented segments weaken coherence and may mislead the LLM’s predictions, resulting in poorer performance compared to StreamingLLM. When the span size reaches 32, the performance on LongBench peaks. Beyond this point, due to the constraint of pre-trained position embedding, the number of extracted segments decreases, hindering the effective capture of critical contextual information. We also compare the influence of the top-$k$ number and find that top-4 yields the best results. A smaller top-$k$ number makes it difficult for LLM to identify critical information, while a larger top-$k$ number may introduce irrelevant content that interferes with judgment. Finally, we analyze the effect of local size. Smaller local size compromises LLM’s ability to maintain semantic coherence, and excessive selection beyond the local context may skew the KV cache during the prefilling stage. Therefore, we set the local size to half of the pre-training context length for LLaMA3-8B-8K and apply this hyper-parameter configuration to other LLMs.

### 4.2 Postion-Agnostic Cache Selection

Figure: (a) Attention distribution of a correct case with position embedding.
Refer to caption: x22.png

In ReAttention, the full-context cache selection is based on the dot product between the query and key vectors without position embedding. This may create a gap between the selection of the KV cache and the self-attention distribution. Therefore, here raises the question of whether using semantic vectors without position information can effectively locate the critical information in the context. To analyze this, we select a correct case and a wrong case from the results of Needle-In-A-Haystack evaluation(Contributors, ; ) on InternLM2-7B-200K(Cai et al., ) with full attention in 32K context length. For each case, we calculate both the attention distribution with position embedding, i.e., the real self-attention distribution, and the attention distribution without position embedding.

The result is shown in Figure 8. For the correct case, the attention distributions, both with and without position embedding, can locate the position of the ”needle” in the ”haystack”. However, the attention distribution with position embedding appears more dispersed, and this effect becomes more pronounced in the higher layers of the model. For the wrong case, the attention distribution with position embedding is also very dispersed and highly influenced by a particular attention head misled by irrelevant information from the lower layers. This ultimately results in the model’s failure.

Interestingly, when observing the attention distribution without position embedding, we find that the model can identify the position of the ”needle” using the inner product of semantic vectors and effectively filter out a large amount of irrelevant noise. Practically, the ReAttention-enhanced InternLM2-7B-200K can successfully locate the ”needle” in this case. Therefore, the inner product of semantic vectors in the cache selection is reasonable, and it proves to be more effective in finding the relevant context compared with position-aware inner products. Additionally, due to the absence of position embedding in the KV cache, ReAttention can achieve an effective context extension.

### 4.3 Observations on Synthetic Tasks

Figure: Table 3: The performance of Dynamic NTK, InfLLM, and ReAttention on RULER benchmark in 8K and 16K context length. S3 and MK3 are the short forms of NIAH-Single3 and NIAH-Multikey3 respectively.
Refer to caption: extracted/5903171/figures/tsne/mixed.jpg

The above analysis has demonstrated ReAttention’s good performance in various long-context scenarios, suggesting that it can extrapolate to infinite context lengths. However, these evaluations primarily focus on common natural contexts and lack currently widely-discussed benchmarks(Kuratov et al., ; Li et al., ), such as RULER(Hsieh et al., ). Unfortunately, ReAttention performs poorly on the RULER benchmark. In fact, the extrapolation methods not based on full attention, including ReAttention and InfLLM, fail to pass the RULER benchmark. For instance, using the LLaMA3-8B-8K(Meta, ), we compare Dynamic NTK extrapolation with InfLLM(Xiao et al., ) and ReAttention at 8K and 16K context lengths. The results are shown in Table 3. Only Dynamic NTK achieves effective extrapolation, while both ReAttention and InfLLM exhibit more pronounced performance degradation as the context length increases. Specifically, We focus on reporting the results of two subtasks from RULER: NIAH-Single3 and NIAH-MultiKey3(Hsieh et al., ). Both tasks involve extracting key information from a context that contains misleading items. The difference is that the context for Single3 consists of natural text, where the key information is a string of random alphanumeric characters, whereas MultiKey3 features a stack of key-value pairs composed of those strings. We find that while all three methods can succeed in the Single3 test, only full-attention-based Dynamic NTK can effectively handle MultiKey3.

To uncover the mechanisms behind the observation, we perform a t-SNE visualization(Van der Maaten & Hinton, ) of the K cache in LLaMA3-8B-8K using Dynamic NTK for both Single3 and MultiKey3, using the K cache in the last layer as an example. The results are shown in Figure 9. For Single3, which primarily consists of natural text, the reduced K cache exhibits a single manifold with a local cluster and an overall coherent feature as the input length increases. In contrast, for MultiKey3, which is dominated by chaotic text(Lv et al., ), the reduced K cache reveals multiple overlapping manifolds rather than an extension along a single curve. This results in the fact that during the K cache selection in MultiKey3, regardless of whether we use the same index encoding or direct sequential encoding for the selected segments, the subsequent self-attention fails to identify which manifold branch the segments belong to. Only by encoding all K caches according to their original position indices can LLM implicitly recognize(Chi et al., ; Kazemnejad et al., ) which manifold branch it resides on. This is why only full attention can effectively handle such tasks.

Nevertheless, the synthetic chaotic long texts are quite rare in practical scenarios(Lv et al., ). In the common natural long text benchmarks we have tested, ReAttention consistently demonstrates strong performance. Therefore, ReAttention still remains competitive for application in extending the context length of LLMs to infinity. We provide an in-depth discussion of the characteristics reflected in the QK sequence extension patterns between synthetic and natural texts in Appendix B.

## 5 Related Work

Length extrapolation is an important issue for LLMs(Press et al., ), namely training in a short context, and maintaining good performance in a longer context. The mainstream extrapolation research mainly focuses on adjusting the Rotary Position Embedding (RoPE)(Su et al., ). For example, Linear PI(Chen et al., ) first achieves the length extrapolation in LLMs by scaling the position indices to the pre-training range with little fine-tuning. The NTK method(bloc97, ; ; Peng et al., ) then adjusts the rotary base in RoPE(Su et al., ) to achieve plug-and-play length extrapolation. Subsequently, amplifying the rotary base and training on longer lengths has become the dominant approach for length extrapolation(Rozière et al., ; Xiong et al., ; Liu et al., ; Ding et al., ), but these methods all have an explicit extrapolation upper bound. In addition, ReRoPE(Su, ), Self-Extend(Jin et al., ), and ChunkLLaMA(An et al., ) also achieve plug-and-play extrapolation by limiting the relative position. However, all of the above methods are based on full attention, facing the problem of attention entropy soaring with input length, and thus fail to achieve infinite context length(Han et al., ; Wang et al., ).

In contrast, another line of research has tried to extend the context length of models through sparse attention. Considering the self-attention distribution tends to focus on the global and local context, StreamingLLM and LM-Infinite proposed the $\Lambda$-shaped attention window to achieve almost unlimited input length(Xiao et al., ; Han et al., ). However, since the input is accompanied by discarding the previous context, it still fails to extend the context length(Dong et al., ). Based on StreamingLLM, InfLLM(Xiao et al., ) and LongHeads(Lu et al., ) attempt to extend context through chunkwise retrieval from the middle cache, but the semantic fragmentation and block representation issues also affect downstream task performance(Luo et al., ). Recently, MInference(Jiang et al., ) and RetrievalAttentionLiu et al. () utilize a dynamic cache selection to achieve significant speedup, but not attempt context extrapolation. In contrast, we propose ReAttention, which regards the cache selection as the preceding attention before the ordinary self-attention and achieves an infinite context with a finite attention scope, making LLMs free from the challenge of length extrapolation. We also minimize the top-$k$ attention kernel-wise like FlashAttention in self-attention(Dao et al., ; Dao, ).

## 6 Conclusion

In this paper, we introduce ReAttention, using a finite attention window to realize an infinite context length in each inference step. We evaluate ReAttention is on par with full attention in performance with LongBench, L-Eval, and InfiniteBench. Furthermore, ReAttention has been shown to successfully extend the context of mainstream LLMs, including the LLaMA series and Mistral, to up to 1M tokens and even expand the context length of LLaMA3.2-3B-chat by 128$\times$ to 4M without any further training in Needle-In-A-Haystack tests. We also improve the efficiency of ReAttention with Triton and achieve an efficient extrapolation without additional overhead.

## Appendix A Detailed Evaluation Results

**Table 4: Results of existing mainstream LLMs, including LLaMA Series and Mistral on LongBench and L-Eval. ReAttention achieves a consistent superiority over StreamingLLM and shows comparable performance with LLMs with full attention.**
|  | LongBench | L-Eval |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
|  | En. | Zh. | Code | Avg. | Close-<br>Ended | Open-<br>Ended | Avg. |
| LLaMA3-8B-8K* | 33.91 | 18.98 | 69.80 | 35.25 | 46.58 | 2.66 | 17.30 |
| + DynamicNTK | 34.27 | 19.95 | 69.89 | 35.48 | 46.11 | 2.79 | 17.23 |
| + StreamingLLM | 29.48 | 13.22 | 70.27 | 32.49 | 36.52 | 1.36 | 13.08 |
| + ReAttention | 33.42 | 18.74 | 70.38 | 35.03 | 44.94 | 2.43 | 16.60 |
| LLaMA3.1-8B-128K | 36.38 | 28.64 | 70.63 | 38.79 | 45.30 | 0.50 | 15.44 |
| + StreamingLLM | 33.41 | 24.24 | 70.50 | 37.06 | 42.36 | 0.38 | 14.37 |
| + ReAttention | 36.14 | 28.56 | 70.48 | 38.63 | 45.25 | 0.50 | 15.42 |
| LLaMA3.1-70B-128K | 46.00 | 32.92 | 74.89 | 46.02 | 60.78 | 0.15 | 20.36 |
| + StreamingLLM | 41.07 | 29.60 | 75.06 | 43.20 | 55.32 | 0.16 | 18.54 |
| + ReAttention | 46.09 | 32.99 | 75.26 | 46.15 | 61.11 | 0.18 | 20.49 |
| LLaMA3.2-3B-128K | 48.03 | 21.84 | 55.73 | 40.76 | 55.75 | 20.63 | 32.33 |
| + StreamingLLM | 38.86 | 15.86 | 54.16 | 35.08 | 43.27 | 18.66 | 26.87 |
| + ReAttention | 46.16 | 19.45 | 55.17 | 39.43 | 53.09 | 20.12 | 31.11 |
| Mistral-v0.3-7B-32K | 35.74 | 18.59 | 65.13 | 35.61 | 41.97 | 4.33 | 16.88 |
| + StreamingLLM | 33.05 | 16.23 | 64.85 | 34.01 | 36.49 | 4.45 | 15.13 |
| + ReAttention | 35.45 | 18.14 | 65.32 | 35.48 | 41.95 | 4.09 | 16.71 |
| InternLM2.5-7B-1M | 55.49 | 41.46 | 52.62 | 47.27 | 62.96 | 24.99 | 37.64 |
| + StreamingLLM | 49.77 | 38.98 | 52.97 | 44.03 | 57.97 | 23.30 | 34.85 |
| + ReAttention | 55.60 | 41.58 | 52.72 | 47.35 | 63.47 | 24.36 | 37.40 |
| Qwen2-7B-128K | 43.47 | 31.23 | 67.65 | 41.34 | 53.43 | 15.41 | 28.08 |
| + StreamingLLM | 39.83 | 28.55 | 67.39 | 39.16 | 50.07 | 14.30 | 26.22 |
| + ReAttention | 42.97 | 30.92 | 67.56 | 41.01 | 53.18 | 15.35 | 27.96 |
| Qwen2-72B-128K | 61.94 | 41.85 | 71.15 | 54.71 | 68.17 | 25.44 | 39.68 |
| + StreamingLLM | 54.03 | 38.12 | 70.91 | 50.08 | 60.43 | 22.38 | 35.06 |
| + ReAttention | 61.53 | 42.00 | 71.39 | 54.53 | 67.31 | 25.34 | 39.33 |
| Qwen2-1B-32K | 34.65 | 23.65 | 57.98 | 33.21 | 41.28 | 15.59 | 24.16 |
| + StreamingLLM | 33.12 | 21.56 | 58.32 | 32.40 | 37.46 | 15.31 | 22.69 |
| + ReAttention | 34.94 | 24.12 | 58.58 | 33.51 | 41.00 | 16.10 | 24.40 |

Table 4 includes all the data previously shown in Figure 2, along with some additional data, providing a detailed comparison of various LLMs including LLaMA(Meta, ; Dubey et al., ; Meta, ), Mistral(mistralai, ), InternLM(Cai et al., ; InternLM, ), and Qwen(Yang et al., ) series across LongBench(Bai et al., ) and L-Eval(An et al., ).

## Appendix B Analysis of Cache Feature

### B.1 The Method for Obtaining t-SNE Results

We begin by extracting a lengthy sample (approximately 16K tokens) of text and feeding it into the LLaMA3-8B-8K(Meta, ). Subsequently, we capture the $\bm{\mathcal{Q}},\bm{\mathcal{K}}$ states at each layer, both before and after position embedding, allowing for a clearer understanding of the impact of position embedding on data representation. Next, the captured matrices are reshaped to (seqlen, nhead * headdim), and we apply t-SNE(Van der Maaten & Hinton, ) for dimension reduction, mapping the high-dimensional data into two dimensions with a final shape of (seqlen, 2).

### B.2 Cache Feature of Long Natural Language Texts

Figure: (a) t-SNE results of $\bm{\mathcal{Q}},\bm{\mathcal{K}}$ from a data sample from the PG19 training set in the final layer of LLaMA3-8B-8K
Refer to caption: extracted/5903171/figures/tsne/PG19_T_qk_layer31.png

Figure: (a) t-SNE results of $\bm{\mathcal{Q}},\bm{\mathcal{K}}$ from a data sample from LongBench GovReport in the 5^th layer of LLaMA3-8B-8K with NTK
Refer to caption: extracted/5903171/figures/tsne/ntk_gov_04_.jpg

The $\bm{\mathcal{Q}},\bm{\mathcal{K}}$ states of long natural language text preserves coherence, which can be visualized as a curve extending through high-dimensional space, encompassing the entire text. When the model extracts contextual information, it effectively accesses specific points along this line. The visualization in Figure 10 and Figure  11 reveals several key features:

- •
Local Semantic Similarity: Locally, the semantic meaning of individual text segments is preserved, forming small clusters.(Zandieh et al., )
- •
Global Coherence: These local clusters, however, are not isolated but part of a larger linear manifold representing the text’s global coherence. This coherence becomes more apparent after position embedding is applied.(Zandieh et al., )
- •
Out-of-Distribution Position Embedding: When the position embedding exceeds 8K, Llama3-8B struggles to extract semantic information, as seen in the diminished clustering.(Liu et al., ) Nonetheless, even when the position embedding is out-of-distribution(OOD), the text representation continues to align with a linear manifold.

### B.3 From Natural Language Texts to Chaotic Texts

In contrast, Ruler’s MultiKey3 subset contains unnaturally long texts that deviate from typical linguistic patterns, lacking the structure and semantic coherence of natural language. Here in this subset, the text length is artificially extended to 16k tokens, not by model generation but by repeating specific patterns

> One of the special magic UUIDs for A is: B.

like

> One of the special magic UUIDs for eab8d5be-b62c-49bf-a4f7-3a3017a15a60 is: 960e9c1a-b4c0-4d18-a5cd-7fd162649ce8.

Moreover, large portions of these patterns consist of non-linguistic, non-logical elements like UUIDs.

Figure: (a) t-SNE results of $\bm{\mathcal{Q}},\bm{\mathcal{K}}$ from a sample in RULER Multikey3 in the 15^th layer of LLaMA3-8B-8K with NTK
Refer to caption: extracted/5903171/figures/tsne/llamantk_gpu_qk_layer15.png

When applying t-SNE to reduce the dimension of MultiKey3 data(Figure 12), the linear manifold observed in natural language text vanishes, highlighting the impact of unnatural patterns on data representation. Key observations include:

- •
Local Semantic Similarity: By examining $\bm{\mathcal{Q}}$ after position embedding, we can clearly see that the UUIDs and natural language components are completely separated, even in the deep purple section at the start. In $\bm{\mathcal{K}}$ after position embedding, adjacent tokens (i.e., data points of similar colors) are dispersed across multiple lines, thus making the term ’local similarity’ questionable.
- •
Global Coherence: The assumption that the entire data representation lies on a linear manifold no longer holds. Unnaturally extended texts, unlike typical natural language, fail to maintain global structural dependencies.

The effectiveness of dynamic sparse methods in replacing the full attention mechanism hinges on the collaboration between global and local information. However, when text coherence is lacking, even adjacent tokens in the $\bm{\mathcal{Q}},\bm{\mathcal{K}}$ cache become dispersed, disrupting the required linear manifold structure that LLM relies on. This suggests that LLM focuses its top-$k$ attention on natural language segments and successfully extracts critical information. Yet, when the top-$k$ values are spread across multiple manifolds, LLM fails to utilize this information efficiently. This contrasts sharply with the typical feature of the $\bm{\mathcal{Q}},\bm{\mathcal{K}}$ cache in natural language, where the representation forms a continuous, intertwined structure across the text in high-dimensional space. It is precisely the absence of this characteristic feature that causes the cache generated during prefilling with ReAttention to be biased when processing RULER Multikey3, leading to the output of non-existent UUIDs.

## Peer Reviews

confidence: 5
contribution: 1
initial_rating: 5
presentation: 2
questions: 1.	The idea appears similar to token eviction methods like H2O and FastGen. How does ReAttention compare with these methods in terms of accuracy and efficiency?
soundness: 3
strengths: 1.	The method is intuitive. ReAttention maintains KV caches of tokens that are more frequently attended by the latest tokens, which reduces the memory overheads and remove redundant contextual segments from being attended. 
2.	The evaluation is comprehensive. Authors provide evaluation for many models and benchmarks. 
3.	The performance improvement is notable, compared to InfLLM and full attention.
summary: This paper proposes ReAttention, a a training-free approach enabling LLM based on the self-attention mechanism to support an infinite context with a finite attention scope under sufficient memory resources. ReAttention splits the tokens into three parts: initial tokens, middle tokens, and local tokens. It maintains the initial and local tokens (their KV caches), select a subset of middle tokens, and re-concatenates them with positional index calibration, and then conduct attention operation.
weaknesses: 1.	The two-round attention operation introduces non-trivial overheads, which may result in latency similar to full attention. 
2.	The efficiency in terms of latency and throughput can be the major drawbacks of this method. 
3.	The method requires sufficient memory resources where full attention is also feasible. The advantage over full attention lies in evicting irrelevant tokens, similar to token-dropping methods, which improves performance. However, efficiency in long-sequence tasks remains a concern. When full attention isn't feasible, the two-round attention calculation might be costly, as the first round still requires full-attention calculations. 
4.	In long-sequence generation, the query tokens can change dynamically. How does ReAttention handle middle-cache selection in this setting? Does it re-select middle caches periodically?

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: How does ReAttention perform on more structured tasks that require precise positional understanding (e.g., code completion, mathematical reasoning)?
soundness: 3
strengths: - The problem is highly relevant and timely, as context length remains a critical bottleneck for LLMs. The paper clearly articulates three key conditions for infinite context extension (lines 040-046).
- The proposed method is simple and training-free. The two-stage attention approach (position-agnostic selection followed by regular attention) is intuitive and well-motivated. 
- The empirical results are impressive. They demonstrate: (1) Comparable or better performance vs full attention across multiple benchmarks (Fig 2); (2) Successful extension of LLaMA3.2-3B context by 128x to 4M tokens (Section 3.3); (3) Strong performance on challenging tasks like LongBench, L-Eval, and InfiniteBench.
- The experimental evaluation is comprehensive, including multiple model architectures (LLaMA, Mistral, Qwen series) and various benchmarks (LongBench, L-Eval, InfiniteBench). It also includes detailed ablation studies on hyperparameters (Table 2) and thorough efficiency analysis with Triton optimization in Section 3.4.
summary: This paper proposes ReAttention, a training-free approach to extend the context window of Large Language Models (LLMs) through a position-agnostic top-k attention mechanism. The key idea is to perform cache selection before regular self-attention, treating the extraction of critical context info as an additional attention process. The method enables models to handle theoretically infinite contexts with a finite attention scope. The authors validate their approach on multiple benchmarks and models, showing strong performance and efficiency gains through custom Triton kernel optimization.
weaknesses: - The paper has several writing issues and typos that should be addressed: e.g., Llama vs LLaMA and the wrong quotation in line 067.
- The related work section omits several relevant recent papers on efficient attention and KV-cache optimization, such as:
SnapKV [1] for efficient cache management, PyramidKV [2,3] for hierarchical cache structures, GemFilter [4] for attention-filtering, and many so on. These works tackle similar challenges and a comparison would strengthen the paper. It would be good to discuss them. 

[1] Li, Yuhong, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. "Snapkv: Llm knows what you are looking for before generation." arXiv preprint arXiv:2404.14469 (2024).

[2] Zhang, Yichi, Bofei Gao, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Baobao Chang, Junjie Hu, and Wen Xiao. "PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling." arXiv preprint arXiv:2406.02069 (2024).

[3] Yang, Dongjie, XiaoDong Han, Yan Gao, Yao Hu, Shilin Zhang, and Hai Zhao. "PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference." arXiv preprint arXiv:2405.12532 (2024).

[4] Shi, Zhenmei, Yifei Ming, Xuan-Phi Nguyen, Yingyu Liang, and Shafiq Joty. "Discovering the gems in early layers: Accelerating long-context llms with 1000x input token reduction." arXiv preprint arXiv:2409.17422 (2024).

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: - *What is the performance of the baselines in the needle-in-a-haystack experiments.*
- *In Figure 7, is the FullAttn* implementation FlashAttention?
- *How exactly does ReAttention differ from InfLLM? Is the only difference that position embeddings are not applied during the lookup step?* (I recommend the authors update the paper to make this comparison more clear.)
- *Minor improvements:*
    - In Figures 3 and 4, the axes and color bars are too small to be legible.
soundness: 2
strengths: *The method is motivated by an interesting observation.* “*We also find that the third condition can be satisfied through the attention score without positional embedding.” This insight that attention scores without positional embeddings can effectively identify salient tokens is quite interesting and, to my knowledge, novel.  

*Method works on pre-trained models.* Unlike approaches like Mamba which require training models from scratch with new architectures, ReAttention can be applied directly to existing pre-trained models. This makes it much more practical for immediate deployment and use with current state-of-the-art models.

*Claim 1 seems to be well-supported by the results.*   The empirical evaluation in Figure 2 and Table 1  across multiple benchmarks (LongBench and InfiniteBench) and model architectures shows that ReAttention matches or outperforms regular attention.
- Note: in the weaknesses, I highlight some missing baselines.
summary: This paper proposes a training-free method (ReAttention) for improving the ability of pre-trained language models to process long context lengths. The method is motivated by the insight that attention scores without positional embeddings are still useful for identifying salient tokens.  

The main claims of the paper are that:

1. ReAttention matches or outperforms the performance of traditional self-attention in long-context scenarios. 
2. ReAttention matches or outperforms the performance of other training-free methods long-context methods (*e.g.* InfLLM). 
3. ReAttention extends the context length of existing LLMs (*e.g.* 8B and 3B llama) “to at least 1 million tokens”. 
4. ReAttention adds no computational overhead and less memory usage than regular self-attention.
weaknesses: *Baselines in the long context evals are missing (Claim 2). W*hy is StreamingLLM included in the LongBench results but not the InfiniteBench results? And why is InfLLM included in the InffiniteBench results but not the LongBench results?

*Baselines in million-scale context seem to be missing. (Claim 3)* For the needle in a haystack experiments supporting Claim 3, the paper should compare against other methods capable of handling very long contexts like InfLLM and other recent approaches. Without these comparisons, it's hard to assess the relative advantages of ReAttention at extreme context lengths.

*Missing token-throughput comparisons (Claim 3).* The paper only compares time to first token (*i.e.* the cost of prefill).  How does it fare against regular attention in terms of efficiency?

*Apparent tradeoff in memory consumption (Claim 4).* The authors state that ReAttention uses less memory than self-attention. However, based on Figure 7, it appears that this is only true after ~100k in context length. The authors should highlight this tradeoff. 

*Method description lacks details.* It’s unclear exactly how the method works from the description in Section 2. For example, what does it mean that ReAttention “adopts chunked streaming input during the prefilling stage.” I recommend that the authors provide a detailed description of how prefill with ReAttention differs from decoding. It would be great if the authors included a pseudo-code description of both the prefill and decoding stages (either in the main paper or the appendix). 

*The "infinite" in the title is misleading.* It’s not clear from the experiments that ReAttention actually supports infinite context. Further, there are still practical limitations from memory constraints that would prevent the model from really achieving infinite context length. The paper should be more precise about what "infinite" means in this context and acknowledge the real-world constraints.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 2
questions: see weaknesses
soundness: 2
strengths: 1. The use of Triton to optimize the attention mechanism reduces computational overhead and memory usage.
2. The method supports significant context length extensions, showcasing its potential for real-world applications.
summary: The paper introduces ReAttention, a training-free approach for enabling large language models (LLMs) to handle infinite context lengths with a finite attention scope. The method involves a position-agnostic top-k attention mechanism, which precedes traditional self-attention, to select the most relevant context segments. This approach addresses the limitations of context length in LLMs without the need for additional training. The authors validate ReAttention across multiple benchmarks, demonstrating its effectiveness and efficiency compared to existing methods.
weaknesses: 1. The paper uses the top-K method to select KV, but this has been studied in H2O. It feels like the paper is a combination of StreamingLLM+H2O, and then rewrites Triton to make up for the calculation time of QK. I don't see any novelty. It would be helpful to provide more specific details on how ReAttention differs from or combines elements of StreamingLLM and H2O. 

2. The rewrite of the Triton operator is indeed very effective, but it is more of an engineering contribution.

3. The paper is mainly an improvement on KV. There are so many optimizations on KV, but there are so few baselines added for comparison. It is recommended to select some baselines from SnapKV, PyramidKV, LongGen, InfiniPot, DuoAttention and other methods for comparison.

4. Many experiments in the paper show good results, but there is no intuitive or formal explanation for why this method can achieve such good results. For example, theoretical analysis, or visualizations that might help explain the performance gains.

Meta-review: The paper proposes ReAttention, a novel training-free method that enables large language models (LLMs) to handle effectively infinite context lengths with a finite attention scope. The key innovation lies in performing a position-agnostic top-k attention before the regular self-attention mechanism, allowing ReAttention to dynamically manage and retrieve essential context tokens. The authors validate the approach across benchmarks, demonstrating significant performance gains in long-context scenarios. ReAttention stands out for extending context lengths of mainstream LLMs like LLaMA and Mistral to unprecedented lengths (up to 4M tokens) without requiring additional training. Additionally, the Triton-based implementation highlights the engineering contributions, which optimize computational efficiency.

Reviewers generally commended the ability of ReAttention to extrapolate context length significantly beyond pre-trained limits while maintaining competitive or superior performance. The engineering contributions, including a highly efficient Triton kernel, further solidify its applicability for real-world scenarios. The method’s flexibility in maintaining context coherence without positional embeddings is both conceptually and empirically validated. ReAttention’s capacity to extend LLMs' context length while preserving retrieval accuracy in challenging tasks is a compelling advancement in efficient and scalable LLM inference.

Overall, the reviewers converged on the paper’s merit as an impactful contribution to the field, with key innovations and broad applicability. Thus I recommend acceptance. Meanwhile, I do concur with Reviewer fRQF that the word "infinite" in the title is proper because the authors **did not demonstrate it**. The authors are advised to pick a more solid title in the final revision.

Decision: Accept (Poster)