Understanding and Improving Length Generalization in Hierarchical Sparse Attention Models

## Abstract

Abstract Effectively processing long contexts is a critical challenge for language models. While standard Transformers are limited by quadratic complexity and poor length extrapolation, alternative architectures like sliding window attention and state space models sacrifice the ability to effectively utilize the full context due to their fixed-size memory. Chunk-based sparse attention has emerged as a promising paradigm for extreme length generalization, yet the key architectural principles underpinning its success are not yet fully understood. In this work, we present a systematic dissection of these models to identify the core components driving their performance. Through a unified framework and comprehensive ablation studies, we demonstrate that a combination of three design principles is critical: (1) an expressive, non-linear Chunk Encoder with a dedicated CLS token to produce representations for retrieval; (2) a Bypassing Residual Path to stably integrate retrieved global information without it being overridden by the local residual stream; and (3) enforced selection sparsity during pre-training to bridge the train-test distribution gap. We provide a theoretical motivation for intra-chunk information processing and landmark generation. By combining these principles, we establish a new state-of-the-art for training-free length extrapolation, successfully generalizing models trained on a 4K context to 32 million tokens on RULER and BABILong. Our findings provide a clear and empirically-grounded set of design principles for developing future, highly-capable long-context language models.

## 1 Introduction

The recent proliferation of Large Language Models (LLMs) has catalyzed an urgent demand for models capable of processing long contexts (Brown et al., 2020; Touvron et al., 2023; OpenAI et al., 2024). However, the performance of standard Transformer architectures (Vaswani et al., 2023), which serve as the universal backbone, is well-documented to degrade sharply on out-of-distribution context lengths (Hsieh et al., 2024; Xiao et al., 2024; Lu et al., 2024; Kuratov et al., 2024). Concurrently, naively increasing the context window during training is often infeasible due to the quadratic complexity of the attention mechanism, which leads to prohibitive computational costs. This has spurred the community to investigate novel architectures designed explicitly for length extrapolation.

We posit that ideal length extrapolation necessitates two key properties: (1) the ability to train on shorter sequences while achieving stable or even lower perplexity on longer sequences during inference, and (2) the capacity to effectively utilize the entire context. Significant research has focused on the first property.
For instance, architectures like Sliding Window Attention (Beltagy et al., 2020) and recurrent models can maintain a nearly constant perplexity as the context extends beyond the training length (Ruiz & Gu, 2025).
However, these methods fall short on the second property. The former is constrained by a fixed-size local receptive field, while the latter compresses the entire history into a fixed-size state, creating an information bottleneck.
As revealed by in-context retrieval tasks like RULER(Hsieh et al., 2024), the inability to access and utilize arbitrary information from long contexts remains a critical limitation for many applications, from multi-round chatbots to agent-based systems (Liu et al., 2023; Pan et al., 2025; Guo et al., 2023).

Fortunately, recent methods based on retrieval and sparse attention (Mohtashami & Jaggi, 2023; Hu et al., 2025b; a) can maintain a near-constant perplexity on out-of-domain long sequences and achieve high accuracy on simple retrieval tasks like passkey retrieval.
Their approach involves dividing the entire sequence into chunks, selecting the K most similar chunks and attending to the tokens within those chunks.
However, on more complex retrieval tasks, their accuracy still degrades as context length increases. Moreover, a comprehensive analysis of why these models extrapolate so effectively—and what role their different designs play in this capability—is largely missing.

To address this gap, we present a systematic dissection of chunk-based sparse attention mechanisms, identifying the core architectural principles, theoretical motivations, and sparsity configurations that enable extreme generalization. Our main contributions are:

1. A Unified Framework and Systematic Ablation. We present a unified framework for chunk-based sparse attention and conduct a systematic ablation to identify the critical architectural components for generalization. We empirically demonstrate that an optimal combination of three design choices is essential: a non-linear Chunk Encoder for expressive representations, a dedicated CLS token for disentangling retrieval and content information, and a Bypassing Residual Path for stable information integration.

2. Theoretical Motivation and Diagnostic Analysis. We provide a theoretical motivation for the Chunk Encoder, framing it as a necessary non-linear approximator for full attention scores. We support our architectural claims with in-depth diagnostic analysis that correlates intermediate retrieval accuracy with final task performance, revealing how each component improves generalization by enhancing either retrieval prominence or information integration.

3. State-of-the-Art Extrapolation and Insights on Sparsity. By combining these optimal design principles, we establish a new state-of-the-art, generalizing a model trained on a 4K context to 32M tokens. We further provide a detailed analysis on sparsity, showing that both a large context for contrastive learning and enforced selection sparsity during pre-training are crucial for generalization.

## 2 Related Work

Approaches to long-context modeling can be broadly categorized. The first comprises local methods like Sliding Window Attention (Beltagy et al., 2020; Xiao et al., 2024). While effective at maintaining perplexity, they are fundamentally limited by a fixed-size local receptive field, preventing access to information beyond their local window. A second category includes recurrent architectures  (Gu & Dao, 2024; Dao & Gu, 2024; Sun et al., 2023; Qin et al., 2023; Peng et al., 2023a; Yang et al., 2025). Recent work shows that zero-initialized recurrent models struggle to generalize to longer sequences; it attributes the failure to models entering unexplored states and introduces State Passing and Truncated Backpropagation Through Time (Sutskever, 2013; Williams & Peng, 1990) to improve length generalization, achieving stable perplexity on out-of-distribution lengths (Ruiz & Gu, 2025). However, on challenging retrieval tasks (Hsieh et al., 2024), performance still drops significantly beyond the training length. This is because the entire context history is compressed into a fixed-size state, sacrificing the ability to recall specific, distant information losslessly.
The third category seeks to extend full attention by modifying positional encodings (Press et al., 2022; Peng et al., 2023b; Chen et al., 2023; Tan et al., 2025). Their training-free extrapolation gains are modest, typically limited to a small multiple of training length.

In pursuit of more scalable extrapolation, chunk-based sparse attention has emerged as a distinct and highly effective paradigm. Landmark Attention (Mohtashami & Jaggi, 2023) pioneered this by using dedicated landmark tokens to represent chunks for retrieval. While innovative, its length exptrapolation capability is limited to about 64x training length. NSA (Yuan et al., 2025) proposed an efficient hardware-aligned implementation of chunk selection but has very limited length extrapolation (Hu et al., 2025a). State-of-the-art training-free extrapolation has been achieved by Differentiable Retrieval-based Transformers (DRT) (Hu et al., 2025b) and RAMba (Hu et al., 2025a), which can generalize up to 1000x their training length on simple retrieval tasks by introducing a learnable hierarchical chunkwise sparse attention for retrieval. These methods also differ in their retrieval frequency, with DRT operating per-chunk while RAMba, NSA, and Landmark retrieve per-token. Our work provides a systematic analysis to dissect and identify the specific architectural components responsible for the extreme generalization capabilities of this emerging paradigm.

## 3 Preliminary

### 3.1 Random Context Access: The Key Ability to Utilize Full Context

Effective long-context reasoning hinges on the ability to flexibly retrieve information from any part of the preceding context.
This capability, which we term Random Context Access (RCA), is critical for models to excel at tasks requiring the synthesis of distal information. We define it as follows:

Definition:At any given step $t$, Random Context Access is the ability of a model to attend to any subset of tokens from the preceding context $\{x_{0},\dots,x_{t-1}\}$ in a data-dependent and lossless manner.

Under this definition, different architectures exist on a spectrum of capability. The standard Transformer, with its full-attention mechanism, perfectly exemplifies RCA by granting unrestricted access to the entire KV cache. In stark contrast, many efficient architectures trade this property for computational gains by imposing structural constraints. These include: locality constraints, in which models like Sliding Window Attention are confined to a local neighborhood; information bottlenecks, created by State Space Models (SSMs) that compress history into a lossy, fixed-size state; and static access patterns (e.g., Child et al. (2019); Beltagy et al. (2020)), where data-independent sparse attention predetermines information pathways, with more details in Apx. [A.1](https://arxiv.org/html/2510.17196v1#A1.SS1). These architectural bottlenecks fundamentally limit a model’s capacity for true long-range reasoning.

### 3.2 Dynamic Chunkwise Sparse Attention

Chunkwise sparse attention offers a practical and scalable approximation of RCA.
The principle of dynamic chunkwise sparse attention mirrors an efficient strategy for an open-book exam.
Rather than exhaustively reading the entire source material (akin to full attention), one first identifies key pages using high-level bookmarks (landmarks). Queries are then resolved by first consulting these landmarks to locate the relevant pages (chunks) and only then reading their detailed contents (tokens). This hierarchical retrieval drastically reduces the search space, with its success contingent on the quality of the landmarks.

Following this principle, our method provides access to the global context by dynamically selecting a fixed number of chunks. While this imposes a chunkwise access pattern, it aligns with empirical findings that attention scores often exhibit a blockwise structure (Yuan et al., 2025) and that most queries only require a local subset of the context. This approach thereby offers a compelling trade-off between unrestricted context access and computational feasibility.

### 3.3 Hierarchical Sparse Attention and Model Design

Figure: Figure 1: DRT Architecture. The DRT (Hu et al., 2025b) model is primarily composed of Transformers with Sliding Window Attention (SWA), with layers divided into upper layers and lower layers. Lower decoders are composed of SWA followed by an FFN. At the architectural midpoint, a chunking layer splits representations from lower decoders into chunks, and encode them by chunk to build a global memory, each chunk of which contains a landmark vectors ($\mathbf{lmk}$) and encoded chunks ($\mathbf{E}$). Upper decoders each contains a local self-attention layer for local context, followed by an HSA to attend selected global memory and then an FFN.
Refer to caption: x1.png

As shown in Figure [1](https://arxiv.org/html/2510.17196v1#S3.F1.fig1), the DRT model treats the global memory in a chunkwise approach.
During decoding, the chunking layer gathers hidden representations into a buffer and encodes them as complete chunks.
To maintain causality, the chunk containing the current token, as well as all subsequent chunks, are not accessible for retrieval.
Following the notation of Sun et al. (2023), we use indices with $[\cdot]$ to indicate that it is indexed by chunk rather than by token. For instance, $\mathbf{H}_{[t]}:=\mathbf{h}_{t\cdot C+1:(t+1)\cdot C}\in\mathbb{R}^{C\times d}$ is the chunked hidden states. At chunk selection layer, a query vector $\mathbf{q_{t}}\in\mathbb{R}^{d_{\text{Rtrv}}}$ computes a dot-product similarity score on landmark $\mathbf{lmk}_{[i]}\in\mathbb{R}^{d_{\text{Rtrv}}}$, formulated as $s_{t,i}=\mathbf{q_{t}}\cdot\mathbf{{lmk}}_{[i]}\in\mathbb{R}$. The indices of the top-$N$ chunks are then selected, forming set $\mathcal{I}_{t}$.

A critical design choice lies in how these selected chunks are weighted, as this determines how the retrieval signal is backpropagated. The weighting function $w_{t,i}$ is zero for all non-selected chunks. For the selected chunks, the weights are assigned according to distinct strategies for NSA (Eq. [1a](https://arxiv.org/html/2510.17196v1#S3.E1.1)), DRT (Eq. [1b](https://arxiv.org/html/2510.17196v1#S3.E1.2)), and RAMba (Eq. [1c](https://arxiv.org/html/2510.17196v1#S3.E1.3)). Let $\mathcal{S}_{t}=\{s_{t,i}\}_{i\in\mathcal{I}_{t}}$.

This comparison highlights a clear progression: from the implicit binary weighting of NSA, to the proportional softmax weighting in DRT, and finally to the recency-biased allocation, stick-breaking process (^1^11Specifically, let $\mathcal{I}^{\prime}_{t}=\text{sort}(\mathcal{I}_{t})$ be the selected chunk indices sorted by their scores, then $w_{t,k}=\sigma(s_{t,\mathcal{I}^{\prime}_{t,k}})\prod_{j<k}(1-\sigma(s_{t,\mathcal{I}^{\prime}_{t,j}}))$ where $\sigma$ is the sigmoid function.)  (Tan et al., 2025) used in RAMba.

In the second stage, the selected chunk indices $\mathcal{I}_{t}$ and their corresponding weights $w_{t}$ are used to compute the final attention output. The HSA layer takes the query $\mathbf{Q}_{t}$ and the selected keys and values, $[{\mathbf{K}}_{[\mathcal{I}_{t}]},{\mathbf{V}}_{[\mathcal{I}_{t}]}]$, and performs a weighted-sum over standard attention operations:

$$ $\mathbf{O_{t}}=\sum_{i\in\mathcal{I}_{t}}w_{t,i}\cdot\text{Attention}(\mathbf{Q}_{t},{\mathbf{K}}_{[i]},{\mathbf{V}}_{[i]})=\sum_{i\in\mathcal{I}_{t}}w_{t,i}\cdot\text{Softmax}\left(\frac{\mathbf{Q}_{t}{\mathbf{K}}_{[i]}^{T}}{\sqrt{d_{k}}}\right){\mathbf{V}}_{[i]}$ (2) $$

The direct participation of the weights $w_{t,i}$ in the forward computation (Eq. [2](https://arxiv.org/html/2510.17196v1#S3.E2)) creates a differentiable pathway where chunks deemed more important contribute more significantly to the final output. Crucially, in the backward pass, the gradients flowing to each chunk’s parameters are scaled by their weight $w_{t,i}$. This enables the model to perform effective credit assignment, concentrating learning on the most relevant retrived chunks.

## 4 Systematical study of retrieval and intra-chunk processing

### 4.1 Chunk Information preprocess

As shown in Fig. [1](https://arxiv.org/html/2510.17196v1#S3.F1.fig1), $\mathbf{h}^{\frac{L}{2}}_{t}$ serves dual purposes, (1) passed to upper decoder layer as input, (2) gathered with adjacent hidden states to form the chunked hidden states $\mathbf{H}^{\frac{L}{2}}$ to generate global memory (chunked KV and landmarks).
We define these operations through a pair of functions: $f(\cdot)$ for landmark aggregation and $g(\cdot)$ for intra-chunk processing. The final outputs are then derived via linear projections:

$$ $\mathbf{lmk}_{[i]}=\text{Linear}(f({\mathbf{H}}_{[i]}))\in\mathbb{R}^{d_{\text{Rtrv}}}\quad\text{and}\quad[{\mathbf{K}}_{[i]},{\mathbf{V}}_{[i]}]=\text{Linear}(g({\mathbf{H}}_{[i]}))\in\mathbb{R}^{h_{kv}\times(d_{k}+d_{v})}$ $$

**Table 1: Unified Comparison of Chunk Processing: The Encoder is a bidirectional Transformer encoder shared by $f(\cdot)$ and $g(\cdot)$. For the “w/ CLS” variant, $[{0}]$ selects the output corresponding to the prepended $\mathbf{x}_{\text{CLS}}$ token, and $[{1:}]$ selects the outputs for the original chunk hidden states $\mathbf{H}_{[i]}$.**
| Configuration | ${f(\mathbf{H}_{[i]})}$ | ${g(\mathbf{H}_{[i]})}$ |
| --- | --- | --- |
| NSA | $\text{MeanPool}(\mathbf{H}_{[i]})$ | $\mathbf{H}_{[i]}$ |
| HSA w/o Encoder | $\text{MeanPool}(\text{Norm}(\mathbf{H}_{[i]}))$ | $\text{RMSNorm}(\mathbf{H}_{[i]})$ |
| HSA w/ Encoder w/o CLS | $\text{MeanPool}(\text{Encoder}(\mathbf{H}_{[i]}))$ | $\text{Encoder}(\mathbf{H}_{[i]})$ |
| HSA w/ Encoder w/ CLS | $\text{Encoder}([\mathbf{x}_{\text{CLS}};\mathbf{H}_{[i]}])[0]$ | $\text{Encoder}([\mathbf{x}_{\text{CLS}};\mathbf{H}_{[i]}])[1:\ ]$ |

The different architectural configurations we investigate, summarized in Table [1](https://arxiv.org/html/2510.17196v1#S4.T1), can be expressed as joint definitions of $(f,g)$.
In the “w/ CLS” variant, we prepend a learnable token, $\mathbf{x}_{\text{CLS}}$, to the input chunk $\mathbf{H}_{[i]}$, as shown in Fig. [2](https://arxiv.org/html/2510.17196v1#S4.F2). The Encoder processes this combined sequence, and its output corresponding to the $\mathbf{x}_{\text{CLS}}$ position is used to form the landmark, while the remaining outputs form the KV cache.

Chunked Hidden States: Disentangling Representations for Retrieval and Prediction.
Considering the dual purposes of $\mathbf{h}^{\frac{L}{2}}_{t}$, we posit that simply operating on these raw hidden states conflates these two distinct functional objectives. The introduction of non-linear layers aims to disentangle these two roles. By processing the raw states, these components can produce refined representations specifically optimized for retrieval, separate from the immediate demands of next-token prediction.
A RMSNorm provides a simple mechanism to introduce non-linearity, while the Encoder takes this a step further by explicitly learning to re-encode the chunk’s information.

Landmark: Approximating Full Attention via Learnable Summarization.
We frame HSA as a structured approximation of full attention. The goal is for the HSA weight $\hat{\alpha}_{t,i}$(Eq.  [3b](https://arxiv.org/html/2510.17196v1#S4.E3.2)) to be a faithful approximation of the full attention weight $\alpha_{t,i}$ (Eq. [3a](https://arxiv.org/html/2510.17196v1#S4.E3.1)).

Formally, the ideal chunk weight $w_{t,c_{i}}$ should approximate the ratio of its chunk’s attention mass to the global attention mass, as shown in Eq. [4a](https://arxiv.org/html/2510.17196v1#S4.E4.1). However, since a landmark is generated using only local chunk information, it is impossible to compute the global denominator. Therefore, the model must learn a practical proxy: the unnormalized selection score $s_{t,c_{i}}$ must be made proportional to the chunk’s unnormalized attention mass, a relationship defined in Eq. [4b](https://arxiv.org/html/2510.17196v1#S4.E4.2).

Figure: Figure 2: Design of Encoder: (a): Encoder w/o CLS (b): Encoder with a learnable CLS token.
Refer to caption: x2.png

The core challenge is thus to generate a landmark $\mathbf{lmk}_{[i]}$ that satisfies the relationship in Eq. [4b](https://arxiv.org/html/2510.17196v1#S4.E4.2). The right-hand side of this equation is a highly nonlinear function of the keys within the chunk. A simple linear operator like MeanPool is insufficient for this task. A multilayer Chunk Encoder, as a powerful learnable function, is essential for learning this complex relationship. Adding a CLS token and using its output as landmark, further improve the nonlinearity and learnability compared with mean-pooling of the Chunk Encoder outputs.

### 4.2 Skip Connection Design for Cross-Layer Information Fusion

Some HSA layers in our model can retrieve information to the shared long-range memory, by attending to the key-value states at the output of the chunking layer. This design introduces a challenge: how to effectively fuse the more abstract information of the current upper layer, $\mathbf{x}_{\text{in}}$, with the more literal, lower-level context retrieved by the HSA module, $\mathcal{H}(\mathbf{x}_{\text{in}})$.

We investigate two alternative skip connection strategies for integrating the HSA layer within the upper decoder blocks. In both designs, the output of the HSA layer is added to the input residual to form an intermediate representation, $\mathbf{x}^{\prime}$ (Eq. [5](https://arxiv.org/html/2510.17196v1#S4.E5)). This representation is then processed by the subsequent MLP block, $\mathcal{M}(\cdot)$. The key distinction between the two strategies lies in the final residual connection. The Standard Sequential Path (Eq. [6a](https://arxiv.org/html/2510.17196v1#S4.E6.1)) follows the conventional Transformer design, adding the MLP’s output back to the intermediate representation $\mathbf{x}^{\prime}$. In contrast, the Bypassing Residual Path (Eq. [6b](https://arxiv.org/html/2510.17196v1#S4.E6.2)) provides a more controlled mechanism, ensuring the direct output from the HSA module *bypasses* the final residual addition.

We hypothesize that the Bypassing Residual Path is particularly suited for our cross-layer attention architecture. The direct addition of cross-layer information $\mathcal{H}(\mathbf{x}_{\text{in}})$ to the highly-processed residual stream, as in the Standard Sequential Path may disrupt the information flow. On the contrary, the Bypassing Residual Path tasks the MLP with explicitly learning how to resolve the information gap between network depths.

## 5 Experiments

### 5.1 Experimental Setup

Figure: Figure 3: BabiLong Evaluation Results: Our best DRT model demonstrates robust long-context extrapolation, maintaining high accuracy up to 8M tokens. In stark contrast, baseline models exhibit clear limitations. Full-attention methods collapse to near-zero accuracy shortly after their training length. Prior sparse methods like Landmark Attention show better generalization but also fail decisively at longer contexts ($\geq$64k). Sliding Window Attention plateaus at a low accuracy consistent with random guessing, while Mamba2’s performance degrades substantially.
Refer to caption: x3.png

Our experiments are conducted on a series of models based on the DRT architecture detailed in Figure [1](https://arxiv.org/html/2510.17196v1#S3.F1.fig1). All model variants contain approximately 240M parameters and are trained with a context length of 4K. For our analysis, we replace the original GCA mechanism with our HSA module, utilizing a stick-breaking weighting strategy (Eq. [1c](https://arxiv.org/html/2510.17196v1#S3.E1.3)) to enable robust per-token retrieval.

Key hyperparameters for this setup are fixed across experiments. The sliding window size for the lower layers is 512 tokens, and the HSA chunk size is 64. Unless otherwise noted, we use a Top-K of 8 for chunk selection, giving the HSA module an effective receptive field of 512 tokens. A comprehensive description of our training recipes and further hyperparameter details is provided in Appendix [B](https://arxiv.org/html/2510.17196v1#A2).

Benchmark. We evaluate our models on two long-context benchmarks: BabiLong (Kuratov et al., 2024) and RULER (Hsieh et al., 2024). BabiLong is designed to simulate complex question-answering scenarios over long documents. It extends the classic bAbI dataset (Weston et al., 2015) by embedding the original short stories and factual sentences into a long distractor context. This design challenges a model to first perform robust, long-range information retrieval to locate all relevant facts, and then apply the reasoning required by the original task. Consequently, BabiLong tests retrieval and reasoning capabilities simultaneously.

To complement this and more directly assess a model’s core retrieval ability in isolation, we also use RULER. From RULER, we select three representative sub-tasks: Single-Needle (S-N), Multi-Query Needle (MQ-N), and Variable Tracking (VT). Collectively, these tasks provide a targeted evaluation of a model’s capacity for precise, long-range information retrieval and tracking simple logical chains. As RULER requires accurately retrieve several tokens, it provides a strict test of models’ random context access ability.

Baselines. We compare our method against several baselines. Our primary full-attention baseline is Llama with YaRN and Mamba2 with State Passing (Ruiz & Gu, 2025), which is trained on a 4k context length. To ensure a fair comparison with other efficient attention mechanisms, we include Landmark Attention, configured to match our model’s training FLOPs during training by using a sequence length of 756. Furthermore, to benchmark against methods with a similar computational budget, we include Stick-breaking Attention and Sliding Window Attention. Both of these are trained with a 1k context window to keep the receptive field same to our approach.

### 5.2 Length Extrapolation Results

**Table 2: RULER evaluation results. Our DRT best model maintains high accuracy up to 32M tokens, significantly outperforming all baselines in long-context scenarios. In Model Specs, ‘Enc’ denotes the number of encoder layers, ‘CLS’ indicates the use of a CLS token, and ‘Bypass’ refers to the Bypassing Residual Path.**
| Model | Enc | CLS | Bypass | S-N | MQ-N | VT | Avg. | S-N | MQ-N | VT | Avg. | S-N | MQ-N | VT | Avg. |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Model Specs | ctx-len=4K | ctx-len=32K | ctx-len=128K |  |  |  |  |  |  |  |  |  |  |  |  |
| Mamba2 | 0 | — | — | 99.54 | 15.96 | 80.80 | 65.43 | 1.48 | 1.49 | 0.40 | 1.12 | 0.00 | 0.00 | 0.00 | 0.00 |
| NSA | 0 | — | — | 22.82 | 3.15 | 12.43 | 12.80 | 0.00 | 0.00 | 4.48 | 1.49 | 0.00 | 0.00 | 0.00 | 0.00 |
| Llama-Yarn | 0 | — | — | 92.39 | 69.11 | 93.51 | 85.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
| Stick-Break | 0 | — | — | 76.72 | 22.17 | 37.20 | 45.36 | 73.88 | 24.63 | 31.34 | 43.28 | 33.33 | 18.18 | 30.30 | 27.27 |
| Landmark | 0 | — | — | 83.00 | 37.00 | 40.00 | 53.33 | 83.00 | 15.00 | 20.00 | 39.33 | 0.00 | 0.00 | 0.00 | 0.00 |
| DRT | 0 | no | no | 84.40 | 61.30 | 37.02 | 60.91 | 78.97 | 38.80 | 12.14 | 43.30 | 77.49 | 35.14 | 7.80 | 40.14 |
| DRT | 1 | no | no | 81.20 | 68.40 | 87.80 | 79.13 | 81.00 | 66.20 | 87.40 | 78.20 | 77.20 | 62.60 | 86.60 | 75.47 |
| DRT | 1 | yes | no | 79.80 | 70.40 | 90.00 | 80.07 | 81.00 | 65.40 | 86.40 | 77.60 | 75.60 | 62.20 | 84.80 | 74.20 |
| DRT | 2 | no | no | 77.39 | 67.52 | 89.24 | 78.05 | 78.58 | 65.75 | 87.66 | 77.33 | 74.83 | 64.76 | 87.27 | 75.62 |
| DRT | 2 | yes | no | 86.28 | 71.96 | 93.98 | 84.07 | 87.46 | 68.11 | 92.89 | 82.82 | 85.69 | 67.32 | 93.48 | 82.16 |
| DRT | 0 | no | yes | 90.33 | 71.77 | 72.26 | 78.12 | 88.06 | 71.08 | 69.20 | 76.11 | 84.21 | 69.00 | 65.75 | 72.99 |
| DRT | 1 | no | yes | 89.20 | 77.00 | 93.80 | 86.67 | 89.40 | 75.60 | 90.80 | 85.27 | 84.40 | 73.40 | 90.20 | 82.67 |
| DRT | 1 | yes | yes | 91.20 | 75.20 | 88.80 | 85.07 | 89.20 | 72.80 | 89.80 | 83.93 | 88.20 | 74.00 | 87.80 | 83.33 |
| DRT | 2 | no | yes | 91.71 | 72.56 | 93.68 | 85.98 | 88.15 | 72.36 | 92.20 | 84.24 | 86.97 | 71.17 | 92.99 | 83.71 |
| DRT | 2 | yes | yes | 91.51 | 78.78 | 93.48 | 87.92 | 89.14 | 73.84 | 91.61 | 84.86 | 88.94 | 76.21 | 91.81 | 85.65 |
| Models | Enc | CLS | Bypass | S-N | MQ-N | VT | Avg. | S-N | MQ-N | VT | Avg. | S-N | MQ-N | VT | Avg. |
| Model Specs | ctx-len=1M | ctx-len=8M | ctx-len=32M |  |  |  |  |  |  |  |  |  |  |  |  |
| DRT | 0 | no | no | 72.28 | 21.78 | 3.96 | 32.67 | 61.39 | 6.93 | 1.98 | 23.43 | 61.80 | 1.12 | 0.00 | 20.97 |
| DRT | 1 | no | no | 72.73 | 58.18 | 72.73 | 67.88 | 69.09 | 34.55 | 58.18 | 53.94 | 70.91 | 47.27 | 43.64 | 53.94 |
| DRT | 1 | yes | no | 63.64 | 40.00 | 54.55 | 52.73 | 69.09 | 38.18 | 49.09 | 52.12 | 54.55 | 21.82 | 30.91 | 35.76 |
| DRT | 2 | no | no | 63.37 | 63.37 | 72.28 | 66.34 | 58.42 | 48.51 | 66.34 | 57.76 | 67.42 | 42.70 | 70.79 | 60.30 |
| DRT | 2 | yes | no | 83.17 | 46.53 | 82.18 | 70.63 | 78.22 | 28.71 | 80.20 | 62.38 | 82.02 | 24.72 | 61.80 | 56.18 |
| DRT | 0 | no | yes | 80.20 | 64.36 | 45.54 | 63.37 | 71.29 | 46.53 | 15.84 | 44.55 | 82.02 | 31.46 | 3.37 | 38.95 |
| DRT | 1 | no | yes | 69.09 | 56.36 | 45.45 | 56.97 | 58.18 | 30.91 | 10.91 | 33.33 | 63.64 | 36.36 | 12.73 | 37.58 |
| DRT | 1 | yes | yes | 89.09 | 69.09 | 80.00 | 79.39 | 72.73 | 78.18 | 76.36 | 75.76 | 85.45 | 69.09 | 69.09 | 74.54 |
| DRT | 2 | no | yes | 85.15 | 74.26 | 74.26 | 77.89 | 78.22 | 65.35 | 65.35 | 69.64 | 82.02 | 66.29 | 41.57 | 63.29 |
| DRT | 2 | yes | yes | 88.12 | 76.24 | 93.07 | 85.81 | 79.21 | 77.23 | 90.10 | 82.18 | 82.02 | 68.54 | 88.76 | 79.77 |

As shown in Figure [3](https://arxiv.org/html/2510.17196v1#S5.F3), the results on BabiLong highlight the stark differences in extrapolation capability across architectures. The benchmark’s classification-like nature establishes a random-guess baseline around 0.3 accuracy, a level to which Sliding Window Attention predictably converges due to its limited context. Other baselines also fail to generalize, with full-attention and Landmark Attention collapsing at various points beyond their training regimes. Crucially, the figure also reveals the importance of our key design choices within the DRT framework. The DRT variants with a learnable chunk encoder consistently outperform those without.

To further validate these findings and provide a more granular analysis of retrieval performance, we turn to the RULER benchmark, with detailed results in Table [2](https://arxiv.org/html/2510.17196v1#S5.T2). The trends observed here strongly corroborate our conclusions from BabiLong. First, comparing our DRT models against the baselines reveals a significant performance gap in extrapolation. Llama with Yarn (Llama-Yarn), despite its strong in-domain performance at 4K, fails catastrophically at 32K, with their scores dropping to zero. Stick-breaking Attention (Stick-Break) and Landmark Attention demonstrate some extrapolation ability but exhibit significant performance decay and are consistently outperformed by our DRT models on longer contexts.

Second, while all DRT variants showcase robust long-context performance, the results also highlight a clear performance hierarchy among our different architectural configurations. Our full model (DRT with Enc=2, CLS=yes, Bypass=yes) consistently achieves state-of-the-art results. It not only secures the highest average score of 87.92% at the 4K training length but also degrades remarkably gracefully, maintaining an average score of 79.77% even at an extreme context length of 32M. This indicates that the specific combination of our proposed components is critical for unlocking maximum performance, an interaction we will analyze in detail in the subsequent section.

### 5.3 In-depth Analysis of RULER Retrieval

Figure: (a) S-N Top-K Retrieval Accuracy
Refer to caption: x4.png

To further understand the mechanisms behind our architectural designs, we move beyond final task scores to conduct a diagnostic analysis of the intermediate retrieval process. Specifically, we investigate the relationship between the accuracy of chunk selection and the correctness of the final model output on the S-N and MQ-N tasks. In these retrieval-centric tasks, we can unambiguously identify the critical “needle” chunk(s) containing the information required to answer the prompt. We then measure the retrieval recall for these essential chunks and correlate it with final answer accuracy. This allows us to disentangle two primary failure modes: a failure to retrieve the relevant context versus a failure to utilize the context once it has been retrieved. Given that the evaluation context length far exceeds the local sliding window, successful task performance becomes contingent on accurate long-range retrieval in most cases.

Our analysis reveals that even at an extreme context length of 8M, most model configurations, with the notable exception of the poorly performing baseline (without an encoder or bypass path), successfully retrieve the correct “needle” chunk within their top-k(k=8) selections. The significant performance gap, therefore, does not originate from a complete failure of retrieval, but rather from two more subtle factors:
Retrieval Prominence: Superior models assign a significantly higher rank and selection weight to the correct chunk within the top-k set, indicating a more precise and confident retrieval score, as indicated in Fig. [4(a)](https://arxiv.org/html/2510.17196v1#S5.F4.sf1) and Fig. [4(b)](https://arxiv.org/html/2510.17196v1#S5.F4.sf2). As shown in Fig. [4(c)](https://arxiv.org/html/2510.17196v1#S5.F4.sf3), the model can integrate information more accurately if the target information is assigned higher importance among the selected chunks. Information Integration: After successfully retrieving the relevant chunk(s), better-performing models are generally more effective at integrating this information to produce the correct final answer.

Our experiments empirically validate the role of our proposed architectural components in improving both these aspects: 1. The Bypassing Residual Path is crucial for effective information integration. We observe that the bypassing path (‘Bypass=yes‘) consistently and significantly improves final answer accuracy, even when the correct chunks are among selected chunks.
This provides strong evidence for our hypothesis that retrieved information requires dedicated modulation before being integrated into the main residual stream. The bypassing design proves to be a highly effective mechanism for this modulation. 2. The Chunk Encoder and CLS token improve retrieval prominence. Our results show a direct positive correlation between using the encoder and CLS token and the rank of the correct chunk. These models have the same total parameter count; Models with one more encoder layer have one less lower-decoder layer. The fact that this architectural trade-off leads to superior performance on retrieval-intensive tasks demonstrates that the encoder is a more parameter-efficient mechanism for learning robust retrieval representations and adding a CLS token to generate landmark can improve the parameter efficiency of the chunk encoders. Based on these findings, we also significantly improve length generalization of RAMba, detailed in Appendix. [C](https://arxiv.org/html/2510.17196v1#A3).

### 5.4 The Role of Sparsity

**Table 3: Ablation study on sparsity using the RULER benchmark. The interplay between training context length and sparsity is critical for extreme extrapolation. A model trained on a 4K context with high selection sparsity (Top-K=8) ultimately achieves the best performance at 32M tokens.**
| Model | Model Specs | ctx-len=4K | ctx-len=32K | ctx-len=128K |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| TPR | Top-K | Train-Length | S-N | MQ-N | VT | Avg. | S-N | MQ-N | VT | Avg. | S-N | MQ-N | VT | Avg. |  |
| DRT | 1 | 8 | 1024 | 38.68 | 2.50 | 11.60 | 17.59 | 51.49 | 0.75 | 0.00 | 17.41 | 36.36 | 0.00 | 0.00 | 12.12 |
| DRT | 1 | 8 | 2048 | 95.27 | 67.90 | 56.86 | 73.34 | 96.27 | 71.64 | 10.45 | 59.45 | 87.88 | 63.64 | 18.18 | 56.57 |
| DRT | 1 | 8 | 4096 | 91.51 | 78.78 | 93.48 | 87.92 | 89.14 | 73.84 | 91.61 | 84.86 | 88.94 | 76.21 | 91.81 | 85.65 |
| DRT | 1 | 8 | 8192 | 98.70 | 90.54 | 96.75 | 95.33 | 95.52 | 92.54 | 97.01 | 95.02 | 96.97 | 87.88 | 96.97 | 93.94 |
| DRT | 1 | 64 | 4096 | 94.97 | 88.85 | 97.73 | 93.85 | 95.56 | 86.38 | 97.63 | 93.19 | 91.51 | 87.96 | 98.72 | 92.73 |
| DRT | 8 | 8 | 4096 | 93.69 | 31.82 | 89.89 | 71.80 | 93.28 | 24.63 | 88.06 | 68.66 | 93.94 | 6.06 | 63.64 | 54.55 |
| Model | Model Specs | ctx-len=1M | ctx-len=8M | ctx-len=32M |  |  |  |  |  |  |  |  |  |  |  |
| TPR | Top-K | Train-Length | S-N | MQ-N | VT | Avg. | S-N | MQ-N | VT | Avg. | S-N | MQ-N | VT | Avg. |  |
| DRT | 1 | 8 | 4096 | 88.12 | 76.24 | 93.07 | 85.81 | 79.21 | 77.23 | 90.10 | 82.18 | 82.02 | 68.54 | 88.76 | 79.77 |
| DRT | 1 | 8 | 8192 | 97.50 | 84.00 | 87.00 | 89.50 | 97.50 | 75.00 | 65.00 | 79.17 | 95.00 | 45.00 | 45.00 | 61.67 |
| DRT | 1 | 64 | 4096 | 78.18 | 90.91 | 96.36 | 88.48 | 51.31 | 76.36 | 94.55 | 74.07 | 50.91 | 69.09 | 89.09 | 69.70 |

The effectiveness of our model hinges on a nuanced application of sparsity. We identify and analyze three critical dimensions of sparsity, using our best-performing DRT configuration for a controlled study (Table [3](https://arxiv.org/html/2510.17196v1#S5.T3)).

#### Sufficient Training Length.

This dimension refers to the sequence length available to the sparse attention mechanism during training. A longer context provides a larger and more diverse pool of chunk candidates, enabling the model to learn a more robust retrieval policy through contrastive learning. Our results confirm its importance: models trained on a 4K context extrapolate successfully up to 8000x their training length, whereas models trained on a 1K context fail to generalize. Increasing the training length to 8K yields further improvements, establishing a clear trend.

#### Selection Sparsity (Top-K).

This controls how many of the available chunks the model can attend to at each step. A large Top-K (e.g., 64) allows near-dense global access, while a small Top-K (e.g., 8) enforces a sparse retrieval policy, forcing the model to be highly selective. Enforcing this sparsity is crucial for generalization. At extreme lengths (32M), the model trained with Top-K=8 significantly outperforms its Top-K=64 counterpart, as the sparse policy learned during training better matches the demands of inference on extremely long sequences.

#### Retrieval Frequency (Tokens Per Retrieval).

This dimension defines how often the retrieval mechanism (TPR) is invoked—every token (per-token) or less frequently (e.g., every 8 tokens). While less frequent retrieval (per-8-tokens) shows marginal gains on the simple single-retrieval task (S-N), its performance degrades substantially on complex tasks requiring multiple lookups (MQ-N, VT). Therefore, per-token retrieval proves to be the more robust and general-purpose choice, which we use in our final model.

## 6 Conclusion

In this work, we conducted a systematic investigation into the architectural principles underpinning extreme length generalization in chunk-based sparse attention models. Through a unified framework and comprehensive ablation studies, we demonstrated that a combination of three design principles is critical: (1) an expressive, non-linear Chunk Encoder with a dedicated CLS token to generate disentangled representations for retrieval and content; (2) a Bypassing Residual Path to stably integrate retrieved global information without disrupting the local processing stream; and (3) enforced selection sparsity during pre-training to align the model’s learned retrieval policy with the demands of inference on unseen, longer contexts. Our in-depth analysis revealed precisely how these components improve performance by enhancing either retrieval prominence or information integration. By combining these principles, we developed a model that generalizes from a 4K training context to 32 million tokens without fine-tuning. Ultimately, our findings distill the complex design space into a clear, empirically-grounded set of guidelines for developing future, highly-capable long-context language models.

## Ethics Statement

The authors have read and adhere to the ICLR Code of Ethics. This work is foundational, focusing on the architectural principles of language models rather than a specific application. To uphold the principles of scientific transparency and reproducibility, our experiments were conducted exclusively on the publicly available datasets, and we commit to releasing our source code upon de-anonymization to allow for verification and future research. We believe that transparent and reproducible research is crucial for the responsible development of future AI systems.

## Reproducibility Statement

To ensure the reproducibility of our findings, we have provided comprehensive details of our experimental setup. The proposed architectural modifications are described thoroughly, with a detailed architectural diagram and conceptual overview in Figure [1](https://arxiv.org/html/2510.17196v1#S3.F1.fig1). All hyperparameters and training procedures, including the specific recipes for pre-training and supervised fine-tuning on the RULER and BabiLong benchmarks, are thoroughly documented in Appendix [B](https://arxiv.org/html/2510.17196v1#A2). The datasets used are publicly available, and our evaluation setup is described in Section [5.1](https://arxiv.org/html/2510.17196v1#S5.SS1). Furthermore, we commit to releasing our source code upon de-anonymization to allow for full verification of our results and to support future research in this area.

## References

- Ainslie et al. (2020)
Joshua Ainslie, Santiago Ontanon, Chris Alberti, Vaclav Cvicek, Zachary Fisher, Philip Pham, Anirudh Ravula, Sumit Sanghai, Qifan Wang, and Li Yang.
Etc: Encoding long and structured inputs in transformers, 2020.
URL [https://arxiv.org/abs/2004.08483](https://arxiv.org/abs/2004.08483).
- Beltagy et al. (2020)
Iz Beltagy, Matthew E. Peters, and Arman Cohan.
Longformer: The long-document transformer, 2020.
URL [https://arxiv.org/abs/2004.05150](https://arxiv.org/abs/2004.05150).
- Black et al. (2022)
Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach.
GPT-NeoX-20B: An open-source autoregressive language model.
In *Proceedings of the ACL Workshop on Challenges & Perspectives in Creating Large Language Models*, 2022.
URL [https://arxiv.org/abs/2204.06745](https://arxiv.org/abs/2204.06745).
- Botev et al. (2024)
Aleksandar Botev, Soham De, Samuel L Smith, Anushan Fernando, George-Cristian Muraru, Ruba Haroun, Leonard Berrada, Razvan Pascanu, Pier Giuseppe Sessa, Robert Dadashi, Léonard Hussenot, Johan Ferret, Sertan Girgin, Olivier Bachem, Alek Andreev, Kathleen Kenealy, Thomas Mesnard, Cassidy Hardin, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Armand Joulin, Noah Fiedel, Evan Senter, Yutian Chen, Srivatsan Srinivasan, Guillaume Desjardins, David Budden, Arnaud Doucet, Sharad Vikram, Adam Paszke, Trevor Gale, Sebastian Borgeaud, Charlie Chen, Andy Brock, Antonia Paterson, Jenny Brennan, Meg Risdal, Raj Gundluru, Nesh Devanathan, Paul Mooney, Nilay Chauhan, Phil Culliton, Luiz Gustavo Martins, Elisa Bandy, David Huntsperger, Glenn Cameron, Arthur Zucker, Tris Warkentin, Ludovic Peran, Minh Giang, Zoubin Ghahramani, Clément Farabet, Koray Kavukcuoglu, Demis Hassabis, Raia Hadsell, Yee Whye Teh, and Nando de Frietas.
Recurrentgemma: Moving past transformers for efficient open language models, 2024.
URL [https://arxiv.org/abs/2404.07839](https://arxiv.org/abs/2404.07839).
- Brown et al. (2020)
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.
Language models are few-shot learners, 2020.
URL [https://arxiv.org/abs/2005.14165](https://arxiv.org/abs/2005.14165).
- Chen et al. (2023)
Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian.
Extending context window of large language models via positional interpolation, 2023.
URL [https://arxiv.org/abs/2306.15595](https://arxiv.org/abs/2306.15595).
- Child et al. (2019)
Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever.
Generating long sequences with sparse transformers, 2019.
URL [https://arxiv.org/abs/1904.10509](https://arxiv.org/abs/1904.10509).
- Dai et al. (2019)
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V. Le, and Ruslan Salakhutdinov.
Transformer-xl: Attentive language models beyond a fixed-length context, 2019.
URL [https://arxiv.org/abs/1901.02860](https://arxiv.org/abs/1901.02860).
- Dao & Gu (2024)
Tri Dao and Albert Gu.
Transformers are ssms: Generalized models and efficient algorithms through structured state space duality, 2024.
URL [https://arxiv.org/abs/2405.21060](https://arxiv.org/abs/2405.21060).
- De et al. (2024)
Soham De, Samuel L. Smith, Anushan Fernando, Aleksandar Botev, George Cristian-Muraru, Albert Gu, Ruba Haroun, Leonard Berrada, Yutian Chen, Srivatsan Srinivasan, Guillaume Desjardins, Arnaud Doucet, David Budden, Yee Whye Teh, Razvan Pascanu, Nando De Freitas, and Caglar Gulcehre.
Griffin: Mixing gated linear recurrences with local attention for efficient language models, 2024.
URL [https://arxiv.org/abs/2402.19427](https://arxiv.org/abs/2402.19427).
- Gao et al. (2020)
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy.
The pile: An 800gb dataset of diverse text for language modeling, 2020.
URL [https://arxiv.org/abs/2101.00027](https://arxiv.org/abs/2101.00027).
- Glorioso et al. (2024a)
Paolo Glorioso, Quentin Anthony, Yury Tokpanov, Anna Golubeva, Vasudev Shyam, James Whittington, Jonathan Pilault, and Beren Millidge.
The zamba2 suite: Technical report, 2024a.
URL [https://arxiv.org/abs/2411.15242](https://arxiv.org/abs/2411.15242).
- Glorioso et al. (2024b)
Paolo Glorioso, Quentin Anthony, Yury Tokpanov, James Whittington, Jonathan Pilault, Adam Ibrahim, and Beren Millidge.
Zamba: A compact 7b ssm hybrid model, 2024b.
URL [https://arxiv.org/abs/2405.16712](https://arxiv.org/abs/2405.16712).
- Gu & Dao (2024)
Albert Gu and Tri Dao.
Mamba: Linear-time sequence modeling with selective state spaces, 2024.
URL [https://arxiv.org/abs/2312.00752](https://arxiv.org/abs/2312.00752).
- Guo et al. (2023)
Daya Guo, Canwen Xu, Nan Duan, Jian Yin, and Julian McAuley.
Longcoder: A long-range pre-trained language model for code completion, 2023.
URL [https://arxiv.org/abs/2306.14893](https://arxiv.org/abs/2306.14893).
- Hou et al. (2025)
Haowen Hou, Zhiyi Huang, Kaifeng Tan, Rongchang Lu, and Fei Richard Yu.
Rwkv-x: A linear complexity hybrid language model, 2025.
URL [https://arxiv.org/abs/2504.21463](https://arxiv.org/abs/2504.21463).
- Hsieh et al. (2024)
Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg.
Ruler: What’s the real context size of your long-context language models?, 2024.
URL [https://arxiv.org/abs/2404.06654](https://arxiv.org/abs/2404.06654).
- Hu et al. (2025a)
Xiang Hu, Jiaqi Leng, Jun Zhao, Kewei Tu, and Wei Wu.
Random long-context access for mamba via hardware-aligned hierarchical sparse attention, 2025a.
URL [https://arxiv.org/abs/2504.16795](https://arxiv.org/abs/2504.16795).
- Hu et al. (2025b)
Xiang Hu, Zhihao Teng, Jun Zhao, Wei Wu, and Kewei Tu.
Efficient length-generalizable attention via causal retrieval for long-context language modeling, 2025b.
URL [https://arxiv.org/abs/2410.01651](https://arxiv.org/abs/2410.01651).
- Kitaev et al. (2020)
Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya.
Reformer: The efficient transformer, 2020.
URL [https://arxiv.org/abs/2001.04451](https://arxiv.org/abs/2001.04451).
- Kuratov et al. (2024)
Yuri Kuratov, Aydar Bulatov, Petr Anokhin, Ivan Rodkin, Dmitry Sorokin, Artyom Sorokin, and Mikhail Burtsev.
Babilong: Testing the limits of llms with long context reasoning-in-a-haystack, 2024.
URL [https://arxiv.org/abs/2406.10149](https://arxiv.org/abs/2406.10149).
- Lieber et al. (2024)
Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Meirom, Yonatan Belinkov, Shai Shalev-Shwartz, Omri Abend, Raz Alon, Tomer Asida, Amir Bergman, Roman Glozman, Michael Gokhman, Avashalom Manevich, Nir Ratner, Noam Rozen, Erez Shwartz, Mor Zusman, and Yoav Shoham.
Jamba: A hybrid transformer-mamba language model, 2024.
URL [https://arxiv.org/abs/2403.19887](https://arxiv.org/abs/2403.19887).
- Liu et al. (2023)
Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang.
Lost in the middle: How language models use long contexts, 2023.
URL [https://arxiv.org/abs/2307.03172](https://arxiv.org/abs/2307.03172).
- Loshchilov & Hutter (2019)
Ilya Loshchilov and Frank Hutter.
Decoupled weight decay regularization, 2019.
URL [https://arxiv.org/abs/1711.05101](https://arxiv.org/abs/1711.05101).
- Lu et al. (2025)
Enzhe Lu, Zhejun Jiang, Jingyuan Liu, Yulun Du, Tao Jiang, Chao Hong, Shaowei Liu, Weiran He, Enming Yuan, Yuzhi Wang, Zhiqi Huang, Huan Yuan, Suting Xu, Xinran Xu, Guokun Lai, Yanru Chen, Huabin Zheng, Junjie Yan, Jianlin Su, Yuxin Wu, Neo Y. Zhang, Zhilin Yang, Xinyu Zhou, Mingxing Zhang, and Jiezhong Qiu.
Moba: Mixture of block attention for long-context llms, 2025.
URL [https://arxiv.org/abs/2502.13189](https://arxiv.org/abs/2502.13189).
- Lu et al. (2024)
Yi Lu, Jing Nathan Yan, Songlin Yang, Justin T. Chiu, Siyu Ren, Fei Yuan, Wenting Zhao, Zhiyong Wu, and Alexander M. Rush.
A controlled study on long context extension and generalization in llms, 2024.
URL [https://arxiv.org/abs/2409.12181](https://arxiv.org/abs/2409.12181).
- MiniMax et al. (2025)
MiniMax, Aonian Li, Bangwei Gong, Bo Yang, Boji Shan, Chang Liu, Cheng Zhu, Chunhao Zhang, Congchao Guo, Da Chen, Dong Li, Enwei Jiao, Gengxin Li, Guojun Zhang, Haohai Sun, Houze Dong, Jiadai Zhu, Jiaqi Zhuang, Jiayuan Song, Jin Zhu, Jingtao Han, Jingyang Li, Junbin Xie, Junhao Xu, Junjie Yan, Kaishun Zhang, Kecheng Xiao, Kexi Kang, Le Han, Leyang Wang, Lianfei Yu, Liheng Feng, Lin Zheng, Linbo Chai, Long Xing, Meizhi Ju, Mingyuan Chi, Mozhi Zhang, Peikai Huang, Pengcheng Niu, Pengfei Li, Pengyu Zhao, Qi Yang, Qidi Xu, Qiexiang Wang, Qin Wang, Qiuhui Li, Ruitao Leng, Shengmin Shi, Shuqi Yu, Sichen Li, Songquan Zhu, Tao Huang, Tianrun Liang, Weigao Sun, Weixuan Sun, Weiyu Cheng, Wenkai Li, Xiangjun Song, Xiao Su, Xiaodong Han, Xinjie Zhang, Xinzhu Hou, Xu Min, Xun Zou, Xuyang Shen, Yan Gong, Yingjie Zhu, Yipeng Zhou, Yiran Zhong, Yongyi Hu, Yuanxiang Fan, Yue Yu, Yufeng Yang, Yuhao Li, Yunan Huang, Yunji Li, Yunpeng Huang, Yunzhi Xu, Yuxin Mao, Zehan Li, Zekang Li, Zewei Tao, Zewen Ying, Zhaoyang Cong, Zhen
Qin, Zhenhua Fan, Zhihang Yu, Zhuo Jiang, and Zijia Wu.
Minimax-01: Scaling foundation models with lightning attention, 2025.
URL [https://arxiv.org/abs/2501.08313](https://arxiv.org/abs/2501.08313).
- Mohtashami & Jaggi (2023)
Amirkeivan Mohtashami and Martin Jaggi.
Landmark attention: Random-access infinite context length for transformers, 2023.
URL [https://arxiv.org/abs/2305.16300](https://arxiv.org/abs/2305.16300).
- OpenAI et al. (2024)
OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo
Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Łukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Łukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan
Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O’Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez,
Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia
Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph.
Gpt-4 technical report, 2024.
URL [https://arxiv.org/abs/2303.08774](https://arxiv.org/abs/2303.08774).
- Pan et al. (2025)
Zhuoshi Pan, Qianhui Wu, Huiqiang Jiang, Xufang Luo, Hao Cheng, Dongsheng Li, Yuqing Yang, Chin-Yew Lin, H. Vicky Zhao, Lili Qiu, and Jianfeng Gao.
On memory construction and retrieval for personalized conversational agents, 2025.
URL [https://arxiv.org/abs/2502.05589](https://arxiv.org/abs/2502.05589).
- Peng et al. (2023a)
Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, Kranthi Kiran GV, Xuzheng He, Haowen Hou, Jiaju Lin, Przemyslaw Kazienko, Jan Kocon, Jiaming Kong, Bartlomiej Koptyra, Hayden Lau, Krishna Sri Ipsit Mantri, Ferdinand Mom, Atsushi Saito, Guangyu Song, Xiangru Tang, Bolun Wang, Johan S. Wind, Stanislaw Wozniak, Ruichong Zhang, Zhenyuan Zhang, Qihang Zhao, Peng Zhou, Qinghua Zhou, Jian Zhu, and Rui-Jie Zhu.
Rwkv: Reinventing rnns for the transformer era, 2023a.
URL [https://arxiv.org/abs/2305.13048](https://arxiv.org/abs/2305.13048).
- Peng et al. (2023b)
Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole.
Yarn: Efficient context window extension of large language models, 2023b.
URL [https://arxiv.org/abs/2309.00071](https://arxiv.org/abs/2309.00071).
- Press et al. (2022)
Ofir Press, Noah A. Smith, and Mike Lewis.
Train short, test long: Attention with linear biases enables input length extrapolation, 2022.
URL [https://arxiv.org/abs/2108.12409](https://arxiv.org/abs/2108.12409).
- Qin et al. (2023)
Zhen Qin, Songlin Yang, and Yiran Zhong.
Hierarchically gated recurrent neural network for sequence modeling, 2023.
URL [https://arxiv.org/abs/2311.04823](https://arxiv.org/abs/2311.04823).
- Ren et al. (2025)
Liliang Ren, Yang Liu, Yadong Lu, Yelong Shen, Chen Liang, and Weizhu Chen.
Samba: Simple hybrid state space models for efficient unlimited context language modeling, 2025.
URL [https://arxiv.org/abs/2406.07522](https://arxiv.org/abs/2406.07522).
- Roy et al. (2020)
Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier.
Efficient content-based sparse attention with routing transformers, 2020.
URL [https://arxiv.org/abs/2003.05997](https://arxiv.org/abs/2003.05997).
- Ruiz & Gu (2025)
Ricardo Buitrago Ruiz and Albert Gu.
Understanding and improving length generalization in recurrent models, 2025.
URL [https://arxiv.org/abs/2507.02782](https://arxiv.org/abs/2507.02782).
- Soman et al. (2024)
Karthik Soman, Andrew Langdon, Catalina Villouta, Chinmay Agrawal, Lashaw Salta, Braian Peetoom, Gianmarco Bellucci, and Orion J Buske.
Zebra-llama: A context-aware large language model for democratizing rare disease knowledge, 2024.
URL [https://arxiv.org/abs/2411.02657](https://arxiv.org/abs/2411.02657).
- Sun et al. (2023)
Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei.
Retentive network: A successor to transformer for large language models, 2023.
URL [https://arxiv.org/abs/2307.08621](https://arxiv.org/abs/2307.08621).
- Sutskever (2013)
Ilya Sutskever.
*Training recurrent neural networks*.
PhD thesis, 2013.
- Tan et al. (2025)
Shawn Tan, Songlin Yang, Aaron Courville, Rameswar Panda, and Yikang Shen.
Scaling stick-breaking attention: An efficient implementation and in-depth study, 2025.
URL [https://arxiv.org/abs/2410.17980](https://arxiv.org/abs/2410.17980).
- Team et al. (2024)
Jamba Team, Barak Lenz, Alan Arazi, Amir Bergman, Avshalom Manevich, Barak Peleg, Ben Aviram, Chen Almagor, Clara Fridman, Dan Padnos, Daniel Gissin, Daniel Jannai, Dor Muhlgay, Dor Zimberg, Edden M Gerber, Elad Dolev, Eran Krakovsky, Erez Safahi, Erez Schwartz, Gal Cohen, Gal Shachaf, Haim Rozenblum, Hofit Bata, Ido Blass, Inbal Magar, Itay Dalmedigos, Jhonathan Osin, Julie Fadlon, Maria Rozman, Matan Danos, Michael Gokhman, Mor Zusman, Naama Gidron, Nir Ratner, Noam Gat, Noam Rozen, Oded Fried, Ohad Leshno, Omer Antverg, Omri Abend, Opher Lieber, Or Dagan, Orit Cohavi, Raz Alon, Ro’i Belson, Roi Cohen, Rom Gilad, Roman Glozman, Shahar Lev, Shaked Meirom, Tal Delbari, Tal Ness, Tomer Asida, Tom Ben Gal, Tom Braude, Uriya Pumerantz, Yehoshua Cohen, Yonatan Belinkov, Yuval Globerson, Yuval Peleg Levy, and Yoav Shoham.
Jamba-1.5: Hybrid transformer-mamba models at scale, 2024.
URL [https://arxiv.org/abs/2408.12570](https://arxiv.org/abs/2408.12570).
- Team et al. (2025)
Tencent Hunyuan Team, Ao Liu, Botong Zhou, Can Xu, Chayse Zhou, ChenChen Zhang, Chengcheng Xu, Chenhao Wang, Decheng Wu, Dengpeng Wu, Dian Jiao, Dong Du, Dong Wang, Feng Zhang, Fengzong Lian, Guanghui Xu, Guanwei Zhang, Hai Wang, Haipeng Luo, Han Hu, Huilin Xu, Jiajia Wu, Jianchen Zhu, Jianfeng Yan, Jiaqi Zhu, Jihong Zhang, Jinbao Xue, Jun Xia, Junqiang Zheng, Kai Liu, Kai Zhang, Kai Zheng, Kejiao Li, Keyao Wang, Lan Jiang, Lixin Liu, Lulu Wu, Mengyuan Huang, Peijie Yu, Peiqi Wang, Qian Wang, Qianbiao Xiang, Qibin Liu, Qingfeng Sun, Richard Guo, Ruobing Xie, Saiyong Yang, Shaohua Chen, Shihui Hu, Shuai Li, Shuaipeng Li, Shuang Chen, Suncong Zheng, Tao Yang, Tian Zhang, Tinghao Yu, Weidong Han, Weijie Liu, Weijin Zhou, Weikang Wang, Wesleye Chen, Xiao Feng, Xiaoqin Ren, Xingwu Sun, Xiong Kuang, Xuemeng Huang, Xun Cao, Yanfeng Chen, Yang Du, Zhen Yang, Yangyu Tao, Yaping Deng, Yi Shen, Yigeng Hong, Yiqi Chen, Yiqing Huang, Yuchi Deng, Yue Mao, Yulong Wang, Yuyuan Zeng, Zenan Xu, Zhanhui Kang, Zhe Zhao,
ZhenXiang Yan, Zheng Fang, Zhichao Hu, Zhongzhi Chen, Zhuoyu Li, Zongwei Li, Alex Yan, Ande Liang, Baitong Liu, Beiping Pan, Bin Xing, Binghong Wu, Bingxin Qu, Bolin Ni, Boyu Wu, Chen Li, Cheng Jiang, Cheng Zhang, Chengjun Liu, Chengxu Yang, Chengzhong Xu, Chiyu Wang, Chong Zha, Daisy Yi, Di Wang, Fanyang Lu, Fei Chen, Feifei Liu, Feng Zheng, Guanghua Yu, Guiyang Li, Guohua Wang, Haisheng Lin, Han Liu, Han Wang, Hao Fei, Hao Lu, Haoqing Jiang, Haoran Sun, Haotian Zhu, Huangjin Dai, Huankui Chen, Huawen Feng, Huihui Cai, Huxin Peng, Jackson Lv, Jiacheng Shi, Jiahao Bu, Jianbo Li, Jianglu Hu, Jiangtao Guan, Jianing Xu, Jianwei Cai, Jiarong Zhang, Jiawei Song, Jie Jiang, Jie Liu, Jieneng Yang, Jihong Zhang, Jin lv, Jing Zhao, Jinjian Li, Jinxing Liu, Jun Zhao, Juntao Guo, Kai Wang, Kan Wu, Lei Fu, Lei He, Lei Wang, Li Liu, Liang Dong, Liya Zhan, Long Cheng, Long Xu, Mao Zheng, Meng Liu, Mengkang Hu, Nanli Chen, Peirui Chen, Peng He, Pengju Pan, Pengzhi Wei, Qi Yang, Qi Yi, Roberts Wang, Rongpeng Chen, Rui Sun,
Rui Yang, Ruibin Chen, Ruixu Zhou, Shaofeng Zhang, Sheng Zhang, Shihao Xu, Shuaishuai Chang, Shulin Liu, SiQi Wang, Songjia Feng, Songling Yuan, Tao Zhang, Tianjiao Lang, Tongkai Li, Wei Deng, Wei Li, Weichao Wang, Weigang Zhang, Weixuan Sun, Wen Ouyang, Wenxiang Jiao, Wenzhi Sun, Wenzhuo Jia, Xiang Zhang, Xiangyu He, Xianshun Ren, XiaoYing Zhu, Xiaolong Guo, Xiaoxue Li, Xiaoyu Ma, Xican Lu, Xinhua Feng, Xinting Huang, Xinyu Guan, Xirui Li, Xu Zhang, Xudong Gao, Xun Luo, Xuxiang Qi, Yangkun Chen, Yangyu Tao, Yanling Xiao, Yantao Mai, Yanze Chen, Yao Ding, Yeting Yang, YiFan Song, Yifan Yang, Yijiao Zhu, Yinhe Wu, Yixian Liu, Yong Yang, Yuanjun Cai, Yuanlin Tu, Yue Zhang, Yufei Huang, Yuhang Zhou, Yuhao Jiang, Yuhong Liu, Yuhui Hu, Yujin Lin, Yun Yang, Yunhao Wang, Yusong Zhang, Zekun Wu, Zelong Zhang, Zhan Yu, Zhaoliang Yang, Zhe Zhao, Zheng Li, Zhenyu Huang, Zhiguang Liu, Zhijiang Xu, Zhiqing Kui, Zhiyin Zeng, Zhiyuan Xiong, Zhuo Han, Zifan Wu, Zigang Geng, Zilong Zhao, Ziyan Tang, Ziyuan Zhu, Zonglei Zhu,
and Zhijiang Xu.
Hunyuan-turbos: Advancing large language models through mamba-transformer synergy and adaptive chain-of-thought, 2025.
URL [https://arxiv.org/abs/2505.15431](https://arxiv.org/abs/2505.15431).
- Touvron et al. (2023)
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas
Scialom.
Llama 2: Open foundation and fine-tuned chat models, 2023.
URL [https://arxiv.org/abs/2307.09288](https://arxiv.org/abs/2307.09288).
- Vaswani et al. (2023)
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.
Attention is all you need, 2023.
URL [https://arxiv.org/abs/1706.03762](https://arxiv.org/abs/1706.03762).
- Weston et al. (2015)
Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M. Rush, Bart van Merriënboer, Armand Joulin, and Tomas Mikolov.
Towards ai-complete question answering: A set of prerequisite toy tasks, 2015.
URL [https://arxiv.org/abs/1502.05698](https://arxiv.org/abs/1502.05698).
- Williams & Peng (1990)
Ronald J Williams and Jing Peng.
An efficient gradient-based algorithm for on-line training of recurrent network trajectories.
*Neural computation*, 2(4):490–501, 1990.
- Wu et al. (2022)
Yuhuai Wu, Markus N. Rabe, DeLesley Hutchins, and Christian Szegedy.
Memorizing transformers, 2022.
URL [https://arxiv.org/abs/2203.08913](https://arxiv.org/abs/2203.08913).
- Xiao et al. (2024)
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis.
Efficient streaming language models with attention sinks, 2024.
URL [https://arxiv.org/abs/2309.17453](https://arxiv.org/abs/2309.17453).
- Yang et al. (2025)
Songlin Yang, Bailin Wang, Yu Zhang, Yikang Shen, and Yoon Kim.
Parallelizing linear transformers with the delta rule over sequence length, 2025.
URL [https://arxiv.org/abs/2406.06484](https://arxiv.org/abs/2406.06484).
- Yuan et al. (2025)
Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, Y. X. Wei, Lean Wang, Zhiping Xiao, Yuqing Wang, Chong Ruan, Ming Zhang, Wenfeng Liang, and Wangding Zeng.
Native sparse attention: Hardware-aligned and natively trainable sparse attention, 2025.
URL [https://arxiv.org/abs/2502.11089](https://arxiv.org/abs/2502.11089).
- Zaheer et al. (2021)
Manzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed.
Big bird: Transformers for longer sequences, 2021.
URL [https://arxiv.org/abs/2007.14062](https://arxiv.org/abs/2007.14062).

## Appendix

## Appendix A Additional Related Work

In the main body of our paper, we focus on dynamic sparse attention methods specifically designed for length extrapolation. Here, we provide a broader overview of the rich literature on sparse attention, which has historically focused on a different goal: reducing the quadratic complexity of attention for long, but still in-distribution, sequences. We categorize these methods into static and dynamic approaches, and for each, we explain why their design principles do not typically lend themselves to training-free length extrapolation.

### A.1 Static Sparse Attention

Static sparse attention methods reduce computational complexity by restricting each token to a predefined, fixed subset of other tokens. These data-independent patterns are highly efficient but lack the flexibility required for length extrapolation.
Early work like Sparse Transformer (Child et al., 2019) introduced fixed strided and dilated patterns, a foundational concept that influenced many subsequent models. This idea evolved into more complex topologies that combine local windowed attention with some form of global connectivity. For instance, Longformer (Beltagy et al., 2020) and ETC (Ainslie et al., 2020) designated a few “global” tokens to act as information hubs for the entire sequence. BigBird (Zaheer et al., 2021) generalized this by combining local, global, and random connections to approximate the small-world properties of a dense graph. These methods, while successfully extending the manageable context length for tasks like long-document processing, are fundamentally tied to the positional relationships learned during training. Their fixed patterns and reliance on learned positional biases prevent them from generalizing to sequence lengths orders of magnitude beyond what they were trained on.

### A.2 Dynamic Sparse Attention

Unlike static methods, dynamic sparse attention mechanisms determine attention patterns adaptively based on the input content, aiming to approximate the expressiveness of full attention by focusing computation on relevant tokens.
Early approaches often relied on heuristic-based grouping or clustering. Reformer (Kitaev et al., 2020) pioneered the use of locality-sensitive hashing (LSH) to bucket tokens for attention, while the Routing Transformer (Roy et al., 2020) used online k-means clustering. These methods were a significant step towards content-aware sparsity. Another line of work focused on memory, with Transformer-XL (Dai et al., 2019) introducing a recurrent memory and Memorizing Transformers (Wu et al., 2022) using a kNN index for retrieval from an external long-term memory.
While these methods are “dynamic”, their selection mechanisms are still parameterized and optimized for the context lengths seen during training. The learned LSH functions, cluster centroids, or memory management policies are not inherently length-agnostic and thus do not extrapolate well. More recent hardware-aware methods like NSA (Yuan et al., 2025) and MoBA (Lu et al., 2025) have achieved significant practical speedups, but their design goal remains computational efficiency within a known context distribution, not training-free generalization to extreme lengths. Our work, in contrast, focuses specifically on learning a retrieval policy that is robust to drastic changes in sequence length, a distinct and complementary objective.

## Appendix B Training Recipe

### B.1 Pre-training Recipe

Our pre-training recipe was designed for our models of approximately 240M parameters and is summarized in Table [4](https://arxiv.org/html/2510.17196v1#A2.T4). All models were pre-trained on a tokenized version of the deduplicated Pile dataset (Gao et al., 2020) using the GPT-NeoX-20B tokenizer (Black et al., 2022). The models were trained for approximately 8.05 trillion tokens using a global batch size of 1.05 million tokens. We used the AdamW optimizer (Loshchilov & Hutter, 2019) with a cosine learning rate schedule, a peak learning rate of $1\times 10^{-3}$, and a minimum learning rate of $2\times 10^{-4}$.

### B.2 Supervised Fine-Tuning Recipes

Following pre-training, separate fine-tuning runs were conducted for the RULER and BabiLong benchmarks. All SFT runs were initialized from their respective pre-trained checkpoints.

#### SFT for RULER.

The SFT data was constructed dynamically. For each sample, one of the three RULER sub-tasks (S-N, MQ-N, or VT) was randomly selected. The task’s “needle” was then inserted at a random position within a background context sampled from the Pile. The model was trained to generate the correct answer. This stage ran for approximately 1.0 trillion tokens with a peak learning rate of $2\times 10^{-4}$.

#### SFT for BabiLong.

In a similar fashion, the SFT data for BabiLong was created by dynamically embedding tasks from the original bAbI dataset (Tasks 1-20) into a background context from the Pile. For each training sample, one of the 20 bAbI tasks was randomly chosen, and its factual sentences were embedded into a context. The model was then trained on the associated question-answering pair. The hyperparameters were identical to the RULER SFT, with the primary difference being a slightly longer training schedule of approximately 1.07 trillion tokens.

**Table 4: Key hyperparameters for Pre-training and SFT stages.**
| Hyperparameter | Pre-training | SFT (RULER) | SFT (BabiLong) |
| --- | --- | --- | --- |
| Common Parameters |  |  |  |
| Model Parameters | $\approx$ 240M |  |  |
| Optimizer | AdamW |  |  |
| LR Schedule | Cosine with 2% warmup |  |  |
| Global Batch Size | 1.05 Million tokens |  |  |
| Stage-Specific Parameters |  |  |  |
| Training Tokens | $\approx$ 8.05 Trillion | $\approx$ 1.0 Trillion | $\approx$ 1.07 Trillion |
| Peak Learning Rate | $1\times 10^{-3}$ | $2\times 10^{-4}$ | $2\times 10^{-4}$ |
| Minimum Learning Rate | $2\times 10^{-4}$ | $4\times 10^{-5}$ | $4\times 10^{-5}$ |

## Appendix C Hybrid Architectures with SSM

Figure: Figure 5: RAMba Architectures. An MLP is inserted into upper decoder layer in RAMB-MLP as Mamba does not have explicit MLP layer.
Refer to caption: x7.png

The recent success of hybrid architectures, which synergistically combine different components to balance performance, efficiency, and long-context reasoning, marks a significant trend in sequence modeling. This paradigm has been prominently validated in models built upon the Mamba architecture, such as Jamba (Lieber et al., 2024), Zamba/Zamba2 (Glorioso et al., 2024b; a), Samba (Ren et al., 2025), and others (Team et al., 2025; Soman et al., 2024). The principle extends to hybrids combining attention with linear RNNs or other mechanisms, as seen in Griffin (De et al., 2024), RWKV-X (Hou et al., 2025), RecurrentGemma (Botev et al., 2024), and large-scale MoE models (MiniMax et al., 2025; Team et al., 2024).

A notable model in this space is RAMba (Hu et al., 2025a), which integrates an SSM with a HSA and demonstrates 64x length generalization on RULER. We hypothesize that our architectural principles can further enhance the performance of such hybrid models. To test this, we apply our components to the RAMba architecture. The original RAMba design does not include a distinct MLP block due to Mamba’s block design, which is necessary for a controlled study of our Bypassing Residual Path, as shown in Figure [5](https://arxiv.org/html/2510.17196v1#A3.F5). We therefore introduce an MLP after each HSA block, enabling us to isolate the effects of our proposed encoder, CLS token, and bypass mechanism on a state-of-the-art SSM-based foundation. We systematically investigate on RULER, detailed results in Table [5](https://arxiv.org/html/2510.17196v1#A3.T5).

**Table 5: RULER results for the RAMba hybrid architecture. The results confirm that the design principles identified for DRT generalize effectively to an SSM-based model. Our full RAMba configuration (w/ Encoder, CLS, Bypass) once again achieves the best performance, particularly at extreme context lengths ($\geq 1$M), validating the critical and synergistic role of these components across different local architectures.**
| Model | Enc.Layers | CLS | Bypass | S-N | MQ-N | VT | Avg. | S-N | MQ-N | VT | Avg. | S-N | MQ-N | VT | Avg. |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Model Specs | ctx-len=4K | ctx-len=32K | ctx-len=128K |  |  |  |  |  |  |  |  |  |  |  |  |
| Mamba2 | 0 | — | — | 99.54 | 15.96 | 80.80 | 65.43 | 1.48 | 1.49 | 0.40 | 1.12 | 0.00 | 0.00 | 0.00 | 0.00 |
| Mamba2 w/ NSA | 0 | — | — | 64.94 | 1.48 | 35.62 | 34.01 | 4.48 | 0.00 | 5.97 | 3.48 | 0.00 | 0.00 | 15.15 | 5.05 |
| RAMBA (original) | 2 | no | no | 90.17 | 76.16 | 94.81 | 87.05 | 82.84 | 74.63 | 88.81 | 82.09 | 84.85 | 72.73 | 96.97 | 84.85 |
| RAMBA | 2 | yes | no | 94.06 | 78.20 | 94.99 | 89.08 | 87.31 | 84.33 | 98.51 | 90.05 | 81.82 | 84.85 | 90.91 | 85.86 |
| RAMBA-MLP | 0 | no | yes | 84.69 | 55.47 | 71.43 | 70.53 | 85.82 | 44.03 | 35.07 | 54.97 | 75.76 | 15.15 | 15.15 | 35.35 |
| RAMBA-MLP | 2 | no | yes | 95.08 | 79.04 | 94.99 | 89.70 | 94.78 | 60.45 | 85.82 | 80.35 | 100.00 | 45.45 | 81.82 | 75.76 |
| RAMBA-MLP | 2 | yes | no | 93.32 | 81.35 | 96.66 | 90.44 | 84.33 | 78.36 | 96.27 | 86.32 | 84.85 | 66.67 | 96.97 | 82.83 |
| RAMBA-MLP (ours) | 2 | yes | yes | 95.36 | 83.95 | 96.64 | 91.98 | 97.76 | 88.06 | 97.76 | 94.53 | 100.0 | 87.88 | 100.0 | 95.96 |
| Models | Enc | CLS | Bypass | S-N | MQ-N | VT | Avg. | S-N | MQ-N | VT | Avg. |  |  |  |  |
| Model Specs | ctx-len=1M | ctx-len=8M |  |  |  |  |  |  |  |  |  |  |  |  |  |
| RAMBA (original) | 2 | no | no | 33.66 | 51.49 | 43.56 | 42.90 | 6.93 | 24.75 | 22.77 | 18.15 |  |  |  |  |
| RAMBA | 2 | yes | no | 57.43 | 52.48 | 32.67 | 47.53 | 17.82 | 22.77 | 17.82 | 19.47 |  |  |  |  |
| RAMBA-MLP | 0 | no | yes | 68.32 | 5.94 | 30.69 | 34.98 | 48.51 | 1.98 | 16.83 | 22.44 |  |  |  |  |
| RAMBA-MLP | 2 | no | yes | 92.17 | 14.81 | 60.00 | 55.66 | 80.00 | 10.00 | 45.00 | 45.00 |  |  |  |  |
| RAMBA-MLP | 2 | yes | no | 48.56 | 65.49 | 48.18 | 54.08 | 15.17 | 48.28 | 10.34 | 24.60 |  |  |  |  |
| RAMBA-MLP (ours) | 2 | yes | yes | 100.00 | 80.00 | 72.00 | 84.00 | 90.00 | 72.00 | 63.64 | 75.21 |  |  |  |  |

## Appendix D LLM Usage Statement

In accordance with the disclosure made during the submission process, large language models (LLM) were utilized as a writing assistant to aid in polishing the text of this manuscript. The LLM’s role was strictly limited to improving the clarity, conciseness, and narrative flow of author-written content. All scientific contributions, including the core research ideas, experimental design, and analysis of results, are the sole work of the human authors. The authors have reviewed and edited all text and take full responsibility for the final content of this paper.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: see weakness section
soundness: 3
strengths: 1.The paper offers a clear conceptual story. They provide a nice way to unify many previous chunk-based designs with the proposed concept, and the proposed concept is simple and effective.
2. The empirical results are strong: models trained on 4k contexts can extrapolate to tens of millions of tokens. 
3.The ablation study is especially thorough—it covers encoder depth, presence of CLS, bypass design, Top-K size, and training length, giving a convincing picture of what really matters.
summary: The paper try to interpret why chunk-based sparse attention models, for examples DRT and RAMbe, tend to generalize well to longer contexts than they were trained on. The authors argue that these models implicitly approximate a property they call Random Context Access (RCA)—the ability to flexibly retrieve any relevant part of the past without full attention. They identify three key design choices that make this work. 1. A non-linear chunk encoder with a CLS token, which learns better chunk summaries than simple mean pooling. 2. Bypassing residual path, which helps integrate retrieved global information without it being drowned out by local residuals. 3. Training with small Top-K retrieval, which avoids train–test mismatch.
They build on a 240M-parameter DRT backbone trained at 4k tokens and show strong length extrapolation—up to 32M tokens—on RULER and BabiLong, outperforming full attention and other long-context baselines.
weaknesses: 1.The theory part is more intuition-driven than rigorous; the RCA argument isn’t formally analyzed.
2. The experiments more rely on synthetic or controlled tasks. The experiment on realistic long-context applications is limited.
3. The detailed efficiency analysis of the method is not discussed.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 2
questions: 1. Precisely delineate what is new.
   The paper combines chunk encoders, a CLS landmark, and the bypass residual within a unified HSA view, but it is not crystal-clear what is truly novel relative to DRT, RAMba, and NSA beyond the specific encoder+CLS objective and the bypass wiring. Please provide a one‑page “delta table” contrasting your equations and learning signals with DRT’s retrieval and RAMba’s stick‑breaking setup, and quantify the marginal gains when retrofitting only your two advertised contributions (encoder+CLS and bypass) into prior baselines without other recipe changes. A tight novelty delineation would materially strengthen the paper. See the unified comparison in Table 1 (page 5), Eq. 6a/6b (page 5), and the baseline list in Section 5.1 (page 6). 

2. Landmark theory: can you give a formal guarantee or falsifiable test?
   Eq. 4a/4b (page 5) proposes that the landmark score should track a chunk’s total attention mass. Could you provide either: (i) a bound on how well a finite encoder can approximate this mass under simple distributions (e.g., mixture of von Mises–Fisher keys), or (ii) a calibration plot showing correlation between your selection score and the true within‑chunk mass across lengths? A short, synthetic study or correlation analysis would make the claim more than heuristic. 

3. Bypassing residual: mechanism and gradients.
   The bypass changes the residual addition point (Eq. 6a vs 6b, page 5), and Figure 4 suggests improved utilization. Can you show gradient‑flow or activation‑scale diagnostics demonstrating that the bypass prevents residual stream “overwrites”? A simple ablation with a learned gate or scalar on the bypass path and a rank‑1 sensitivity analysis would help attribute the gain to integration rather than retrieval side effects. Cite Figure 4 (page 8) and Eq. 6a/6b (page 5). 

4. Baseline fairness and apples‑to‑apples choices.
   Landmark Attention is trained on shorter sequences to match FLOPs, whereas your models train at 4k; retrieval frequency and Top‑K also differ across methods. Please report a “length‑matched” Landmark baseline at 4k and harmonize retrieval frequency (per‑token) and Top‑K across baselines. Adding one strong positional baseline beyond YaRN (e.g., ALiBi/PI) at 4k would reduce the chance that your positional baseline is a soft target. The request is small at 32k/128k (not 32M). See Section 5.1 (page 6) and Table 2 (page 7). 

5. Compute and efficiency at long lengths.
   The results emphasize accuracy up to 32M tokens, but the paper lacks wall‑clock, memory, and throughput numbers for chunk selection vs token attention. Please add a figure with end‑to‑end latency and peak memory versus length for Top‑K in {4, 8, 16}, chunk size 64, and the hardware used. Even a 32k→128k→1M sweep would make the practical story credible. Table 4 (page 17) shows large training/SFT budgets; efficiency data would help separate algorithmic benefits from sheer scale. 

6. Scale and backbone generality.
   Most main results use ~240M parameters. Appendix C shows encouraging RAMba results, but still at small scale and with a RAMba‑MLP variant. Can you share any existing runs or partial results at ≥1B parameters, or at least trends when moving the chunking boundary earlier/later in depth in the Transformer backbone? Figure 1 (page 3) and Table 5 (page 18) are the relevant anchors. Even partial curves at 32k/128k would be informative.
soundness: 2
strengths: Originality.

* Introduces a clean formalization of *Random Context Access* as the capability a long‑context model actually needs, which sharpens the objective beyond “long perplexity is stable” and motivates the architectural choices thereafter (Section 3.1). 
* Provides a unified lens for *dynamic chunkwise sparse attention* that decomposes chunk selection, weighting, and intra‑chunk processing; the side‑by‑side comparison of weighting rules (NSA, softmax, stick‑breaking) in Eq. 1a–1c and the f/g functions for landmark vs KV formation is a helpful synthesis that organizes a messy literature (Section 3.3; Table 1 on page 5). 
* Offers a simple but convincing theoretical target for landmark design: landmarks should approximate a chunk’s total attention mass, which in turn argues for a nonlinear chunk encoder and a CLS‑based summarizer rather than linear pooling (Eq. 4a–4b on page 5). 
* Proposes a *Bypassing Residual Path* that changes how retrieved global information is fused, with a concrete formulation (Eq. 6b on page 5) and a diagnostic that separates retrieval recall from integration precision (Figure 4 on page 8). 
* Demonstrates transfer of the principles to an SSM hybrid (RAMba) by inserting an explicit MLP to test the bypass mechanism and encoder/CLS design in a different backbone (Appendix C; Table 5 on page 18). 

Quality.

* Careful experimental design: all DRT variants hold parameter count (~240M) fixed and standardize key knobs (window 512, chunk size 64, Top‑K 8) so ablations isolate the proposed components rather than incidental capacity changes (Section 5.1). 
* Strong and transparent baselines: full‑attention with RoPE scaling (Llama‑YaRN), Landmark Attention, sliding‑window, stick‑breaking, and Mamba2 with State Passing are included; training FLOPs or receptive fields are matched where relevant (page 6). 
* Evaluations cover both retrieval‑only skills (RULER: S‑N, MQ‑N, VT) and retrieval‑plus‑reasoning (BabiLong), reducing the chance of overfitting to a single metric. The curves on BabiLong reach up to 8M tokens (Figure 3 on page 6), and RULER tables extend to 32M (Table 2 on page 7). 
* Diagnostics move beyond end accuracy: retrieval recall@Top‑K and *marginal integration precision* clarify that the bypass path mostly helps utilization while the encoder/CLS improves ranking prominence (Figure 4 on page 8). This is rare and genuinely informative. 
* Thoughtful sparsity study: clean ablations over training length, Top‑K, and tokens‑per‑retrieval show why selection sparsity during pre‑training and per‑token retrieval matter at extreme lengths (Table 3 on page 9). 

Clarity.

* The architectural flow is easy to follow: the DRT diagram highlights lower vs upper decoders and the chunking layer that produces landmarks and encoded chunks (Figure 1 on page 3). The two encoder variants are visually contrasted (Figure 2 on page 5). 
* Mathematical definitions are compact and targeted: Eq. 2 stitches chunk weights into standard attention; Eq. 4b states the landmark learning objective in one line; Eq. 6a–6b precisely defines the bypass change. The paper ties each equation to an experiment that tests it, which keeps the narrative crisp. 
* Results are presented with the right granularity: a single multi‑length RULER table makes degradation profiles legible (Table 2 on page 7), BabiLong’s averaged curve makes plateau/failure modes obvious (Figure 3 on page 6), and Appendix B lays out hyperparameters and training schedules in a reproducible table (Table 4 on page 17). 

Significance.

* Establishes training‑free extrapolation far beyond common practice: models trained at 4K hold up to 32M on RULER with graceful decay and maintain strong BabiLong accuracy to 8M, while RoPE‑style baselines collapse shortly after the training length (Table 2 on page 7; Figure 3 on page 6). This pushes the empirical frontier and sets a new reference point for “extreme” long‑context capability. 
* Distills actionable design rules that others can adopt: use a nonlinear chunk encoder with a CLS landmark, fuse via a bypass that lets the MLP mediate cross‑layer information, enforce selection sparsity during pre‑training, and retrieve per token. These prescriptions are validated again in a distinct hybrid (RAMba), which broadens the paper’s impact beyond a single architecture (Table 5 on page 18). 
* Clarifies what it means to “use the whole context” by centering Random Context Access and measuring retrieval prominence vs integration. That conceptual shift is likely to influence how future long‑context work frames objectives and ablations, not just how it reports numbers (Sections 3–5). 

Overall, the paper combines a tidy theoretical target, disciplined ablations, and record‑scale evidence to produce design guidance that is both novel and broadly useful to the long‑context community.
summary: Paper summary

The paper studies why hierarchical, chunk-based sparse attention models can extrapolate to extreme context lengths and how to make them more reliable. Using a unified view of “dynamic chunkwise sparse attention,” it decomposes these systems into (i) how chunks are encoded and landmarks are formed for retrieval, (ii) how retrieved global information is fused back into the model, and (iii) how sparsity is enforced during training. A theoretical argument shows landmarks must approximate a chunk’s total attention mass, which requires a nonlinear chunk encoder rather than simple pooling. Empirically, models trained on 4K tokens generalize, without further tuning, to contexts up to 32M on RULER and to 8M on BabiLong. Figure 1 (page 3) diagrams the architecture; Figure 3 (page 6) shows BabiLong accuracy remaining high as length grows; Table 2 (page 7) reports strong RULER results at 32M; Table 3 (page 9) details sparsity ablations. 

Key contributions

* Unified framework + ablations: Formalizes chunk processing as functions over hidden states and compares weighting schemes (NSA, softmax, stick‑breaking) and encoder designs (with/without CLS), isolating which parts drive extrapolation. 
* Theory for landmark design: Shows that effective landmarks must approximate a chunk’s total attention mass, motivating a nonlinear Chunk Encoder and a dedicated CLS token for learnable summarization instead of mean pooling. 
* Cross‑layer fusion mechanism: Introduces a Bypassing Residual Path so retrieved global context is integrated via the MLP rather than overwritten by the residual stream; diagnostics separate retrieval recall from integration precision. 
* State‑of‑the‑art training‑free length extrapolation: 4K‑trained models reach 32M tokens on RULER with graceful degradation (e.g., ~80% average at 32M in Table 2) and maintain high BabiLong accuracy up to 8M tokens (Figure 3). 
* Principled sparsity insights: Finds that selection sparsity during pre‑training (small Top‑K like 8), sufficient training length (≥4K), and per‑token retrieval are critical for extreme generalization (Table 3). 
* Generality to hybrids: Applying the same principles to RAMba (Mamba + HSA) yields consistent gains across long lengths (Appendix C, Table 5).
weaknesses: 1. Positioning and novelty are under‑specified.
   The paper’s core design elements (chunkwise retrieval with landmarks, per‑token retrieval, and hierarchical sparse attention) are largely inherited from DRT/RAMba/NSA; the main additions are the specific *bypass* residual and the non‑linear chunk encoder with a CLS landmark. The text acknowledges these antecedents but doesn’t clearly delimit what is genuinely new vs. what is a careful re‑combination. Tighten the novelty claim by: (i) explicitly contrasting your equations and learning signals with DRT’s GCA and RAMba’s stick‑breaking weighting; (ii) running a head‑to‑head where you *retrofit* prior baselines with your two contributions (encoder+CLS and bypass) without other recipe changes, isolating marginal gains. Table 5 partially does this for RAMba‑MLP; extend the same “surgery” to Landmark and DRT to make the incremental contribution undeniable. 

2. Theory is insightful but stops short of guarantees.
   The “landmark should approximate a chunk’s total attention mass” target (Eq. 4a–4b) is a compelling heuristic, but it is not accompanied by identifiability or approximation bounds. As written, it is unclear when a finite‑depth encoder can satisfy Eq. 4b under distribution shift or adversarial distractors. Strengthen the section by: (i) providing an upper bound on the KL or total‑variation gap between α and α̂ (Eq. 3a vs 3b) under assumptions on within‑chunk key dispersion; (ii) proving that mean pooling fails in a controlled setting (e.g., mixture of von Mises–Fisher keys) and that a 2‑layer encoder with CLS can represent the log‑sum‑exp proxy; (iii) stress‑testing Eq. 4b with synthetic counter‑examples where per‑chunk keys are multi‑modal. Figure 2 and the argument around Eq. 4b are the right hooks; add formal statements and failure cases. 

3. Evaluation is narrow; external validity is uncertain.
   The study leans heavily on RULER S‑N/MQ‑N/VT and BabiLong. These are excellent for retrieval diagnostics, but they are synthetic and cue‑rich; they do not stress discourse coherence, paraphrase invariance, or multi‑hop compositionality across heterogeneous documents. Broaden the evidence by adding: (i) long‑document QA with paraphrase and entity aliasing; (ii) multi‑file code navigation where identifiers are obfuscated; (iii) book‑length summarization with contradiction traps; and (iv) “lost‑in‑the‑middle”‑style probes that deliberately place relevant spans at low prior positions. Figure 3’s averaged BabiLong curve and Table 2’s RULER tables are strong; add at least one real‑world, non‑templated task to bolster claims of “utilize the entire context.” 

4. Comparative fairness has confounds.
   Several baseline choices can be contested: Landmark is trained with shorter sequences to match FLOPs, which handicaps its retrieval policy; LLaMA‑YaRN is known to degrade beyond train length, so a single positional‑scaling baseline is a soft target. Since Section 5.1 already modifies DRT to use stick‑breaking for per‑token retrieval, some of the gains vs “DRT 0” could be due to weighting, not just encoder/CLS/bypass. Make the comparison harder to dispute by: (i) giving baselines the same retrieval frequency (per‑token) and Top‑K; (ii) matching *training sequence length* rather than FLOPs for Landmark; and (iii) adding a second strong positional baseline (e.g., an ALiBi/PI variant) trained with the same 4k context and SFT recipe. See the setup on page 6 and Table 2 on page 7. 

5. Compute and efficiency claims are underspecified.
   The paper emphasizes accuracy at 1M–32M tokens but gives no end‑to‑end latency, memory, or throughput numbers for these lengths, and no complexity constants for chunk selection. Appendix Table 4 signals substantial training/SFT budgets (≈8.05T pre‑training tokens; ≈1T tokens per SFT), which can swamp architectural effects. Add: (i) wall‑clock and peak‑memory curves vs. sequence length for Top‑K∈{4,8,16} and chunk size 64 on a standard accelerator; (ii) ablate retrieval index cost (score computation on landmarks) separately from token‑level attention; (iii) a compute‑normalized comparison (accuracy vs. TFLOPs) at long lengths. This will make the “practical” story credible. 

6. Scale and backbone generality remain open.
   All primary models are ~240M parameters. The transfer to RAMba is helpful, but results are still at small scale and with a custom RAMba‑MLP that inserts an MLP not present in the original block. Demonstrate robustness by: (i) repeating key ablations at ≥1B parameters; (ii) applying the encoder+CLS and bypass to a pure Transformer baseline with different local windows; and (iii) showing the trend holds when the chunking boundary is moved earlier/later in depth (Figure 1 shows a fixed mid‑point). Table 5 provides a good template; broaden it to other backbones and sizes.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 4
questions: 1. How do you initialize the CLS token, and does it matter? Have you experimented with multiple CLS tokens per chunk? 
2. Are there tasks where you find that your method underperforms full attention at shorter contexts, which would demonstrate a fundamental limitation of chunk-based retrieval?
3. Is your method sensitive to the chunk size hyperparameter, and does this vary with context length or task? An ablation on this would be valuable.
soundness: 3
strengths: 1. The paper has a clear presentation and nicely unifies different sparse attention approaches under a single framework. 
2. This method achieves significant training-free length extrapolation gains and retains them consistently on RULER and BABILong, marking strong empirical evidence.
3. The justification for the chunk encoder as approximating full attention scores is compelling.
4. The result in Appendix C showing that this method generalizes to hybrid SSMs is exciting and empirically sound.
summary: This paper analyzes chunk-based hierarchical sparse attention models for extreme length extrapolation, specifically calls out three design choices: a non-linear chunk encoder, a bypassing residual path, and high selection sparsity. By combining these elements, the authors achieve state-of-the-art training-free length extrapolation, generalizing models trained on a 4K window to 32M tokens on the RULER and BABILong benchmarks. The work also provides theoretical motivation for the Chunk Encoder and diagnostic analysis correlating retrieval accuracy with task performance.
weaknesses: 1. The theoretical analysis is informal, and is not derived — there is no formal analysis of when and why this approximation is reasonable to make, although it is intuitive and stated as such. 
2. It would be valuable to have an analysis of computational costs, and any impact on inference speed (e.g. TPS and TTFT statistics) through this method.
3. Evaluation primarily focuses on mostly retrieval tasks and synthetic reasoning, some real-world tasks that are a bit more diverse (e.g. summarization over long-documents) would be valuable to show general utility. 
4. The model scale is limited, and it is unclear whether this extends to larger models; I am willing to overlook this given the results are quite compelling, but for practical utility, this is important to study.

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 3
questions: 1. The authors state that sliding-window attention cannot access information outside its window. Given this limitation, why is its performance on the babilong benchmark at 8M context nearly identical to its performance at 4K? Does this imply the task itself does not require genuine long-range dependencies?

2. Do these long-context modifications harm the model's short-text capabilities? It would be beneficial to evaluate the model on standard short-text benchmarks (e.g., MMLU, Wikitext-2) to verify that the proposed changes do not cause performance regression in non-long-context scenarios.
soundness: 2
strengths: 1. The proposed strategies—such as adding an encoder for global KV representations, using a CLS token as a landmark, and incorporating a bypass residual connection—are simple, yet deliver significant and immediate gains on downstream tasks. This makes the approach highly practical.


2. The paper provides detailed ablation studies that convincingly demonstrate the necessity of each component and the synergistic benefits of their combination.


3. The model is evaluated on well-known long-text benchmarks (RULER, babilong). The choice of benchmarks is reasonable and effectively characterizes the model's long-context capabilities.

4. The authors attempt to provide a rationale for their modifications, such as explaining how the encoder-processed CLS token landmark can better approximate the highly non-linear attention denominator.
summary: This paper explores a method combining chunk-wise attention and retrieval, enabling models trained on short texts to extrapolate to ultra-long contexts. The framework, an enhancement of DRT, investigates improvements like adding encoder for global KV representation, a CLS token for predicting landmark, and bypass residual connection. Experiments on RULER and babilong demonstrate that these incremental modifications yield significantly superior performance.
weaknesses: Despite the strong empirical results, the paper's primary weakness lies in its insufficient theoretical justification for why the proposed modifications are effective. The framework is fundamentally similar to DRT, yet the analysis of the new components is cursory.

1. Ambiguous Role of the Encoder: The distinction in performance with and without the encoder is not substantial. It is unclear if the limited improvement stems from its purported "decoupling" function or simply from increased model capacity via extra parameters. The paper fails to clarify if this addition removes a specific bottleneck, or if adding the same parameter count elsewhere (e.g., in the base model) would yield similar gains.

2. Unclear Mechanism of the Bypass Connection: The large performance boost from the simple bypass residual connection is surprising. The authors' explanation—that "the Standard Sequential Path may disrupt the information flow"—is vague and unconvincing. The paper lacks a sufficient explanation for the mechanism by which this simple "re-wiring" dramatically improves retrieval performance.

Overall, the theoretical arguments for these modifications feel brief and tenuous, which undermines the reliability of the empirical findings.


Decision: Accept (Poster)