Rope to Nope and Back Again: A New Hybrid Attention Strategy
Authors: Bowen Yang
Bharat Venkitesh
Dwaraknath Gnaneshwar
Hangyu Lin
David Cairuz
Phil Blunsom
Acyr Locatelli
Venue: NeurIPS 2025 poster ()
Keywords: LLM
Pretraining
Long Context
Hybrid Attention
Primary area: deep_learning
TL;DR: This paper analyzes existing methods for long-context modeling and compares various attention variants. It introduces a hyrbid architecture that offers a more efficient and scalable solution with better performance for extended sequence processing.

Abstract: Long-context large language models (LLMs) have achieved remarkable advancements, driven by techniques like Rotary Position Embedding (RoPE) (Su et al., 2023) and its extensions (Chen et al., 2023; Liu et al., 2024c; Peng et al., 2023). By adjusting RoPE parameters and incorporating training data with extended contexts, we can train performant models with considerably longer input sequences. However, existing RoPE-based methods exhibit performance limitations when applied to extended context lengths. This paper presents a comprehensive analysis of various attention mechanisms, including RoPE, No Positional Embedding (NoPE), and Query-Key Normalization (QK-Norm), identifying their strengths and shortcomings in long-context modeling. Our investigation identifies distinctive attention patterns in these methods and highlights their impact on long-context performance, providing valuable insights for architectural design.  on long context performance, providing valuable insights for architectural design. Building on these findings, we propose a novel architecture featuring a hybrid attention mechanism that integrates global and local attention spans. This design not only surpasses conventional RoPE-based transformer models with full attention in both long and short context tasks but also delivers substantial efficiency gains during training and inference.

Rope to Nope and Back Again:
A New Hybrid Attention Strategy
Bowen Yang'
Bharat Venkitesh'
Dwarak Talupuru'
Hangyu Lin'
David Cairuz
Phil Blunsom
Acyr Locatelli
Cohere
(bowen, dwarak, john, phil)@cohere.com (bharat, davidcairuz, acyr) acyr)&cohere....
Abstract
Long context large language models (LLMs) have achieved remarkable advance- ments, driven by techniques like Rotary Position Embedding (RoPE) [61] and its By adjusting RoPE parameters and incorporating training data with extended contexts, we can train performant models with considerably longer input sequences. However, existing RoPF-based methods exhibit perfor- mance limitations when applied extended context lengths. This paper presents a comprehensive analysis of various attention mechanisms, including RoPE, No Positional Embedding (NoPE), and Query-Key Key Normalization (QK-Norm), identi- fying their strengths and shortcomings in long context modeling. Our investigation identifies distinctive attention patterns in these methods and highlights their impact on long context performance, providing valuable insights for architectural design. Building these findings, propose novel architecture featuring hybrid attention mechanism that integrates global and local attention spans. This design not only surpasses conventional RoPE-based transformer models with full attention in both long and short context tasks but also delivers substantial efficiency gains during training and inference.
extensions 12 47 54
1 Introduction
Developing language models capable of handling long context lengths poses several challenges. First, as the context length increases, an effective modeling extended input sequences becomes increasingly critical. This often requires advancements in positional encoding [61], extrapolation techniques [23], or architectural innovations [69 34]. Second, training long context large language models with billions of parameters demands significant computational resources. Overcoming this challenge requires scalable algorithms, high-quality datasets, and robust infrastructure. Lastly, deploying these models in real-world applications demands low latency and low memory usage, which requires meticulous optimization of both the model architecture and the serving infrastructure.
On the modeling front, two components of the transformer architecture are particularly crucial for long context capabilities: the attention mechanism and positional embeddings. Recent research has proposed various methods to enhance these components. For instance, Landmark Attention 53 trains attention modules to select relevant blocks using a representative token, referred to as a "landmark token". for efficient retrieval within extended text corpora. Similarly, Focused Transformer [69] adopts a contrastive training approach to prioritize attending the most relevant portions of the input sequence, allowing the model to focus smaller, contextually significant subsets of tokens. Although these approaches improve long context modeling, stabilizing training remains a key challenge for extending transformer capabilities to longer sequences. Query-Key Normalization (QK-Norm) [1] 57) has
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

been introduced to address the stability issue, which normalizes the query-key vectors along the head dimension before computing attention. Although QK-Norm mitigates numerical instability during it may impair long context capabilities.
training and is widely used 666 21 42
In addition to the chosen attention mechanism, positional emheddings play crucial role in long context modeling. Various approaches have been proposed to improve their effectiveness. Popular methods include Absolute Position Embedding (APE)
[70], Relative Position Embedding (56). ALiBi Among these, RoPE has gained significant
[551, and Rotary Position Embedding (RoPE) [1].
adoption in large language models (LLMs) 24
75 83 due to its simplicity and effectiveness.
In particular, it has the ability to extrapolate context lengths by adjusting RoPE A values during training 45 2 [8]. Other techniques, such as relative bias [55] [6) and contextualized position embeddings 26]. introduce distance-basee basee bias terms or condition the position information imput semantics. These methods often affect attention distribution by incorporating auxiliary information, such a positional indices or explicit recency bias. However, whether certain information or biases are beneficial to long context modeling or overall performance remains less explored. Additionally, the concept of No Positional Embedding (NoPE) has been explored by [38], suggesting that removing explicit positional embeddings and relying solely on implicit positional information derived from the causal mask can enhance long context performance.
Despite advances in long context modeling, training and serving such models remain challenging due to the quadratic complexity of standard attention. Techniques like Sliding Window Attention mitigate this by restricting each token's 's attention a local window, reducing compute
(SWA) 36 67 while maintaining local coherence. Sparse attention methods [14] 7 64
extend this by introducing structured sparsity, including random 78] and dilated patterns 22 More recent approaches further compress attention, such as activation beacon [80) which sequentially summarizes local keys and values, and attention sink (73), which stabilizes long-sequence training by preserving early tokens with a sliding window. On the serving side, KV cache trimming methods [8] 41 [0] selectively discard cached states based on heuristics to reduce memory usage and boost inference efficiency. However, these gains often come with trade-offs in model quality, emphasizing the need for careful design.
In this paper, we begin analyzing attention patterns of different attention mechanisms, RoPE, NoPE, and QK-Norm and its impacts on long context performance trained up to 750 billion tokens. Building on these insights, we propose novel hybrid attention architecture and extensively pretrain up to trillion tokens, followed by supervised fine-tuning on diverse set of datasets tailored for long context. We show that this architecture surpasses existing state-of-the-art extrapolation-basel RoPE models 47 by large margin, striking a balance between efficiency and performance.
2
Observation
In this section, we assess three models with different attention components on needles-in -inss--------aa in-a-haystack 377 (NIAH) and analyze the attention patterns to understand how these variants affect performance. Analysis in this section guides our architectural design choices throughout this work.
2.1 Experimental Setup
All model variants share a common configuration consisting of 8 billion parameters (including the token embedding parameters), with detailed architectural specifications provided in Table [ For this set of experiments, the model is trained in two stages: a pretraining stage followed by a supervised fine-tuning (SFT) stage. Previous research shows that the SFT stage is necessary for long context evaluations, as it can reduce variance in long context tasks and enables the emergence of long context capabilities that may not manifest in models trained solely through pretraining (25]
We pretrain the model with a batch size of 4 million tokens. We use AdamW with peak learning rate of 7e- -3, linear warmup of 2000 steps and cosine learning rate schedule decaying to 3.5c over 179,000 steps for a total of 750 billion tokens. For the SFT stage, we adopt an interleaved training strategy: we combine short- and long context data in a 3:1 ratio, with context lengths of 8192 and 65536 tokens, respectively. We use a batch size of 0.5 million tokens.
The 3 model variants we test are:

---

Parameters
Emb. Dim
FFN Dim
Non-linearity
Layers
Heads
KV Heads
Vocab Size
Values
4096
28672
swiglu
32
32
8
256000
Table 1: Model architecture details
1. RoPE Model: For this variant, we employ Rotary Position Embedding (RoPE) to encode positional information. During the pretraining stage, the RoPE parameter 0 is set to 10,000. In the subsequent supervised fine-tuning (SFT) stage, 0 is increased to million account for the increased context length. This variant serves as the baseline model configuration, maintaining an architecture similar to that of most existing models [24 36 17
2. QK-Norm Model:
Layer normalization 4 is applied to both the query and the key vectors before performing the angular rotation used in RoPE. All other hyperparameters, including the values and training methodology, remain identical to those of the RoPE variant.
3. NoPE Model:
Previous research 772 (29 has demonstrated that transformer variants trained without positional embeddings (NoPE) can perform effectively on long context tasks. How- ever, these models often exhibit inferior performance in terms of perplexity and downstream task evaluations within the trained sequence length (29). In our study, the NoPE variant does not have QK-Norm. This variant is trained using the same methodology the other two variants.
2.2
Evaluation and Attention Analysis
2.2.1
Evaluations
We evaluate the variants a set of core evaluation benchmarks, including MMLU (30, HellaSwag [79]. CommonsenseQA [63], ARC [15] for core model capabilities and NIAH benchmark [37]
for long context capability. NIAH evaluates model's ability retrieve information accurately from a specific sentence (the "needle") embedded within a lengthy document (the "haystack"). The needle is randomly placed varying depths within the sequence to examine performance across different context lengths. To improve robustness, we modify the original NIAH benchmark, where each context-depth combination tested 16 times with different random seeds, creating diverse context compositions for comprehensive evaluation. The results of all standard benchmark evaluations and results with 65536 context length needles are presented in Table 2 Although prior research has emphasized the limitations of NIAH (74] for evaluating deeper and more general context understanding, our focus solely on testing basic long context capabilities and gaining insights on model architecture design, for which this benchmark is sufficient.
Table 2 shows that the RoPE and QK-Norm variants exhibit comparable performance on standard benchmarks, while the NoPE variant lags behind. This finding is consistent with previous studies
38 72). For long context evaluations, QK-Norm performs the worst among the three variants, despite its decent performance in other capabilities. This observation is consistent with the results from the comparisons between Command R and Command R+, where Command R, despite being significantly smaller model, outperforms Command R+ overall on longer context benchmarks [32]. Although the NoPE variant has slightly lower needles score compared to the RoPE variant, it is decent given that its base capabilities is relatively low.
2.2.2 Attention Pattern Analysis
To better understand the impacts of different architectures. we also analyze the attention patterns within each model. This approach is inspired by previous studies [77 39 where attention scores assigned different parts of the context are closely examined.
We still utilize the NIAH task by first dividing the context into four segments:
Begin: The first 10 tokens. This part of the context is also often referred to as the "attention sink" (73), where a disproportionately large amount of attention is typically allocated in trained transformer model.
3

---

Model
Val Loss
MMLU
HellaSwag 73.74
CommonsenseeQ 68.30
ARC-E
ARC-C
Needles 65k
RoPE
1.52
48.55
81.05
39.13
9.82
QK-Norm 1.53
48.21
73.68
68.23
80.54
38.98
7.93
NoPE
1.58
47.61
72.16
66.42
76.94
37.12
9.03
Table 2: Comparison of model performance across range of benchmarks, All evaluations are based on the outputs of the SFT models. Red cells indicate lower performance.
Contoet Longth
Cortzost Length = 32k
Contest Length 128k
p200
Figure 1: Attention Distribution across context lengths of each variant
Needle: The tokens of the inserted needle sentence. Ideally, a well-trained model should assign relatively high amount of attention this part of the context.
End: The query and completion tokens, which can represent recency bias.
Context: The remainder of the context, typically consisting of noise or irrelevant content.
We position the needles at approximately 50% depth within the context to increase the complexity of the task, as most models suffer from the set-in-the-middie problem, highlighted in previous works [46] [5) For each model, we first calculate the attention scores between the query tokens of "End" segment and the key tokens of all four segments across all heads and layers. The attention scores are summed within each segment and then aggregated across all heads and layers to obtain the average attention weight for each segment. These scores are further averaged across multiple samples at sequence lengths of 8,000 tokens, 32,000 tokens, and 228000 tokens. We refer to this metric as "attention mass" in the following sections. The results of each variant are visualized in Figure [
We begin by comparing attention distributions across different sequence lengths from Figure [ We observe consistent decrease in attention mass allocated to the "Needle" segment and a corresponding increase in attention mass on the "Context" segment as sequence length increases. This trend indicates that retrieving relevant information becomes increasingly difficult as the context grows longer. When comparing across model variants, the NoPE variant consistently allocates the highest attention mass the Needle" segment, followed by the RoPE variant, while the QK-Norm variant assigns the least attention to this segment. Furthermore, the QK-Norm variant exhibits markedly lower attention mass on the "Begin" " segment and substantially higher attention mass on the "Context" segment relative to other variants. This indicates that models trained with the QK-Norm component exhibit weak attention sink and are more susceptible interference from noisy content. These patterns are consistent with QK-Norm's 's poor performance on the NIAH task. We argue that QK-Norm has this effect because the normalization operation mitigates magnitude information from the dot product of Query and Key vectors which tends to result in attention logits being closer in terms of magnitude and flatter in terms of distribution. A more detailed analysis of why QK-Norm is detrimental to long-context modeling is provided in Appendix B

---

RoPE Layers - 32x
0.7
Attention Weights
0.5
Model
RoPE
0.4
0.354
RNOFE-10E
RNoPE 100K
0..
ANOPE-2M
0.223
0.2
0.192
0.111 0.096
0.1
0.005
0.024
0.000 0.001
0.0
Begin
Needle Segment
NoPE Layers 32k
0.8
0.7
Attention Weights
0.5
Model
RNoPE-10K
0.4
0.326
0.325
FNcPE 2M
RNoPE-4r
DNoPE-1OK--n
0.2
0.076 0.078 0.071
0.074
0.037
0.017
0.0
Begin
Needle Segment
End
Figure 2: Attention Distribution at 32k context length for rope baseline and hybrid variants.
2.2.3
Hybrid Model
Building on the findings above, we are inspired combine RoPE and NoPE layer-wise leverage the strengths of both variants. NoPE provides an effective attention mechanism for retrieving information based on vector similarity, while RoPE explicitly models positional information and introduces recency bias. By combining them, we aim to enhance overall performance. We propose new variant that alternates between NoPE and Rotary Position Embedding (RoPE) layers. Specifically, the two position-embeddiin in the next. To ensure consistency and enable meaningful comparisons, the RoPE parameter is initially set to 10,000 during pre- rre-training. The pre-training procedure follows the setup described in Section in terms of data and training protocol. Subsequently, we perform multiple fine-tuning runs with varying values-10.000 across different configurations, using the same training steps and data as in Section 2.1 We refer to these models RNoPE- 10k, ANoPE--101. RNoPE-2M and RNoPE-4M with the specific RoPE value used for each during supervised fine-tuning (SFT).
strategies are interleaved, with NoPE applied in one layer and RoPE
100,000, 2 million, and 4 million- to evaluate performance
Model
RoPE
RNoPB-10K
100K
RNoPE-2M
RNoPE-4M
RNOPE-10K-swa
Needles-1222 128k Score
7.40
8.04
7.46
7.02
6.20
9.56
Table Needles score at 128k for all model variants
For the RoPE baseline model and all RNoPE variants, we report the needles evaluation score of all
SFT models sequence length of 128,000 in Table 3] We also display the attention mass of all
variants in Figure 2 The attention mass is aggregated separately for all RoPE and NoPE layers. For
5

---

simplicity, we present results based on samples with 32,000 sequence lengths, with the complete table for all sequence lengths provided in Appendix
The results in Table 3 reveal that increasing the RoPE parameter 0 during fine-tuning of the RNoPE variants- NoPE and RoPE layers are interleaved-leade to a decline in the model's long- context capability. In contrast, previous studies on pure RoPE architectures [50] 45 have shown that using a larger RoPE during pre-training or fine-tuning enhances long-context performance and expands the effective attention receptive field. To investigate this discrepancy, we compare the attention mass across different model variants from Figure 2
First, when comparing the NoPE and RoPE layers across all RNoPE variants, we observe a clear divergence in their behavior. The NoPE layers exhibit strong retrieval capabilities, characterized by pronounced spike in attention mass on the "Needle" segment and a phenomenon of attention sink 773 on the "Begin" segment. Additionally, these layers show a significantly weaker recency bias indicated by the low attention mass on the "End" segment. In contrast, the RoPE layers within the RNoPE variants demonstrate extremely weak retrieval performance, evidenced by near-zero attention mass on the "Needle" segment and only modest attention on the "Begin" ingment-indicating an absence of an attention sink. However, these RoPE layers exhibit a much stronger recency bias compared to the pure RoPE baseline. In summary, interleaving RoPE and NoPE layers induces a spontaneous "division of labor" phenomenon, where RoPE layers focus on local information aggregation and NoPE layers specialize in long-range retrieval. Remarkably, this functional specialization emerged naturally during training. without any explicit training constraints, data augmentation strategies, or specific loss function designs.
Next, we examine the RNoPE variants fine-tuned with different @ values. As & increases, we observe a consistent decline in the recency bias of the RoPE layers. Specifically, the attention mass on the "End" segment drops from 0.314 in RNoPE-1OK 0.223 in RNoPE-4M, indicating a flatter attention distribution that reaches deeper into the context. This aligns with prior findings and theoretical analyses suggesting that larger values expand the effective receptive field of the attention mechanism [50]. However, our empirical results indicate that this expanded receptive field in the RoPE layers introduces noise into the overall architecture, which disrupts the NoPE layers' ability to compute similarities and perform retrieval effectively. This degradation is reflected in both attention mass and task performance: the needle attention mass in the NoPE layers drops from 0.0765 to 0.0369, and the needle evaluation score decreases from 8.036 to 6.203 as O increases from 10,000 to 4 million. These findings further underscore that the model spontaneously develops a "division of labor" mechanism during training, with distinct roles emerging between RoPE and NoPE layers.
From these observations, we draw the following insights:
1. Division of Labour: Combining NoPE and RoPE layers yields complementary strengths, with each type naturally assuming specialized roles after training. NoPE layers are adapted to information retrieval, while RoPE layers become effective at modeling local context due to their inherent recency bias.
2. Potential Efficiency Gains: Restricting the effective attention span of the RoPE layers in RNoPE models can reduce noise in the attention distribution and reinforce the functional specialization of each layer. Additionally, it lowers the computational cost (FLOPs) during for longer context teheths-thereby improving training efficiency while maintaining or even enhancing performance.
Guided by these insights, we fine-tune a new variant, RNoPE- where "swa" denotes sliding window attention. This modification imposes a hard limit on the attention span of RoPE layers, operationalizing the second insight above. Specifically, the sliding window size for RoPE layers is set to 8, 192,, while the NoPE layers retain full attention to support long-context retrieval. All other training configurations remain identical to the RNoPE- 10k variant, including the use of # = 10,000. Evaluation results, presented in Table 3, show a marked improvement. The RNoPE- 10k-swa variant achieves a meedles-128k score of 9.562, surpassing both the baseline and the original RNoPF- model. Moreover, its NoPE layers exhibit well-structured attention pattern with high attention mass on both the "Begin" and "End" segments from 2 reflecting strong long-context retrieval capabilities.

---

NoPE Full Attention Span
RoPE SWA
RoPE SWA
RoPE SWA
NoPE Full Attention Span
RoPE SWA
RoPE SWA
RoPE + SWA
NoPE Full Attention Span
Figure 3: RNope-SWA Model Architecture. SWA stands for sliding window attention.
3 Model Architecture
Based on the analysis above, we make the following design choices on top of the Command R+ architecture [17]. First, we remove the QK-Norm component due poorly shaped attention patterns, which adversely impacted long context performance. Second, NoPE layers with a full attention span are introduced to enhance the model's 's retrieval capabilities. Third, a sliding window size of 4,096 is applied to RoPE embeddings, leveraging RoPE's inherent recency bias to improve performance on moort-to-medium context ranges. particular, the bliding-window approach has been employed in several prior works [6]]]]]]]]]]]]]]]]]]]]]]]]] 11. Regarding the number of layers, we perform an ablation study on the interleaving ratio of full attention and sliding window layers, testing the configurations of 1:1, 1:3, and 1:7. The results show that 1:3 ratio strikes an optimal balance between computational efficiency and performance. We position the full attention layers at the end of each interleaving group, preceded by three sliding window layers. All other hyperparameters for the model architecture remain consistent with those outlined Table A visualization is shown Figure]
For additional context, the resulting design shares similarities with other well-explored long-sequence For example, 48 and
architectures, such as Mega 48 and state-space models (SSMs) 28 27
[49 introduced a multi dineiiiaimenionnll damped version of an exponential moving average component conjunction with the gated attention unit (GAU) architecture B3 aiming to balance local and long-term dependencies, a common challenge in time-series modeling. Similarly, previous studies have proposed a diagonal variant of the $4 architecture (28), incorporating an exponentially decaying measure to enhance the model's s ability to capture long-range dependencies
655
Earlier transformer variants, such as GPT-J [71] and GPT-NeoX (8), explored hybridizing RoPE and NoPE by applying rotational embeddings to a subset of the head dimensions. The work of P-RoPE 66] further advanced this line of research by analyzing RoPE in the frequency domain, revealing that low-frequency components capture semantic relationships while high-frequency components encode positional information. Similar prior Partial-RoPE approaches, p- p-RoPE removes selected low-frequency components along the head dimensions. These findings align with our observation that NoPE primarily facilitates retrieval (semantic focus), whereas RoPE exhibits a stronger recency bias (positional focus). However, prior works did not identify the cross-layer patterns underlying the "division of labor" phenomenon described in Section 2.2.3] which enables our approach achieve both improved performance and practical engineering benefits.
7

---

Model
MMLU
HellaSwag
ARC-E
ARC-C
SATEn
SATMath
GSM8K
Winogrande
MBPP
Baseline
57.5
75.8
84.6
48.5
70.0
30.9
40.9
68.5
39.1
RNope-SWA
59.5
76.2
82.5
48.8
71.9
30.5
42.7
69.5
39.3
Table 4: Comparison of models on a variety of benchmarks. All the evaluations are based on the performance of the SFT-models.
4 Experiments
In this section, we detail the experiments conducted on the model architectures, covering the stages of pretraining, cooldown, supervised fine-tuning, and evaluations. Alongside long context evaluations, we provide a comprehensive assessment of short-context benchmarks, a dimension often disregarded in other long context studies. We train two models: one with the RoPE architecture a baseline and another employing the architecture introduced in Section B
Pretraining and Cooldown. The models are pretrained for 5 trillion tokens of diverse data with batch size of 8 million tokens using FP8 precision format [51]. We use cosine learning rate schedule of 5c-3 peak learning rate and 5% end learning rate with 8,000 linear warmup steps. From the pre-trained model, we linearly anneal the learning rate from 2.5c-4 to le-6 for 50,000 steps in BF16 precision. The context length was initially maintained at 8k for the first 35,000 steps, then extended to 32k and 128k for 10,000 steps and 5,000 steps, respectively. For the baseline model, the RoPE 0 values were increased 1,000,000 for 32k and 8,000,000 for 128k contexts during the length extrapolation phase, while remaining constant for the RNope- same interleaved training strategy outlined in Section 2.1
variant. Both models utilized the
Supervised Finetuning. We supervise fine-tune on top of the pretrained models. As the primary focus of this study is to evaluate the impact of architectural design on downstream tasks, preference training is deferred to future work. To preserve the long context capabilities of the model, the fine- tuning process utilizes interleaved datasets containing 8k and 128k prompt-response pairs. The long context SFT data at 128k includes Eatrieval-Augmentee Generation (RAG) 40 datasets, multilingual translation datasets with extended passages, long code instruction datasets, and long context retrieval datasets. Training was performed for two epochs across the entire dataset.
Experimental Results
Our evaluation contains comprehensive analysis over standard benchmarks below 8k context length such as MMLU (30). HellaSwag (79), ARC [13], SAT [82], GSM8k (16), Winograde [58] and MBPP [], well popular long context benchmarks with needles-in -asss-iii----htteeee [37] and the retrieval and QA portion of Ruler (32). We test the context lengths up to 256k so we can examine the impact of these choices in the extrapolation capability of the model. We denote the baseline model trained with RoPE scaling as "Baseline" and the architecture with interleaved attention span and position embeddings as "RNope-SWA"..
5.1 Standard Benchmarks
In this set of evaluations, we evaluate baseline and RNope-SWA on a standard LLM benchmark covering various language, math and code capabilities. The results shown in Table 4 We observe that the model is better or on-par on most of the metrics compared to the baseline and has some gains over the baseline numbers certain benchmarks (+2.0% on MMLU and + +1.8% on GSM8k). This set of results also indicates that although RNope-SWA explicitly removed position embeddings from 25% of all its layers, positional information is retained by the interleaving attention span and captured by RoPE from previous layers. RNoop-SWA does not have the performance loss due to the removal
of explicit position embeddings, as previous works have shown 38 72
5.2 Long Context Benchmarks
To evaluate the long context performance of these models, we use NIAH and the retrieval and QA components of Ruler [32]. To better understand how architectural choices affect long context

---

Model Needles-1288
Needles-256k
Baseline
9.99
8.25
RNope-SWA
9.99
9.97
Table Needles Score of Baseline and RNoPE-SWA up 256k sequence length.
performance, also evaluate with context lengths extending beyond training sequence length. This allows us to assess how well these models can interpolate as well as extrapolate to unseen context lengths and how specific architectural decisions influence these capabilities.
5.2.1 NIAH Evaluations
Following the settings of section in Table The figure indicates that although both models are able to get close to perfect scores below the context length achieves almost no loss up to 256k context length while the baseline fails to extrapolate well - despite using high RoPE 9 value of million. We attach the graphical visualization with depth and length dimension expanded in Appendix
run NIAH test 256k context lengths. The scores are reported
during training, RNope-SW,, SWA has better extrapolation capabilities and
5.2.2 Ruler Evaluations
Model
8k
16k
32k
64k
128k
256k
Baseline
96.6
94.4
95.1
89.1
83.0
57.1
RNope-SWA
96.1
96.1
94.9
92.0
90.0
74.8
Table 6: Ruler Retrieval Evaluation
Model
8k
16k
32k
64k
128k
256k
Baseline
53.5
50.0
52.5
45.5
36.0
30.0
RNope-SWA
55.5
52.5
55,5
49.0
46.0
42.5
Table 7: Ruler QA Evaluation
First introduced 33). the Ruler benchmark aims to provide a set of more difficult tasks than NIAH. It covers wider range of retrieval under Multi- Bullti-uuery/Key/Vaal
settings, more realistic tasks with a long-context Question-Amsn Answering format and more. Although our modification of NIAH introduced more context variants and proves be difficult than the vanilla version, still cannot test the limits of the model. Therefore, we evaluate our models on the retrieval and QA portion of the Ruler can better separate their performance.
From the results, we can observe that the baseline model with RoPE scaling approach suffers from a sharp drop in the longer context range, especially 64k and longer. Comparing the difference between the scores obtained at 8k and 256k, the baseline model dropped from 96.6 to 57.1 1 (about 41%) on retrieval and from 53.5 to 30.0 (about 44%) on QA, whereas the RNope-SWA model dropped from 96.1 to 74.8 (about 22. 1%) on retrieval and from 55.5 to 42.5 (about 23.4% on QA respectively. From the original Ruler Paper [32,, models that adopt similar RoPE scaling approaches have shown similar degradation over longer context lengths 22 75 as the baseline.
5.3 Impacts on Training and Inference
We also report the differences in training and inference speed, as well as memory requirements, of RNope-SWA compared to the baseline model. Let S denote the sliding window size and L represent full training context length. During training. 75% all layers now operate with computational complexity of O(SL), rather than the quadratic complexity of O(I.)). This results in the model being approximately 50% faster than the baseline at a 64k context length and nearly 2x faster 128k in
terms of training throughput, using flash attention 20
19 59 and requence-paralle scheme similar

---

or its variants [9]. sliding window adoption can reduce key-v block communication overhead if carefully sharded along the sequence dimension, potentially improving speed. In theory, KV cache memory and time complexity of RNNOPESWAA can be reduced by up to 75%. Empirically, we observed a 44% latency reduction at 132k input and 96 output tokens, and nearly 70% at 990k input and 8 output tokens-appen appenn-appon the theoretical limit as sequence length grows. Increasing the ratio of sliding window to full attention layers can further improve speed and memory efficiency.
to [35 76
With alternative implementations, such as Ring Attention 143 44
6 Discussions and Future Work
In this paper, we introduced RNope-SWA, an architecture that interleaves NoPE and RoPE position embeddings with varying attention spans (RNope-SWA). RNope-SWA is able to strike a balance between effective attention modeling and computational efficiency, achieving nearly 4x reduction in KV cache size and significantly boosting both training and inference speeds without compromising performance. The integration of NoPE layers with full attention spans enhances long context capabilities, eliminating the need for RoPE scaling. This simplification improves the stability of training and delivers excellent long context performance.
Our findings align with recent work, such as YoCo (62), Jamba-1.5 668, and MiniMax-01 [52). which demonstrate that hybrid attention mechanisms generally outperform full attention mechanisms in handling long contexts. However, the underlying reasons behind this seemingly counterintuitive observation remain largely unexplored. This opens an intriguing area of study, particularly as models push towards muult-million-te components of transformer architectures, such as attention mechanisms, may become essential to accommodating these extreme requirements. Recent works
context lengths. Re-visiting and re-thinking the foundational
[77 39 60 have begun to explore this direction by focusing reducing attention noise across large context windows, a promising approach to refine the performance of attention modules.
References
[1] M. Abdin, J. Aneja, H. Awadalla, A. Awadallah, A. A. Awan, N. Bach, A. Bahree, A. Bakhtiari, J. Bao, H. Behl, A. Benhaim, M. Bilenko, J. Bjorck, S. Bubeck, M. Cai, Q Cai, V. Chaudhary, D. Chen, D. Chen, W. Chen, Y.-C. Chen, Y.-L. Chen, H. Cheng, P. Chopra, X. Dai, M. Dixon, R. Eldan, V. Fragoso, J. Gao, M. Gao, M. Gao, A. Garg, A. D. Giorno, A. Goswami, S. Gu- nasekar, E. Haider, J. Hao, R. J. Hewett, W. Hu, J. Huynh, D. Iter, S. A. Jacobs, M. Javaheripi, X. Jin, N. Karampatziakis, P. Kauffmann, M. Khademi, D. Kim, Y. J. Kim, L. Kurilenko, J. R. Lee. Y. T. Lee, Y. Li, Y. Li, C. Liang, L. Liden, X. Lin, Z. Lin, C. Liu, L. Liu, M. Liu, W. Liu, X. Liu, C. Luo, P. Madan, A. Mahmoudzadeh, D. Majercak, M. Mazzola, C. C. T. Mendes, A. Mitra, H. Modi, A. Nguyen, B. Norick, B. Patra, D. Perez-Becker, T. Portet, R. Pryzant, H. Qin, M. Radmilac, L. Ren, G. de Rosa, C. Rosset, S. Roy, O. Ruwase, O. Saarikivi, A. Saied, A. Salim, M. Santacroce, S. Shah, N. Shang, H. Sharma, Y. Shen, S. Shukla, X. Song, M. Tanaka, Wang, G. Wang. L. Wang, S. Wang. X. Wang. Y. Wang. R. Ward, W. Wen, P. Witte, H. Wu, X. Wu, M. Wyatt, B. Xiao, C. Xu, J. Xu, W. Xu, J. Xue, S. Yadav, Yang, Yang, Y. Yang, Yang. D. Yu, L. Yuan, C. Zhang, C. Zhang, J. Zhang, L. L. Zhang, Y. Zhang, Y. Zhang, Y. Zhang, and X. Zhou. Phi-3 technical report: A highly capable language model locally on your phone, 2024. htRt::tts:://are
A. Tupini, P. Vaddamanu,
org/ abs/ //0// 14219
[2] AI, A. Young, B. Chen, C. Li, C. Huang, G. Zhang, G. Zhang, H. Li, J. Zhu, J. Chen, J. Chang, K. Yu, Liu, Q. Liu, S. Yue, S. Yang, S. Yang, T. Yu, W. Xie, W. Huang, X. Hu, X. Ren, X. Niu, P. Nie, Y. Xu, Y. Liu, Y. Wang. Y. Cai, Z. Gu, Z. Liu, and Z. Dai. Yi: Open foundation models by O1.ai, 2024. URL htt.htttp//xxiiiii http:///xxvv.oog//aaa wwt./bo//////a..eo..eo. 240.a///2.....m 2etps//44000466222
[3] J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang. C. Cai, M. Terry. Q. Le, and C. Sutton. Program synthesis with large language models, 2021. URL https //arxiv. ptg/abs//108.001
[4] I.. Ba, R. Kiros, and G. E. Hinton. Layer normalization, 2016. URL https : //arxiv org/ abs/ 1600/ 06450
10

---

[5] G. A. Baker, A. Raut, S. Shaier, L. E. Hunter, and K. von der Wense. Lost in the middle, and ne-beeween: Enhancing language models' ability to reason over long contexts in multi-hop qa. 2024. URL (RR.https: h 10079
[6] F. Barbero, A. Vitvitskyi, C. Perivolaropoulos, R. Pascanu, and P. Veličković. Round and round we go! what makes rotary positional encodings useful?, 2025. URL httpt:htts/:xiiiiii org/
abs abs/2410. 0es5/410006205 06205
[7] Beltagy, M. E. Peters, and A. Cohan. Longformer: The long-document transformer, 2020.
httpstt:: //wrxvvavvvv////////a//aaa/aaava org/ absaabss//00 05150
[8] S. Black, S. Biderman, E. Hallahan, Q. Anthony, L. Gao, L. Golding, H. He, C. Leahy, K. McDonell, J. Phang, M. Pieler, U. S. Prashanth, S. Purohit, L. Reynolds, J. Tow, B. Wang, and S. Weinbach. GPT-NeoX-20B: An open-source autoregressive language model. In A. Fan, S. Ilie, T. Wolf, and M. Gallé, editors, Proceedings of BigScience Episode #5 - Workshop on Challenges & Perspectives in Creating Large Language Models, pages 95-136, Dubbunl+ May 2022. Association for Computational Linguistics. doi: 10.18663/11/22222bggeee URL httpps://aclanthoooogyy org/ wttp:///0......ccceeee--........ bigscience-1.99
1.9.
[9] W. Brandon, A. Nrusimha, K. Qian, Z. Ankner, T. Jin. Z. Song. and J. Ragan-Kelley. Striped attention: Faster ring attention for causal transformers, 2023. URL https://arxiv/ org/ abs/2311.0943)
[10] Z. Cai, Y. Zhang, B. Gao, Y. Liu, T. Liu, K. Lu, W. Xiong, Y. Dong, B. Chang, J. Hu, and W. Xiao. Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling, org/ abs abw////b....../////// .20669
2024. URL WtRLhhtps:///rrtt
[11] Character.AL. Optimizing ai inference character.ai, 2024.
[12] S. Chen, Wong, L. Chen, and Y. Tian. Extending context window of large language models via positional interpolation, 2023. URL https://///// wwtp://wwww..rraa//////////0/ 15595
[13] T.-C. Chi, T.-H. Fan, P. Ramadge, and A. I. Rudnicky, Kerple: Kemelized relative positional embedding for length extrapolation, 2022. URL URtpssts://arxi http://aaxxvvvog//////// argg/bs///ae htta//225.09999
[14] R. Child, S. Gray, A. Radford, and I. Sutskever. Generating long sequences with sparse transformers, 2019. URL https: arxiv. wwwrxxv.oog//abeee /ttp://www..//////////....00./.0 10509
[15] Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018. URL https //arxiv. org/ satrivv.orr//bs/8003 05457
[16] K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman. Training verifiers solve math word problems, 2021. URL https://s///r wtt/ 14168
[17] Cohere For AL. clai-coomman---plus (revision 432fac1), 2024. URL/tttss//huugigggaacc co/CohereForAle wow.aa--ooonnddd--ll.aaa
[18] J. Dang, S. Singh, D. D'souza, A. Ahmadian, A. Salamanca, M. Smith, A. Peppin, S. Hong, M. Govindassamy, T. Zhao, S. Kublik, M. Amer, V. Aryabumi, A. Campos, Y.C. Tan, T. Kocmi, F. Strub, N. Grinsztajn, Y. Flet-Berliac, A. Locatelli, H. Lin, D. Talupuru, B. Venkitesh, D. Cairuz, B. Yang. T. Chung, W.-Y Ko, S. S. Shi, A. Shukayev, S. Bae, A. Piktus, R. Castagné, F. Cruz-Salinas, F. Kim, L. Crawhall-Stein, A. Morisot, S. Roy, P. Blunsom, I. Zhang, A. Gomez, N. Frosst, M. Fadaee, B. Ermis, A. Üstün, and S. Hooker. Aya expanse: Combining research breakthroughs for multilingual frontier, 2024. BRRLLttpp:://arri.oo/////////// 04261
[19] T. Dao. Flashattention-2: Faster attention with better parallelism and work partitioning, 2023. URL https::////// wttp:/iww.rr/ab//////a// www.o////../..oo/.////// 08691
[20] Dao, Fu, S. Ermon, A. Rudra, and C. Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness, 2022. htLLstt:::// //arxivv http://..wrr///////05./../ 14135

---

[21] M. Dehghani, J. Djolonga, B. Mustafa, P. Padlewski, J. Heek, J. Gilmer, A. Steiner. M. Caron,
R. Geirhos. I. Alabdulmohsin, R. Jenatton, L. Beyer, M. Tschannen, A. Amab. X. Wang.
C. Riquelme, M. Minderer, J. Puigcerver, U. Evei, M. Kumar, S. van Steenkiste, G. F. Elsayed, A. Mahendran, F. Yu, A. Oliver, F. Huot, J. Bastings, M. P. Collier, A. Gritsenko, V. Birodkar, C. Vasconcelos, Y. Tay, T. Mensink, A. Kolesnikov, F. Pavetić, D. Tran, T. Kipf, M. Lučié,
X. Zhai, D. Keysers, J. Harmsen, and N. Houlsby, Scaling vision transformers to 22 billion
parameters, 2023. w
https:// /ttpa:///axvv../////
[22] Ding, S. Ma, L.. Dong, X. Zhang, Huang, W. Wang, N. Zheng, and F. Wei. Longnet: Scaling transformers to 1,000,000,000 tokens, 2023.
http:///axxvvvv.o org/ abs http/2307.00200
[23] Y. Ding, L. Zhang, C. Zhang, Y. Xu, N. Shang, J. Xu, F. Yang, and M. Yang. Longrope: Extending Ilm context window beyond 2 million tokens, 2024. URL https:htttssiiiii org/ ibs/2402.133 13753
[24] A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, org/
A. Yang, and e. Angela Fan. The llama 3 herd of models, 2024. URL
abs/2401 21783
[25] T. Gao, A. Wettig, H. Yen, and D. Chen. How to train long-context language models (effectivelyy. 2024. Utttttts:://aan //axttps://xxi...../ wwwaaiiiw.../raaaeeeee hwrg/abb/2410 000.//0620.00000000
[26] O. Golovneva, T. Wang. J. Weston, and Sukhbaatar. Contextual position encoding: Learning to count what's important, 2024. URL http:://axxx. http:////aaxxi.oooaaaa/////s//// wwt.:///..ab.......///// 18719
[27] A. Gu and T. Dao. Mamba: Linear-time sequence modeling with selective state spaces, 2024. URL
[28] A. Gu, K. Goel, and C. Ré. Efficiently modeling long sequences with structured state spaces, wwtgg/bb//11111.1 00396
2022. URL Uttphttss:///////
[29] A. Haviv, O. Ram, O. Press, P. Izsak, and O. Levy. Transformer language models without positional encodings still learn positional information. In Y. Goldberg, Z. Kozareva, and Y. Zhang, editors, Findings of the Association for Computational Linguistics: EMNLP 2022, pages 1382-1390, Abu Dhabi, United Arab Emirates, Dec. 2022. Association for Computational -erddings-emml.9..... URL BuRLatttps///allattoloogg org/
Linguistics. doi: (0111653/11/222..mmll
2022. findings-nnnnninn www.p..mmml..9999.
[30] D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding, 2021. URL https://arxiv wwwpp/i...rr/bb//////oo/a 03300
[31] A. Henry, P. R. Dachapally, S. S. Pawar, and Y. Chen. Query-key normalization for transformers. In T. Cohn, Y. He, and Y. Liu, editors, Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4246-4253, Online, Nov. 2020. Asso- ciation for Computational Linguistics. doi: 11116653/1/2022.fnntt findingg-eemll.379. URL www.llogy.orr//// www./020.000.....oo http:/222..f.nnnnnnn..aef findings-emnlln 379/
[32] C.-P. Hsich, S. Sun, S. Kriman, S. Acharya, D. Rekesh, F. Jia, Y. Zhang, and B. Ginsburg. Ruler: What's the real context size of your long-context language models?, 2024. URL https://arxiv. org/ 200o://244.0000 06654
[33] W. Hua, Z. Dai, H. Liu. and Q V. Le. Transformer quality in linear time, 2022. URL https://arxi: org/ abs/ /00p://20...2222 10447
[34] Y. Huang, J. Xu, J. Lai, Z. Jiang, T. Chen, Z. Li, Y. Yao, X. Ma. L. Yang, H. Chen, Li, and P. Zhao. Advancing transformer architecture in long-context large language models: A comprehensive survey, 2024. URL httph: /tttpss//arxiv..oo atgg/bbs/23111 12351
[35] S. A. Jacobs, M. Tanaka, C. Zhang, M. Zhang. S. I.. Song, S. Rajbhandari, and Y. He. Deepspeed ulysses: System optimizations for enabling training of extreme long sequence transformer models, 2023. URL sittps://arxxn wttp:/iivvrrg/b/////0000 abtpa//bb///////////// 14509
12

---

[36] A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford. D. S. Chaplot, D. de las Casas, F. Bressand, G. Lengyel. G. Lample, L. Saulnier, L. R. Lavaud, M.-A. Lachaux, Stock, T. Scao. T. Lavril.
T. Wang. T. Lacroix, and W. E. Sayed. Mistral 7b, 2023. https:t::/// pttttp::/aaxxv.orrter /bs/
2310.. 2310.06825
[37] G. Kamradt, Needle in haystack-pressure testing 1ms,, 2023.
[38] A. Kazemnejad, I. Padhi, K. N. Ramamurthy, P. Das, and S. Reddy, The impact of positional encoding on length generalization in transformers, 2023. htttps:::///rtrt org/ abs/
2305. 19466)
[39] Y. Leviathan, M. Kalman, and Y. Matias. Selective attention improves transformer, 2024. URL https://arxiv. org/ abs 2000///2000020
[40] P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W. tau Yih. T. Rocktäschel, S. Riedel, and D. Kiela. Retrieval-auggmente generation for knowledge- intensive nlp tasks, 2021. URL http:: hates://arrxii www./ogg////ooooooo http/bbb/200555 11401
[41] Y. Li, Y. Huang, B. Yang, B. Venkitesh, A. Locatelli, H. Ye, T. Cai, Lewis, and D. Chen. Snapkv: Lim knows what you are looking for before generation, 2024. URL https: //arxiv org/abs/2404. 14469
[42] Z. Li, J. Zhang, Q. Lin, J. Xiong. Y. Long, X. Deng, Y. Zhang. X. Liu, M. Huang, Z. Xiao, and D. C. et al. Hunyuan-dit: A powerful multi-resolution diffusion transformer with fine-grained chinese understanding, 2024. URL
08748
[43] H. Liu and P. Abbeel. Blockwise parallel transformer for large context models, 2023, URL https: hopps://arxivv org/ abs 2305. 19370
[44] H. Liu, M. Zaharia, and P. Abbeel. Ring attention with blockwise transformers for near-infinite context, 2023. URL http::ttt:///t http:///wwwwww...oraa/ra////
01889.
[45] H. Liu, W. Yan, M. Zaharia, and P. Abbeel. World model on millioonlengthh video and language with blockwise ringattention, 2024. URL htttttpp://ai...oor///oaa// 08262
[46] N. F. Liu, K. Lin, J. Hewitt A. Paranjape, M. Bevilacqua, Petroni, and P. Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157-173, 2024.
doi: 10.1162/ucc a 00638. URL https
//aclanthology .rg 2024. 2ttp:/2222aaaac..999
[47] X. Liu, H. Yan, S. Zhang, C. An, X. Qiu, and D. Lin. Scaling laws of rope-based extrapolation, 2024. Uttttttps://////e //arxiv argg/bbs//11//0 /2300000000009
[48] X. Ma, C. Zhou, X. Kong, J. He, L. Gui, G. Neubig, J. May, and L. Zettlemoyer. Mega: Moving average equipped gated attention, 2023. URL htttp::/aaxiisi wwwpexiivv..rraraeoe hwwpg/abb/2209... 10655
[49] X. Ma, X. Yang. W. Xiong, B. Chen, L. Yu, H. Zhang, J. May, L. Zettlemoyer, O. Levy, and
C. Zhou. Megalodon: Efficient I1m pretraining and inference with unlimited context length,
2024. JtLhhttps::/ //axxiv org/ abs// 088001
[50] X. Men, M. Xu, B. Wang, Q. Zhang, H. Lin, X. Han, and W. Chen. Base of rope bounds context length, 2024.
org/ org/abs/ abs//405..1 14591
[51] P. Micikevicius, D. Stosic, N. Burgess, M. Cornea, P. Dubey, R. Grisenthwaite, S. Ha, A. Hei- necke, P. Judd, J. Kamalu, N. Mellempudi, S. Oberman, M. Shoeybi, M. Siu, and H. Wu. Fp8 formats for deep learning, 2022. URL htth::ts: http://xxvvv.oooo org/ absaaaaa..////////////// 05433
[52] MiniMax, A. Li, B. Gong, B. Yang, B. Shan, C. Liu, C. Zhu, C. Zhang, C. Guo, D. Chen, D. Li, E. Jiao, G. Li, G. Zhang. H. Sun, H. Dong, Zhu, Zhuang. J. Song. J. Zhu, J. Han, J. Li,
J. Xie, J. Xu, J. Yan, K. Zhang, K. Xiao, K. Kang, L. Han, L. Wang, L. Yu, L. Feng. L. Zheng,
L. Chai, L. Xing, M. Ju, M. Chi, M. Zhang, P. Huang, P. NIu, P. Li, P. Zhao, Q. Yang, Q. Xu,
Q. Wang, Q. Wang. Q. Li, R. Leng, S. Shi, S. Yu, S. Li, S. Zhu, T. Huang, T. Liang, W. Sun, W. Sun, W. Cheng, W. Li, X. Song, X. Su, X. Han, X. Zhang, X. Hou, X. Min, X. Zou, X. Shen,
Y. Gong, Y. Zhu, Y. Zhou, Y. Zhong, Y. Hu, Y. Fan, Y. Yu, Y. Yang, Y. Li. Y. Huang, Y. Li,
13

---

Y. Huang, Y. Xu, Y. Mao. Li, Z. Li, Z. Tao, Z. Ying, Z. Cong, Z. Qin, Z. Fan, Z. Yu, Z. Jiang, and Z. Wu. Minimax-01. 01: Scaling foundation models with lightning attention, 2025. URL https://arxiv www.
08313
[53] A. Mohtashami and M. Jaggi Landmark attention: Random-accen infinite context length for transformers, 2023. http://ps///axi..orrrraaa//////a
16300
[54] B. Peng, J. Quesnelle, H. Fan, and E. Shippole. Yam: Efficient context window extension of large language models, 2023.
[55] O. Press, N. A. Smith, and M. Lewis. Train short, test long: Attention with linear biases enables input length extrapolation, 2022. URL
12409
[56] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer, 2023. URL https://arxiv httpe://arriv..0g//bb///9110 10683
[57] O. Rybakov, M. Chrzanowski, P. Dykas, J. Xue, and B. Lanir. Methods of improving llm training stability, 2024. URL httth:tttarrritii org/ abs abs///a/...././///// 16682
[58] K. Sakaguchi, L. Bras, C. Bhagavatula, and Y. Choi. Winogrande: An adversarial winograd schema challenge at scale, 2019. URL https:tt/////iii/ www.aaawww.....raa/aaaaaaaaaaaa/ httg//bs/1900/ 10611
[59] J. Shah, G. Bikshandi, Y. Zhang, V. Thakkar, P. Ramani, and T. Dao. Flashattention-3: Fast and accurate attention with asynchrony and low-precision, 2024. URL htttps:///axx org/ abs/2407 .08608
[60] Y. J. Soh, H. Huang, Y. Tian, and J. Zhao. You only use reactive attention slice for long context retrieval, 2024. httpss:s:://rrt //aniiv.//xvvvv/// http://axxxvv...aaaaa./ www.oogg////aeeee www.aabee/24000. 13695
[61] J. Su, Y. Lu, S. Pan, A. Murtadha, B. Wen, and Y. Liu, Roformer: Enhanced transformer with rotary position embedding. 2023. URL htttps:////xxx www..orggggboeoa htsp/bbb/22004 09864
[62] Y. Sun, L. Dong, Y. Zhu, S. Huang, W. Wang, S. Ma, Q. Zhang, J. Wang, and F. Wei. You only cache once: Decoder-decoder architectures for language models, 2024. URL https
//arxiv. org/ Saazxi..rrg/bb//200 abs///bb/24455 05254
[63] A. Talmor, J. Herzig, N. Lourie, and J. Berant. Commonsensequ: A question answering challenge targeting commonsense knowledge, 2019. URL httt:p:://axxiv.orr/bbbs.. 00937
[64] Y. Tay, D. Bahri, L. Yang, Metzler, and D.-C. Juan. Sparse sinkhorn attention, 2020. URL https://arxiv: org/ abs/2002. 11296
[65] Y. Tay, M. Dehghani, S. Abnar,
Y. Shen, D. Bahri, P. Pham, J. Rao, L. Yang, S. Ruder, and D. Metzler. Long range arena: A benchmark for efficient transformers, 2020. URL
httpp://arxiv.orgg httgg/ab/2201111 04006
[66] C. Team. Chameleon: Mixed-modee modal early-fusion foundation models, 2024. URL https
laaxiiv.rrg//a www./oogg//////eeeeeeoo 09818
[67] G. Team, M. Riviere, S. Pathak, P. G. Sessa, C. Hardin, S. Bhupatiraju, L. Hussenot, T. Mesnard, and e. a. Bobak Shahriari. Gemma 2: Improving open language models a practical size, 2024. URL http:::/t/artttt
abg//bbb/2408 00118
[68] J. Team, B. Lenz, A. Arazi, A. Bergman, A. Manevich, B. Peleg, B. Aviram, C. Almagor, C. Fridman, D. Padnos, D. Gissin, D. Jannai, D. Muhlgay, D. Zimberg, E. M. Gerber, E. Dolev, E. Krakovsky, E. Safahi, E. Schwartz, G. Cohen, G. Shachaf, H. Rozenblum, H. Bata, I. Blass, I. Magar, I. Dalmedigos, Osin, J. Fadlon, M. Rozman, M. Danos, M. Gokhman, M. Zusman, N. Gidron, N. Ratner, N. Gat, N. Rozen, O. Fried, O. Leshno, O. Antverg, O. Abend, O. Lieber, O. Dagan, O. Cohavi, R. Alon, R. Belson, R. Cohen, R. Gilad, R. Glozman, S. Lev, Meirom, T. Delbari, T. Ness, T. Asida, T. B. Gal, T. Braude, U. Pumerantz, Y. Cohen, Y. Belinkov, Globerson, P. Levy, and Y. Shoham. Jamba- 15: Hybrid mransformer-mante models at scale, 2024. URL https: //tp/vvavvv/////////v///aa//aaaa www.oosg//////eeoooo/ooooeeeeeet has///bb////..... /408 12570
14

---

[69] S. Tworkowski, K. Staniszewski, M. Pacek, Y. Wu, H. Michalewski, and P. Mitos. Focused transformer: Contrastive training for context scaling, 2023. hattps::/arxiv.orggre.
2307. 03170
[70] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS), pages 6008. 2017. URL PapessaaaPPppp.... pdf
newriss.eurpppp. cc/
[71] B. Wang and A. Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model.
May 2021.
[72] Wang, T. Ji, Y. Wu, H. Yan, I. Gui, Q. Zhang, X. Huang, and X. Wang. Length generalization of causal transformers without position encoding, 2024. URL
2404. 12224
[73] G. Xiao, Y. Tian, B. Chen, S. Han, and M. Lewis. Efficient streaming language models with attention sinks, 2024. https://:///a wwtp:oog//aaba//g/oo 17453
[74] X. Xu. Q. Ye, and X. Ren. task haystack, 2024. URL
honp:/ssstnc.ooooooooooootccctlo language models with lifelong icl and
[75] A. Yang, B. Yang, B. Hui, B. Zheng, B. Yu, C. Zhou, C. Li, C. Li, D. Liu, F. Huang, G. Dong,
H. Wei, H. Lin, Tang, J. Wang, J. Yang, Tu, J. Zhang, J. Ma, J. Yang. J. Xu, J. Zhou,
J. Bai, J. He, J. Lin, K. Dang, K. Lu, K. Chen, K. Yang. M. Li, M. Xue, N. Ni, P. Zhang, Wang. R. Peng, R. Men, R. Gao, R. Lin. S. Wang, Bai. S. Tan, T. Zhu, T. Li, T. Liu, W. Ge,
X. Deng, X. Zhou, X. Ren, X. Zhang. X. Wei, X. Ren, X. Liu, Y. Fan, Y. Yao, Y. Zhang, Y. Wan,
Y. Chu, Y. Liu, Z. Cui, Z. Zhang, Z. Guo, and Z. Fan. Qwen2 technical report, 2024. URL
https://arx/: org/ absaass//aaa 2407 10671
[76] A. Yang, J. Yang, A. Ibrahim, X. Xie, B. Tang, G. Sizov, J. Reizenstein, J. Park, and J. Huang. Context parallelism for scalable million-token inference, 2024. https:tss: org/ 0bs/2441.00777
[77] T. Ye, L. Dong, Y. Xia, Y. Sun, Y. Zhu, G. Huang, and F. Wei. Differential transformer, 2024. URL hhttps:/ h org/ abs /4100 05258
[78] M. Zaheer, G. Guruganesh, A. Dubey, J. Ainslie, C. Alberti, S. Ontamon, P. Pham, A. Ravula, Q. Wang, L. Yang, and A. Ahmed, Big bird: Transformers for longer sequences, 2021. URL
https://arre
org/ abs/ 14062
[79] R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi. Hellaswag: Can machine really finish your sentence?, 2019. URL https: http://axxvvvo..aaar// 1900s, 19830
[80] Zhang, Z. Liu, S. Xiao, N. Shao, Q. Ye, and Z. Dou. Long context compression with activation beacon, 2024. httlhtttp//aar/ http://aaaxai..rr//bb////00.....
[81] Z. Zhang, Y. Sheng, T. Zhou, T. Chen, L. Zheng, R. Cai, Z. Song, Y. Tian, C. Ré, C. Barrett, Z. Wang, and B. Chen. H20: Heavy-hitter oracle for efficient generative inference of large language models, 2023. https:/// http://arxiivooor////////33
abrg/abs/2300
14048
[82] W. Zhong, R. Cui, Y. Guo, Y. Liang, S. Lu, Y. Wang, A. Saied, W. Chen, and N. Duan. Agieval: A numan-centric benchmark for evaluating foundation models, 2023. URL https //arxiv. htrg/abs/200 2304. 06364
[83] A. Üstün, V. Aryabumi, Z.-X. Yong, W.Y Ko, D. D' D'souza, G. Onilude, N. Bhandari, S. Singh, H.-L. Ooi, A. Kayid, F. Vargus, P. Blunsom, S. Longpre, N. Muennighoff, M. Fadaee, J. Kreutzer, and S. Hooker. Aya model: An instruction finetuned open-access multilingual language model, https: //tpxiv/..iiviavaa org/ abs/ abs//402. 07827
2024. URL
15

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No] or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the
relevant information is Not Available.
Please provide short (1-2 sentence) justificatin right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]" ", is perfectly acceptable to answer "[No] "provided proper justification is given (c.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
general, answering
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: The abstract clearly states the contributions of this paper is analyzing ex- isting attention approaches and proposing new hybrid attention architecture with better performance and efficiency.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
16

---

Justification:
The approach described by this paper is robust and verified with ample training and evaluations. Computational efficiency also covered in Section 5.3. We discuss limitation in Section 6 where the reason behind hybrid architectures working better than traditional dense architecture is still underexplored.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-texi system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
This paper is empirical and involve no proof on results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
17

---

This paper gives detailed explanation on the model architecture, training methodology and type of data used. Although we don't 't open source the data or the model, the process should be easy to follow for people who work on this area.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (c.g.. with an open-soure dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification:
We did not open source the data or code for training.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
mablic/guides Sruiddes/ooeSuuiissseeeeeeeeeeee (oodeSummissioooooleee sosiiyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https for more details.
//nips. http:pppwwippppieeeeeeeeeeeaaaaa
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
18

---

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper covers details on training steps, learning rates, nlyper-parmmett optimizers used and the essential information on the data used in Section 2 and Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[NA]
Justification:
The results of this paper is very clear and conspicuous among model variants.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (c.g. negative error rates).
If error bars are reported in tables or plots. The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [No]
19

---

Justification:
Compute resources required to train varies depending on the type of hardware and frameworks used. It is also not very relevant the paper's focus.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: : http///eerripp.eeeee httpc//cccccthticcc.....oeollll
Answer: [Yes]
Justification: This paper completely conforms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This is not related this paper.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation. generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
20

---

Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Not related to this paper.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: This is done properly.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwpppswii thoode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
No assets introduced in this paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
21

---

14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper doesn't involve the crowdsourcing or human subjects).
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based on the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Not applicable this paper.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMss if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification:
The core content of this research does not involve LLMs.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our L.LM for what should should not be described.
22

---

A
Attention Distribution of All Lengths
This table contains the attention distribution of RoPE and RNoPE variants from Section 2.2.2 over 8k, 32k and 128k sequence lengths.
NoPE Layers
RoPE Layers
Context Length
Model
Begin
Needle
Context
End
Begin
Needle
Context
End
RoPE
0.3863
0.0328
0.3809
0.2000
RNoPE-10K
0.3900
0.0952
0.4736
0.0412
0.1255
0,0102
0.5340
0.3302
RNoPE-100K 0.3854
0.0932
0.4783
0.0430
0.2135
0.0136
0.4558
0.3171
8k
RNoPE-2M 0.3775
0.0902
0.4874
0.0449
0.2041
0.0126
0.4952
0.2881
RNoPE-4M 0.4153
0.0546
0.5072
0.0229
0.1389
0.0136
0.6162
0.2313
RNoPE-1OK-swa
0.3830
0.1025
0.4702
0.0443
0.2040
0.0110
0.5938
0.1911
RoPE
0.3541
0.0201
0,4343
0.1915
RNoPE-101 10k
0.3275
0.0765
0.5672
0.0287
0.0049
0.0004
0.6805
0.3142
RNoPE-100K 0.3263
0.0778
0.5633
0.0327
0.0241
0,0005
0.6782
0,2972
32k
RNoPE-2M 0.3250
0.0712
0.5735
0.0303
0.1111
0.0046
0.6233
0.2611
RNoPE-4M 0.3486
0.0369
0.5981
0.0165
0.0960
0.0039
0.6774
0.2227
RNoPE-1OK-swa
0.3303
0.0742
0.5634
0.0321
RoPE
0.3463
0.0010
0,4751
0.1776
RNoPE-10K
0.2991
0.0444
0.6430
0.0135
0.0000
0.0001
0.7230
0.2769
RNoPE-1000 0.2454
0.0419
0.7016
0.0111
0.0001
0.0000
0.7749
0.2250
128k
RNoPE-2M
0.2600
0.0401
0.6836
0.0162
0.0417
0.0008
0.7516
0.2059
RNoPE-4M
0.2949
0.0307
0.6635
0.0109
0.0663
0.0022
0.7115
0.2230
RNoPE-1Ok-swa
0.2760
0.0467
0.6615
0.0159
Table 8: Needles Attention Patter:: RoPE and RNoPE variants
B Attention Distribution of RoPE and QK-Norm variants
In this section, we further investigate the suboptimal performance of the QK-Norm variant. We present three plots comparing the attention distribution between the RoPE and QK-Norm variants across sequence lengths of 8k, 32k, and 128k on needle samples, following the setup outlined in Section 2.2.2 Additionally, we provide the aggregated attention entropy for each variant to quantitatively support the arguments.
To enhance the clarity of the distribution plots, preprocess the attention distribution array by removing the first 10 tokens and the last 3% of tokens from each sequence. This preprocessing step mitigates the disproportionate attention mass resulting from the attention sink effect and the recency bias observed in RoPE, thereby making the attention patterns more interpretable. We then compute moving average with window size of 100 tokens and average the results across all samples and layers to generate the final distributions.
Model
8k
32k
128k
RoPE
6.02
6.95
7.62
QK-Norm
10.71
12.46
14.14
Table 9: Entropy values of aggregated attention distribution
23

---

2.0000
CK Norm
Aggiegatee Attertninn Scones
5.0000
0.00020
0.0005
0.00015
0.0003
0.00100
0.007
5.0001
0.0000
0.90000
0
1000
2000
3000
2000
5000
60C0
7000
2000
3000
30003
15000
70000
25000
30000
Contest Length (max 913
Conteat Length (max 32x
(a) Context Length 8k
(b) Context Length 32k
Jm-3
More
Aggreqptee Artortion
40000
60000
20000
120000
Dontest Length (maa 1283)
(c) Context Length 128k
Figure 4: Attention Distribution Across Sequence lengths
From Figure we observe that the QK-Norm variant exhibits lower spike on needle tokens but distributes more attention mass across context tokens. However, it also demonstrates a stronger recency bias compared the RoPE variant. This characteristic results in lower signal-to-noise ratio for the QK-Norm variant, which hampers its ability to effectively retrieve relevant information from long contexts. To further quantify this observation, we calculate the entropy values of the attention distributions for both variants, averaging across samples and layers at each sequence length. The results, listed in Table 9 show that the QK-Nom variant has significantly higher entropy values than the RoPE variant. This aligns with its weaker performance in long context retrieval tasks, as higher entropy reflects a more dispersed and less focused attention distribution.
C Needles Score 256k
24

---

rating
20
Depth percent
40
60
80
100
50k
100k
150k
200k
250k
Token limit
(a) Baseline
rating
20
40
Depth
60
80
100
Token
(b) RNoPE-SWA
Figure 5: Needle Evaluation of Baseline and RNoPE-SWA on 256k sequence length
25


Reviews:
[{"review_id": "BNJRxktlHq", "reviewer": "Reviewer_qtcq", "summary": "The paper presents a comprehensive analysis of different attention and positional encoding mechanisms in large language models, specifically for handling long contexts. The authors begin by empirically evaluating the performance and attention patterns of models using Rotary Position Embedding (RoPE), No Positional Embedding (NoPE), and Query-Key Normalization (QK-Norm). They identify that NoPE excels at information retrieval over long distances by relying on semantic similarity, but lacks in general performance. Conversely, standard RoPE models with extensions struggle with very long contexts, and QK-Norm, while stabilizing training, degrades long-context performance by creating noisy, unfocused attention patterns.", "questions": "See weakness", "limitations": "No limitations", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n\n- The paper is exceptionally well-written and structured. The narrative flows logically from a clear problem statement, through a comprehensive analysis of existing methods, to the proposal and validation of a new solution. The use of attention pattern visualizations (Fig. 1, 2) provides strong, intuitive evidence for the paper's claims about the behavior of RoPE, NoPE, and QK-Norm, making the motivation for the hybrid model very convincing.\n- The contribution is highly timely. Improving long-context performance while simultaneously increasing computational efficiency is an important goal in LLM research. The proposed RNope-SWA architecture offers a practical and effective solution that addresses both performance and efficiency, a rare combination. The results, particularly the near-perfect scores on NIAH at 256k and the substantially smaller performance drop on Ruler (Table 6), are impressive and demonstrate a clear state-of-the-art advancement over the baseline.\n- While the components themselves (RoPE, NoPE, SWA) are not new, their combination into a functionally specialized, interleaved architecture is novel and insightful. The core idea that different layers can be specialized for different tasks (long-range retrieval vs. local modeling) is powerful. The paper provides strong empirical backing for this \"division of labor\" hypothesis, moving beyond simple engineering to offer a new architectural principle.\n\nWeakness\n\n- The authors state that they performed an ablation study on the interleaving ratio of NoPE to RoPE+SWA layers, testing 1:1, 1:3, and 1:7, and found 1:3 to be optimal (lines 217-219). However, the full results of this important ablation is not presented in the paper or appendix. This is a key design parameter, and its justification currently relies on a statement rather than evidence.\n- The paper makes a key observation that a large RoPE theta value in the hybrid model \"introduces noise into the overall architecture, which disrupts the NoPE layers' ability to...perform retrieval effectively\" (lines 184-186). While the data supports this observation, the paper could benefit from a deeper explanation or hypothesis as to why this occurs. Is it due to the flatter attention distributions from RoPE interfering with the sharp, similarity-based attention of NoPE? A more detailed discussion would strengthen the theoretical underpinnings.", "was_revised": true, "final_justification": "I think this is a great paper, and we should accept it.", "rebuttal": "Thank you for the valuable feedback.\n\n1. Interleaving ratio ablation omitted:\nThank you for highlighting this. We agree that this is an important point, and we will include the full ablation results in the appendix in the revised version. Briefly, our findings are as follows:\n\nIn Section 2, we presented the RNoPE-10k-SWA model using a 1:1 interleaving ratio. We later ablated different ratios and found that 1:3 offered a favorable trade-off between performance and the number of sparse layers. In contrast, a 1:7 ratio introduced excessive sparsity given the model size and number of layers, leading to a noticeable drop in the Needles score. Interestingly, this trend shifted when scaling the model further: larger models with more layers tolerated higher levels of sparsity, which aligns with our intuition that deeper networks can better distribute and specialize functionality.\n\nWe initially omitted these details because we felt they were tangential to the core contributions. Regardless of the exact ratio, any 1:X interleaving configuration yields a theoretical compute cost of 1/(1 + X) compared to a dense baseline model at very long sequence lengths. Thus, the “free lunch” phenomenon—achieving reduced compute with improved accuracy—holds across a range of interleaving ratios.\n\n-----------------------------------------------------------------------------------------------------------------------------------------------\n\n2. Thanks for bringing this up, we think this is also one of the very interesting findings of the paper. We actually gave this lots of thinking but ended up not including too much details about this in the original version of the paper. We will try to add as much as possible in the revised version. Our ideas are actually similar to yours (flatter attention distributions from RoPE interfering with the sharp, similarity-based attention of NoPE):\n\n- This paper has shown that the original dense, rope-based transformer architecture has lots of noise in its attention mechanism after training. These noise prevents the model from forming useful attention patterns, especially at longer context. We think transformer models try to \"fight\" this design flaw during gradient descent but is not able to completely denoise its attention. This could be why when we visualize the attention distributions of the needles, there are still significant mass on the irrelevant contextual tokens (ex. with QK-Norm being the highest).\n\n- From our understanding, the \"division of labor\" that happens in Figure 2 indicates that by constructing the model with interleaved NoPE/RoPE layers, gradient descent will \"push\" the NoPE layers to learn information retrieval more than RoPE layers since NoPE doesn't have recency bias built in at all, whereas the RoPE layers will be \"pushed\" to aggregate local information. We think how well this system can be trained following this setting depends on how much inductive bias we give to each type of layers. increasing RoPE’s theta value expands its receptive field, increasing its tendency to attend to distant tokens and introducing more noise into the system. Cutting the context off with sliding window is strictly imposing this inductive bias to RoPE layers to make sure they solely focus on recent contexts and introduces less obstacle for the model to follow the \"division of labor\" phenomenon during training.\n\nWe believe this phenomenon warrants further theoretical analysis, including the development of formal bounds or derivations to better explain the observed effects. Some related insights can be found in the minimax-text-01 report (https://arxiv.org/pdf/2501.08313), particularly Section 2.2.4, which we found to be highly thought-provoking.\n\n\nWe sincerely appreciate your thoughtful review and look forward to incorporating this feedback to strengthen our paper."}, {"review_id": "5kRTg51HLA", "reviewer": "Reviewer_V4bg", "summary": "The paper proposed to mix NoPE and RoPE layer by layer interleaved so that it can get the advantages from both methods. NoPE for better retrieval ability, while RoPE for better local ability. They made some observations on the attention distribution of the models, and then use the proposed method to show improvements on the downstream tasks of long context retrieval. Performance gains is more obvious when the context length goes up.", "questions": "- What does qc mean in the figure 2? If it’s “query and completion”, please explicitly say it.\n- Why is NoPE not included as a baseline?\n- Can you add NoPE and NoPE+SWA to the ablation?\n- Can you include more ablation on the design choices?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 1, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Weaknesses:\n- Why didn't the experiment show NoPE as a baseline? I guess NoPE will still outperform NRoPE on some retrieval tasks, and it will also be good at short context tasks.\n- Insufficient ablations: a good ablation of the method will include the following ablated variants:\n    1. RoPE\n    2. RoPE with swa\n    3. NoPE\n    4. NoPE with swa.  \nFor both accuracy performance and efficiency performance angles to discuss. The current experiments are only with a single “Baseline” which is not professional and not able to prove the effectiveness/efficiency of RNoPE-swa. If efficiency is important, you should change section 5.3 into a table to compare the efficiency of all different variants.\n- The evenly stacking of RoPE and NoPE one by one is not well supported by ablations either. In my opinion, the first few of transformer early layers should be encoding more local informations, so it’s more useful to put RoPE-swa, while the later layers can encode more of the global information that requires NoPE.\n- Again, the ablation of how to mix RoPE/NoPE layers is only in a sentence of “we perform an ablation study on the interleaving ratio of full attention and sliding window layers, testing the configurations of 1:1, 1:3, and 1:7.” Where are your ablation results? How can I trust this sentence without the numbers/significance differences to back it up? There are no tables/numbers for it. The whole paper so far makes me feel like a very non-professional work from an undergrad course project.", "was_revised": true, "final_justification": "The authors addressed most of my concerns, and I raised the score accordingly.", "rebuttal": "Thank you for your thoughtful feedback. Below we clarify several misunderstandings and explain the scope and focus of our work as well as answering questions.\n\nFirst we want to emphasize that the goals of this paper are:\n- Illustrate with analysis that **different position embeddings can demonstrate different attention distributions**. We then use this insight to hypothesize that mixing them may combine their strengths.\n- Understanding these patterns and combining some position embeddings in a certain way (RNoPE-SWA) can easily outperform the prevailing SOTA long-context scaling method based on RoPE-NTK-Scaling and its variants. And RNoPE-SWA can achieve this with much less compute during long-context training and serving. Overall, RNoPE-SWA without much hyper-parameter tuning easily outperforms SOTA baseline with much less compute required -- a **\"free-lunch\"** phenomenon observed in other hybrid attention works (minimax-text-01, jamba, yoco, etc.) as well.\n- Note that our aim is **not to provide exhaustive ablations** for RNoPE‑SWA, but to demonstrate the interesting phenomenon that reduced compute can yield better results under some design choices.\n\nTo give a brief summary of how we approached the problem:\n- In section 2.1 and 2.2, we analyzed the evaluation score and attention distributions of variants with RoPE, NoPE and QK-Norm. Based on the analysis, we formed some intuitions and made some hypothesis on how we can combine different types of position embeddings to **potentially** improve the model performance.\n- In section 2.2.3, we showed that a model with interleaved NoPE and RoPE layers demonstrate very interesting attention distribution and distinct attention characristics among layers with different position embedding types. The combination also showed excellent long-context performance which corresponds to our analysis and conjectures that models with this attention distribution might work better on long-context tasks than existing SOTA baselines trained with RoPE-NTK. We also shrinked the attention span of RoPE layers based on the observations we had on the attention layers to signifcantly bring down compute requirement during long context training and serving.\n- **Based on the above analysis**, we chose to interleave NoPE and RoPE with a certain ratio and sliding window attention as our final model design.\n- We compare our model architecture with the prevailing method (a dense architecture with RoPE-NTK training) where most open-sourced SOTA models adopted more or less (llama3, Qwen-2.5, DeepSeek-v3, Command R, etc.)\n\nNote that our final method and architecture were derived not from exhaustive ablations but from empirical observations and analyses in several small‑scale pre‑experiments. The primary goal of this paper is to demonstrate that this specific approach outperforms the prevailing SOTA long‑context extension training recipe based on RoPE NTK scaling. Determining whether other combinations or hyper-parameter tuning can surpass our proposed method is beyond this work’s scope and left for future research.\n\n---------------------------------------------------------------------------------------------------------------------\n\nWe will answer the questions with the above paper goals in mind:\n\n1. Why didn't the experiment show NoPE as a baseline?\n\nPrior work (e.g., [37], [71]) already shown pure NoPE lags behind RoPE on standard benchmarks, despite strong extrapolation ability. In Sec. 2.1–2.2 we independently confirm that NoPE underperforms RoPE‑NTK on both retrieval accuracy and perplexity, which motivated focusing on mixed architectures rather than repeating a known weak baseline. We eliminated QK-Norm from subsequent experiments due to the same reason.\n\n---------------------------------------------------------------------------------------------------------------------\n\n2. The paper should include all the below variants in the ablation:\n- RoPE\n- RoPE + SWA\n- NoPE\n- NoPE + SWA\n\nNote that we derived our final architecture RNoPE-SWA based on analysis from section 2 rather than exhaustive architecture searches on architecture combinations. During the analysis, we pointed out a few interesting attention patterns and already eliminated some bad design choices like pure NoPE or QK-Norm.\nThere could potentially exist other combinations of position embeddings that achieve on-par oir better performance and same or different attention patterns compared to RNoPE-SWA. Several potential variants can be tested such as RoPE + SWA, Alibi+SWA, partial RoPE or Yarn + NoPE + SWA and we think they are good to be explored in the future. However, the goal of the paper is to illustrate on the findings from the analysis and the experimental results that even with much less compute, this particular combination -- RNoPE-SWA -- can easily outperform the SOTA methods which uses a dense architecture and RoPE-NTK. Note that his \"free-lunch\" phenomenon also corresponds to findings of other recent works on hybrid attention and the analysis conducted in the paper can serve as a step towards explaining this behavior.\n\nTo sum up the reasons that prevents us from including some experiments:\n\n- Resource constraints: Pretraining models from scratch or fine-tuning major variants with SWA is computationally expensive and falls beyond our current resource budget.\n\n- Low practical impact of pure NoPE‑only variants: In our preliminary experiments, models using only NoPE—regardless of whether with SWA—showed significantly worse performance, including large increases in perplexity. This suggests limited practical value for pure NoPE baselines in long‑context regimes.\n\n- Focus on core contributions: Rather than prove RNoPE‑SWA is optimal among all hybrids, we aim to show it can already surpass SOTA long‑context extension architectures/recipes.\n\n---------------------------------------------------------------------------------------------------------------------\n\n3. The paper should include why evenly stacking RoPE and NoPE instead of assigning early layers with RoPE and later layers with NoPE\n\nSimilar to our reasoning above, this can be one of the explorations for fututre work. It may or may not work but comparing all available variants is not the goal of this paper.\n\n---------------------------------------------------------------------------------------------------------------------\n\n4. Layer interleaving ratio ablations are not shown explicitly in the paper.\n\nThanks for pointing this out. We will include more information about this in the Appendix. We originally left them out because we felt they were not central to our core message, for the following reasons:\n- At the end of section 2, we resutled in RNoPE-10k-SWA with 1:1 interleaving. We ablated and found 1:3 to be a good number with a decent level of sparse layers. Going with 1:7 seems to be streching too much given the model size and number of layers. This turns out to be different again when we further scaled the model parameters later on. It seems that the model can allow higher level of sparsity as it grows bigger with more layers, which makes sense intuitively as well. However, this is less relevant to the core concept the paper wants to convey.\n- Any 1:X interleaving yields a theoretical compute cost of 1/(1 + X) compared to a dense model at very long sequence lengths. Thus, the “free‑lunch” trade‑off—reduced compute with better accuracy —holds regardless of the exact ratio.\n\nWhile exploring how far we can push sparsity is an interesting avenue (and subject to future work), omitting these details does not undermine our main narrative: that a simple sparse‑dense interleaving outperforms dense RoPE‑NTK baselines without needing to fine‑tune hyperparameters for optimal sparsity.\n\n---------------------------------------------------------------------------------------------------------------------\n\nAnswer to Questions:\n1. What does qc mean in the figure 2? -- Thanks for spotting this, we will rectify the typo accordingly in the figure.\n2. Can you include more ablation on the design choices? -- With similar answers from above, we refined our design and obtained the final architecture based on analysis in section 2 and part of section 3 instead of massive ablations.\n\nWe appreciate your thoughtful feedback and look forward to further discussions to enhance our work."}, {"review_id": "3IYdnjRCqG", "reviewer": "Reviewer_qPgF", "summary": "The paper focuses on position embedding for better long-context modeling. Experiments are conducted to compare three position embedding methods, including RoPE, Query-Key normalization, and NoPE. Based on the empirical results, the paper proposes combining rotary position embedding (RoPE) and no position embedding (NoPE) for better long-context performance.", "questions": "* As mentioned in the Weakness part, more comparisons between the proposed method and various baseline variants (various $\\theta$ values for RoPE,  RoPE with sliding window attention) could further validate the effectiveness of the proposed method. I look forward to further results with more fair comparisons.\n\n* It would be better if the authors could provide a more comprehensive analysis of why choosing NoPE and RoPE for the hybrid position embedding. A deeper analysis of the motivation and insights could strengthen the paper, as it currently only considers three position embedding variants in Section 2.\n\nI look forward to the authors' reply and will adjust my rating accordingly.", "limitations": "The authors could address more of the limitations of the proposed method. Currently, there seems to be little discussion of the limitation.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strength**:\n* The proposed hybrid position embedding method seems effective in improving model performance.\n\n* Generally, the paper is clear and easy to follow.\n\n**Weakness**:\n* It lacks a deep analysis or sufficient supportive empirical evidence for the choice of hybrid position embedding with both RoPE and NoPE. While the paper provides observations in Section 2, there is no analysis on why the RoPE and NoPE are complementary. After all, as shown in Section 2.1 (Figures 1 and 2, Table 2), RoPE and NoPE are similar when the model employs a single position embedding.\n\n* It requires more comparison to the baseline models to truly verify the effectiveness of the proposed method. While a larger $\\theta$ enhances the long-context performance for RoPE, it would only be fair to compare the proposed hybrid position embedding to  RoPE with varying $\\theta$. Meanwhile, the proposed method uses RoPE with sliding window attention; the baseline should also include models with sliding window attention.", "was_revised": true, "final_justification": "This paper proposes to mix the RoPE and NoPE for better long-context modeling. Experiments are conducted to compare three position embedding methods, including RoPE, Query-Key normalization, and NoPE.", "rebuttal": "Thank you for the valuable feedback. We address the comment as follows:\n\n1. Analysis of RoPE, NoPE, and QK-Norm, and rationale for interleaving RoPE and NoPE for improved performance:\n\n    In Figure 1, we analyzed the attention mass on the needle tokens across sequence lengths and model variants. From these observations, we infer that NoPE exhibits an advantage in its ability to focus on relevant information, based on the following points:\n    \n- Attention mass differences at fixed sequence length:\nFor example, in Figure 1 (sequence length 32k), the attention mass of RoPE, NoPE, and QK-Norm is 0.02, 0.03, and 0.01, respectively. While the absolute values are small, the relative differences are notable: RoPE ≈ 2× QK-Norm, and NoPE ≈ 3× QK-Norm.\n\n- Attention mass differences across sequence lengths:\nWhen comparing the attention mass over needle tokens across sequence lengths, we observe that QK-Norm drops the most (from 0.017 to 0.005, >3× reduction), RoPE drops from 0.033 to 0.015 (>2× reduction), while NoPE drops from 0.045 to 0.025 (<2× reduction). This suggests that NoPE degrades less when sequence length increases.\n\n- Needle score and prior findings:\nFrom Table 2 (and corroborating prior studies), QK-Norm performs poorly on long-context tasks, likely due to its reduced ability to attend to relevant tokens—a hypothesis we further analyze in Appendix B.\n\nThese differences led us to hypothesize that RoPE and NoPE could be complementary where RoPE explicitly models positional information and recency bias and NoPE improves long-context retention and reduces degradation across long sequences.\n\nMotivated by this, we explored interleaving RoPE and NoPE to combine their strengths. This design decision were made as an exploration. We also conducted several other attempts to improve the model's capabilities, which we did not include as they either performed poorly or were less relevant. As we also stated in the paper, other attempts were also made by prior work like GPT-J/Neox of combing NoPE and RoPE (partial RoPE) but not across layers.\n\nThank you for pointing this out. We will make our reasoning clearer in the final version by including numeric comparisons and percentage differences.\n\n-------------------------------------------------------------------------------------\n\n2. Comparing with baseline with larger RoPE theta values and sliding window.\n\nThank you for this important suggestion. We plan to include experiments utilizing varying θ values for RoPE in the final version. Due to computational constraints, these may be limited to the fine-tuning (SFT) stage.\nOur current baseline uses a θ of 8 million, which aligns with industry-standard long-context training regimes. For example:\n\n- LLaMA 3 adopts θ == 500 k for support of an 8 k context window \n- Qwen‑2.5, supporting up to 128 k tokens, employs θ around 10 million \n- Cohere Command R, also designed for 128 k context length, uses θ on the order of 8 million\n\nWhile we acknowledge that these models also incorporate techniques like Dual‑Chunk Attention or the Yarn mechanism to enhance long-context performance, the overarching trend of scaling θ to a few million for extended context remains consistent.\n\n-------------------------------------------------------------------------------------\n\n3. On including baseline models with sliding‑window attention\n\nThank you for this perceptive suggestion. We agree that adding baseline comparisons that also utilize sliding‑window attention (SWA) would strengthen the completeness and fairness of our evaluation. However, we did not include those in the current version for the following reasons:\n\n- Resource constraints: Pretraining models from scratch or fine-tuning major variants with SWA is computationally expensive and falls beyond our current resource budget.\n\n- Low practical impact of pure NoPE‑only variants: In our preliminary experiments, models using only NoPE—regardless of whether with SWA—showed significantly worse performance, including large increases in perplexity. This suggests limited practical value for pure NoPE baselines in long‑context regimes.\n\n- Focus on core contributions: We prioritized experiments directly aligned with our hypothesis: the hybrid combination of RoPE and NoPE. As such, we ran QK‑Norm with sliding‑window attention—but omitted it from the paper because it did not illuminate the central mechanisms we study.\n\nThe goal of this paper is that we want to show:\n-  Analysis that different attention mechanisms and combinations across layers can result in interesting attention patterns and further understanding this pattern can help us improve the model.\n- Experiments that focus on demonstrating that RNoPE‑SWA can easily outperform prevailing long-context recipes with lower compute and serving overhead without tuning the θ values. This \"free-lunch\" phenomenon also corresponds to findings of other recent works on hybrid attention.\n- We understand that there are possibly other ways to combine different mechanisms to improve, even just between NoPE and RoPE. We think it will be good for further work to explore and refine on top of this approach.\n\n-------------------------------------------------------------------------------------\n\nWe appreciate your thoughtful feedback and look forward to further discussions to enhance our work."}, {"review_id": "5h6DONjYpm", "reviewer": "Reviewer_LEho", "summary": "This paper presents a new hybrid attention strategy combining Rotary Position Embedding (RoPE) and No Positional Embedding (NoPE) for long-context large language models (LLMs). The authors analyze existing attention mechanisms and highlight their limitations, then propose an architecture that alternates between RoPE and NoPE layers. This hybrid model outperforms traditional RoPE-based methods on long-context tasks and is efficient in both training speed and memory usage, providing a more effective solution for handling long input sequences.", "questions": "See Weaknesses.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "## Strengths\n1. The paper introduces a unique hybrid attention mechanism, combining RoPE and NoPE, offering an innovative solution for long-context modeling.\n2. It provides a clear comparison of existing attention mechanisms, making the analysis accessible and well-structured.\n3. The proposed model significantly improves performance and efficiency, making it a valuable contribution to the development of large language models.\n\n## Weaknesses\n1. The paper adjusts the model architecture but lacks an analysis of the training overhead introduced by these adjustments.\n2. The findings in this paper are similar to those in [1], but the paper does not explain the differences.\n3. The paper uses only RoPE as the baseline, lacking validation of the method's effectiveness; it could benefit from a comparison with p-RoPE [1].\n\n[1] Round and Round We Go! What makes Rotary Positional Encodings useful?", "was_revised": true, "final_justification": "Thank you for the response. I have raised my score accordingly.", "rebuttal": "Thank you for the valuable feedback. We address the comments as follows:\n\n1. Training overhead analysis:\nWe discuss relevant aspects in Section 5.3, Impacts on Training and Inference. Our findings indicate that introducing hybrid attention can significantly accelerate both training and serving. Additional benefits arise when combining it with specialized kernels such as ring attention, which reduces communication overhead.\n\n2. Comparison with p-RoPE:\nThank you for pointing this out. In the second paragraph of Section 3, we briefly mention variants that combine NoPE and RoPE along the head dimensions (e.g., GPT-J/NeoX). The work of p-RoPE further analyzes RoPE, focusing on the impact of different frequency domains and concluded that low/high frequencies focus on semantic and positonal attention respectively, which is similar to our findings that NoPE focuses more on information retrieval and RoPE more on recency bias. The difference is that p-RoPE removes low frequencies in the head dimensions (like partial rope used in prior work) and we combine RoPE/NoPE across layers (which has more engineering benefits as pointed out in section 5.3). \n\nWe are probably not able to conduct another pretraining from scratch with this variant as the resource constraint is very demanding for this experiment (similar to why we didn't do NoPE + SWA). However, we will include a reference to the p-RoPE paper and expand our discussion of its relationship to our method in the same paragraph in the final version.\n\n\nWe appreciate your thoughtful feedback and look forward to further discussions to enhance our work."}]

Author rebuttal:

Meta-review: This paper presents a thorough analysis of attention mechanisms—RoPE, NoPE, and QK-Norm—highlighting their strengths and limitations in long-context language modeling. It uncovers distinctive attention patterns that influence performance and offers architectural insights for improving model design. Building on these findings, the authors propose a novel hybrid attention mechanism that outperforms RoPE-based transformers on long-context tasks while remaining competitive on shorter ones.

After the discussions between authors and reviewers the major concerns were:

1. Insufficient experiments and/or ablations (R-V4bg, R-qPgF, R-qtcq); and
2. Missing distinction between the work in [1] and the submission (R-LEho).

However, the authors replied to these points and most reviewers were satisfied w/ the authors’ replies. In the end, all reviewers support the acceptance of this work. We encourage the authors to include beneficial material discussed w/ the reviewers and clearly state the distinction of this work and [1].
Decision: Accept (poster)