Title: RAT: Bridging RNN Efficiency and Attention Accuracy via Chunk-based Sequence Modeling
Venue: NeurIPS 2025
Accepted: yes
Keywords: language modeling

attention

recurrence
Abstract: Transformers have become the cornerstone of modern large-scale language models, but their reliance on softmax attention poses a computational bottleneck at both training and inference. Recurrent models offer high efficiency, but compressing the full sequence into a fixed-size and holistic representation can suffer from memory degradation in long contexts and limit fine-grained retrieval. To address this, we propose RAT, an intermediate design that bridges the efficiency of RNNs and capacity of attention. RAT partitions the input into chunks, applies recurrence within each chunk for local dependencies, and softmax-based attention across chunks for long-range interactions. This design mitigates memory degradation and enables direct access to distant tokens, while retaining computational efficiency. Empirically, with a chunk size of 16, the RAT block achieves a $7\times$ improvement in training speed for 100K sequence length and $9\times$ in generation at the 4K position, while maintaining similar performance compared to standard attention. We demonstrate this by training 1.3B parameter models from scratch and performing large-scale evaluations, including short- and long-context benchmarks, as well as supervised fine-tuning (SFT). We further propose a hybrid architecture that interleaves RAT with local attention. By combining efficient long-range modeling with strong local interactions, this hybrid design not only improves inference speed and reduces cache memory usage, but also consistently enhances performance and shows the overall best results. Code is available at \url{https://github.com/CLAIRE-Labo/RAT}.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces RAT (Recurrent-Attention Tradeoff), a novel architecture designed to bridge the efficiency of RNNs with the accuracy of attention mechanisms in language modeling. RAT segments input sequences into chunks, applying a lightweight linear recurrence within each chunk for local modeling and using softmax attention across chunks to capture long-range dependencies. This hybrid structure enables flexible trade-offs via chunk size, interpolating between RNN-like and attention-like behavior. The authors demonstrate that RAT significantly improves training and inference speed, up to 10×, without substantial loss in accuracy. Through extensive pretraining and evaluation at the 1.3B parameter scale, RAT matches or exceeds standard attention on various tasks, especially in long-context settings. A further enhancement interleaves RAT with sliding-window attention to boost local precision, showing consistent performance gains across commonsense reasoning, code, and summarization tasks. The study highlights RAT's efficiency, architectural simplicity, and strong empirical results, positioning it as a promising alternative for scalable language models.\n\nSTRENGTHS AND WEAKNESSES:\nQuality: The paper presents a technically solid and well-executed study, with claims supported by strong empirical results across pretraining, evaluation, and fine-tuning settings. The proposed RAT architecture is carefully implemented and benchmarked against attention and RNN baselines. However, theoretical analysis is limited, and some performance gaps (e.g., in tasks like WinoGrande or few-shot QA) are acknowledged but not deeply investigated. The work is complete and honestly evaluated, though larger model scaling and ablations at higher capacity would strengthen the evidence.\n\nClarity: The paper is mostly clear and well-organized, with helpful diagrams and consistent terminology. Key ideas such as intra-chunk recurrence and inter-chunk attention are explained effectively. However, some descriptions (e.g., implementation efficiency or positional encoding strategies) are overly condensed and require more unpacking. Presentation could be improved with clearer separation between core contributions and peripheral optimizations.\n\nSignificance: RAT targets a critical challenge in scaling language models efficiently, and the results suggest meaningful practical benefits in speed and memory while maintaining accuracy. The ability to interpolate between attention and recurrence is valuable, and the hybrid design with sliding-window attention shows promise. While the impact at larger scales remains untested, the demonstrated improvements at 1.3B suggest the approach is relevant and could influence future model design.\n\nOriginality: The idea of chunked recurrence combined with sparse cross-chunk attention is a novel architectural middle ground, distinct from both state-space models and purely linear attention. While the components are not new individually, their integration is original and motivated by practical trade-offs. The paper provides a new way to think about temporal mixing that departs from both strict attention and RNN paradigms, though more conceptual framing and discussion of prior hybrid efforts would make the novelty clearer.\n\nQUESTIONS:\n1. How well does RAT scale to larger models like 7B or 13B? Please comment on any challenges with parallelism, memory, or throughput at scale. Concrete results or estimates would help assess broader relevance.\n\n2. RAT underperforms on tasks like WinoGrande and few-shot QA. Can you explain whether this is due to chunking, attention granularity, or training setup? A deeper analysis would clarify limits of generalization.\n\n3. The reversed design (attention within chunk, recurrence across) is said to perform worse. Can you provide more insight or experiments to show why? Understanding this failure would help validate your design choices.\n\n4. How does RAT compare directly to recent efficient models like Mamba or DeltaNet under similar compute or FLOP budgets? A head-to-head baseline is needed to contextualize the claimed efficiency and accuracy trade-offs.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAttempts to bridge the efficiency of RNNs (actually gated linear recurrence) and the accuracy of attention mechanisms for language modeling.\n - RAT divides input sequences into chunks, applying lightweight gated linear recurrence within each chunk to capture local dependencies and softmax attention across chunks for long-range interactions.\n- RAT interpolates between RNN and attention behaviors, balancing computational cost and performance.\n- RAT achieves significant efficiency gains: up to 7× training speedup for 1M-token sequences and 9× generation speedup at 4K tokens, with reduced KV cache usage\n- Empirically, RAT matches or surpasses standard attention on benchmarks like LongBench (e.g., 24.9 vs. 18.6 on GovReport) and commonsense reasoning tasks.\n- A hybrid RAT-SWA variant, interleaving RAT with sliding-window attention, further boosts performance (e.g., +4 points on code tasks, +1 Rouge-L on summarization).\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- Achieves up to 7× training speedup (1M tokens) and 9× generation speedup (4K tokens), with reduced KV cache usage, minimizing OOM risks. However, these efficiency gains may fade at a larger scale when MLP computation dominates.\n\nWeaknesses:\n- One big issue I have is that Eqn. 2. is a specific case of Mamba, and that this proposal for a new sequence mixing method is a mix of attention and this EMA construction. \n  - A more meaningful comparison then should also be between Mamba-Transformer Hybrid architectures, both in terms of performance on downstream tasks, and on efficiency metrics. \n- Because of the EMA-style construction of the \"RNN\" part, this also does not afford the model the capability to do state-tracking: https://arxiv.org/pdf/2404.08819.\n   Since I suspect the speedups gained in the paper may fade with 1) different implementation comparisons, or 2) scale of model, I think there is little gain in capability if we consider the gains from using a RAT-Attention hybrid vs a Mamba-Attention hybrid.\n\nQUESTIONS:\n- I am surprised at the 7x training speedup and 9x generation speedup, and I am curious what the baseline comparisons are. Was flash attention used?\n\nLIMITATIONS:\nNone.", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 1, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes RAT (Recurrent-Attention Transformer), an architectural design that bridges RNN efficiency and Transformer attention accuracy for language modeling. RAT partitions the input sequence into fixed-size chunks, applies a lightweight gated RNN within each chunk for capturing local dependencies, and then performs softmax attention across chunks to model long-range interactions. This design interpolates between standard RNNs and full attention depending on the chunk size, allowing a trade-off between efficiency and expressiveness.\n\nRAT models, especially with a chunk size of 16, achieve substantial speedups—up to 9× faster generation at 4K length—while matching or surpassing attention baselines on both pretraining perplexity and downstream tasks. Moreover, a hybrid variant interleaving RAT with sliding-window attention improves both efficiency and accuracy across commonsense reasoning, long-context understanding (LongBench), and supervised fine-tuning tasks, outperforming full-attention models in several benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. Proposes a new intermediate architecture (RAT) that blends the efficiency of RNNs with the accuracy of attention mechanisms.\n\n2. The hybrid variant (RAT interleaved with sliding-window attention) further boosts performance, showing strong results on reasoning, summarization, and code tasks.\n\nWeaknesses:\n\n1. Missing strong baselines: While the paper discusses the landscape of linear and local attention methods, it only benchmarks against vanilla RNNs and standard attention. This makes the advantages of RAT less convincing, as prior work—such as hybrid architectures mixing linear and softmax attention (e.g., Hymba [1])—has shown similar trade-offs. Including such comparisons is necessary to establish RAT’s utility over existing efficient attention methods.\n\n2. Sensitivity to chunk size: RAT’s performance declines with larger chunk sizes, requiring careful tuning to balance accuracy and efficiency, particularly for long-context or noisy inputs.\n\n3. Lack of large-scale evaluation: The study is limited to 1.3B parameter models, leaving open questions about how well RAT scales to larger models used in practical deployment.\n\n[1] Hymba: A Hybrid-head Architecture for Small Language Models\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces RAT (Recurrence-Attention Transition), a novel architecture that interpolates between the efficiency of RNNs and the expressiveness of attention by chunking sequences. \nWithin each chunk (size L,16 in the paper), a lightweight gated linear RNN captures local dependencies, while softmax attention spans chunks to model long-range interactions. \nThis design reduce the burdens of the quadratic complexity of full self-attention (O(T²)) by reducing inter-chunk attention to O(C·D) flops, where C = T/L is the number of chunks. RAT flexibly trades off efficiency and accuracy via L: L=1 mimics attention, while L=T collapses to an RNN.\n\nExperiments on 1.3B-parameter models demonstrate 7× faster training (1M-token sequences) and 9× faster generation (4K-length) with L=16, while maintaining perplexity within 0.06 of full attention. \nOn short-context benchmarks (e.g., HellaSwag, PIQA), RAT w/ L 16 trails attention by <2 points but improves with hybrid RAT-SWA (sliding-window attention + RAT), gaining +1 point on commonsense tasks. On long-context LongBench, RAT-SWA outperforms attention on 12/14 tasks, including +4 points on code completion.\n\nSTRENGTHS AND WEAKNESSES:\nI find the RAT (Recurrence-Attention Transition) architecture to be an elegant and natural solution for balancing expressiveness and efficiency in long-context modeling. Unlike recent works (e.g., Log-Linear Attention), which impose hierarchical inductive biases with O(log N) memory growth, RAT’s chunking strategy feels more intuitively grounded:\n* At 1M context, RAT with L=16 retains a fixed 64K KV cache, a constant-sized compression that scales gracefully without the logarithmic overhead of alternatives (log₂(1M)/log₂(4K) ≈ 2× growth in Log-Linear Attention).\n* Hybridization with SWA further reinforces its practicality: local attention handles short-range dependencies (where RNNs might underperform), while RAT’s inter-chunk attention preserves global coherence. This synergy outperforms pure attention or SSMs on LongBench, suggesting the architecture generalizes well beyond synthetic benchmarks.\n\nAnd I am particularly eager to see the results of RAT in terms of param and context scaling (if possible):\n* will RAT’s efficiency gains persist under larger model regimes like 7B/14B?\n* how does RAT fare against other RNNs like Mamba-2/HGRN2/DeltaNet/GDN on retrieval-heavy tasks?\n\nQUESTIONS:\n* I'm curious if the author try other recent RNN variants in RAT, for example, HGRN and RWKV4?\n* In terms of param allocation, if I understand correctly, the final layer would have 6d^2 params with shared q/k? Did the authors tied key and forget gate param allocation as in HGDN2?\n* It would make the paper more comprehensive if the authors add some discussions with GAU in terms of local/global chunk design, and Log Linear Attention in terms of O(log N) caching. Some recent works, e.g., GSA, also makes some efforts in combing attention with RNNs\n* I'm especially curious what if combining RAT with some state expansion techniques like in Mamba or Hedgehog, which intuitively could make the 1d RNNs more powerful once expanded to 16x or larger state size\n\nHGRN: https://arxiv.org/abs/2311.04823\n\nGAU: https://arxiv.org/pdf/2202.10447\n\nGSA: https://arxiv.org/abs/2409.07146\n\nHedgehog: https://arxiv.org/abs/2402.04347\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-12"}