Title: Blending Complementary Memory Systems in Hybrid Quadratic-Linear Transformers
Venue: NeurIPS 2025
Accepted: yes
Keywords: sequence processing

transformers

linear transformers

fast weight programmers

key-value memory

memory modeling

hybrid transformers
Abstract: We develop hybrid memory architectures for general-purpose sequence processing neural networks, that combine key-value memory using softmax attention (KV-memory) with fast weight memory through dynamic synaptic modulation (FW-memory)---the core principles of quadratic and linear transformers, respectively. These two memory systems have complementary but individually limited properties: KV-memory offers precise retrieval but is constrained by quadratic complexity in sequence length, while FW-memory supports arbitrarily long sequences and enables more expressive computation but sacrifices precise recall. We propose and compare three methods to blend these two systems into a single memory system, differing in how and when input information is delivered to each system, to leverage the strengths of both. We conduct experiments on general language modeling and retrieval tasks by training 340M- and 1.3B-parameter models from scratch, as well as on synthetic algorithmic tasks designed to precisely illustrate the benefits of certain hybrid methods over others. We also evaluate our hybrid memory systems on reinforcement learning in partially observable environments. Overall, we demonstrate how a well-designed hybrid can overcome the limitations of its individual components, offering new insights into the design principle of neural memory systems.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Hybrid Quadratic-Linear Transformers (HQLTs) that combine key-value (KV) memory from standard transformers with fast-weight (FW) memory from linear transformers, specifically using DeltaNet. The authors argue these systems have complementary properties: KV-memory provides precise retrieval but suffers from quadratic complexity, while FW-memory supports unbounded sequences with higher expressivity but lower retrieval precision. The authors proposed 3 strategies to blend these two systems together: Delayed-Streaming (old KV pairs feed into FW-memory), Delayed-Chunk (chunk-wise processing), and Synchronous (both memories process inputs simultaneously). The authors evaluate on language modelling (340M and 1.3B parameters trained on 15B tokens), synthetic algorithmic tasks (parity, modular arithmetic), and retrieval tasks (FDA, SWDE, SQuAD), concluding that the Synchronous variant performs best.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n**Quality:**\n- The paper includes a comprehensive experimental evaluation across diverse tasks (language modelling, algorithmic reasoning, retrieval).\n- The authors tried different model scales (340M, 1.3B) with consistent training protocols. Of course, more scales could be useful, but I do recognise that this requires very large computation capabilities that are not always available.\n- The paper also contains thoughtful ablations on mixing strategies, window sizes, and FW-memory choices.\n- The authors provided their source code, which aids reproducibility and is commendable.\n- Finally, there is a clear mathematical formulation of the three proposed hybrid variants.\n\n**Clarity:**\n- The paper is well-structured with clear motivation and background.\n- It contains helpful visual illustrations (Figure 1).\n- It also contains explicit equations for each variant\n\n**Originality:**\n- The paper updates prior hybrid attempts by using the (relatively) state-of-the-art and very interesting DeltaNet model instead of vanilla linear attention.\n- It also includes the systematic exploration of three conceptually distinct blending strategies mentioned above.\n- Finally, the evaluation on expressivity tasks to test memory synergy is (to my knowledge) novel.\n\n### Weaknesses\n**Quality:** I believe there are a few very important issues here which the authors need to clearly address to have their work accepted in a conference like NeurIPS.\n1. **Misappropriation of established terminology**: The first big issue is that the paper invokes \"Complementary Learning Systems\" (CLS) from neuroscience but proposes an entirely different dichotomy, as far as I understand. Classical CLS (McClelland et al., 1995) describes hippocampal-neocortical differences in learning rates and memory types (episodic vs. semantic), not computational trade-offs between precision and efficiency. The authors do not justify this properly which leads me to believe that it is a misuse of the term, thus raising concerns about theoretical grounding. However, there might be a rationale behind this that is not clear from the current version of the manuscript, in which case I would kindly ask the authors to elaborate.\n2. **Core claims not supported by results:**\n\t* On standard language tasks, improvements are typically at least <1% over baselines. The only notable gain is ~15% on LAMBADA perplexity, which appears task-specific. Given the added architectural complexity, these minimal gains don't justify the approach.\n\t* Table 4 shows HQLT requires large windows (1024 tokens) to approach Transformer performance. With small windows (64), performance is dramatically worse (19.8 vs 44.7 avg accuracy). This suggests the hybrid isn't truly leveraging complementary strengths but rather defaulting to brute-force quadratic attention.\n3. **Missing critical computational analysis**: Despite computational efficiency being a primary motivation, the paper provides no FLOPs analysis or throughput comparisons. The appendix dismisses this with \"the actual computational cost is negligible\" (line 459), which is inadequate. How does HQLT with 1024-token window compare to Transformer++ with 2048 tokens?\n4. **The expressivity results don't demonstrate synergy**: Synchronous HQLT solving parity/modular arithmetic merely shows DeltaNet works when included (see Table 3 and Appendix Table 7), not that the hybrid achieves something neither component could alone.\n\n**Clarity:** (Minor issues)\n- Inconsistent terminology (HQLT vs HQTL)\n- The LAMBADA anomaly (strong performance with small windows) is unexplained\n- Defers crucial efficiency discussion to appendix\n\n**Significance:**\n1. **Limited practical impact**: If large windows are still required for retrieval, the hybrid approach fails to solve the fundamental limitation it targets\n2. **Algorithmic task results are unsurprising**: As mentioned above, showing that Synchronous HQLT (which includes DeltaNet) solves tasks that DeltaNet already solves doesn't demonstrate synergy.\n3. **No comparison to other long-context solutions** (Transformer-XL, EM-LLM, Compressive Transformer, etc.)\n\n**Originality:**\n- Core hybridization concept exists in prior work (Arora et al., Munkhdalai et al.)\n- Main distinction is using DeltaNet, which feels incremental\n- The successful \"Synchronous\" approach is conceptually straightforward\n\nQUESTIONS:\n1. **On theoretical grounding**: Why invoke \"Complementary Learning Systems\" when your dichotomy (efficiency vs. precision) fundamentally differs from the established neuroscientific concept (episodic vs. semantic memory with different learning rates)?\n2. **On efficiency claims**: Table 4 shows HQLT needs 1024-token windows for competitive retrieval. The appendix claims costs are \"negligible\" but provides no analysis. Can you provide concrete FLOPs/throughput comparisons? Specifically: HQLT-1024 vs Transformer-2048?\n3. **On LAMBADA**: Why does HQLT dramatically outperform on LAMBADA with just 64 tokens when other retrieval tasks fail? This contradicts your other results and deserves investigation.\n4. **On synergy**: How do the algorithmic task results demonstrate synergy rather than just \"DeltaNet works when included\"? What can the hybrid do that neither component could achieve alone?\n5. **On comparisons**: Why no comparison to other long-context solutions (Mamba, Transformer-XL, EM-LLM, etc.)? How does HQLT compare to simply using Flash Attention with longer contexts?\n\n_Clear criteria for score improvement_:\n\n- Provide thorough computational analysis showing efficiency gains\n- Demonstrate retrieval working with small windows (<256)\n- Show true synergistic capabilities beyond individual components\n- Correct theoretical framing, better justify your choice or remove CLS references\n- Compare against modern long-context baselines\n\nLIMITATIONS:\nThe authors acknowledge some limitations (e.g., large windows still needed for retrieval) but don't adequately address:\n- Computational overhead of dual memory systems\n- Questionable practical advantages over simpler alternatives\n- Sensitivity of results to hyperparameters\n- Missing comparisons to other long-context solutions", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Hybrid Quadratic-Linear Transformers (HQLTs), a new class of sequence models that blend two complementary neural memory systems: quadratic key-value (KV) memory, known for precise retrieval but limited context due to quadratic complexity, and linear fast-weight (FW) memory, known for unbounded context and expressivity but imprecise recall. The authors explore three methods to integrate these systems - Delayed-Streaming, Delayed-Chunk, and Synchronous - and conduct extensive experiments to evaluate them across language modeling, retrieval, and synthetic algorithmic tasks. Results show that the synchronous variant, which allows both memory systems to operate concurrently on the same input, outperforms others by effectively combining the strengths of each memory system. The work establishes that such hybridization not only overcomes individual limitations but also enables new capabilities in memory system design, especially when advanced FW memory models like DeltaNet are used.\n\nSTRENGTHS AND WEAKNESSES:\nQuality: The paper is technically solid and presents a complete piece of work. The proposed hybrid models are carefully evaluated across a wide range of tasks, and the use of advanced fast-weight methods like DeltaNet adds credibility. The authors are honest about limitations and support their claims with solid empirical evidence. However, the performance improvements are sometimes small, and the architectural blending techniques feel more incremental than transformative. Some choices, like gating mechanisms, would benefit from further theoretical justification.\n\nClarity: The writing is mostly clear, and the paper is well-organized. The motivation and high-level ideas are easy to follow, but certain sections - especially those describing the hybrid mechanisms - are dense and could be made more accessible. The notation can be overwhelming at times, and clearer visualizations or example-driven explanations would help. Overall, the reader can understand the core contributions, but digesting the technical details takes effort.\n\nSignificance: The paper tackles a meaningful problem in transformer architecture design - merging long-context capability with precise memory. The proposed models are relevant to practical scenarios and could influence future work on efficient attention. That said, the real-world impact is still limited by modest gains and unresolved issues with retrieval. The contribution is useful, but not groundbreaking, and the adoption by the broader community will depend on future extensions.\n\nOriginality: The idea of combining quadratic and linear attention is not new, but this paper brings a more thorough and updated evaluation, especially by using stronger fast-weight models. The synchronous design and detailed empirical analysis are clear advances over prior work. Still, the methodological novelty is limited - the core ideas are largely recombinations of known mechanisms. The paper is more valuable for its careful integration and insight than for introducing radically new techniques.\n\nQUESTIONS:\n1. Can you show a clear case where hybridization gives a meaningful win over both standalone Transformer and DeltaNet, even with tuning or scaling? This would help justify the added complexity.\n\n2. The choice between sum, scalar, and vector mixing seems arbitrary. What guides this design? Is there a principled way to adapt mixing across tasks or contexts?\n\n3. Retrieval tasks still rely heavily on the KV-memory window size. Could you explore a mechanism where FW-memory actively contributes to retrieval, not just as a backup store?\n\n4. Comparison to other recent hybrids like Infini-attention or Mamba-based models is missing. Can you provide head-to-head results to clarify what your method uniquely improves?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the well-known limitations of quadratic and linear attention mechanisms in sequence modeling. The quadratic attention (i.e., standard softmax-based key-value attention) provides precise token retrieval but suffers from quadratic time and memory complexity with respect to sequence length. In contrast, linear attention (i.e., fast-weight programmer) scales linearly with sequence length and supports long-context processing, but struggles to precisely recall specific tokens, especially when the number of tokens to remember exceeds the hidden state dimension. To tackle these limitations, this paper proposes **Hybrid Quadratic-Linear Transformers (HQLTs)**, which combine the precise retrieval capabilities of quadratic attention and the expressive state-tracking of linear attention (specifically, DeltaNet). In particular, this paper introduces and compares three blending strategies (Delayed-Streaming, Delayed-Chunk, and Synchronous) and memory mixing and gating mechanisms to effectively integrate KV-memory and FW-memory outputs. Comprehensive experiments on language modeling, retrieval, and algorithmic tasks demonstrate the potential of HQLTs to mitigate the limitations of each memory system and offer new insights into the design of hybrid neural memory architectures.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n- **S1**. The paper clearly motivates the complementarity of quadratic and linear transformers, drawing an interesting connection to the theory of complementary learning systems.\n- **S2**. It systematically explores multiple blending strategies that are possible architectural designs for hybridization and empirically evaluates them on diverse tasks, including language modeling, retrieval, and algorithmic benchmarks.\n- **S3**. The proposed HQLT framework is compatible with any linear attention variant, making it extensible to future advances in fast-weight or linear attention methods.\n- **S4**. The broad range of experiments demonstrates the effectiveness of the hybrid model compared to using either quadratic attention alone or linear attention alone.\n\n### Weaknesses\n- **W1**. The concept of hybrid approaches that combine quadratic and linear attention is not entirely novel; prior works such as [1] have explored similar ideas.\n- **W2**. As shown in Table 3, on specific tasks (e.g., parity and modular arithmetic), all HQLT variants underperform compared to standalone DeltaNet, suggesting that hybridization can sometimes weaken the strengths of the fast-weight memory.\n- **W3**. While the Synchronous HQLT generally outperforms other variants, it is notable that in the 1.3B setting, the Delayed-Stream variant achieves better results on the ARC tasks. This indicates that we could face additional difficulty finding the most appropriate blending strategy for a given use case.\n\n[1] Transformer quality in linear time, ICML 2022\n\nQUESTIONS:\n- **Q1**. Regarding the W1, could the authors clarify what distinguishes HQLT from existing hybrid models that also combine quadratic and linear attention mechanisms?\n- **Q2**. This paper blends quadratic and linear attention within a single layer. Could the authors elaborate on why this within-layer hybridization is preferable to a layer-wise separation strategy (e.g., using quadratic attention in earlier and linear attention in later layers, or vice versa)?\n- **Q3**. Is there potential for an intermediate approach with partial overlap between the Delayed-Streaming (no overlap) and Synchronous (full overlap) strategies? I would like to hear the authors’ perspective on this idea, including any expected benefits and drawbacks.\n\nLIMITATIONS:\nThe authors adequately addressed the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a hybrid quadratic-linear architecture that combines two types of memory systems: key-value memory (used in transformers for precise recall but with quadratic complexity) and synaptic memory (used in linear transformers for efficient computation). The authors design and compare three hybrid models—Delayed-Streaming, Delayed-Chunk, and Synchronous—and show that the Synchronous hybrid best leverages the strengths of both memory types. Through experiments on language modeling, expressivity tasks, and in-context retrieval tasks, the paper demonstrates the hybrid model's enhanced performance and scalability.\n\nSTRENGTHS AND WEAKNESSES:\nstrengths\n\n1. In-layer hybrid architecture series validated on linear attention with delta rule. Although incremental, it is still worthwhile to explore the combination of two very different memory mechanisms.\n\n2. Experiments on different hybrid methods on multiple tasks and their detailed ablations.\n\n3. The author's discussion of this work is valuable. The fixed-size memory of linear attention is difficult to accurately handle retrieval tasks, but the infinitely expanded kv memory is not necessarily optimal. Although the author did not provide a sufficiently novel solutions in the article, how to better combine the two types of memory is an important research topic.\n\nweakness\n\n1. The abstract and introduction lack a brief introduction to the method. The authors should have elaborated on the \"Three blending methods\"(i.e. various types of HQLT).\n\n2. I thought that Delayed-Streaming HQLT and Delayed-Chunk HQLT just correspond to the serial and chunk parallel forms in linear attention, and the calculation results are the same. But from the authors' experiment, Delayed-Chunk HQLT is different from Delayed-Streaming HQLT in both calculation process and results, which confuses me. I hope the author can explain why Delayed-Chunk HQLT is a hybrid method independent of Delayed-Streaming HQLT.\n\n3. How does the training efficiency of hybrid architectures compare to linear models and transformers? A supplementary experiment may answer my concerns.\n\n4. The authors mentioned in the limitation that \"Investigating such a mechanism is not straightforward in the current era of model development, which sets hardware-efficiency as a hard requirement.\" I value this point of view of the authors. Recently, Log-linear attention [1] has achieved a balance in exploring the utilization of intermediate-scale memory and computational efficiency. I wonder what the authors think of this work. (This is just an open question)\n\nminors:\n5. The word \"synaptic memory\" used in the abstract is confusing, as the authors does not provide any explanation in the paper. Either explain it in the text, or use FW memory instead, which may be easier to understand.\n6. Some HQLTs are misspelled as HQTLs. Use ctrl+F to find them. \n\n[1] Guo, et al. Log-Linear Attention.\n\nQUESTIONS:\nsee weakness.\n\nLIMITATIONS:\nsee strength 3.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-07"}