Title: Improving Bilinear RNN with Closed-loop Control
Venue: NeurIPS 2025
Accepted: yes
Keywords: Sequence Modeling
Abstract: Recent efficient sequence modeling methods, such as Gated DeltaNet, TTT, and RWKV-7, have achieved performance improvements by supervising the recurrent memory management through the Delta learning rule. Unlike previous state-space models (e.g., Mamba) and gated linear attentions (e.g., GLA), these models introduce interactions between the recurrent state and the key vector, resulting in a bilinear recursive structure. In this paper, we first introduce the concept of Bilinear RNNs with a comprehensive analysis on the advantages and limitations of these models. Then based on the closed-loop control theory, we propose a novel Bilinear RNN variant named Comba, which adopts a scalar-plus-low-rank state transition, with both state feedback and output feedback corrections. We also implement a hardware-efficient chunk-wise parallel kernel in Triton and train models with 340M/1.3B parameters on a large-scale corpus. Comba demonstrates its superior performance and computation efficiency on both language modeling and vision tasks.


Peer reviews:
comments: SUMMARY:
## Summary

In this article, the authors provide a detailed literature review on recursive models for efficient processing of sequential data. This synthesis highlights the strengths and weaknesses of each model and thus allows them to propose a novel RNN architecture that aims to incorporate dependencies on the previous state and the output while remaining efficient during inference. In addition to the detailed literature summary, the authors support their new architecture with experiments that show that the new model is competitive with the state of the art, and in a number of tasks outperforms the state of the art.

STRENGTHS AND WEAKNESSES:
## Strengths
- The article is very well written, well -illustrated and clear.
- The authors have a deep knowledge of literature and know exactly how their work is positioned.
- The methodology of numérical experiments is rigorous.

## Weaknesses
- There is a lack of theory to support the claims. Validations are purely empirical.
- As the authors themselves said, due to infrastructure limitations, they were unable to conduct very large-scale experiments.

QUESTIONS:
## Questions
Anglais

Do you have any idea if it is theoretically possible to show the expressiveness of your model and if possible place it in the ecosystem of existing models?

LIMITATIONS:
Due to the unavailability of these state-of-the-art models, the authors were unable to compare their models to the latest state-of-the-art models. Due to infrastructure limitations, they were unable to conduct very large-scale experiments.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-18

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new nonlinear recurrent neural network (RNN) model called Comba, which is inspired by control theory. The authors review recent nonlinear RNNs that use Delta learning rules to improve memory control. Unlike earlier models, Comba uses a scalar-plus-low-rank (SPLR) state update and adds feedback both to the state and output, forming a closed-loop system. It is also designed to be hardware-efficient by supporting chunk-wise parallel computation. The authors test Comba on various language and vision tasks, showing it has better performance and faster training compared to other models. They also explore hybrid designs and provide detailed experiments and analysis.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper introduces Comba, a new nonlinear RNN architecture that incorporates closed-loop control by combining state feedback and output correction.\n2. The authors provide an optimized chunk-wise parallel implementation in Triton, achieving a good speed improvement in forward pass. This makes the method more practical for large-scale training.\n3. Comba is rigorously evaluated across multiple benchmarks, including language modeling, commonsense reasoning, long-context understanding (LongBench), and etc.. Results consistently show competitive or superior performance.\n4. The work nicely bridges the gap between SSMs, gated linear attention, and Delta-rule-based updates. It also provides a unified perspective on “Nonlinear RNNs” and positions Comba as a foundation model candidate.\n5. The authors conduct comprehensive ablations on architecture variants (e.g., SPLR vs. IPLR), feedback mechanisms, and output correction terms, giving credibility to their design choices.\n\nWeaknesses:\n1. The paper is not easy to follow for readers who are not already familiar with recent state-space models like Mamba, or with LLM-centric efficient RNNs. \n2. The term “Nonlinear RNN” in the title may be confusing for researchers coming from classical RNN/statistical learning backgrounds, where RNNs are already nonlinear in most cases by design. It might benefit from a subtitle or clarification.\n3. The notation is dense and not well-initialized early in the paper. Symbols  are used heavily but not clearly introduced or contextualized for new readers. This affects the clarity of theoretical sections.\n\nQUESTIONS:\n1. I find the discussion on whether models like Mamba2 are considered open-loop or closed-loop somewhat confusing. From a general perspective, most RNNs, including Mamba2, are inherently “closed-loop” in the sense that the current state S_t depends on the previous state S_{t-1}, which serves as a form of feedback. This contrasts with autoregressive models like Transformers (with KV cache), which are more “open-loop” because they do not incorporate such recurrent state feedback in the same way. However, in Section 3, the paper seems to classify Mamba2 as an open-loop control system. Does this mean that the term “closed-loop” in this work specifically refers to the inclusion of negative feedback mechanisms, such as corrective terms that subtract projections from previous states (e.g., v_t - P_t(S_{t-1}))? If so, it would be helpful to explicitly distinguish this interpretation from the broader control-theoretic or RNN-centric notions of feedback.\n\n2.The experimental results convincingly demonstrate that the proposed model achieves strong performance. However, I still find it unclear whether the negative feedback mechanism is truly the key contributing factor. Could the authors provide more explanation, either theoretical or empirical, on why negative feedback helps in this setting? For instance, does it improve stability, memory precision, or long-range dependency handling in a measurable way? A deeper discussion or ablation focusing specifically on the feedback terms would be very helpful.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents Comba, a nonlinear RNN model with SPLR state transition and output correction to improve the model performance. Using chunkwise parallelization, Comba can be efficiently trained, with 40% speedup compared to Gated DeltaNet. The effectiveness of two proposed techniques is confirmed by experimental results on language and vision tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. Sec. 2 and Table. 1 provide an informative overview of the development of linear and nonlinear RNNs and the motivations behind these designs.\n\n2. The effectiveness of SPLR and output correction are verified through extensive ablation studies; several variants of Comba are explored to justify the design decisions.\n\n3. Open access to the code is provided.\n\nWeaknesses:\n\n1. It is claimed in the title and Sec. 3 that the design of Comba is based on closed-loop control. However, it is quite difficult to identify the key components (i.e., controller, plant, error detector, feedback element, etc.) of a closed-loop control system from the equations in Table 3 and Eq. 5. In addition, it is difficult to see how closed-loop control helps to improve model performance, besides the general claim that negative feedback improves system adaptability (Line 148). I believe providing a block diagram (as commonly seen in the control literature) for each column of Table 3 would provide better insight to Comba.\n\n2. The output correction seems to be a feed-forward instead of a feedback path, because the output $\\boldsymbol{o}_t$ is still a linear transforma of the state $\\boldsymbol{S}_t$, just replacing the projection vector from $\\boldsymbol{q}_t$ to $\\boldsymbol{q}_t-d\\boldsymbol{k}_t$, and both $\\boldsymbol{q}_t$ & $\\boldsymbol{k}_t$ depend on the instantaneous input (maybe I misunderstood what ‘feedback’ exactly means in Comba – see W.1). In this case, it might be incorrect to claim that TTT/DeltaNet/RWKV-7 are not strictly ‘closed-loop systems’ (Line 155) and that Comba has two-stage feedback (Line 149).\n\n3. The initial value of $d$ (for output correction) seems to affect the model performance substantially, and the optimal choice of the initial value seems to heavily depend on the model size (50x difference in $d$ for 3x difference in model size). Investigating the model performance for other values of $d$ (between 0.02 and 1) would help practitioners in choosing this hyperparameter.\n\\end{enumerate}\n\nQUESTIONS:\nQuestions:\n\n1. What is the required memory size for different sequence lengths during training? How does it compare with the Transformer and other modern RNN variants?\n\n2. Eq. 3 seems to only hold for $\\alpha=1$. Is this intended?\n\n3. In Table 2, the input dimension of Mamba2 is said to be $2D/H$. Should it be $2D$?\n\n4. Appendix A.1 is empty. Is this intended?\n\n5. Based on Table 1, the difference between the state transition equations of Gated DeltaNet and Comba seems to be the parametrization of $\\alpha_t$ and $\\beta_t$. In this case, why is Comba sufficiently faster than Gated DeltaNet? Could you apply the method in Sec. 3.2 to Gated DeltaNet similarly so that only one matrix inverse is computed?\n\nLIMITATIONS:\nYes: the authors have adequately addressed the limitations and potential negative societal impact of their work", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Comba as a new model to the series of sub-quadratic architectures like Mamba-2 \\[1] and Gated-DeltaNet \\[2]. It makes the following contributions:\n\n1. It improves the Gated DeltaNet recurrence,\n\n   $$\n    \\mathbf{S_t}  = \\mathbf{S_{t-1}} {\\alpha_t} (\\mathbf{I} - {\\beta_t} \\mathbf{k_t}\\mathbf{k_t}^T) + \\beta_t \\mathbf{v_t}\\mathbf{k_t}^T\n   $$\n\n   to\n\n   $$\n     \\mathbf{S_t}  = \\mathbf{S_{t-1}} (\\alpha_t \\mathbf{I} - \\tilde{\\beta}_t \\mathbf{k_t}\\mathbf{k_t}^T) + \\beta_t \\mathbf{v_t}\\mathbf{k_t}^T\n   $$\n\n   untying the two $\\beta$s and relocating $\\alpha$ (motivated by control theory).\n\n2. It uses UT transforms for its kernel, making it 40 % more efficient than the WY transform.\n\n3. It shows improvements in performance on language modeling, recall tasks, Long Bench, and ImageNet classification.\n\n\n-----\n\n[1]: Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality. Tri Dao, Albert Gu\n\n[2]: Gated Delta Networks: Improving Mamba2 with Delta Rule. Songlin Yang, Jan Kautz, Ali Hatamizadeh\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n1. The paper is well written and the proposed change is straightforward to follow.\n2. The PPL improvements on Wiki and LAMBADA over Gated DeltaNet are substantial.\n3. The kernel achieves a 40 % speedup compared to the WY transform.\n\n### Weaknesses\n\n1. The switch from WY to UT transform for Delta Rule–based methods appears to have been introduced in the DeltaNet paper \\[3]. Could the authors clarify how Comba’s algorithm differs from \\[3] beyond the expected updates to the $W$ and $T$ matrices?\n\n2. In Table 6 (main LM results), I think important baselines seem to be under-tuned/missing:\n\n   * **Gated DeltaNet**:\n\n     * DeltaNet \\[3] reports Wiki/LAMBADA PPLs of $(28.24\\,37.37)$ and $(16.87\\,12.21)$ for the 340M/1.3B models on SlimPajama (15 B/100 B tokens).\n     * The authors report $(26.47\\,45.46)$ and $(17.14\\,18.80)$ for Gated DeltaNet. These results are worse than DeltaNet itself, even though Gated DeltaNet is known to more performant than DeltaNet.\n     * Comba achieves $(24.15\\,39.91)$ and $(16.01\\,12.68)$.\n      \n     Under this light, the results become much more mixed than they seem to be. Could the authors please comment on this discrepancy\n\n   * **Mamba-2 vs. Mamba**:\n     I recommend using Mamba-2 \\[1] rather than the original Mamba \\[4] in Table 6, as Mamba-2 is the more recent and better-performing variant.\n\n   * **Transformer++**:\n     I understand that the authors have used a previous paper for Transformer++ results, but I would recommend training this model in-house. The reported PPL numbers strongly suggest that this baseline has been poorly tuned. For reference, please see that the gap is much less pronounced in the numbers reported in Table 2 in MesaNet \\[5]\n\n\nTo sum up, I believe Gated DeltaNet, Mamba-2, and Transformer++ are the most important baselines, and they appear under-tuned/missing. I would recommend better-tuning these baselines to more reliably understand Comba’s improvements.\n\n----\n\n[3]: Parallelizing Linear Transformers with the Delta Rule over Sequence Length. Songlin Yang, Bailin Wang, Yu Zhang, Yikang Shen, Yoon Kim\n\n[4]: Mamba: Linear-Time Sequence Modeling with Selective State Spaces. Albert Gu, Tri Dao\n\n[5]: MesaNet: Sequence Modeling by Locally Optimal Test-Time Training. Johannes von Oswald, Nino Scherrer, Seijin Kobayashi, Luca Versari, Songlin Yang, Maximilian Schlegel, Kaitlin Maile, Yanick Schimpf, Oliver Sieberling, Alexander Meulemans, Rif A. Saurous, Guillaume Lajoie, Charlotte Frenkel, Razvan Pascanu, Blaise Agüera y Arcas, João Sacramento\n\nQUESTIONS:\nPlease see weaknesses\n\nLIMITATIONS:\nPlease see weaknesses", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}