Title: Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Adaptive Computation

Dynamic Routing

Early-exiting

Parameter Sharing

Language Model

Efficiency
Abstract: Scaling language models unlocks impressive capabilities, but the accompanying computational and memory demands make both training and deployment expensive. Existing efficiency efforts typically target either parameter sharing or adaptive computation, leaving open the question of how to attain both simultaneously. We introduce Mixture-of-Recursions (MoR), a unified framework that combines the two axes of efficiency inside a single Recursive Transformer. MoR reuses a shared stack of layers across recursion steps to achieve parameter efficiency, while lightweight routers enable adaptive token-level thinking by dynamically assigning different recursion depths to individual tokens. This allows MoR to focus quadratic attention computation only among tokens still active at a given recursion depth, further improving memory access efficiency by selectively caching only their key-value pairs. Beyond these core mechanisms, we also propose a KV sharing variant that reuses KV pairs from the first recursion, specifically designed to further decrease memory footprint. Across model scales ranging from 135M to 1.7B parameters, MoR forms a new Pareto frontier: at equal training FLOPs and smaller model sizes, it significantly lowers validation perplexity and improves few-shot accuracy, while delivering higher throughput compared with vanilla and existing recursive baselines.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Mixture-of-Recursions, a  that aims to integrate two mainstream model efficiency techniques—parameter sharing and dynamic computation (i.e., adaptive depth) into a single, unified architecture. The core idea is to reuse a shared block of layers within a recursive Transformer to achieve parameter efficiency, while simultaneously employing a lightweight, token-level router to dynamically assign different recursion depths to each token.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n-  The idea of combining parameter sharing and dynamic computation within a single recursive framework is highly novel and forward-looking. Developing more efficient large language models without sacrificing performance is a core challenge in the field, and this paper's research direction is well-aligned with this goal, making it highly significant.\n\n- The design of the MoR framework is well-considered, encompassing not only dynamic routing but also a corresponding KV cache strategy to form a complete solution. The paper's explanation of the methodology is clear.\n\n- The authors evaluate their model across multiple scales and computational budgets and conduct several ablation studies (e.g., on parameter sharing and routing strategies), which demonstrates the effort invested in validating their method.\n\n**Weaknesses**\n\n- A core claim of the paper is that MoR is more efficient to train, but this is primarily justified using theoretical FLOPs rather than actual wall-clock time. The dynamic routing and conditional computation logic introduced by MoR will incur additional overhead, which may offset or even negate the FLOPs savings on real hardware. The advantage of a truly efficient architecture should ultimately be reflected in end-to-end training time. The lack of a comparison based on actual training time undermines the claim of efficiency.\n\n-  The paper's main results rely on the expert-choice routing strategy, but the ablation study shows that this method requires an auxiliary loss to function correctly, which adds complexity and potential instability to training. Meanwhile, the simpler \"Token-choice\" strategy performs significantly worse. This raises doubts about whether the success of the MoR framework is overly dependent on a specific, finicky routing mechanism rather than a clean, robust design.\n\n- The comparison between the M-Cyc and Cyc strategies may fail to properly control for variables. M-Cyc has independent first and last layers, meaning its shared recursive module has fewer parameters than that of Cyc. Therefore, it is unclear whether M-Cyc's superiority stems from its structural advantage or simply from its smaller, easier-to-optimize recursive module. A more rigorous ablation study is needed to disentangle these factors.\n\nQUESTIONS:\nPlease refer to my weakness part.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper aims at improving the efficiency of recursive transformers, where the inputs is looped over the model for several recursive steps. The efficiency gains are obtained across three main axis. First, a routing mechanism allocates tokens to different loops. Two variants are proposed, by sequentially dropping tokens at each loop, or by allocating the tokens to different number of loops prior to start each of them. Second, by proposing two alternatives for caring the activations, specifically tailored for the recursive computation performed by the model. Third, different ways of grouping layers. The results show that recursive transformers with these modifications become as compute efficient as vanilla transformers despite using much less parameters. Several ablations compare all the different proposed alternatives in the setup.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n- **Strong improvements over baselines**: The proposed MoR architecture significantly outperforms other recursive baselines across compute budgets, achieving lower validation loss and higher few-shot accuracy (e.g., Table 3) while maintaining efficiency through dynamic recursion. It is impressive that it is as FLOPs efficient as a vanilla transformer despite using much less parameters. \n- **Efficient use of compute**: The model demonstrates compute optimality consistent with scaling laws (e.g., Chinchilla scaling). The models and baselines seem well tuned. I verified with quick calculations that with the FLOPs budget of $16.5 * 10^{18}$ the vanilla model with the 730M parameter should be the optimal, which indeed appears to be true from Figure 3.\n- **Advances in recurrent efficiency**: The focus on improving efficiency in recurrent architectures, which are typically FLOP-intensive, addresses an important practical bottleneck.\n- **Pareto improvements in throughput**: MoR delivers substantial throughput gains, positioning it as a practical solution for high-speed inference, which is also a remarkable achievement. \n\n### Weaknesses\n\n- **More recursion is detrimental?**: Across the results in the paper, such as in iso-flops comparisons, fixed token setting and throughout plots, it seems that more recursive steps give worse performance. This seems to indicate some issues with the experimental set up. I would be expecting performance to saturate, but not to degrade, with more recursions. In this context, trainability issues do not seem to be addressed. Or could it be that the combination of routing mechanism + high number of recursions causes signal degeneracies? Some of these issues were addressed in related works, see e.g. \"Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach\".\n- **Test-time scaling interpretation**: The conclusion that “allocating more recursion steps at inference can improve generation quality” is overstated. The results suggest that across models, fewer recursion steps are generally better, and additional recursion helps only within a given model. \n- **Shallow semantic analysis**: The discussion of how the router aligns recursion depth with semantic importance is somewhat superficial. The mechanism by which token context determines exit decisions could be better examined. See also questions on this point.\n- **Vanilla still outperforms at iso-parameter**: In iso-parameter settings, the vanilla Transformer continues to outperform recurrent variants, including MoR, as seen in some configurations in Figure 4. While not a major flaw given the paper’s focus on efficiency, this limitation could be acknowledged more explicitly.\n- Minor: **Decoding logic clarity**: The description of the decoding process—particularly how tokens that exit early are dynamically substituted, and how masking and caching are handled—is insufficiently detailed. This is important for interpreting the throughput results of Figure 4a and understanding practical deployment implications.\n- Minor: **Caching mechanism explanation**: The prose explaining the caching mechanism (e.g., lines 143–149) could be clearer, especially in detailing the logic behind the claimed memory savings and consistency of KV caches. Right now, it feels a bit too compressed. \n\nOverall, I am in favor of acceptance, although I will consider increasing my score based on the author's rebuttal, especially on the issue of why more recursions does not improve performance.\n\nQUESTIONS:\n- **Effective FLOPs-parameter trade-off**: For fixed FLOPs, MoR with $N_r = 2$ strikes an optimal balance, while deeper recursion degrades performance as parameter count is reduced disproportionately. Can the authors expand on this point? Is the model saturating its capacity with too many recursions until becoming inefficient?\n\n- Does the model quickly decode because the prediction is easier, i.e. the model is more confident about predicting? What if the a token was forced to be decoded despite the router allocating it to the next step?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the authors present Mixture-of-Recursions (MoR), a unified framework for improving both the training and inference efficiency of Transformer models. MoR combines two key ideas: parameter sharing through recursive reuse of a shared stack of layers, and adaptive computation over the tokens (i.e, not all tokens proceed to full depth/receive full compute). This allows the model to allocate computation proportionally to importance, reducing redundant computation and significantly lowering overall cost compared to conventional baselines. MoR is shown to dominate the Pareto frontier in iso-FLOP settings, consistently achieving lower validation perplexity with better efficiency. Additionally, this conditional token computation delivers strong inference speedup on several model sizes. These results highlight MoR as a compelling direction for scaling model quality without incurring full-scale computational costs.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. Clarity and Presentation: This paper is well written, with diagrams and figures that clearly convey the core ideas. In particular, Figure 1 effectively illustrates how computation is allocated per output token during the decoding stage, providing intuitive insight into the mechanism of Mixture-of-Recursions (MoR).\n\n2. KV Caching Design: The approach to key-value caching is both novel and well-motivated. The authors show thoughtful design by selectively storing KV entries only for tokens that pass through routed blocks, minimizing unnecessary memory use. Meanwhile, the layer-sharing component ensures full-context awareness during generation, demonstrating a strong understanding of the importance of maintaining contextual integrity in autoregressive decoding.\n\n3. Strong Empirical Results: The experimental results are compelling. MoR consistently dominates the compute-performance Pareto frontier across model scales and benchmarks. The ablations on cache strategy, throughput, and router configuration provide valuable insights into the contribution of each component and the rationale behind the final design choices. The detailed router analysis further supports the effectiveness of the method.\n\nWeaknesses:\nWhile the paper is overall well-executed and well-motivated, I believe the methodology could benefit from deeper discussion in relation to prior work on conditional depth computation. In particular, \"Mixture-of-Depths: Dynamically Allocating Compute in Transformer-Based Language Models\" (arXiv 2024) is notably relevant. That paper also explores token-level conditional routing using depth-based computation and introduces a similar routing heatmap to Figure 1 in this work.\n\nGiven these similarities, it would be valuable for the authors to clarify how MoR goes beyond a naive combination of Mixture-of-Experts (MoE) and Mixture-of-Depths (MoD). Without this, one might interpret MoR as simply integrating known ideas (layer skipping + routing) under a recursive framing. An in-depth comparison or positioning in the related work would help distinguish the unique contributions of this paper.\n\nQUESTIONS:\nThe main question I have is whether the authors could explicitly contrast MoR with a conceptual combination of MoE and MoD. Since the layer router could be viewed as a generalization of expert selection, and the token-router can be viewed similarly as depth allocation, it would be useful to understand what architectural or algorithmic elements of MoR are fundamentally new, beyond being a combination of existing approaches (refer to weaknesses).\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

comments: SUMMARY:
This paper proposes a new method named mixture-of-recursive, which considers both parameter and computation efficiency. It combines the (expert-level or token-level) router with recursive layers, and also KV cache sharing strategies to improve model performance, can achieve similar or even better performance than vanilla transformer with 1/3 parameters when training from scratch.

STRENGTHS AND WEAKNESSES:
Strengths:
1. The ideas are clear and effective, and the writing is excellent and very easy to follow
2. The experimental results are comprehensive and impressive. Utilizing recursive layers could save the number of parameters but will always cause a little drop of performance. While here utilizing token router can let model learn better on important tokens and improve performance. This paper considers different scales of models and computations and different combinations of strategies, and show consistent advantage compared with vanilla recursive layers.
3. Doing many ablations studies to show how to combine different routers and archs, etc.

Weakness:
1. maybe giving the calculation process of Tab2 (right) in appendix and refer it in Tab 2 would be helpful for readers, because they are dependent on some specific capability factor sequence (N_r/N_r, ..., 1/N_r)
2. It seems that as increasing compute budget, the performance gain from MoR is decreasing. And on small models (like 135M), it perform worse than vanilla models.

QUESTIONS:
could you give some intuitions about the example in Fig.1 Right bottom, why words like "---" or "and" "those" will be pass twise while words like "comfortable" "views" only once?

LIMITATIONS:
authors mention their limitations in appendix A
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03