Title: Recursive Inference Scaling: A Winning Path to Scalable Inference in Language and Multimodal Systems
Venue: NeurIPS 2025
Accepted: yes
Keywords: Recursive inference

scaling inference time

self-similarity

language models

SigLIP

multimodal systems.
Abstract: Inspired by recent findings on the fractal geometry of language, we introduce Recursive INference Scaling (RINS) as a complementary, plug-in recipe for scaling inference time in language and multimodal systems. RINS is a particular form of recursive depth that significantly outperforms +55 other variants, including the recent "repeat-all-over" (RAO) strategy in Mobile LLM (Liu et al., 2024) and latent recurrent thinking (Geiping et al., 2025). Unlike prior works, we carry out our comparisons on a compute-matched regime, and demonstrate that for a  fixed model size and training compute budget, RINS substantially improves language modeling performance. It also generalizes beyond pure language tasks, delivering gains in multimodal systems, including a +2% improvement in 0-shot ImageNet accuracy for SigLIP-B/16. Additionally, by deriving data scaling laws, we show that RINS improves both the asymptotic performance limits and the scaling exponents. More importantly, with light-weight (linear) adapters (comprising <1% of model parameters) and stochastic dropout, RINS offers a no-regret strategy, meaning that RINS-enabled pretraining improves performance in language modeling even when recursive depth is not applied at inference time. This corresponds to improving performance on a training compute-, parameter-, and inference-matched regime, suggesting its potential as a viable component of LLM pretraining!


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes Recursive INference Scaling (RINS), a recursive architectural design aimed at improving inference performance in language and multimodal models. RINS recursively applies early parts of the model (Block A) multiple times before forwarding to the later blocks (Block B), with the idea of leveraging the self-similar (fractal) structure of language. The authors benchmark this against other recursive methods like Repeat-All-Over (RAO) and show gains in both language modeling and multimodal tasks under compute-matched settings.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- Compute-Controlled Experiments: The paper controls for training compute, which is commendable and eliminates a common confounding factor in model comparisons.\n\n- Multimodal Evaluation: The application to SigLIP and multimodal benchmarks adds empirical breadth.\n\n- Stochastic RINS Design: Introduces inference flexibility and reduces regret via lightweight adapters, potentially improving deployment flexibility.\n\n\n\n\n**Weaknesses**\n\n***Overhyped Novelty:***\n- The idea of recursion or repeated inference is not new—RAO, latent recurrent thinking, and universal transformers have explored similar ground.\n\n- RINS is presented as a novel and superior alternative, but it essentially tunes recursive patterns and partitions, lacking substantial conceptual innovation.\n\n***Heavily Empirical, Light on Insight:***\n\n- While the experiments are abundant, they mostly confirm what could be expected: more compute during inference can help, especially with training-time regularization.\n\n- There is little theoretical grounding or mechanistic insight into why RINS specifically works better than RAO, aside from the fractal analogy.\n\n***Limited Generalization Beyond Specific Settings:***\n\n- The improvements are only observed in language and vision-language setups. In pure vision tasks, RINS offers no benefit—this suggests the method is brittle and task-specific.\n\n- It is unclear if RINS scales well to larger LLMs (e.g., >7B), especially given the overhead of recursive inference.\n\n***Deployment Realism Unclear:***\n\n- While the paper proposes stochastic variants to mitigate inference costs, real-world deployment often requires strict latency budgets, and recursive inference is not easily compatible with that.\n\n- The actual wall-clock inference latency and memory overhead are not reported, which is crucial for a method claiming \"scalable inference\".\n\n**Lack of Simpler Baseline Comparisons:**\n\n- Comparisons are primarily made against RAO and long-sequence baselines, but not against simpler strategies like output sampling ensembles or basic CoT prompting under same FLOPs.\n\n- It's unclear if the gains come from architectural recursion or simply from deeper, repetitive computation paths.\n\nQUESTIONS:\n- Would RINS still outperform simpler sampling-based ensemble methods if evaluated under equal wall-clock time constraints?\n\n- How do you recommend partitioning a large pretrained model into A and B blocks in real applications—what are the criteria?\n\n- Can you provide evidence that RINS remains beneficial when scaling to models >1B parameters with real deployment constraints?\n\nLIMITATIONS:\nNo — The discussion on societal impact and practical deployment tradeoffs is underdeveloped. Authors should include analysis of runtime cost, carbon footprint, or fairness implications of enabling RINS, especially for multimodal systems used in global-scale platforms.", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nRecursive Inference Scaling (RINS) work has taken inspiration from the self-similar(fractal) nature of language modality and seeks to exploit scale-invariant decoding for better generalization. The methodology decomposes a model into two sequential blocks (A & B), and then recursively applies first block A multiple times (A^rB structure). This helps in increasing the inference time depth without increasing parameter count. An exhaustive ablation and empirical study have been performed using a compute-matched evaluation framework. RINS consistently outperforms most of the baselines.\nRINS also showcase better scaling exponents and asymptotic limits, demonstrating superior sample efficiency and generalization.\nFurther, the paper introduces stochastic RINS, where recursion depth is randomized during training, and shows that incorporating lightweight linear adapters(<1%) enables no-regret-training. RINS trained models retain improved performance even when recursive inference is skipped at test time. Additionally, KVcache sharing study has been performed, to reduce memory cost while maintaining performance gains. Overall, RINS aims to provide plug and play method for scaling inference in compact models, with benefits validated across both language and multimodal tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths-\n1. The paper provides a highly empirical study with evaluation on more than 55 architectural variants, multiple model sizes(300M-1B), and tasks. All experiments are compute and parameter matched, ensuring fair comparisons. It is great to see alot of common downstream tasks were covered including but not limited to OpenBookQA, PIQA, 0-shot Imagenet classification.\n2. The authors have presented the method, motivation and results in a well-structured and accessible manner. The recursion signature, degree, adapter use, KVcache sharing provide insights into the trade offs and behavior of RINS. The notation, taxonomy and diagrams aid understanding.\n3. The training details, datasets, FLOPs and failure cases(OOM for some models) are explicitly detailed.\n4. The use case is highly relevant for deployment constrained environments.\n5. The no-regret training setup, that provides training time benefits even without using recursion at test time is an important and practical innovation that lowers adoption cost. \n6. The motivation from fractal hypothesis of language and the idea of recursively applying a subset of model instead of full model seems quite novel and interesting.\n\nWeakness-\n1. While the paper is extremely insightful and provides strong empirical justification, it is not clear what is the deep theoretical framework explaining why RINS work so great. It isn't very clear about what exactly the recursive A block learning or refining over iterations.\n2. While inference compute(FLOPs) is matched, real world wall-time or memory benchmarks which are crucial for deployment are not presented.\n3. Most of the experiments are limited to 300M-1B parameter model, would it provide gains for actual LLM use case models.\n\nQUESTIONS:\n1. It would be nice to get some theoretical insights on what exactly RINS is benefitting from, or what is the recursive blocks optimizing or learning.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

comments: SUMMARY:
This paper introduces Recursive INference Scaling (RINS), a method for scaling inference time in language and multimodal models by recursively applying a subset of model layers. The paper splits a model into two blocks A and B, where block A is applied recursively r times to its own output before passing to block B (A^r B). Further combining with stochastically dropping and linear adapter, RINS-enabled pretraining improves performance in language modeling even when recursive depth is not applied at inference time. The paper evaluates RINS against 55+ other parameter-sharing strategies under compute-matched conditions and demonstrate consistent improvements in language modeling and multimodal tasks.

STRENGTHS AND WEAKNESSES:
Strengthes:
- The paper proposes a simple, elegant and effective A^r B parameter-sharing method, and systematically studies the scaling law of it, across model size, modalities, tasks, and data size.
- The comparison with the control of training budget is fair and meaningful.
- Provide a stochastic RINS which improves performance even when recursion isn't used at inference, making the method more practical feasible.
- Analyze the KV cache sharing for reduced memory footprint.

Weaknesses:
- The paper targets the deployment environments with stringent memory limitations, but does not include latency analysis or profiling memory footprint.
- Evaluations are focused primarily on C4/SlimPajama and perplexity for language modeling.

QUESTIONS:
- Could you explain more the hypothesize about self-similar geometry and how it supports RINS theoretically.
- Why some results show very large error bars?

LIMITATIONS:
See weaknesses.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03

comments: SUMMARY:
This paper proposes that the (A^r)B recurrent structure is the best structure for parameter sharing. However, I find several parts of the article confusing and difficult to understand. I hope the author can clarify these points during the rebuttal phase.

STRENGTHS AND WEAKNESSES:
I list some of the points that I find confusing as follows:
- Why it is a winning path to scalable inference in Language and **Multimodal Systems**, regarding that this paper (Appendix G) concludes that "parameter-sharing techniques, including RINS, do not confer any advantage in supervised image classification. The non-recursive baseline surpasses all other methods. This starkly differs from the result observed in language modeling." Though Section 6 provides some other experiments about multimodality. It is still unclear what the relationship between these two parts is.
- How is this paper related to **inference scaling**? The experiments focus on the PPL and training FLOPs. I don't find contents related to scalable inference.
- The experiments are conducted on 300M/600M models. We can hardly say it is a **winning path** based on observations from such small models.

QUESTIONS:
My questions are listed above.

LIMITATIONS:
YES
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-05