Title: Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach
Venue: NeurIPS 2025
Accepted: yes
Keywords: test-time scaling

large language models

recurrence

reasoning

depth

universal transformers
Abstract: We study a novel language model architecture that is capable of scaling test-time computation by implicitly reasoning in latent space.  Our model works by iterating a recurrent block, thereby unrolling to arbitrary depth at test-time. This stands in contrast to mainstream reasoning models that scale up compute by producing more tokens. Unlike approaches based on chain-of-thought, our approach does not require any specialized training data, can work with small context windows, and can capture types of reasoning that are not easily represented in words. 
We train a proof-of-concept model from scratch with 3.5 billion parameters and 800 billion tokens. We show that this model can effortlessly use varying levels of compute, significantly improving with additional compute especially on reasoning tasks, such as math and coding. Further, this architecture naturally reduces compute costs via zero-shot per-token adaptive compute, KV-cache sharing and speculative decoding.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper designs a recurrent transformer architecture and evaluates it on language modeling tasks. The model is trained with a varying number of recurrences. This enables adaptive test-time compute by applying the recurrent block a varying number of times. The authors compare to a nonrecurrent model of the same architecture. The recurrent model with high test time compute performs slightly better. In addition, the authors provide an analysis of the latent space trajectories. \"Overall, we believe this may be the largest language model training run to completion in terms of number of devices used in parallel on an AMD cluster, as of time of writing\"\n\nSTRENGTHS AND WEAKNESSES:\nThe main strength of this paper is that the authors train a really large model, with a novel architecture to see how good the results are. \nThe biggest value in this paper is Table 4, specifically the first three rows of it. Here, we can see how good the recurrent model is compared to the non-recurrent baseline. This is very scientifically useful. \n\nThe main weakness of the paper is that the \"message\" of the paper is very confused. The authors are trying to argue that the recurrent approach could have advantages, but the results of this large scale experiment seem to indicate that the current recurrent model approach is not worth. I think the paper would be much stronger with a clearer message. It feels like the authors threw a lot of random things in the paper to try to make it better, but these all feel like distractions compared to the main point of this paper, which is experimenting with a huge recurrent model training run. \n\nI think of this paper as a \"negative result\" for recurrent architectures, that is still worth being published. It's a huge experiment and this work should clearly be published. However, I think the framing of the paper obfuscates the most valuable lessons from this paper. I think the authors probably threw in a lot of other analyses to make the paper seem stronger, but I would rather have it more clearly present the negative result to make it easier for others to build on it.\n\nTable 4: this table shows that compared to a fixed-depth baseline, the recurrent model only performs a bit better when using R=32 recurrences. The paper states that the recurrent model \"consumes flops close to what a 32B parameter transformer would consume during pretraining\" and the R=32 test time compute is comparable in flops to a 50B parameter fixed depth. So this approach seems really not worth it — almost 10x parameters to the baseline, and hardly higher in performance.\n\nThe remaining comparisons, such as table 1, I don't find to be very useful, given how many flops the model consumes in comparison to the other models.\n\n## Strengths\n- The paper trains a really large model. \"Overall, we believe this may be the largest language model training run to completion in terms of number of devices used in parallel on an AMD cluster, as of time of writing\" — wow!\n- The recurrent model seems well-designed and is described well in the paper.\n- The authors provide a ton of various analyses of the model, and a thorough appendix with their training methodology. \n\n## Weaknesses\n- Many of the figures are far too small to read when the paper is printed out. (4, 5, 6, and many in the appendix) — need to make sure the axes and labels can be read.\n- Many of the sections of the paper are not very enlightening. Table 4 is the main payday. \n  - Section 4.4 is just about ensembling to improve metrics.\n  - Section 6 tries to analyze the mechanisms in the recurrent model, but it's unclear whether the analysis uncovers anything that's not already present in fixed-depth models (see questions section)\n  - Figure 4 is very poorly explained (see questions section)\n\nQUESTIONS:\nQ1. Is it fair to compare the recurrent model, when evaluated at R=32, to a 32B parameter model in terms of training Flops, or 50B for test flops? Or are there other metrics that are useful for comparing the model to baselines \"of equal ability\" ? Maybe it would be useful to show the number of flops for each model in tables such as Table 1 and Table 4.\n\nQ2. Section 5: the KL divergence between two successive steps is used for adaptive compute. What exactly is the KL divergence between? The latent state? Figure 4 is really confusing... what is Continuous CoT? what is Default? These labels are never described.\n\nQ3. It's unclear to me whether there's any insight in the analysis in section 6. Can the same analysis be applied to nonrecurrent models, but looking at how the latent state between layers evolves over time? Have such analyses been done, and do the results here differ? The paper does mention that similar multi-dimensional orbits have been observed in fixed-depth transformers for arithmetic tasks.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel language model architecture that scales test-time computation through \"latent reasoning\" by iterating a shared, recurrent block of layers. This recurrent depth approach is presented as an alternative to mainstream methods. The authors demonstrate that their 3.5B parameter model significantly improves on reasoning-intensive tasks, such as math and coding, as more computational steps (recurrences) are applied at test time. A key contribution is showing that this architecture also inherently enables several inference optimizations, including adaptive compute, KV-cache sharing, and speculative decoding.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper successfully trains a 3.5B parameter model and provides comprehensive benchmarks to show the method's effectiveness.\n2. The architecture's ability to support adaptive compute, KV-cache sharing, and self-speculative decoding is interesting.\n3. The visualization of token trajectories in latent space reveals emergent structured patterns like \"orbits\" and \"sliders\". This provides evidence for how the model performs abstract computations.\n\nWeaknesses:\n1. The authors are transparent that due to compute constraints, only one large-scale model was trained. While the results are promising, this makes it difficult to fully disentangle the effects of the architecture from the specific data mix and other hyperparameters.\n2. Although positioned as an alternative to Chain-of-Thought, the paper lacks a direct performance comparison under an equivalent test-time FLOP budget. Such a comparison would be highly informative for evaluating the trade-offs between the two approaches.\n\nQUESTIONS:\n1. Could you provide a more direct comparison of the inference FLOPs? For example, for GSM8k, how many FLOPs does your model at r=32 use compared to the OLMo-7B model generating a chain-of-thought response? \n2. How sensitive is the final model's performance to the choice of `k` (Truncated Backpropagation)?\n\nLIMITATIONS:\nSee weakness and questions.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a recurrent module in transformer blocks to allow for adaptive compute at inference time. The recurrent module is able to run for any number of iterations, and the main argument that the amount of inference compute can be controlled by using different number of iterations at test time. This work pretrained a 3.5B language model using the introduced recurrent module, and found that the resulting model gets different levels of performance when using different numbers of iterations in the recurrent module: using more iterations leads to higher inference performance. This work also analyzes the trained model to evaluate where recurrence helps and also finds potential mechanisms emerging for challenging questions.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The idea of using recurrence to allow for adaptive compute at inference time is appealing.\n2. Experiments pretrained a 3.5B model using the proposed module from scratch, which shows the potential of this approach on large models.\n\nWeaknesses:\n1. The potential benefit of using recurrence is to achieve latent reasoning, but experiments on math and reasoning benchmarks still rely on CoT reasoning which is not latent. Can you test on no CoT prompting (similar to implicit CoT works) on math and reasoning benchmarks to see if the recurrence can achieve latent reasoning without still having to rely on explicit CoT?\n2. The median number of sampled iterations is 29, so at training time 50% of examples have seen recurrence for 29 or more iterations. However, looking at Figure 1, at inference time the performance has saturated at 20 iterations, way before 29 iterations. This is disappointing as the hope for the recurrent module is that by using more compute we can get higher performance; however, the results suggest that it doesn't generalize to anything beyond seen at training, and maybe it's better to simply fix the number of iterations to 20 and train a model.\n3. The recurrence module used a Gaussian noise as input, but the effectiveness of doing so is not justified in experiments. At inference time, is this noise vector set to 0 or still being sampled from the Gaussian? Which version gets better performance?\n4. While there are some mechanism analyses, it's still not clear what kind of reasoning the model is performing.\n\nQUESTIONS:\n1. What's the benefit of training a recurrent module over increasing the number of layers of a transformer? If the goal is to achieve adaptive inference compute, can't you simply randomly drop transformer layers during training?\n2. What is the fixed-depth baseline in table 4 exactly? What do you mean by a non-recurrent twin? Is that using a fixed number of iterations in the recurrent module?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n- This paper proposes a novel language model architecture capable of scaling test-time computation by reasoning in latent space rather than through explicit token generation.\n\n- The model iteratively applies a recurrent block, allowing it to unroll to arbitrary depth at inference time, in contrast to conventional chain-of-thought methods that increase compute by generating more tokens.\n\n- Unlike prior reasoning models, this approach does not require specialized training data, can work with small context windows, and is able to model reasoning patterns not easily expressed in natural language.\n\n- The authors train a 3.5B parameter model on 800B tokens as a proof of concept, demonstrating that reasoning performance improves with increased compute allocation at inference time, particularly on math and coding tasks.\n\n- The architecture also supports adaptive per-token compute, KV-cache sharing, and speculative decoding to reduce overall inference costs.\n\nSTRENGTHS AND WEAKNESSES:\n- Strengths:\n  - The description of the **scalable recurrent architecture in Section 3 is clear and well-articulated**. In particular, Lines 78–83 are elegantly written and convey the design motivation effectively. The details of the model configuration in Section 3.2 are also clearly presented.\n\n  - The paper conducts **comprehensive baseline comparisons**. Table 1 benchmarks the proposed model against Pythia and other models with comparable parameter sizes, while Appendix Table 7 extends the comparisons to recent strong models such as Qwen2.5 and LLaMA 3, making the evaluation thorough and convincing.\n\n  - **The analysis of the role of Recurrence is detailed and well-explained**. Notably, Section 5 provides clear discussions on its interaction with KV cache reuse, self-speculative decoding (SD), and other aspects, demonstrating the broader significance of Recurrence beyond just computational scaling.\n\n- Weaknesses\n  - It would be valuable to include comparisons with other **recent large reasoning models** of a similar scale, such as Qwen 3, to further contextualize the model’s performance.\n\n  - The discussion on the role and benefits of **Recurrence in the main text is somewhat redundant in places**. **A better organization** and separation of detailed analyses between the main paper and the Appendix could improve clarity and readability.\n\nQUESTIONS:\nSee Strengths And Weaknesses.\n\nLIMITATIONS:\nSee Strengths And Weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-06"}