Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach
Authors: Jonas Geiping
Sean Michael McLeish
Neel Jain
John Kirchenbauer
Siddharth Singh
Brian R. Bartoldson
Bhavya Kailkhura
Abhinav Bhatele
Tom Goldstein
Venue: NeurIPS 2025 spotlight ()
Keywords: test-time scaling
large language models
recurrence
reasoning
depth
universal transformers
Primary area: deep_learning
TL;DR: We show that recurrent-depth transformers can be scaled to be effective language models, with particularly strong gains through additional compute for reasoning tasks.

Abstract: We study a novel language model architecture that is capable of scaling test-time computation by implicitly reasoning in latent space.  Our model works by iterating a recurrent block, thereby unrolling to arbitrary depth at test-time. This stands in contrast to mainstream reasoning models that scale up compute by producing more tokens. Unlike approaches based on chain-of-thought, our approach does not require any specialized training data, can work with small context windows, and can capture types of reasoning that are not easily represented in words. 
We train a proof-of-concept model from scratch with 3.5 billion parameters and 800 billion tokens. We show that this model can effortlessly use varying levels of compute, significantly improving with additional compute especially on reasoning tasks, such as math and coding. Further, this architecture naturally reduces compute costs via zero-shot per-token adaptive compute, KV-cache sharing and speculative decoding.

Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach
Jonas Geiping Sean M
Neel Jain
John Kirchenbauer
Siddharth Singh ?
Brian R. Bartoldson
Bhavya Kailkhura²
Abhinav Bhatele?
Tom Goldstein
'ELLIS Institute Tübingen, Max-Planck Institute for Intelligent Systems, Tübingen AI Center University of Maryland, College Park *Lawrence Livermore National Laboratory amas@tue.ellis. itomg@umd. edu
Abstract
We study a novel language model architecture that is capable of scaling test-time computation by implicitly reasoning in latent space. Our model works by iterating a recurrent block, thereby unrolling to arbitrary depth at test-time. This stands in contrast to mainstream reasoning models that scale up compute by producing more tokens. Unlike approaches based on chain-of- thouuhh our approach does not require any specialized training data, can work with small context windows, and can capture types of reasoning that are not easily represented in words. We train proof-of-concept model from scratch with 3.5 billion parameters and 800 billion tokens. We show that this model can effortlessly use varying levels of compute, significantly improving with additional compute especially on reasoning tasks, such as math and coding. Further, this architecture naturally reduces compute costs via zero-shot per-token adaptive compute, KV-cache sharing and speculative decoding.
Scaling by Thinking in Continuous Space
Humans naturally expend more mental effort solving some problems than others. While humans are capable of thinking over long time spans by verbalizing intermediate results and writing them down, substantial amount of thought happens through complex, recurrent firing patterns in the brain, before the first word of an answer is uttered.
Early attempts at increasing the power of language models focused on scaling model size, a practice that requires extreme amounts of data and computation. More recently, researchers have explored ways to enhance the reasoning capability of models by scaling test time computation. The mainstream approach involves post-training on long thain-of-thougee examples develop the model's ability to verbalize intermediate calculations in its context window and thereby externalize thoughts.
However, the constraint that expensive internal reasoning must always be projected down single verbalized next token appears wasteful; is plausible that models could be more competent if they were able natively "think" in their continuous latent space. One way unlock this untapped dimension of additional compute involves adding recurrent unit to a model. This unit runs in loop, iteratively processing and updating its hidden state and enabling computations be carried on indefinitely. While this is not currently the dominant paradigm. this idea is foundational to machine learning and has been (re- rrecvdicc in every decade, for example as recurrent neural networks, diffusion models, feature recycling, and as universal or looped transformers.
In this work, show that depth-recurrent language models learn effectively, be trained in an efficient manner, and demonstrate significant performance improvements under the scaling of test-time compute. Our proposed transformer architecture is built upon latent depth-recurrent
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Scaling up Test-Time Compute with Recurrent Depth
Matertalized d Parameters
Performance Gains through Recurreat Depth
8.38
14.8R21.08 3368 52,68
103.28
50.0
50
K.N2
40
ABCxecy
64
Accrray (95
Hellww
30
MRPC
ABS-batoong
20
Open 0814
Mattrmndd (L. hond
challenge rage
OpenBoooOOA
GSMBK CUT
COMiE sta
Randor Chorses
Perteesanne (rw))
De: Thi Tixe Compute
6
12
20
32
48
64
Test-Time Compute Recurrnnce
Accuracy (75)
Figure 1: We train 3.5B parameter language madel with depth recurrence. Left: can iterate longer to use more compute and improve its performance. Instead scaling test-time reasoning by "verbalizing" in long Chains-of- ThoThhtt the model improves entirely by reasoning in its own latent space. Right: Simple tasks that require less reasoning like SciQ and BLiMP are solved almost immediately, while tasks like Mastermind (the reasoning game), MMLU, GSM8k or HvaanEval are solved only with extended compute via test time recurrence, separation in difficulty levels that emerges from pretraining the model scale.
At test time, the model
block that is run for randomly sampled number of iterations during training. We show that this paradigm can scale to several billion parameters and over half a trillion tokens of pretraining data. At test-time, the model can improve its performance through recurrent reasoning in latent space, enabling it to compete with other open-source models that benefit from more parameters and training data. Additionally, we show that recurrent depth models naturally support number of features at inference time that require substantial tuning and research effort in non-recurrent models, such as per-token adaptive compute, -selulspeeu decoding, and KV-cache sharing. We finish our study by tracking token trajectories in latent space, showing that number of interesting computation behaviors simply emerge with scale, such as the model rotating shapes in latent space for numerical computations.
2 Why Train Models with Recurrent Depth?
Recurrent layers enable transformer model to perform arbitrarily many computations before emitting a token. In principle, recurrent mechanisms provide a simple solution for test-time compute scaling, Compared more standard approach of long context reasoning OpenAl (2024); DeepSeck-AI et al. (2025), latent recurrent thinking has several advantages.
Latent reasoning does not require construction of bespoke training data. Chain-of-thoughee reasoning requires the model to be trained on long, domain-spectie demonstrations. Our proposed latent reasoning models can train with variable compute budget, using standard training data with no specialized demonstrations, and enhance their abilities at test-time if given additional compute. Latent reasoning models require less memory for training and inference than chain-of-ther -hain-hf-thought reasoning models. Because the latter require extremely long context windows, specialized training methods such as patelpaarlllllizatoon Liu et al. (2023a) may be needed.
Recurrent-depth networks perform more FLOPs per parameter than standard transformers, signifi- cantly reducing communication costs between accelerators at scale. This especially enables higher device utilization when training with slower interconnects.
By constructing an architecture that is compute-heavy and small in parameter count, we hope to set a strong prior towards models that solve problems by "thinking", i.c. by learning meta-strategies, logic and abstraction, instead of memorizing. The strength of recurrent priors for learning complex algorithms has already been demonstrated in the "deep thinking" literature Schwarzschild et al. (2021b); Bansal et al. (2022); Schwarzschild et al. (2023).
On a more philosophical note, we hope that latent reasoning captures facets of human reasoning that defy verbalization, such as spatial thinking, physical intuition or (motor) planning. Over many iterations of the recurrent process, reasoning in high-dimensional vector space would enable the deep exploration of multiple directions simultaneously, instead of linear thinking, leading to a system capable of exhibiting novel and complex reasoning behavior.

---

Scaling compute in this manner is not at odds with scaling through extended (verbalized) inference scaling (Shao al., 2024), or scaling parameter counts in pretraining (Kaplan al., 2020), we argue it may build third axis which to scale model performance.
3 A Scalable Recurrent Architecture
In this section we will describe our proposed architecture for a transformer with latent recurrent depth, discussing design choices and small-scale ablations. A diagram of the architecture be found in Figure 2. We always refer to the sequence dimension as n, the hidden dimension of the model as h, and its vocabulary as the set V.
3.1 Macroscopic Design
The model primarily structured around decoder-only transformer blocks (Vaswani et al., 2017; Radford et al., 2019). However these blocks are structured into three functional groups, the prelude P. which embeds the input data into a latent space using multiple transformer layers, then the core Raxh,
recurrent block R. which is the central unit of recurrent computation modifying states and finally the coda C, which un-embeds from latent space using several layers and also contains the prediction head of the model. The core block is set between the prelude and coda blocks, and by looping the core we can put an indefinite amount of verses in our song.
Given a number of recurrent iterations and a sequence of input tokens x E V these groups used in the following way to produce output probabilities p E [R*x [V]
e = P(x)
So ~
S; R(c,s,-1)
for i (1....,,]
p = C(s,),
where some standard deviation for initializing the random state. This process is shown in Figure 2. Given an init random state So, the model repeatedly applies the core block R, which accepts the latent state s and the embedded input e and outputs a new latent state s,. After finishing all iterations, the coda block processes the last state and produces the probabilities of the next token. This architecture based on deep thinking literature, where it shown that injecting the inputs in every step (Bansal et al., 2022) and initializing the latent vector with random state stabilizes the recurrence and promotes convergence steady state independent of initialization, i.e. path independence (Anil et al., 2022).
3.2 Microscopic Design
Within each group, broadly follow standard transformer layer design. Each block contains multiple layers, and each layer contains a standard, causal self-attention block using RoPE (Su et al., 2021) with a base of 50000, and a gated SiLU MLP (Shazeer, 2020). We use RMSNorm (Zhang and Sennrich, 2019) as our normalization function. The model has learnable biases on queries and keys, and nowhere else. To stabilize the recurrence, we order all layers in the following "sandwich" format, using norm layers n4, related to Ding et al. (2021); Team Gemma et al. (2024):
X =22 (x + (atn(n1(xx(1)) (&t + MLP(na(X:))
While at small scales, most normalization strategies, e.g. pre-norm, post-norm and others, work almost equally well, ablate these options and find that this normalization is required train the recurrence at scale.
Given embedding matrix E' and embedding scale y, the prelude block first embeds input tokens x as Ex)), and then to applies (p many prelude layers with the layout described above. Our core recurrent block R starts with an adapter matrix : 122h - R mapping the concatenation of s; and into the hidden dimension h (Bansal et al., 2022). While re-incorporation of initial embedding features via addition rather than concatenation works equally well for smaller models, we find that concatenation works best at scale. This is then fed into (g transformer layers, At the end of the core

---

0.00
Dessity
0.00
"Hello"
"World"
0.01
5
R
0.00
Sampled:
Prelude
Block
Recurrent
Coda
Input Injection Residual Stream
Mean 33.0
Dunelly
Medo 24.0
Median
Figure 2: Left: A visualization of the Architecture, as described in Scction 3. Each block consists of a number of sub-layers. The blue prelude block embeds the impuls into latent space, where the green shared recurrent block is block of layers that is repeated compute the final latent state, which is decoded by the layers of the coda block. Right: We use a log-normal Poisson Distribution sample the number recurrent iterations for each training step.
block the output is again rescaled with RMSNorm Ne. The coda contains lo layers, normalization by ne, and projection into the vocabulary using tied embeddings E"
summarize the architecture by the triplet (1p, (c), describing the number of layers in each stage, and by the number recurrences r. which may vary in each forward pass. We train number of small-scale models with shape (1, 4, 1) and hidden size h 1024. in addition to large model with shape (2, 4, 2) and = 5280. This model has only 8 "real" layers, but when the recurrent block is iterated, e.g. 32 times, it unfolds constructing computation chains that can be deeper than even the largest fixed-depth transformers (Levine al., 2021; Merrill et al., 2022; Saunshi et al., 2024).
In summary,
effective depth of 2 4r + 132 layers,
3.3 Training Objective
Training Recurrent Models through Unrolling. ensure that the model can function when we scale up recurrent iterations test-time, we randomly sample iteration counts during training, assigning random number of iterations
every input sequence (Schwarzschild et al., 2021b). We optimize the expectation of the loss function L over random samples from distribution X and random iteration counts y from distribution A.
(()) Xx Ry [ (mm(xx,)) x').
Here, m represents the model output, and x' is the sequence x shifted left, i.c., the next tokens in the sequence x. We choose A to be a log-normal Poisson distribution. Given a targeted mean recurrence + and variance that we set 3.4 we can sample from this distribution via
((log(()) -0002000 00000
r ~~ P(e))
(2)
given the normal distribution N and Poisson distribution P, see Figure 2. The distribution most often samples values less than F, but it contains a heavy tail of occasional events in which significantly more iterations are taken.
Truncated Backpropagation. To keep computation and memory low at train time, we backprop- agate through only the last & iterations of the recurrent unit. This enables to train with the heavy-tailed Poisson distribution A, as maximum activation memory and backward compute is now independent of r. We fix & 8 in main experiments. At small scale, this works well as sampling & uniformly, but it equalizes the overall memory usage in each step of training. Note that the prelude block still receives gradient updates in every step, its output is injected in every step. This setup resembles truncated backpropagation through time, as commonly done with RNNs, although our setup recurrent in depth rather than time (Williams and Peng, 1990; Mikolov al., 201 Truncated backpropagation can also be understood as approximation of objectives based on differentiating fixed point of the recurrence, as discussed in Geng al. (2021).
3.4 How to Train Large-Scale Recurrent-Dee Depth Model In the Wild
After verifying that we can reliably train small test models up to 10B tokens, we move on to larger- scale runs. Given our limited compute budget, could either train multiple tiny models too small to

---

show emergent effects or scaling, train a single medium scale model. Based on this, we prepared single large-scale run. We train a 3.5B parameter variant of the proposed architecture on mix of generic text, code and scientific data, with data-parallel training with batch size of 16 million tokens. We provide comprehensive information on all training details in Appendix C.
Benchmark Results
We ultimately train the final model for 800B tokens, and non-recurrent baseline comparison for 180B tokens. We evaluate these checkpoints against other open-source models trained on fully public datasets (like ours) of similar size. We compare against Amber (Liu al., 2023c), Pythia (Biderman et al., 2023) and a number of OLMo 1&2 variants (Groeneveld et al., 2024; A12, 2024; Team OLMo et al., 2025). We execute all standard benchmarks through the Im-eval harness (Biderman et al., 2024) and code benchmarks via bigcode bench (Zhuo et al., 2024).
4.1 Standard Benchmarks
Overall, is not straightforward to place our model in direct comparison to other large language mod- els, all of which are small variations of the standard fixed-depth transformer architecture. While our model has only 3.5B parameters and hence requires only modest interconnect bandwidth during pre- training, it consumes FLOPs (but not memory bandwidth) close to what 32B parameter transformer would consume during pretraining, and can continuously improve in performance with test-time scaling up to FLOP budgets equivalent to what would be standard 50B parameter fixed-depth trans- former. Finally, the model trained on only 800B tokens, while large comparison to older fully open-source models such as the Pythia series, is small in comparison to modern open-source efforts such as OLMo, and tiny in comparison to the datasets used to train industrial open- gpen-weight models.
We collect results for established benchmark tasks (Team OLMo et al., 2025) in Table 1 and show all open-source models side-by-side. In direct comparison we see that our model outperforms the older Pythia series and is roughly comparable to the first OLMo generation, OLMo-7B in most metrics, lags behind the later OLMo models trained larger, more carefully curated datasets. For the first recurrent-depth model for language to be trained at this scale, and considering the limitations of the training run, we find these results promising and certainly suggestive that further research into latent recurrence as an approach to test-time scaling is warranted.
4.2 Math and Coding Benchmarks
We also evaluate model on math and coding. For math, we evaluate GSMSK (Cobbe al., 2021) (as 5-shot and in the 8-way CoT setup), MATH ((Hendrycks et al., 2021b) with the Minerva evaluation
Table 1: Results on zero-shot evaluations across open-source models. We show ARC (Clark et al., 2018),
HellaSwag (Zellers al., 2019), MMLU (Hendrycks et al., 2021a), OpenBookQA (Mihaylov et al., 2018), PiQA (Bisk et al., 2020), SciQ (Johannes Welbl, 2017), and WinoGrande (Sakaguchi al., 2021). We report normalized accuracy on PiQA, OBQA, ARC-C and HellaSwag.
Model
Param Tokens ARC-E
ARC-C HellaSwag
MMLU
OBQA
PiQA
SciQ
WinoGr
random
25.0
25.0
25.0
25.0
25.0
50.0
25.0
50.0
Amber
7B
1.2T
65.70
37.20
72.54
26.77
41.00
78.73
88.50
63.22
Pythia-2.8b
2.8B
0.3T
58.00
32.51
59.17
25.05
35.40
73.29
83.60
57.85
Pythia-6.9b
6.9B
60.48
34.64
63.32
25.74
37.20
75.79
82.90
61.40
Pythia-12b
12B
0.3T
63.22
34.64
66.72
24.01
35.40
75.84
84.40
63.06
OLMo-1B
1B
3T
57.28
30.72
63.00
24.33
36.40
75.24
78.70
59.19
OLMo-7B
7B
2.5T
68.81
40.27
75.52
28.39
42.20
80.03
88.50
67.09
OLMo-7B-0424
7B
2.05T
75.13
45.05
77.24
47.46
41.60
80.09
96.00
68.19
0LMM-778-072
7B
2.75T
74.28
43.43
77.76
50.18
41.60
80.69
95.70
67.17
OLMo-2-
7B
82.79
57.42
80.50
60.56
46.20
81.18
96.40
74.74
32B
0.8T
79.46
54.69
79.85
59.49
48.00
81.07
92.70
75.69
Ours, 4)
3.5B
49.07
27.99
43.46
23.39
28.20
64.96
80.00
$5.24
Ours, (r 8)
3.5B
0.8T
65.11
35.15
58.54
25.29
35.40
73.45
92.10
55.64
Ours, (r 16)
3.5B
0.8T
69.49
37.71
64.67
31.25
37.60
75.79
93.90
57.77
Ours, (r 32)
3.5B
0.8T
69.91
38.23
65.21
31.38
38.80
76.22
93.50
59.43
5

---

Table 2: Benchmarks of math. reasoning and understanding. We
Table 3: Evaluation code benchmarks, report flexible and strict match for GSM8K and GSM8K COT. MBPP and HvmanEval. We report pass @ extracted match for Minerva Math, and acc norm, for MathQA.
for both datasets.
Model
GSMRK
GSMSK CoT
Minerva MATH MathQA
Model
Param
Takens
MBPP
HumanEval
Random
0.00
0.00
0.00
20,00
Random
0.00
0.00
Amber
1,59/2
3,94/1,32
3.34/5.16
1.002.81
1.91
1.96
25.26
24.52
staroader--6
starender2-3b
3B
7B
3.7T
3.3T
43,80
43.00
31.20
31.09
Pythin-6.9h
2,05/2443
2 88
1.38
28.96
Amber
78
1.2T
19.60
13:41
12b
3.49/1622
3.54/466
2.56
25.80
Pythia-2.5 Kh
2.88
0.3T
6.70
7.92
OLMo-IB
1.82/2.27
1.59/255
160
23.38
Pythia-4:9h
6.98
0.3T
7.92
5.60
OI MB
4.0224.10
6,0777728
2.12
28.26
Pythia 12b
12B
0.3T
5.60
OLMo-78-0424
27.0 27.07/27.22 29
26 06.23/26.22
5,56
28.48
OLMo- IB
OLMo-2-1124-78
OLMo-7B-0724
56. 66.72/66.99
28.66/28.73
28.89/28.89
94/66.966 19
19:08
5.62
27.84
37,59
0LMo-788-424
OLMo-7B
7B
7B
2.05T
2.5T
21.20
15.6
16.46
12,80
OLMo-2-328-08T
28.43/8.551
26 76/32.3237
5.72
33,90
OJ Mo-78-07724
01---2--114-78
7B
7B
2.75T
4T
25.60
20.12
10.36
Ours (r 32)
28.05/28.20
32.60/34.57
12.58
26.60
0LMo-2-32B-087
0.8T
19.80
17.68
Ours w/ templ. 32)
24.87/98.13
34.87/42.8
11:24
27.97
Ours (1' - 32)
3.5B
0.8T
24.80
23.17
Table 4: Baseline comparison, comparing the recurrent model with non-recurrent (fixed (ixed-depth) model with the same parameter count, trained the same training setup and data. Comparing the recurrent model with its non-ree con-recurrent baseline, we see that even at 180B tokens, the recurrent substantially outperforms on harder tasks.
Model
Tokens|
ARC-E
ARC-C HellaSwag
MMLU OBQA
PIQA SeiQ WinoGr GSM8K CoT
Non-Recurrent Baseline
0.18T
46.42
26.96
37.34
24.16
29.60
64.47
73.20
51.78
1.82/2.20
Ours, early ckpt. (r = 32)
0.18T
53.62
29.18
48.80
25.59
31.40
68.88
80.60
52.88
9.02/10.24
Ours, early ckpt. (r 1)
0.18T
34.01
23.72
29.19
23.47
25.60
53.26
54.10
53.75
0.00/0.15
Ours, (r = 32)
0.8T
69.91
38.23
65.21
31,38
38.80
76.22
93.50
59.43
34. 34.8042.08
Ours, (r = 1)
0.8T
34.89
24.06
29.34
23.60
26.80
55.33
47.10
49.41
0.00/0.00
rules (Lewkowyez et al.. 2022)) and MathQA (Amini al., 2019). For coding, we check MBPP (Austin et al., 2021) and HvmanEval (Chen et al., 2021). Here we find that our model significantly surpasses all models except the latest OLMo-2 model in mathematical reasoning, as measured on GSM8k and MATH. On coding benchmarks the model beats all other general-purposs open- open-source models, although it does not outperform dedicated code models, such as StarCoder2 (Lozhkov et al., 2024), trained for several trillion tokens. We also note that while further improvements in language modeling are slowing down, as expected at this training scale, both code and mathematical reasoning continue to improve steadily throughout training, see Figure 3.
4.3 Where does recurrence help most?
How much of this performance can we attribute to recurrence, and how much other factors, such as dataset, tokenization and architectural choices? In Table 4, we compare our recurrent model against its non-recurrent twin, which we trained 180B tokens in the exact same setting. In direct comparison of both models at 1 180B tokens, we see that the recurrent model outperforms its baseline with an especially pronounced advantage on harder tasks, such as the ARC challenge set. On other tasks, such as SciQ, which requires straightforward recall of scientific facts, performance of the models is more similar. We observe that gains through reasoning are especially prominent on GSM8K, where the 180B recurrent model is already 5 times better than the baseline at this early snapshot in the pretraining process. We also note that the recurrent model, when evaluated with only a single
32
Hac
Rox
32 Roc
4xx
Ree
Roc
Xix
Reo
Box
Fox
.... 16 Reo
64 64
35
65
30
60
20
55
HelleSeepp
50
HomsnEwwl
1.5
GSMBK
20
65
15
40
10
10
35
0
100
200
300
400 500
600
700
800
100
200
300
400
500
700
800
200
300
400
6.0
700
800
Token (BiBiss)
Tokene Traited (Bilkan)
(Bilise)
Figure 3: GSM8K CoT, HellaSwag, and HumanEval performance over the training tokens with different recurrences at test-time. We evaluate GSM8K CoT with chat template and 8-way few shot as multitum. HellaSwag and HumanEval are zero-shot with no chat template. Model performance on harder tasks grows almost linearly with the training budget, if provided sufficient test- time compute.

---

Sgh.stiest cuthonatio
palainpyy
logiaal Dallcces
0.2
0.3
602
691
25
Stepet KL -alddaed TThrcchldd
Figure 4: Histograms of zero-shot, per-token adaptive exits based on KL difference between steps for questions from MMLU categories. The mean of each distribution given in the legends. The exit threshold is fixed to x 10-1. We see that model converges quicker on high school mathematics than tasks such as logical fallacies or moral scenarios. Further, on some tasks, such as philosophy, the model can also effectively re-use states in its latent CoT (denoted as "cont, compute") and converge quickly on subset of tokens, leading to fewer steps required overall, more details for this inference variant can be found in Appendix D.6.
recurrence, effectively stops improving between the early 180B checkpoint and the 800B checkpoint on hard tasks, showing that further improvements are not built into the fixed, non-recurrent parts but encoded entirely into the iterations of the recurrent block.
4.4 Improvements through Weight Averaging
Due to our constant learning rate, we can materialize further improvements through weight averaging (Izmailov et al., 2018) to simulate the result of a cooldown (Hägele et al., 2024; DeepSeek-Ai al., 2024). We use an exponential moving average starting from our last checkpoint with 0.9. incorporating the last 75 checkpoints with a dilation factor of 7, a modification to established protocols (Kaddour, 2022; Sanyal et al., 2024). We evaluate this EMA model as well, which further improves GMS8k performance to 47.23% flexible (38.59% strict), when tested at r = 64.
5 Recurrent Depth simplifies LLMs
Aside from encouraging performance in mathematical and code reasoning. recurrent-deper models turn out be surprisingly natural tools to support a number of methods that require substantial effort with standard transformers. In the next section, we provide a non-exhaustive overview.
Zero-Shot Adaptive Compute at Test-Time. We have shown that the model is capable of varying compute on a per-query level, running the model in different recurrence modes. This after all also how the model is trained, as in Equation (1), However, it would be more efficient in practice to stop recurring early when predictions are easy, and only spend compute on hard decisions. Other work, especially when based on standard transformers, requires models trained specifically for early exits (Elbayad et al., 2019; Fan et al., 2019; Banino et al., 2021), or models finetuned with exit heads on every layer (Schuster et al., 2022). To test our model's zero-shot exit abilities, choose simple exit criterion to evaluate convergence, the KL-divergence between two the next token probabilities of two successive recurrence steps. If this divergence falls below 5 10 10-4, we stop iterating, sample the output token, and move to generate the next token. This is just one early-exit option, and we experiment with several other schemes in Appendix Table 9.
We show this zero-shot per-token adaptive compute behavior in Figure 4, where we plot the distribu- tion of steps taken before the exit condition is hit. We do this for the first 50 questions from different MMLU categories, asked in free-form chat. Interestingly, the number of steps required to exit differs notably between categories, with the model exiting earlier high school mathematics, but taking on average 3.5 steps more on moral scenarios. We verify on MTBench that this adaptivity does not significantly impact performance in conversational settings (standard: 5.63, early exits: 5.56), and even on hard tasks such as GSM8k, the merged model still reaches 44.8% (at r 32, instead of 46% when exiting early, see Table 9).
Zero-Shot K-ccchhe Sharing. A different avenue to increase efficiency is to reduce the memory footprint of the KV-cache by sharing the cache between layers (character.ai, 2024; Brandon et al., 2024). Typically, transformers must be trained from scratch with this capability. However, as discussed in the previous section, we find that can simply share KV-caches in our model with
7

---

Latest State
for
79
10P
mikng
un
bonh
houadsidd
Suceere
100
schol
mally
wnogg
ne
<wnd
<Baujn bassieff>
100
110
102
Text
Figure 5: Convergence of latent states per token, for every token in a sequence (going top to bottom) and latent iterations (going left to right). Shown is an unsafe question posed to the model. Highly token-specific convergence rates emerge simply from training. surprising the model only trained with constant whole sequences. Convergence is especially slow on the key part the question, really wrong-ed. Not pictured is the model refusing to answer after deliberating the semantics of the question.
minimal impact to performance. We set fixed KV-cache budget for the recurrence every token k and at iteration i, read and write the cache entry mod k. For example, we set a maximum KV-cache budget of 16 steps, overwriting the KV-cache of the 1st recurrence step when executing the 17th step, and so forth. This can be used on its own to reduce KV cache memory, or in combination with per-token adaptive compute discussed above. On MTBench GSM8K, reducing KV-cache memory through sharing does not reduce performance, see Table 9.
Zero-Shot Speculapeu Decoding. Recurrent-depth models can also inherently generate text more efficiently by using speculative decoding (Leviathan et al., 2023) without the need for separate draft model. With standard transformer models, speculative decoding requires external draft model, Medusa heads (Cai et al., 2024), or early-exit adaptation (Zhang et al., 2024b; Elhoushi et al., 2024). Zhang et al. (2024b) implement specuulccu decoding simply through layer skipping, but this does not always result in good draft quality. In comparison, our model can naturally be run with fewer iterations to draft the next N tokens in the generated sequence, which any desired number iterations M > later. Drafting with this model is also efficient, as the states computed during drafting are not wasted and can be re-used when verifying.
then be verified with
6 What Mechanisms Emerge at Scale in Recurrent-Depth Models
Finally, what is the model doing while recurring in latent space? To understand this question better, we analyze the trajectories {s of the model on a few qualitative examples. We are especially interested in understanding what patterns emerge, simply by training this model at scale. In comparison to previous work, such as Bai al. (2019), where the training objective directly encodes prior that pushes trajectories to a fixed point, we only train with truncated unrolling objective.
Figure shows the norm distance ||s, ||s,-s) s*] between each s; trajectory and an approximate limit point s* at r 128. We show the sentence top to bottom and iterations from left to right. We clearly see that convergence behavior depends on context. We see that key parts of the question, and the start of the model response, are "deliberated" much more in latent space. The context dependence can also be seen in the different behavior among the three identical tokens representing each of the three dots. Also note that the distance to s" does not always decrease monotonically (c.g. for school);

---

PCI
PC3 PC4
PCS PCB
10
wrong"
PC1
PC4
PCS PCG
Figure 6: Latent Space trajectories for select tokens. Shown the first 6 PCA directions over the latent state trajectories of tokens sequence. The color gradient going from dark to bright represents steps in the trajectory, center mass marked red. The model has learned to use complex patterns, such as orhits "sliders" to represent and handle more advanced concepts, such as arithmetic or complicated delihcration.
the model may also trace out complicated orbits in its latent trajectory while processing information, though this represented explicitly in training objective.
We look at trajectories for select tokens in more detail in Figure 6. We compute PCA decomposition of latent trajectories over all tokens in sequence, and then show several individual trajectories projected onto the first six PCA directions, with more examples in the appendix. Many tokens simply converge to fixed point. Yet, for harder questions, such in the 1st row', the state of the token quickly falls into an orbit pattern in all three pairs PCA directions. The use multi-dimennsionll orbits like these could serve similar purpose periodic patterns sometimes observed in fixed-depth transformers trained for arithmetic tasks (Nanda et al., 2022), but we find these patterns extend far beyond arithmetic for our model. We often observe the use of orbits tokens such "makes" (see Figure 15) or "thinks" that determine the structure of the response.
Aside from orbits, also observe the model encoding particular key tokens as "sliders", as seen in the middle of the 2nd row in Figure (which is the token "wrong", from the same message as already shown in Figure 5). In these motions the trajectory noticeably drifts in single direction, which the model could use to implement a mechanism to count how many iterations have occurred.
The emergence of structured trajectories in latent space gives a glimpse into how the model performs its computations. Unlike the discrete sequential chain of reasoning seen in verbalized ofain-of-thought approaches. we observe rich geometric patterns including orbits, convergent paths, and drifts means to organize its computational process spatially. This suggests the model independently learning to leverage the high-dimensional nature of its latent space to implement reasoning in new ways.
Related Work Overview
The extent which recurrence is foundational concept of machine learning is hard overstate (Amari, 1972; Hopfield, 1982; Braitenberg, 1986; Gers and Schmidhuber, 2000; Sutskever et al.,
This is the token "3" in GSM8k test question that opens with Claire makes egg omelette.

---

2008). Aside from using recurrence to move along sequences, as in recurrent neural networks, it was understood early to also be the key to adaptive computation (Schmidhuber, 2012; Graves, 2017). For transformers, recurrence was applied in Dehghani et al. (2019), who highlight the aim of recurrent depth to model universal, i.e. Turing-compllee, machines (Graves et al., 2014). was used at scale (but with fixed recurrence) in Lan et al. (2019) and an interesting recent improvement in this line of work are described in Tan et al. (2023); Abnar et al. (2023), Mathur et al. (2024) and Csordás et al. (2024). Schwarzschild et al. (2021b); Bansal et al. (2022); Bear et al. (2024) and McLeish et al. (2024) show that depth recurrence is advantageous when learning generalizable algorithms when training with randomized unrolling and imput injections. Recent work has described depth rectrrrecurrent, looped, transformers and studied their potential benefits with careful theoretical and small-scale analysis (Giannou et al., 2023; Gatmiry et al., 2024; Yang et al., 2024a; Fan et al., 2025; Saunshi et al., 2024). Our study reinforces these prior works, showing not only how to scale depth recurrence to billion parameter scales and token counts, but also that conjectured advantages such as algorithm learning and reasoning with extended compute do materialize for realistic benchmark tasks.
From another angle, these models can be described as neural networks learning fixed-point iteration, as studied in deep equilibrium models (Bai et al., 2019, 2022; Schöne et al., 2025). The variant of latent recurrent depth we discuss in this work is also related to diffusion models (Song and Ermon, 2019), especially latent diffusion models (Rombach et al., 2022), but we note that language diffusion models usually run with per-sequence, instead of per-token, iteration count (Lee et al., 2018). 8).. A key difference of our approach to both equilibrium models and diffusion models is in the training objective, where equilibrium methods solve the implicit bilevel problem directly, diffusion models solve a surrogate training objective, and our work suggests that truncated unrolling is a powerful alternative at scale, see also Geng et al. (2021).
More generally, all architectures that recur in depth can also be understood as directly learning the ana- log to the gradient of latent energy-based model (LeCun and Huang, 2005; LeCun, 2022), to an im- plicitly defined intermediate optimization layer (Amos and Kolter, 2017), or to a Kuramoto layer (Miy- ato et al., 2024). Analogies to gradient descent at inference time also show the connection to test time adaptation (Sun et al., 2020), especially test- test-time adaptation of output states (Boudiaf et al., 2022).
While we consider the proposed recurrent depth approach to be very natural way to learn to reason in continuous latent space from the ground up, the works of Hao et al. (2024); Cheng and Durme (2024) and Liu et al. (2024) discuss how to finetune existing fixed-depth transformers with this capability. These works have similar aim to ours, enabling reasoning in latent space, but approach this goal from separate directions. For additional discussions related to the idea of constructing prior that incentivizes reasoning and algorithm learning at the expense of memorization of simple patterns, we also refer to Chollet (2019), Schwarzschild (2023), Li et al. (2020) and Moulton (2023).
Limitations and Conclusions
While the experiments in this paper demonstrate the viability of (latent) recurrent-depth architectures for language modeling at scale, the models described are ultimately still a proof-of-concee We observe that we can train models that improve with increased test-time scaling via recurrence, improving over a fixed-depth model with the same parameter count by 5x on GSM8K. We observe that in our training recipe, performance saturation depends on task complexity, but is always sigmoidal in the number of iterations. Nevertheless, future work with additional compute still required to allow for precise comparisons to the other forms of scaling, such as training fixtd-depth transformers with the same FLOP count in pretraining, or training verbal CoT models targeting the same FLOP count at test time.
Yet, the interesting behaviors already observable in this work, such as the context-dependent problem- solving speed, with the model learning to solve easy problems with fewer recurrences than harder problems, various zero-shot abilities and emergence of structured thinking in latent space, lead us to believe that latent reasoning is a promising research direction to complement existing approaches for test-time compute scaling. Our work validates the motivations and observations of prior work developed at smaller scales for universal, looped and deep thinking transformers, and we are excited about the potential impact of imbuing generative models with the ability to reason in continuous latent space without the need for specialized data at train time or verbalization at inference time.
10

---

Acknowledgements
An award for computer time was provided by the U.S. Department of Energy's (DOE) Innovative and Novel Computational Impact on Theory and Experiment (INCITE) Program. This research used resources of the Oak Ridge Leadership Computing Facility at the Oak Ridge National Laboratory, which is supported by the Office of Science of the U.S. Department of Energy under Contract Work on the LLNL side was prepared by LLNL under Contract DE- AC52-07NA27344 and supported by the LLNL-LDRD Program under Project No. 24-ERD-010 and 24-ERD-058 CLNL-CCONF-87239), This manuscript has been authored by Lawrence Livermore National Security, LLC under Contract No. TEB-C52-07NA227444 with the U.S. Department of Energy. The United States Government retains a non-exclusive, paid-up, irrevocable, world-wide license to publish or reproduce the published form of this manuscript, or allow others to do so, for United States Government purposes.
No.
JG further acknowledges the support of the Hector II foundation. A large number of small-scale and preliminary experiments were made possible through the support of the MPI Intelligent Systems compute cluster and funding by the Tübingen Al center.
UMD researchers were further supported by the ONR MURI program, DARPA TIAMAT, the National Science Foundation 2212122,,, and the NSF TRAILS Institute (2229885), Commercial support was provided by Capital One Bank, the Amazon Research Award program, and Open Philanthropy. Finally, we thank Avi Schwarzschild for helpful comments on the initial draft.
References
Samira Abnar, Omid Saremi, Laurent Dinh, Shantel Wilson, Miguel Angel Bautista, Chen Huang, Vimal Thilak, Etai Littwin, Jiatao Gu, Josh Susskind, and Samy Bengio. 2023. Adaptivity and Modularity for Efficient Generalization Over Task Complexity. 10.0063.0..8886(css..
A12. 2024. OLMo 1.7-7B: A 24 point improvement on MMLU.
Zeyuan Allen-Zhu Zhu and Yuanzhi Li. 2024. Physics of language models: 3.1. knowledge storage and extraction. In Proceedings of the 41st International Conference on Machine Learning. volume 235 of ICML 224, pages 1067-1077, Vienna, Austria. JMLR.org.
S.-L. Amari. 1972. Learning Patterns and Pattern Sequences by Self-Organizing Nets of Threshold Elements. IEEE Transactions on Computers, C-21(11)1117--1266
AMD. 2021. AMD IMstinct MI250X Accelerators.
Aida Amini, Saadia Gabriel, Peter Lin, Rik Concel-Kedziorsee Yejin Choi, and Hannanch Hajishirzi. 2019. Mathqa: Towards interpretable math word problem solving with operation-based formalisms. arXiv preprint
1905.19995.1311
Brandon Amos and J. Zico Kolter. 2017. OptNet:
Differentiable Optimization as a Layer in Neural Networks. In International Conference on Machine Learning, pages 136-145.
Cem Anil, Ashwini Pokle, Kaiqu Liang, Johannes Treutlein, Yuhuai Wu, Shaojie Bai, J. 7ico Kolter, and Roger Baker Grosse. 2022. Path Independent Equilibrium Models Can Better Exploit Test-Time Computation. In Advances in Neural Information Processing Systems.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and others. 2021. Program synthesis with large language models. arXiv preprint arXiv: 108.0220.
Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Mareo Dos Santos, Stephen Mareus MeAleer, Albert Q. Jiang, Jia Deng, Stella Biderman, and Sean Welleck. 2023. Llemma: An Open Language Model for Mathematics. In The Twelfth International Conference on Learning Representations.
Shaojie Bai, J. Zico Kolter, and Vladlen Koltun. 2019. Deep Equilibrium Models. In Advances in Neural Information Processing Systems, volume 32, Curran Associates, Inc.
Shaojie Bai, Vladlen Koltun, and J. Zico Kolter. 2022. Neural Deep Equilibrium Solvers. In International Conference on Learning Representations.
Yushi Bai, Jiajie Zhang, Xin Lv, Linzhi Zheng, Sigi Zhu, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2024. Long Writer: Unleashing 10,000+ Word Generation from Long Context LLMs. Saki::2408.07055/cr
11

---

Andrea Banino, Jan Balaguer, and Charles Blundell. 2021. PonderNet: Learning to Ponder. In 8th ICML Workshop on Automated Machine Learning (AutoML).
Arpit Bansal, Avi Schwarzschild, Eitan Borgnia, Zeyad Emam, Furong Huang, Micah Goldblum, and Tom Gold- stein. 2022. End-to-end Algorithm Synthesis with Recurrent Networks: Extrapolation without Overthinking. In Advances in Neural Information Processing Systems.
Jay Bear, Adam Prügel-Bennett and Jonathon Hare. 2024. Rethinking Deep Thinking: Stable Learning of Algorithms using Lipschitz Constraints. arxi::4410.2345/ecss
Stas Bekman. 2023. Machine Learning Engineering Open Book. Stasosphere Online Inc.
Loubna Ben Allal, Anton Lozhkov, Guilherme Penedo, Thomas Wolf, and Leandro von Werra. 2024, Smoll MM corpus.
Stella Biderman, Hailcy Schoelkopf, Quentin Anthony, Herbic Bradley, Kyle Brieen, Eric Hallahan, Moham- mad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. 2023. Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling. 2rm.v:2304.01333cc
Stella Biderman. Hailey Schoelkopf, Lintang Sutawika, Leo Gao, Jonathan Tow, Baber Abbasi, Alham Fikri Aji, Pawan Sasanka Ammanamanchi, Sidney Black, Jordan Clive, Anthony DiPofi, Julen Etxaniz, Ben- jamin Fattori, Jessica Zosa Forde, Charles Foster, Jeffrey Hsu, Mimansa Jaiswal, Wilson Y. Lee, Haonan Li, and 11 others. 2024. Lessons from the Trenches on Reproducible Evaluation of Language Models. 2idi::4405.14822(ccm
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2020. Piqa: Reasoning about physical commonsense in natural language. Thirty-Fourth AAAI Conference on Artificial Intelligence.
Malik Boudiaf, Romain Mueller, Ismail Ben Ayed, and Luca Bertinetto. 2022. Parameter- Free Online Test- Time Adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 6344-8353.
Valentino Braitenberg. 1986. Vehicles: Experiments in Synthetic Psychology. MIT press.
William Brandon, Mayank Mishra, Aniruddha Nrusimha, Rameswar Panda, and Jonathan Ragan Kelly. 2024. Reducing Transformer Key-Value Cache Size with Cross-Layer Attention. arxiv:2405. 12..24405..111caa. [cs].
British Library Labs. 2021. Digitised Books. c. 1510 c. 1900, JSONL (OCR Derived Text + Mettadata. British Library.
Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, and Tri Dao. 2024. Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads. In Corty-First International Conference on Machine Learning.
ah 2024. Optimizing Al Inference at Character.AI.
Mark Chen, Jerry Tworek, Hcewoo Jun, Qiming Yuan, Henrique Pondc de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray. Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, and 39 others. 2021. Evaluating large language models trained code. Preprint, arXiv:2107.0337
Jeffrey Cheng and Benjamin Van Durme. 2024. Compressed Chain of Thought: Efficient Reasoning Through Dense Representations. 2tt.:24422.3371fcc.
Euirim Choi. 2023. GoodWiki dataset.
François Chollet. 2019. On the Measure of Intelligence. orxiv:1911.01547(css)
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishck Rao, Parker Baress, Yi Tay, Noam Shazeer, Vinodkumar Prab- hakaran, and 48 others. 2022. PaLM: Scaling Language Modeling with Pathways. arXiv:2204.0231/
[cs].
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. wwXiv1183.00555700
12

---

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert. Training
Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021.
Verifiers to Solve Math Word Problems, arxiv:21 14166//ss)
Owen Colegrove, Vik Paruchuri, and OpenPhi-Team. 2024, Open-phiitexthooks Datasets at Hugging Face.
Róbert Csordás, Kazuki Irie, Jürgen Schmidhuber, Christopher Potts, and Christopher D. Manning. 2024. MoEUT: Mixture-of-Be -xxpprtt Universal Transformers, In The Thirty-eighth Annual Conference on Neural Information Processing Systems.
Gautier Dagan. 2024. Bpeasy.
Gautier Dagan, Gabriel Synnaeve, and Baptiste Rozière. 2024. Getting the most of your tokenizer for pre-training and domain adaptation. arxiv:2400.01035/cs)
Tri Dao. 2023. FlashAttention2:: Faster Attention with Better Parallelism and Work Partitioning. 2xx1:=2307..8011(cs)..
Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022. Memory- Efficient Exact Attention with 10-Awareness. wxdv:2205.14135/cs)
FlashAttention: Fast and
DeepSeek-AL, Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song. Ruoyu Zhang. Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang. Xingkai Yu, Yu Wu, Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, and 181 others. 2025. DeepSeek-RI: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arxiv:2501. 12..0//./99//////
DeepSeek-AT, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang. Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Fangyun Lin, Fucong Dai, and 181 others. 2024. DeepSeek-V V3 Technical Report. (xx1::2412.1433((cs).
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. 2019. Universal Transformers. 1rxiv:1807.038191 stat].
Yuntian Deng. Yejin Choi, and Stuart Shieber. 2024. From Explicit CoT Implicit CoT: Learning Internalize CoT Step by Step. arxiv:2405. 2te.22445..4...8/ 14838/cs).
Hantian Ding. Zijian Wang, Giovanni Paolini, Varun Kumar. Anoop Deoras, Dan Roth, and Stefano Soatto. 2024. Fewer Truncations Improve Language Modeling, In Forty-First International Conference on Machine Learning.
Ming Ding, Zhuoyi Yang. Wenyi Hong. Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, and Jie Tang. 2021. Viewiew: Mastering Texteto-Image Generation via Transformers. In Advances in Neural Information Processing Systems, volume 34, pages 19822-19835. Curran Associates, Inc.
Maha Elbayad, Jiatao Gu, Edouard Grave, and Michael Auli. 2019. Depth-Adaptive Transformer. International Conference on Learning Representations.
Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, Basil Hosmer, Bram Wasti, Liangzhen Lai, Anas Mahmoud, Bilge Acun, Saurabh Agarwal, Ahmed Roman, Ahmed A. Aly, Beidi Chen, and Carole-Jean Wu. 2024. LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding. moxiv:2404.167101
Katie Everett, Lechao Xiao, Mitchell Wortsman, Alexander A. Alemi, Roman Novak, Peter J. Liu, Izzeddin Gur, Jascha Sohl-Dickstein, Leslic Pack Kaelbling, Jachoon Lee, and Jeffrey Pennington. 2024, Scaling Exponents Across Parameterizations and Optimizers. uuxiv:2407.058721cc
Angela Fan, Edouard Grave, and Armand Joulin. 2019. Reducing Transformer Depth on Demand with Structured Dropout, arxiv: 1909. /1556fcs, stat].
Angela Fan. Thibaut Lavril Edouard Grave, Armand Joulin, and Sainbayar Sukhbaatar. 2021. Addressing Some Limitations of Transformers with Feedback Memory. 2tzi::2002.04402.cs. stat).
Ying Fan, Yilun Du, Kannan Ramchandran, and Kangwook Lec. 2025. Looped Transformers for Length Generalization, In The Thirteenth International Conference on Learning Representations.
William Fedus, Barret Zoph, and Noam Shazeer. 2022, Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. arxiv:2101.03361|cc
13

---

Xidong Feng. Yicheng Luo, Ziyan Wang. Hongrui Tang. Mengyue Yang, Kun Shao, David Mguni, Yali Du. and Jun Wang. 2023. ChessGPT: Bridging Policy Learning and Language Modeling. Advances in Neural Information Processing Systems, 36:7216-7262.
Sebastian Gabarain. 2024. Locutusqquehhereeusss-500 Datasets at Hugging Face.
Khashayar Gatmiry, Nikunj Saunshi, Sashank J. Reddi, Stefanie Jegelka, and Sanjiv Kumar. 2024. Can Looped Transformers Learn Implement Multi-step Gradient Descent for In-context Learning?
Jonas Geiping and Tom Goldstein. 2023, Cramming: Training Language Model on single GPU in one day. In Proceedings of the 40th International Conference Machine Learning, pages 11117-11143, PMLR.
Zhengyang Geng, Xin- Yu Zhang, Shaojie Bai, Yisen Wang, and Zhouchen Lin. 2021. On Training Implicit Models. In Advances in Neural Information Processing Systems, volume 34, pages 14247-24260. Curran Associates, Inc.
F.A. Gers and J. Schmidhuber. 2000. Recurrent nets that time and count. In Proceedings of the IEEE-INNS TEEE-INSS-ENSS International Joint Conference on Neural Nenworks. IJCNN 2000. Neural Computing: New Challenges and Perspectives for the New Millennium, volume 3, pages 189-194 vol.3.
Angeliki Giannou, Shashank Rajput, Jy- Yong Sohn, Kangwook Lee, Jason D. Lee, and Dimitris Papailliopoulos 2023. Looped Transformers as Programmable Computers. In Proceedings of the 40th International Conference on Machine Learning, pages 11398-11442 1142. PMLR.
Jonas Golde, Patrick Haller, Fabio Barth, and Alan Akbik, 2025. MastermindFval:: A Simple But Scalable Reasoning Benchmark. 2533v:5503.588//(cs)
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. 2018. Accurate, Large Minibatch SGD: Training ImageNet in Hour. arxiv: 176.::7706.06677[c
Alex Graves. 2017. Adaptive Computation Time for Recurrent Neural Networks. arxiv:1603.88983(cc)
Alex Graves, Greg Wayne, and Ivo Danihelka. 2014, Neural Turing Machines. 141.v:440.55001/6/
Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang. Shane Arora, David Atkinson. Russell Authur, Khy- athi Raghavi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, and 24 others. 2024, OL Accelerating the Science of Language Models. 2tt::2402.0088//cs
Alexander Hägele, Elie Bakouch, Atli Kosson, Loubna Ben Allal, Leandro Von Werra, and Martin Jaggi. 2024. Scaling Laws and Compute-Opümal Training Beyond Fixed Training Durations. In Workshop Efficient Systems for Foundation Models II @ ICML2024.
Shibo Hao. Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason Weston, and Yuandong Tian. 2024. Training Large Language Models to Reason in a Continuous Latent Space. 2xxi:24122.66669/cs
Zexue Hc, Leonid Karlinsky, Donghyun Kim, Julian McAuley, Dmitry Krotov, and Rogerio Feris. 2024. CAMELOT: Towards Large Language Models with Training-Free Consolidated Associative Memory. arxiv:2402.134444 13449(cs].
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021a. Mcasuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR).
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021b. Measuring Massive Multitask Language Understanding. International Conference on Learning Representations.
J J Hopfield. 1982 Neural networks and physical systems with emergent collective computational abilities. Proceedings of the National Academy of Sciences of the United States of America, 79(8):2554-2558.
Jiewen Hu, Thomas Zhu, and Sean Welleck. 2024. miniCTX: Neural Theorem Proving with (Long-)Contextss 003000((sx..
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. 2018. Averaging weights leads to wider optima and better generalization: 34th Conference on Uncertainty in Artificial Intelligence 2018, UAI 2018. 34th Conference on Uncertainty in Artificial Intelligence 2018, UAI 2018, pages 376-885.
14

---

Albert Q. Jiang, Wenda Li, and Mateja Jamnik. 2023. Multilingual Mathematical Autoformalization. axxiv2311..3755(cs))
Matt Gardner Johannes Welbi, Nelson F. Liu. 2017. Crowdsourcing multiple choice science questions.
Jean Kaddour. 2022. Stop Wasting My Time! Saving Days of ImageNet and BERT Training with Latest Weight Averaging. 2209. 1498/.cc stat].
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Beajamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei, 2020. Scaling Laws for Neural Language Models. wrxi::200.088661[cc stat].
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. 2020. Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention. In Proceedings of the 37th International Conference on Machine Learning, pages
PMLR.
Matthew Kenney. 2024. ArXivDLInstruct.
Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig. Moontac I Lee, Kyungjae Lee, and Minjoon Seo. 2024. Prometheus 2: An Open Source Language Model Spe- cialized in Evaluating Other Language Models. In Proceedings of the 2024 Conference Empirical Methods in Natural Language Processing, pages 4334-4353, Miami, Florida, USA. Association for Computational Linguistics.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: Method for Stochastic Optimization. In International Conference on Learning Representations (ICLR), San Diego.
Wojciech Kryšcińíski, Nazneen Rajani, Divyansh Agarwal, Caiming Xiong, and Dragomir Radev. 2022. Book- Sum: A Collection of Datasets for Long-form Narrative Summarization, arxiv:2 000.000sss0
Xin Lai, Zhuotao Tian, Yukang Chen, Senqiao Yang, Xiangru Peng, and Jiaya Jia. 2024, Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMs. arxiv:2406. 186.246o..16ccs
Zhenzhong Lan, Mingda Chen, Sehastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. AL- BERT: A Lite BERT for superviue Learning of Language Representations. In International Conference on Learning Representations.
Yann LeCun. 2022. A Path Towards Autonomous Machine Intelligence. Preprint, Version 0.9.2:62.
Yann LeCun and Fu Huang. 2005. Loss functions for discriminative training of energy- based models. In AISTATS 2005 Proceedings of the 10th International Workshop on Artificial Intelligence and Statistics,
pages 206-213.
Jason Lee, Elman Mansimov, and Kyunghyun Cho. 2018. Deterministic Non-Autore Autoregressive Neural Sequence Modeling by Iterative Refinement. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1173 1182, Brussels, Belgium. Association for Computational Linguistics.
Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2023. Fast Inference from Transformers via Speculative Decoding, In Proceedings of the 40th International Conference Machine Learning, pages 19274-19286 PMLR.
Yoav Levine, Noam Wies, Or Sharir, Hofit Bata, and Amnon Shashua. 2021. The Depth-to- Width Interplay in Self-Attention. arxiv:2006. 00000066.24445 12467fes, stat].
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur- Ari, and Vedant Misra. 2022. Solving quantitative reasoning problems with language models. Preprint, arXiv:2206.1455 14858.
Raymond Li, Loubna Ben Allal, Yangtian Zi Niklas Muennighoff, Denis Kocetkov, Chenghao Mou. Mare Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Joel Lamy-Poirier, Joao Monteiro, Nicolas Gontier, Hong-Ho Yee, and 39 others. 2023. StarCoder: May the source be with you! Transactions on Machine Learning Research.
Yujia Li, Felix Gimeno, Pushmeet Kohli, and Oriol Vinyals. 2020. Strong Generalization and Efficiency in Neural Programs. arxiv:2007.03629//
Omkar Pangarkar Liping Tang, Nikhil Ranjan. 2024. TxT360: A top-quality LLM pre-training dataset requires the perfect blend.
15

---

Hao Liu, Matei Zaharia, and Pieter Abbeel. 2023a. Ring attention with blockwise transformers for near- near-infinite context. arXiv preprint AXXv:2310001889
Luyang Liu, Jonas Pfeiffer, Jiaxing Wu, Jun Xie, and Arthur Szlam. 2024. Deliheration in Latent Space via Differentiable Cache Augmentation. vrxii:2412.17747[cc
Xiao Liu, Hanyu Lai, Hao Yu, Yifan Xu, Aohan Zeng, Zhengxiao Du, Peng Zhang, Yuxiao Dong, and Jie Tang. 2023b. WcbGLM: Towards An Efficient Web-Enhanced Question Answering System with Human Preferences. In Proceedings of the 29th ACM SIGKDD Conference Knowledge Discovery and Data Mining, KDD '23, pages 4569-4560, New York, NY, USA. Association for Computing Machinery.
Zhengzhong Liu, Aurick Qiao, Willie Neiswanger, Hongyi Wang. Bowen Tan, Tianhua Tao, Junbo Li, Yuqi Wang. Suqi Sun, Omkar Pangarkar, Richard Fan, Gu, Victor Miller, Yonghao Zhuang, Guowei He, Haonan Li, Fajri Koto, Liping Tang, Nikhil Ranjan, and 9 others. 2023c. ILM3660: Towards fully transparent open-source LLMS.
Ilya Loshchilov and Frank Hutter. 2017. Decoupled Weight Decay Regularization.
[cs, math].
Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang. Dmytro Pykhtar. Jiawei Liu, Yuxiang Wei, Tiayyang Liu, Max Tian, Denis Kocetkov, Arthur Zucker, Younes Belkada, Zijian Wang, Qian Liu, Dmitry Abulkhanov, Indraneil Paul, and 47 others. 2024. StarCoder 2 and The Stack v2: The Next Generation.
Zimu Lu, Aojun Zhou, Ke Wang, Houxing Ren, Weikang Shi, Junting Pan, Mingjie Zhan, and Hongsheng Li. 2024. MathCoder2: Better Math Reasoning from Continued Pretraining Code. andvv2441000876//css..
Mathematical
Sebastian Majstorovic. 2024. Selected Digitized Books The Library Congress.
Larisa Markeeva, Sean Mchhi Borja Iharz, Wilfried Bounsi, Olga Kozlova, Alex Vitvitskyi, Charles Blundell, Tom Goldstein, Avi Schwarzschild, and Petar Veličković. 2024. The CLRS-Text Algorithmic Reasoning Language Benchmark. wxxi::2406.04229((c
Mrinal Mathur, Barak A. Pearlmutter, and Sergey M. Plis. 2024. MIND over Body: Adaptive Thinking using Dynamic Computation. In The Thirteenth International Conference on Learning Representations.
Sean Michael Mcshish, Arpit Bansal, Alex Stein, Neel Jain, John Kirchenbauer, Brian R. Bartoldson, Bhavya Kailkhura, Abhinay Bhatele, Jonas Geiping, Avi Schwarzschild, and Tom Goldstein. 2024. Transformers Can Do Arithmetic with the Right Embeddings. In The Thirty-eighth Annual Conference on Newral Information Processing Systems.
William Merrill, Ashish Sabharwal, and Noah A. Smith. 2022. Saturated Transformers are Constant- Deethh Threshold Circuits. Transactions of the Association for Computational Linguistics, 10:843-856.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can suit of armor conduct electricity? a new dataset for open book question answering. In EMNLP.
Tomáš Mikolov, Stefan Kombrink, Lukáš Burget, Jan Čemockký, and Sanjeev Khudanpur. 2011. Extensions of recurrent neural network language model. In 2011 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 5528-5531.
Takeru Miyato, Sindy Läwe, Andreas Geiger, and Max Welling. 2024. Artificial Kuramoto Oscillatory Neurons. In The Thirteenth International Conference on Learning Representations, Singapore.
Ryan Moulton. 2023. The Many Ways that Digital Minds Can Know
Niklas Muennighoff, Qian Liu, Armel Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro von Werra, and Shayne Longpre. 2024. OctoPack: Instruction Tuning Code Large Language Models. arxiv:2308.0711
Nam Pham. 2023. Teny-tettbooks (Revision 14de7ba).
Nam Pham. 2024. Tiny-strange-textbooks (Revision 6f304f1),
Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. 2022. Progress measures for grokking via mechanistic interpretability. In The Eleventh International Conference on Learning Representa- tions.
16

---

Lorenzo Noci, Sotiris Anagnostidis, Luca Biggio, Antonio Orvieto, Sidak Pal Singh, and Aurelien Lucchi. 2022. Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapse. In Advances Neural Information Processing Systems.
OpenAI. 2024. New reasoning models: Openai ol or-preveew and ol-mini. http:://ppenai..cm//eeeess
o1-previev- ----prreeeaadd-------- and--a-dddonnn-
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. arxiv:2203.02155(er
Keiran Paster, Marco Dos Santos, Zhangir Azerbayev, and Jimmy Ba. 2023, OpenWebMaiee An Open Dataset of High-Quality Mathematical Web Text. In The Twelfth International Conference on Learning Representations.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodci, and lyya Sutskever. 2019. Language Models are Unsupervised Multitask Learners OpenAl, page 24.
Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, and Timothy P. Lillicrap. 2019. Compressive Trans- formers for Long-Range Sequence Modelling. (rxvv:9111.0500//C
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase. and Yuxiong He. 2020. ZeRO: Memory optimizations Toward Training Trillion Parameter Models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1-16.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022. High-Resolution Image Synthesis With Latent Diffusion Models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 10684-10695.
Keisuke Sakaguchi, Ronan I.e Bras, Chandra Bhagavatula, and Yejin Choi. 2021, WinoGrande: An adversarial winograd schema challenge at scale. Commun. ACM, 64(9):99-106.
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafcai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey. M. Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, and 21 others. 2021. Multitask Prompted Training Enables Zero-Shot Task Generalization. In International Conference on Learning Representations.
Sunny Sanyal, Atula Tejaswi Neerkaje, Jean Kaddour, Abhishek Kumar, and sujay sanghavi. 2024, Early weight averaging meets high learning rates for LLM pre-training. In First Conference Language Modeling.
Nikunj Saunshi, Nishanth Dikkala, Zhiyuan Li, Sanjiv Kumar, and Sashank J. Reddi. 2024. Understanding Reasoning With Looped Model. In The Thirteenth International Conference on Learning Representations.
Juergen Schmidhuber. 2012. Self- Delimiting Neural Networks. 1210v:1200.0118(sss..
Mark Schöne, Babak Rahmani, Heiner Kremer, Fabian Falck, Hitesh Ballani, and Jannes Gladrow. 2025. Implicit Language Models are RNNs: Balancing Parallelization and Expressivity. a5xiv:2502.0882[(css)..
Tal Schuster, Adam Fisch, Jai Gupta, Mostafa Dehghani, Dara Bahri, Vinh Q. Tran, Yi Tay, and Donald Metzler. 2022. Confident Adaptive Language Modeling. In Advances in Neural Information Procesingg Systems.
Avi Schwarzschild, 2023. Deep Thinking Systems: Logical Extrapolation with Recurrent Neural Networks. Ph.D. thesis, University of Maryland, College Park, College Park.
Avi Schwarzschild, Eitan Borgnia, Arjun Gupta, Arpit Bansal, Zeyad Emam, Furong Huang, Micah Gold- blum, and Tom Goldstein. 2021a. Datasets for Studying Generalization from Easy to Hard Examples. 100.0000000000100 [cs)]
Avi Schwarzschild, Eitan Borgnia, Arjun Gupta, Furong Huang, Uzi Vishkin, Micah Goldblum, and Tom Goldstein. 2021b. Can You Learn an Algorithm? Generalizing from Easy to Hard Problems with Recurrent Networks. In Advances Neural Information Processing Systems, volume 34, pages 6695-6706. Curran Associates, Inc.
Avi Schwarzschild, Sean Michael McLeish, Arpit Bansal, Gabriel Diaz, Alex Stein, Aakash Chandnani, Aniruddha Saha, Richard Baraniuk, Long Tran- Thanh, Jonas Geiping, and Tom Goldstein. 2023. Algorithm Design for Learned Algorithms.
17

---

Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural Machine Translation of Rare Words with Subword Units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 1715-1725, Berlin, Germany, Association for Computational Linguistics.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi, Haowei Zhang, Mingchuan Zhang. YK Li. Y Wu, and others. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.055 03300.
Noam Shazeer. 2020. GLU Variants Improve Transformer. anxiv:200.055222(es/
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer.
1711v:170100533(cc1
Siddharth Singh and Abhinav Bhatele. 2022, AxoNN: An asynchronous, message-driven parallel framework for extreme-scale deep learning. In 2022 IEEE International Parallel and Distributed Processing Symposium (IPDPS), pages 606-616.
Siddharth Singh, Prajwal Singhania. Aditya Ranjan, John Kirchenbauer, Jonas Geiping, Yuxin Wen, Neel Jain,
Abhimanyu Hans, Manli Shu, Aditya Tomar. Tom Goldstein, and Abhinav Bhatele. 2024. Democratizing
Al: Open-source Scalable LLM Training on GPU-based Supercomputers. In 2024 SC24: International Conference for High Performance Computing, Networking, Storage and Analysis SC, pages 36 49, IEEE Computer Society.
Daria Soboleva, Faisal Al-Khateeb, Joel Hestness, Nolan Dey, Robert Myers, and Jacob Robert Steeves. 2023. SlimPajama: A 627B token eleaned and deduplicated version of RedPajama.
Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas. Yanai Elazar, Valentin Hofmann, Ananya Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennighoff, and 17 others. 2024. Dolma: An Open Corpus of Three Trillion Tokens for Language Model Pretraining Research. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 15725-15788, Bangkok, Thailand. Association for Computational Linguistics.
Yang Song and Stefano Ermon. 2019. Generative Modeling by Estimating Gradients of the Data Distribution. arXiv: 007000600 [cs, star].
Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. 2021. RoFormer: Enhanced Transformer with Rotary Position Emhedding. arxiv: 104. 04889864441 [cx].
Sainbayar Sukhbaatar, Edouard Grave, Guillaume Lample, Herve Jegou, and Armand Joulin. 2019. Augmenting Self-attention with Persistent Memory, 1907.1900..0177000 stat].
Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexci Efros, and Moritz Hardt. 2020. Test-Time Training with Self-Supervision for Generalization under Distribution Shifts. In Proceedings of the 37th International Conference on Machine Learning, pages 9229-99299999 222999924..99 PMLR.
Ilya Sutskever, Geoffrey E Hinton, and Graham W Taylor. 2008. The Recurrent Temporal Restricted Boltzmann Machine. In Advances in Neural Information Processing Systems, volume 21, Curran Associates, Inc.
Mirac Suzgun, Sebastian Gehrmann, Yonatan Belinkov, and Stuart M. Shieber. 2019. Recurrent Neural Networks Can Learn Generalized Dyck Languages. 191 1.03329fcs).
Mirac Suzgun, Nathan Scales, Nathanacl Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe V. Le, Ed Chi, Denny Zhou, and Jason Wei. 2022. Challenging BIG- -G-Beenh Tasks and Whether Chain-of Thoughh Can Solve Them. 21xiv:2210.0926|(css)
Sho Takase and Shun Kiyono. 2023. arxiv:2104.06222(cs..
Lessons on Parameter Sharing across Layers in Transformers.
Sho Takase, Shun Kiyono, Sosuke Kobayashi, and Jun Suzuki. 2024. Spike No More: Stabilizing the Pre-training of Large Language Models. 2.ti::33/2.16003cc1
Shawn Tan, Yikang Shen, Zhenfang Chen, Aaron Courville, and Chuang Gan. 2023. Sparse Universal Trans- former. arxiv:310.07099/(c:/
Team Gemma, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju. Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen. Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, and 179 others. 2024. Gemma 2: Improving Open Language Models Practical Size. arxiv:2408.0 00118(cs).
18

---

Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang. Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, and 21 others. 2025. 2 OLMo Furious, 0iii:2501.00666/c0
TogetherAT. 2023. Llama-2-7B-324
2-0-----2-----ttttr 1r22K--sstuct---e and fine-tuning for Llama-2 models with Together API.
Shubham Toshniwal, Wei Du Ivan Moshkov, Branislav Kisacanin, Alexan Ayrapetyan, and Igor Gitman. 2024a. OpenMathInstmut-2: Accelerating Al for Math with Massive Open-Source Instruction Data. 2rxiv:2410001560ccr
Shubham Toshniwal, Ivan Moshkov, Sean Narenthiran, Daria Gitman, Fei Jia, and Igor Gitman. 2024b. OpenMathlnstructI:: A Million Math Instruction Tuning Dataset. In The Thirty-eight Conference on Newral Information Processing Systems Datasets and Benchmarks Track.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit. Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. arXiv:1706.03702 [cs].
Zengzhi Wang, Xuefeng Li, Rui Xia, and Pengfei Liu, 2024a. MathPile: A Billion-Tokee Toeoon-Tooe----oee Pretraining Corpus for Math. In The eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track.
Zhilin Wang. Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J. Zhang, Makesh Nar- simhan Sreedhar, and Oleksii Kuchaiev. 2024b. HelpSteer2: Open-source dataset for training lop-performing reward models. arxiv:2406.000 08673ss.)
Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2020. BLiMP: The Benchmark of Linguistic Minimal Pairs for English. Transactions of the Association for Computational Linguisties, 8:377-392.
Maurice Weber, Daniel Y. Fu, Quentin Gregory Anthony, Yonatan Oren, Shane Adams, Anton Alexandrov, Xiaozhong Lyu, Huu Nguyen, Xiaozhe Yao, Virginia Adams, Ben Athiwaratkun, Rahul Chalamala, Kezhen Chen, Max Ryabinin, Tri Dao, Perey Liang, Christopher Re, Irina Rish, and Ce Zhang. 2024. RedPajama: An Open Dataset for Training Large I Language Models. In The Tiiht Conference on Neural Information Processing Systems Datasets and Benchmarks Truck.
Ronald J. Williams and Jing Peng. 1990. An Efficient Gradient-Based Algorithm for On-Line Training of Recurrent Network Trajectories. Neural Computation, 2(4):490-500 -001
Mitchell Wortsman, Tim Dettmers, Luke Zettlemoyer, Ari Morcos, Ali Farhadi, and Ludwig Schmidt. 2023a. Stable and low-precision training for large-scale vision-language models, Advances in Neural Information Processing Systems, 36: 16:11 10299..
Mitchell Wortsman, Tim Dettmers, Luke Zettlemoyer, Ari S. Morcos, Ali Farhadi, and I Ludwig Schmidt. 2023b. Stablc and low-precision training for large-scale vision-language models. In Thirth-Seventh Conference on Neural Information Processing Systems.
Mengshiou Wu and Mark Stock. 2024, Enhancing PyTorch Performance on Frontier with the RCCL OFI-Plugin.
Yuhuai Wu, Markus Norman Rabe, DeLesley Hutchins, and Christian Szegedy. 2022. Memorizing Transformers. In International Conference on Learning Representations.
Zijian Wu, Jiayu Wang. Dahua Lin, and Kai Chen. 2024. LEAN-GitHub: Compiling GitHub LEAN repositories for versatile LEAN prover. arxiv:2407. 17..77777..7
Zhangchen Xu, Fengqing Jiang, Luyao Niu, Yuntian Deng, Radha Poovendran, Yejin Choi, and Bill Yuchen Lin. 2024. Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing. 206.064400.
Kaiyu Yang, Aidan M. Swope, Alex Gu, Rahul Chalamala, Peiyang Song, Shixing Yu, Saad Godil, Ryan Prenger, and Anima Anandkumar. 2023. LeanDojo: Theorem Proving with detrieval-Augmentee Language Models. In Thirty-Sevente Conference on Neural Information Processing Systems Datasets and Benchmarks Track.
Liu Yang, Kangwook Lee, Robert D. Nowak, and Dimitris Papailiopoulos. 2024a. Looped Transformers are Bet- ter at Learning Learning Algorithms. In The Twelfth International Conference on Learning Representations.
Songlin Yang, Bailin Wang. Yu Zhang, Yikang Shen, and Yoon Kim. 2024b. Parallelizing Linear Transformers with the Delta Rule over Sequence Length. In The Thirty-eight Annual Conference on Neural Information Processing Systems.
19

---

Huaiyuan Ying. Zijian Wu, Yihan Geng, Jiayu Wang, Dahua Lin, and Kai Chen. 2024. Lean Workbook: A large-scale Lean problem formalized from natural language math problems. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track.
Longhui Yu, Weisen Jiang. Han Shi, Jincheng Yu. Zhengying Liu, Yu Zhang, James Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. 2023. MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models. In The Twelfth International Conference on Learning Representations.
Pedram Zamirai, Jian Zhang, Christopher R. Aberger, and Christopher Sa. 2021. Revisiting BF Training. 0rxi:22010.0619/(cc stat].
Eric Zelikman, Georges Harik, Yijia Shao, Varuna Jayasiri, Nick Haber, and Noah D. Goodman, 2024. Quiet- STaR: Language Models Can Teach Themselves to Think Before Speaking. arxiv:2403. cxxiv:2403.0062((css)
Rowan Zellers, Ari Holtzman, Yonatan Bisk. Ali Farhadi, Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational
Linguistics.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. 2022. Scaling Vision Transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12104-1211 1144--211..
Biao Zhang and Rico Sennrich. 2019. Root Mean Square Layer Normalization. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
Ge Zhang, Scott Qu, Jiaheng Liu, Chenchen Zhang, Chenghua Lin, Chou Leuang Yu, Danny Pan, Esther Cheng, Jie Liu, Qunshu Lin, Raven Yuan, Tuney Zheng, Wei Pang, Xinrun Du Yiming Liang, Yinghao Ma, Yizhi Li, Ziyang Ma, Bill Lin, and 26 others. 2024a. MAP-Neo: Highly Capable and Transparent Bilingual Large Language Model Series. arxiv:2405. 19327(ss..
Jun Zhang, Jue Wang. Huan Li, Lidan Shou, Ke Chen, Gang Chen, and Sharad Mchrotra. 2024b. Draft& Verify: Lossless Large Language Model Acceleration Speculapee Decoding. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11263-11288 Bangkok, Thailand. Association for Computational Linguistics.
Yifan Zhang, Yifan Luo, Yang Yuan, and Andrew C. Yao. 2024c. Autonomous Data Selection with Language Models for Mathematical Texts. In ICLR 2024 Workshop on Navigating and Addressing Data Problems for Foundation Models.
Tianyu Zheng, Ge Zhang, Tianhao Shen, Xueling Liu, Bill Yuchen Lin, Jie Fu, Wenhu Chen, and Xiang Yue. 2024. OpenCodeenterpreeer: Integrating Code Generation with Execution and Refinement. In Findings of the Association for Computational Linguistics: ACI. 2024, pages 12834-12859, Bangkok, Thailand. Association for Computational Linguistics.
Fan Zhou, Zengzhi Wang, Qian Liu, Junlong Li, and Pengfei Liu, 2024. Programming Every Example: Lifting Pre-training Data Quality like Experts Scale. arxiv:2409.177 171/5sss.
Terry Yue Zhuo Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ramadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, Simon Brunner. Chen Gong, Thong Hoang, Armel Randy Zebaze, Xiaoheng Hong, Wen-Ding Li, Jean Kaddour, Ming Xu, Zhihan Zhang, and 14 others. 2024. BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions.
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We aim to set out to show that recurrent depth architectures can scale, which show in the main set of experiments. We make theoretical claims in the introduction, refering to prior theoretical work on recurrent models where appropriate. Our aspirational goals with this line of work are discussed in separate motivation section titled "Why train models with recurrent depth".
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
See Section "Limitations and Conclusions"
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
21

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: We discuss prior theoretical work, but make no additional theoretical contribu- tions in this work.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide an extensive appendix that comprehensively describes our train- ing run, data mix and setup. Together with our provided code, this allows for perfect reproduction of our training run (albeit given sufficient compute).
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibilityy can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the partieular way they provide for reproducibiliity In the case of closed-source models, it may be that access to the model is limited in
22

---

some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide our code with the supplementary material, which also includes the data processing scripts. We will also publish the processed training dataset and final model to ease reproduction and analysis.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines ftttps://niin cc/pu plic/guides getg//udde//ee// CodeSubmi ssi onmolicy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed run reproduce the results. See the NeurIPS code and data submission guidelines (https:
//nips. cccppblic/geeeeeceeeecceeceeeeee /ecci/guieee weeeddee/ooooubiiiie cobbbmisssinnii ssionPoiccy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Yes, appendix section "Training a large-scale recurrent-depth Language Model".
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We provide error bars for our evaluation results where appropriate, such as in Figure I (left).
Guidelines:
The answer NA means that the paper does not include experiments.
23

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Yes, see appendix section C.2 and C.3. We train on 4096 AMD MI-250X GPUs for about 10 days.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: htttss////narrsss..pbpbt/t/tttt ceabllic/thissuuieeel..
Answer: [Yes]
Justification: Yes.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Yes, see Appendix Section A, "Potential Implications of this Work".

---

Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairesss considerations (e.g., dep.oymely of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
We consider the risk for misuse with the language model pretrained in this work to be small, given the amount of training compute. All data used in this work is publicly available Hugging Face, and we believe this work does not lead to additional risks attached to this data.
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Please refer to the tables detailing all data sources, their owners, and their licenses in Table 11 and Table 12.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
25

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use the package should be provided. For popular datasets, saperswithcods com datassess has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's ' creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The code submission Readme explains how to use the provided code.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the datasct/cod//model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve human subject studies.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve human subject studies.
Guidelines:
26

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA].
Justification:
Only very limited LLM use overall.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should not be described.
27

---

Appendix Overview
Appendix A: Broader Impact
Appendix B: Future Work
Appendix C: Training a large-scale recurrent-depth Language Model
Appendix C.I: Training Setup
* Pretraining Data
* Tokenization and Packing Details
* Architecture and Initialization
* Locked-Step Sampling
Optimizer and Learning Rate Schedule
Appendix C.2: Compute Setup and Hardware
# Device Speed and Parallelization Strategy
Appendix C:3: Importance of Norms and Initializations at Scale
Appendix C.4: Additional Implementation Details
Appendix D: Additional Evaluations and Benchmark Instructions
Appendix D.1: Benchmarking Settings
Appendix D.2: Recurrence Scaling by Tasks
Appendix D.3: Recurrence and Context
Appendix D.3: Comparison to weight Models
Appendix D.5: Regarding KV-cache Misses in Token-Adaptive Early Exits
Appendix D.6: Zero-Shot Continuous Chain-of-Thought
Appendix D.7: Saturation Evaluation
Appendix D.8: Model Stability at Arbitrary Depths
Appendix D.9: Inference Speed Benchmark
Appendix E: Latent Space Visualizations
Appendix F: Pretraining Data
A
Broader Impact
This work describes a novel architecture and training objective for language modeling with promising performance, especially on tasks that require the model reason. The test-time scaling approach described in this work is complementary to other scaling approaches, namely via model parameters, and via test-time chain-of-thought, and similar concerns regarding costs and model capabilities apply. propose is naturally smaller than models scaled by parameter scaling, and this
The architecture may have broader benefits for the local deployment of these models with commodity chips. Finally, while argue that moving the reasoning capabilities of the model into the high- dimensionli continuous latent space of the recurrence beneficial in terms of capabilities, we note that there is concern that this comes with costs in model oversight in comparison verbalized chains of thought, that are currently still human-readable. We provide initial results in Section showing that the high-dimensional state trajectories of our models interpreted. A number of additional visualization of the latent state can be found in the pages of the appendix. Ultimately, from our perspective, this type of thinking in latent space is not dissimilar from the latent computations that already happen in the intermediate layers of current, fixed-depth transformers. The increasing complexity of capability of these models necessitates white-box interpretability even without considering latent thinking, especially as verbal chain-of of-thought traces from frontier models are becoming less interpretable. Analysis techniques that work to understand computations in intermediate layers also apply analogously to latent thinking.
be analyzed and some of their mechanisms
B Future Work
Aside from work extending and analyzing the scaling behaviors of recurrent depth models, there are many questions that remain unanswered. For example, to us, there potentially large number of
28

---

novel post-training schemes that further enhance the capabilities of these models, such as fine-tuning to compress the recurrence or reinforcement learning with data with differrnt hardness levels (Zelikman et al., 2024), or to internalize reasoning from CoT data into the recurrence (Deng et al., 2024).
Another aspect not covered in this work is the relationship to other modern architecture improvements. Efficient sequence mixing operations, especially those that are linear in sequence dimension, such as linear attention (Katharopoulos et al., 2020; Yang al., 2024b), are limited in the number of comparisons that can be made. However, with recurrent depth, blocks containing linear operators can repeat until all necessary comparisons between sequence elements are computed (Suzgun et al., 2019), and recent work in this direction can be found in Schöne et al. (2025), For simplicity, we also focus on a single recurrence, where prior work has considered multiple successive recurrent stages (Takase and Kiyono, 2023; Csordás et al., 2024).
Finally, the proposed architecture is set up be compute-heavy, with more "materialized" parameters than there are actual parameters. This naturally mirrors mixture-of-exper models (MoE), which are parameter-hearyy using fewer active parameters per forward pass than exist within the model (Shazeer et al., 2017; Fedus et al., 2022), We posit that where the recurrent-deple setup excels at learning reasoning patterns, the MoE excels at effectively storing and retrieving complex information. Their complementarity supports the hypothesis that a future architecture would contain both modifications. While in a standard MoE model, each expert can only be activated once per forward pass, or skipped entirely, recurrent MoE model could also refine its latent state over multiple iterations, potentially routing to the same expert multiple times, before switching Csordás et al., 2024). While MoF models are the currently leading solution to implement this type of "memory" in dense transformers, these considerations also hold for other memory mechanisms suggested for LLMs (Sukhbaatar et al., 2019; Fan et al., 2021; Wu et al., 2022; He et al., 2024).
a different one (Tan et al., 2023;
C Training a large-scale recurrent-depth Language Model
In this section we provide a comprehensive report of the large-scale training run that we executed for this work. We discuss data setup, tokenization, optimizer settings, initialization and additional architecture details, machine learning engineering required to train the recurrent-depth model at scale, given the accelerators available to us and practical roadIblocks observed during pretraining.
C.1 Training Setup
We describe the training setup, separated into architecture, optimization setup and pretraining data here. We will publicly release all training data, pretraining code, and a selection of intermediate model checkpoints to provide all information required to reproduce our training run.
Pretraining Data. Given access only enough compute for single large scale model run, we opted for dataset mixture that maximized the potential for emergent reasoning behaviors, not necessarily for optimal benchmark performance. Our final mixture is heavily skewed towards code and mathematical reasoning data with (hopefully) just enough general webtext to allow the model to acquire standard language modeling abilities. All sources are publicly available. We provide an overview in Figure 7. Following Allen-Zhu and Li (2024), we directly mix relevant instruction data into the pretraining data. However, due to compute and time constraints, we were not able to ablate this mixture. We expect that a more careful data preparation could further improve the model's performance. We list all data sources in Appendix F.
Tokenization and Packing Details. We construct vocabulary of 65536 tokens via BPE (Sennrich et al., 2016), using the implementation of Dagan (2024). In comparison to conventional tokenizer training, we construct our tokenizer directly on the instruction data split of our pretraining corpus, to maximize tokenization efficiency on the target domain. We also substantially modify the pre- tokenization regex (e.g. of Dagan al. (2024)) to better support code, contractions and LaTeX. We include a < Deegin_eett, |> token at the start of every document. After tokenizing our pretraining corpus, we pack our tokenized documents into sequences of length 4096. When packing, we discard document ends that would otherwise lack previous context, to fix an issue described as the "grounding problem" in Ding et al. (2024), aside from several long-document sources of mathematical content, which we preserve in their entirety.
29

---

generic-text: 28.71%
code: 25.36%
scientific-text: 18.73%
synthetic-text: 8.14%
longform-text: 7.50%
math: 6.14%
generic-instruct: 2.09%
Q&A-text: 1.58%
math-instruct: 1.51%
writing-instructt 0.12%
misc-reasoning: 0.11%
Figure Distribution of data sources that included during training. The majority of data comprised generic web-text, scientific writing and code.
Architecture and Initialization. We scale the architecture described in Section 3, setting the layers to (2, 4, 2). and train with mean recurrence value of 32. We mainly scale by increasing the hidden size to 5280, which yields 55 heads of size 96. The MLP inner dimension 17920 and the RMSNorm is 10-6, Overall this model shape has about 1.5B parameters in non-recurrent prelude and head, 1.5B parameters in the core recurrent block, and 0.5B in the tied input embedding.
At small scales, most sensible initialization schemes work. However, at larger scales, we use the initialization of Takase al. (2024) which prescribes a variance of 5k We initialize all parameters from truncated normal distribution (truncated at 30) with this variance, except all out-projection layers, where the variance set to aa 5h7 for 1p Fl + le the number of effective layers, which is 132 for this model. As result, the out-projection layers are initialized with fairly small values (Goyal et al., 201 8). The output of the embedding layer is scaled by vh. To match this initialization, the state 80 also sampled from a truncated normal distribution, here with variance o =
Locked-Step Sampling, To enable synchronization between parallel workers, we sample a single depth r for each micro-batch of training, which we synchronize across workers (otherwise workers would idle while waiting for the model with the largest r to complete its backward pass). We verify small scale that this modification improves compute utilization without impacting convergence speed, but note that at large batch sizes, training could be further improved by optimally sampling and scheduling independent steps r on each worker, more faithfully model the expectation over steps in Equation (1).
Optimizer and Learning Rate Schedule. We train using the Adam optimizer with decoupled weight regularization (81 0.9, 82 0.95, Hutter, 2017) 7), modified to include update clipping (Wortsman et al., 2023b) and removal of the constant as in Everett al. (2024). We clip gradients above 1. We train with warm-up and constant learning rate (Zhai et al., 2022; Geiping and Goldstein, 2023), warming up to our maximal learning rate within the first 4096 steps.
10-4) (Kingma and Ba, 2015; Loshchilov and
C.2 Compute Setup and Hardware
We train this model using compute time allocated on HPE Cray EX supercomputer containing compute nodes with AMD MI250X GPUs, connected using Slingshot dragonfly network. The
30

---

Hlideen State
Val PPT, (log)
Recurrence
Loss (log)
Corr, (log)
10
10"
8
10
16
10
10
10
10
10"
10
Optimizer Stop
Optimizer Step
Optimizer Step
64
Main
Bad Run
Bad Run 2
Figure 8: Plots of the initial 10000 steps for the first two failed attempts and the final, successful run ("Main"). Note the hidden state collapse (middle) and collapse of the recurrence (right) in the first two failed runs, underlining the importance of our architecture and initialization inducing a recurrent model and explain the underperformance of these runs in terms of pretraining loss (lcft).
Tokens (log)
Tokens (log)
10
10
10"
10
12
10
10
10d2
Validation Perplexity (log)
Recurrence
10
10"
10²
5
16
32
10
Step (log)
Step (log)
Figure 9: Left: Plot of pretrain loss over the 800B tokens on the main run. Right: Plot of val ppl at recurrent
depths 1, 4, 8, 16, 32, 64. During training, the model improves in perplexity all levels of recurrence.
scheduling system is orchestrated through SLURM. We train in 6 mixed precision using a PyTorch-based implementation (Zamirai al., 2021).
Device Speed and Parallelization Strategy. Nominally, each MI250X chip" achieves 192 TFLOP per GPU (AMD, 2021), 1), For single matrix multiplication, we measure a maximum achievable speed software stack (ROCM 6.2.0, Py PyTorch 2.6 pre-release 11/02)
on these GPUs of 125 TFLOP/s (Bekman, 2023). Our implementation, using extensive PyTorch compilation and optimization of the hidden dimension to h 5280 achieves single-node training speed of 108.75 75 TFLOP/s, i.e. 87% AFU ("Achievable Flop Utilization"). Due to the weight sharing inherent in our recurrent design, even our largest model is still small enough to be trained using only data (not tensor) parallelism, with only optimizer sharding (Rajbhandari et al., 2020) and gradient checkpointing on a per-iteration granularity. With batch size of per GPU we end up with a global batch size of 16M tokens per step, minimizing inter-GPU communication bandwidth.
scale 4096 GPUs, achieve 52-64 TFLOP/s per GPU, i.e. 41%-51% % AFU, or 1-1.2M tokens per second. To achieve this, we wrote a hand-crafted distributed data parallel implementation circumvent critical AMD interconnect issue, which describe in more detail in Appendix C.4. Overall, we believe this may be the largest language model training run to completion in terms of number of devices used in parallel
When
AMD cluster, of time of writing.
Training Timeline. Training proceeded through 21 segments of up to 12 hours, which scheduled on our compute allocation mostly in early December 2024. We also ran baseline comparison, where we train the same architecture but in feedforward manner with only pass through the core/recurrent block. This trained with the same setup for 180B tokens on 256 nodes with a batch size of per GPU. Ultimately, we were able to schedule 795B tokens of pretraining of the main model. Due to our constant learning rate schedule, we were able to add additional segments "on-demand", when an allocation happened be available.
TTechnically, each node contains 4 dual-chip MI250X cards, but its main software stack (ROCm runtime) treats these chips as fully independent.
31

---

C.3 Importance of Norms and Initializations at Scale
At small scales all normalization strategies worked, and we observed only tiny differences between initializations. The same was not true at scale. The first training run we started was set up with the same block sandwich structure as described above, but parameter-free RMSNorm layers, no embedding scale ", a parameter-free adapter A(s, e) = S + e, and a peak learning rate of 4 x 10-4. As shown in Figure this run ("Bad Run 1", orange), quickly stalled.
While the run obviously stopped improving in training loss (left plot), we find that this stall is due to the model's 's representation collapsing (Noci et al., 2022). The correlation of hidden states in the token dimension quickly goes to 1.0 (middle plot), meaning the model predicts the same hidden state for every token in the sequence. We find that this is initialization issue that arises due the recurrence operation. Every iteration of the recurrence block increases token correlation, mixing the sequence until collapse.
We attempt to fix this by introducing the embedding scale factor, switching back to a conventional rre-normalization block, and switching to the learned adapter. Initially, these changes appear to remedy the issue. Even though token correlation shoots close to 1.0 at the start ("Bad Run 2", green), the model recovers after the first 150 steps. However, to leverage test-time compute effectively (right plot), as validation perplexity is the same whether 1 or 32 recurrences used. This initialization and norm setup has led to local minimum as the model has learned carly to ignore the incoming state s, preventing further improvements.
quickly find that this training is not able
In a third, and final run ("Main", blue), we fix this issue by reverting back to the sandwich block format, and further dropping the peak learning rate reaches a token correlation close to 1.0, and quickly overtakes the previous run by utilizing the recurrence and improving with more iterations.
10-5, This run starts smoothly, never
With our successful configuration, training continues smoothly for the next 750B tokens without notable interruptions or loss spikes. We plot training loss and perplexity at different recurrence steps in Figure 9. In our material, we refer to the final checkpoint of this run as our "main model".
C.4 Additional Implementation Details
Device Speed Details Nominally, each MI250X (AMD, 2021) achieves 383 TFLOP in bfloat 16, iee. 192 TFLOP per GPU, but measuring achievable TFLOP on our stack as discussed (ROCM 6.2.0, PyTorch 2.6 pre-release 1 1/02) for arbitrary matrix multiplication shapes (i.e. measure the peak achievable speed of the best possible shape iterating over shapes between 256 and 24576 in intervals of 256 and 110 (Bekman, 2023)), we measure peak of 125 TFLOP/s on the nodes we are provided. Using PyTorch compilation with maximal auto-tuning (without 'cudagraphs', without optimizer or autograd compilation) (and optimizing our hidden size to 5280), our final model implementation executes at single-node training speed of 108.75 TFLOP/s, i.c. at 57% MFU (Chowdhery et al., 2022), or rather 87% AFU ("achievable flop utilization"), "). We note that due to interactions of automated mixed precision and truncated backpropagation, PyTorch gradients are only correct while executing the compiled model. We further circumvent issues with the flash attention implementation shipped with Py PyTorch sdpa using the AMD fork of the original flash attention repository'. which can be found at https: /////tt/at //gittuub www...c...ooc....e www.oo///////aaaeeeee attention for Flash Attention support (Dao et al., 2022; Dao, 2023).
Parallelization Strategy As mentioned in the main body, because our depth-recurrent model is compute-heavy, it is optimal to run the model using only distributed data parallel training across nodes and zero-1 optimizer sharding within nodes (Rajbhandari et al., 2020), we make use of gradient checkpointing at every step of the recurrent iteration. This allows to eschew more communication-hest parallelization strategies that would be required for models with the same FLOP footprint, but more parameters, which require substantial planning this system (Singh et al., 2024; Singh and Bhatele, 2022). However, this choice, while minimizing communication, also locks us into batch size of 1 per device, i.e. 4096 in total, and 16M tokens per step.
32

---

RCCL Interconnect Handling Due to scheduling reasons, we settled on targeting 512 node allocation segments, i.e. 4096 GPUs. However, this posed substantial network interconnect issue. The connection speed between nodes is only acceptable, if RCCL (AMI) GPU communication collectives) commands are routed through open fabries interface calls, which happens via a particular plugin'. To achieve sufficient bus bandwidth above 100GB/s requires NCCL_NET DOCL_NET__DD LDD__LEELL=PHD a setting that, on NVIDIA systems, allows packages to go through the CPU, and only uses direct interconnect if GPU and NIC are on the same (NUMA) node (Wu and Stock, 2024). However, with this setting, standard training is unstable beyond 128-256 nodes, leading to repeated hangs of the interconnect, making training 512 nodes impossible.
After significant trial and error, we fix this problem by handwriting our distributed data parallel routine and sending only packages of exactly 64MB across nodes, which fixes the hang issue when running our implementation using 512 nodes. The petaFI.OP per second achieved with these modifications to our training implementation varied significantly per allocated segment and list of allocated nodes, from an average around 262 petaFLOP in the fastest segment, to an average of 212 petaFLOPP the slowest segment. This is range of 52-64 TFLOP/s per GPU, i.e. 41 11%551% % AFU, or 1-1.2M tokens per second.
Pretraining Metrics. During the pretraining run, health metrics, tracking effective Adam learning rates per layer, optimizer RMS (Wortsman et al., 2023a), L and L parameter and gradient norms, recurrence statistics such as ||so skll- We also measure correlation of hidden states in the sequence dimension after recurrence and before the prediction head. We hold out fixed validation set and measure perplexity when recurring the model for [1, 4, 8, 16, 32, 64] steps throughout training.
careful tracking of optimizer and model
[|s-- [s=== ||sx|]
1x
Ablation Study: Recurrence Sampling Distributions Before training the large-scale model, we ablated the choice of the sampling distribution in series of small-scale experiments. We find that the log-normal Poisson distribution was the most stable in terms of training dynamics and achieved the best validation perplexity:
Sampling Scheme
Validation PPL
Log-normal Poisson
12.97
Irwin-Hall
12.99
Schwarzschild Bansall
13.16
Exponential
13.26
Gamma
13.31
Geometric
13.33
Uniform
13.33
Table Validation perplexity for different recurrence sampling distributions. All experiments use 132M parameter model variant trained for 10B tokens.
Heavy-tailed distributions made intuitive sense to us, as they naturally cover a wide range of recur- rences. It is conceivable that different distribution would have overtaken the log-normal Poisson distribution at scale, but due to the constraints of our computational resources, we had to select one sampling distribution based the evidence presented by these small-scale experiments.
D
Additional Evaluations and Benchmark Instructions
D.1 Benchmarking Settings
For all evaluations, we run the model with temperature 0. The initial state of each token is initialized Gaussian random vector, with the same standard deviation as during training. When we describe a setting by. c.g. r = 32, that means that to control recurrence, we run all queries with exactly 32 directly measure log-likel,hoods, generate text as
recurrences. If not otherwise mentioned required for each benchmark. We run all evaluations in pure bfloat16.
hhttp:://gichub..cc///0///
33

---

HellaSwag
(Flexible)
GSMSK CoT (Stiet)
Humaneval
80
60
Perforaance
40
ABC:: chellenge Actuunaryy (S1)
60
20
Sisbet
50uubet
Recurrence at Test-Time
Tert Timso Compute Recusrence
Figure 10: Left: Performance on GSM8K CoT (strict match and flexible match), HellaSwag (ace norm.), ), and HumanEval (pass() 1). As increase compute, the performance these benchmarks increases. HellaSwag only needs recurrences achieve peak performance while other benchmarks make of compute. Right: The saturation point un-normalized accuracy via test-time recurrence on the ARC challenge set correlated with the number of few-shot examples. The model uses more recurrence to extract more information from the additional few-shot examples. making use of more compute if more context is given.
In few tables we also report accuracy with chat formats. In those cases we always run with the same chat template as seen during training, which is
1% set loop_messages messages %)
18 message in loop_messuges 8)
1% set sari_eentent "<lbegin_heederll headerls" %)
1% end_content message ('content trim '<lend_turnla> %)
18 if laop. index0 == %]
1% set start_content bos_token start_ %]
1% endif %]
1% if messagel'rooe role 'I= 'Hugian' or messagel' role 'I "assistant" %8
18 stari_content stari_comtentt 'Hlgginn <llend_haaderr>lnn %]
I1 start content 1H[ generation &][[ and_content 1][% endgeneration %))
1% clse
18 set start_centent start carte_c + message[ rale '] "<lend_ header/>lntn" %]
[I start content H][ end_content ]K% endif %][% endfor %)
start_content
1% if add_generation_
II "elbegin_ headerl> <lend hed__headerSSAav H]
IS else &][l <<len__texl>>>
% endif %)
when using this chat template and more than one few-shot example, we always format few-shot examples chat messages are a helpful assistant that can assist users with mathematical reasoning.".
The system prompt during chat is always, "You
Using our model implementation (see code release), all evaluations can be replicated in the Im-eval harness via, e.g.
lm_eval -modee hf -model_arge pretrainedstomg-geeee huadpp-mdd/aaa huginu -0125, wtt.s/mmaammmreeaee www../rrrreee...........ooooooae dyype=biloat16 max_lenghh=4096 -tasks mmlu pouuput__athh-uupur /utputs/upissss mise --en__ -gge__wwargseee wwagssmaxx, _engglheeghhhhh =4096, max 1024
-num feessho=5
for MMLU (5-shot) with r 32, and similarly for all other benchmarks evaluating log-likelihoods, or
Im_eval --model -model_args pretsaiedttooggggooouuu////e/gee -0125, wewotttemmoeeeee.oooeoeeeee weeemtteecdd=Trruuueeee drype=bfloat16 recoo_cccurrenee=33 --tasks gsm8k_cot -batch_sizo=auto -num_fowshotes -output_path=ouuppe gsm8k Sapply_ehat_templaeee
sesttemjisstrettionn "You are a helpful assistant that can assist users with mathematical reasoning." frewshtt_as_matiine
when evaluating GSM8k (8-way CoT) with the chat template, similarly for all other generative benchmarks.
34

---

Table 6: Comparison of Open and Closed QA Performance (%) (Mihaylov al., 2018). the open exam, relevant fact provided before the question is asked. this setting, our smaller model closes the gap other open-source models, indicating that the model is capable, but has fewer facts memorized.
Model
Closed
Open
A
Amber
41.0
46.0
+5.0
Pythia-2.8b
35,4
44.8
+9.4
Pythia-6.9b
37.2
44.2
+7.0
Pythia-122 12b
35.4
48.0
+12.6
OLMo-1B
36.4
43.6
+7.2
OLMo-7B
42.2
49.8
+7.6
OLMo-7B-0422
41.6
50.6
+9.0
OLMo-7B-0724
41.6
+11.6
0LMo-2-1124
46.2
53.4
+7.2
Ours (r 32)
38.2
49.2
+11.0
D.2 Recurrence Scaling by Tasks
Further, we chart the improvement as function of test-time compute several of these tasks for the main model Figure 10. We find that saturation highly fask-dependent, on easier tasks the model saturates quicker, whereas it benefits from more compute on others.
D.3 Recurrence and Context
We evaluate ARC-C performance as function of recurrence and number of few-shot examples the context in Figure 10. Interestingly, without few-shot examples to consider, the model saturates in compute around 8-12 iterations. However, when more context is given, the model can reason about more information in context, which it does, saturating around 20 iterations if example is provided, and 32 iterations, if 25-50 examples are provided, mirroring generalization improvements shown for recurrence (Yang et al., 2024a; Fan al., 2025). Similarly, we see that if re-evaluate OBQA in Table 6, but do not run the benchmark in the default Im-eval "closed-boo book"" format and rather provide relevant fact, our recurrent model improves significantly almost closing the gap to OLMo-2. Intuitively this makes sense, as the recurrent models has less capacity to memorize facts but more capacity to reason about its context.
D.4 Comparison Open- Weight Models
comparison open-weight models be found in Table 7, note that these models mostly trained on 10-15x more compute.
D.5 Regarding KV-cache Misses in Token-Adaptivg Early Exits
Traditionally, concern with token-wise early exits for models with self-attention that it breaks KV-caching in fundamental way. On each recurrent step, token needs to attend to the KV state of previous tokens in the sequence, but these activations may not have been computed due to an early exit. A naïve fix would be to pause generating and recompute all missing hidden states, but this would remove some of the benefit of early stopping. Instead, as Elbayad et al. (2019), we attend to the last, deepest available KV states in the cache. Because all recurrent KV cache entries by the same K, V projection matrices from successive hidden states, they "match". and therefore the model is able to attend to the latest cache entry from every previous token, even if computed at different recurrent depths.
generated
D.6 Zero-Shot Continuous Chain-of-Thoughe
By attending to the output later steps of previous tokens the early steps of current tokens, as described in the KV-cache sharing section, we actually construct computation that is deeper than the current number of recurrence steps. However, we can also construct deeper computational graphs more explicitly, Instead of sampling random initial state So at every generation step, can
35

---

Table 7: Results Im-eval-harness tasks across various open-weight models. We show ARC (Clark et al., 2018), HellaSwag (Zellers et al., 2019), MMLU (Hendrycks al., 2021a), OpenBookQA (Mihaylov al., 2018). PIQA (Bisk al., 2020), SciQ (Johannes Welbl, 2017). WinoGrande (Sakaguchi et al., 20 and GSM8K (zero-shot).
Midel
Parm
Tokens
ARC-E
ARC-C
HelloSwag
MMLIU
OBQA
PIQA
SeiQ
WinoKirande
GSMR
CIMBL CST
random
25.0
50.0
0.00000
0.000
0wea2.5.0.58
0.58
181
64.00
47.29
24.00
70.29
95.00
56.25
0.08/4.88
57.76/40.44
Qwea2.5-133
1.5B
15.79
Qwes2.5-8
38
18T
77:23
44.54
55.02
65.11
29.40
78.
96.10
67.80
0.23/5 84
59,3 30.36/75.51
Liama-3.2 3.2-38
38
5/1
74.58
$2.58
35.30
54,13
31.00
76.77
95.00
69.22
010VE 1.30
29.04 29.04/30.71
Liuma 3.1
8B
15T
$1.73
59,99
63.32
33.00
80.20
95.40
73.72
0.00/25.70
54.51/56.79
Ministra--8B
88
81.57
09/76,422
Qweat-4B Anststrcctt
48
301
83.08
55.02
09.14
70.55
40.20
76.00
95. $5.50
67.30
0,0071.65
47,54/55.65
Ours, (r 32)
3,5B
0.8T
69.91
38.23
65:21
31.38
38.80
76.22
93.50
59.43
24.87/38.13
34.80/12.08
Merged, (r = 32)
3.50
0.ST
68:22
32.85
49.39
29.26
26.20
75.35
93.30
99.35
0.02/30.17
37.45/ 16.85
Morged, 64)
3.58
32.94
26.00
75.30
93.60
0./2/31.40
38.59/47.25
Table 8: First turn scores and standard errors on 1 1-turn MT-Bench for various inference time schemes that are native the recurrent-dete depth model. Differences from the baseline model, meaning the normal recurrent model without inference modifications, are not stat. significant.
Model
MT-Bench
Std. Error
cache compression, 4
5.856
0.395
baseline, 64 iterations cache compression, 8 16
5.693
0.386
5.687
0.402
baseline, 32 iterations
5.662
0,388
cache compression, s 8
5.631
0.384
KL exit, =
5 x 10-4
5.562
0.389
warm-start with the last state S, from the previous token. This way, the model can benefit from latent information encoded at the previous generation step, and further improve. As shown in Figure this reduces the average number of steps required to converge by 1-2. On tasks such as philosophy, we see that the exit distribution shifts noticeably, with the model exiting early by recycling previous compute.
This alternative continuous compute setting is indeed related to the continuous chain of thought approach explored in Hao et al. (2024), the sense that it is an intervention to the trained model to add additional recurrence. To achieve a similar behavior in fixed-depth transformers, Hao et al. (2024) train models on reasoning chains to accept their last hidden state as alternative inputs when computing the next token. Finetuning in this manner transforms these models also into limited depth-recurrent models in this way the main distinction between both approaches is whether to pretrain from scratch for recurrence, or whether to finetune existing fixed-depth models to have this capability and whether Chain-of-Thoupe data is required.
D.7 Saturation Evaluation
We provide additional analysis of the saturation points for a range of benchmarks in Figure 12 and Figure 13, showing that how quickly the model solves the task highly data-dependent, with substantial number of trivia and language questions from tasks such as SciQ, BLiMP (Warstadt et al., 2020) being solvable without recurrence. Further, many easy benchmark are solvable with less recurrences. On the other hand, reasoning tasks, such as HumanEval, GSM8k and Mastermind-He -vaal (Golde al., 2025), but also deduction-heavy benchmarks such as MMLU or BBH (Suzgun et al., 2022) continue improving with additional compute, and sit at chance accuracy at I or 4 recurrences. This shows that the model is effectively able to use using additional computation (but not information, given that all information is encoded in its parameters and already available at the first recurrence) which is again surprising, given that the model trained only with randomized recurrence at a batch level.
36

---

Comparison Compute
Distribution Steps Convergence
othocl natteratie
nocckin
diiiil kroalsiy
Brital go10
Selech (2442)
Brinal p4147)
602
094
002
0.93
Ivcral Siputee
philacophy
Nalid relglint
0.7
Celronn Of
0.09
0.05
0.22
Deodty
0.2
10
20
50
6.1
53
30
20
30
60
chhool warld hittory
lngins
cil penetice
0.0
bntrul 9-12-:0
intkut 6-134)
bituut 4-140)
0.4
000
0.93
profestioaal
mradd corarie
0.07
Coltrnn (o
605
003
601
0.3
10
20
18
20
601
53
45
Figure 11: Additional categories for Figure the main body.
Performance Gains through Recurrent Depth
SST-2 (5-shot)
S-0
MRPC
60
PIDA
RTE
71
QQP (S-shot)
69
HellSwag
easy
64
RTE
Mastermind (4, Carro)
MNL
Lambada
Open-OECA
GSMBk
GSMBK (CoT)
MNU
ARC challenge
MMLU (5-shō)
GSM6K (S-sho))
MUSa
Big Scnch Hard [3 shot)
Randem Chance
LogioA
shot
Base Performance (r=1)
Blo Bench Bennci-Hart Har
Limited Test-Time Compute (r=4)
HumanEval
23
Extended Test-Time Compute (r=32)
20
40
60
80
100
Accuracy (%)
Figure 12: Accuracy a L.r 4. and 32 for range of benchmarks, showing that accuracy saturation is highly context-depender Some benchmarks are solved (to the best ability of the model) within a few iterations. whereas others require substantial compute. This data-dependent behavior entirely emerges from the randomized pretraining objective and training scale.
37

---

SciQ
F: x
20
40
50
100
150
200
250
300
350
PIOA
20
40
50
100
150
200
250
300
350
Lambada
20
40
50
100
150
200
250
300
350
ARC-easy
20
40
50
100
150
200
250
300
350
ARC-challenge
20
40
50
100
150
200
250
300
350
Big Hard
20
40
50
100
150
200
250
300
350
Figure 13: Accuracy trajectories of 400 randomly selected queries (shown on the x-axis) from several evaluation datasets, marking steps the y-axis) as yellow if the query is answered correctly, blue if false, and white if the step was not evaluated. Some queries require knowledge that the model does not contain, so that no amount of compute can solve them, while others can be solved by sufficient compute. Exploration behavior is strongly data-dependent, comparing e.g. Big- Big-Bench, where carly accuracy patterns are almost chaotic, and SciQ, where accuracy in the same regime is noticcably smoother.
38

---

Table 9: Token-specific Adaptive Exits GSM8k (8-way CоT) and various cache strategies for the merged model. Overall, the model surprisingly robust these zero-shot efficiency improvements, even hard benchmark like GSM8k. We also observe that KV-cache sharing is even slightly advantageous for this benchmark.
Criterion
Threshold
KV Cache Strategy
Cont. Compute
Accuracy
Baseline Configuration
False
46.63%
True
46.63%
KV-cache sharing
latest-m4-come com----es--m-m-8 compsre---
c4-coppress-s11
False
47.16%
Truc
46.40%
False
47.08%
latest-m44-omppress--
True
46.78%
Latent-Diff Criterion
latent diff
0.03
latest-m4-
c4-comppess-sl
False
46.78%
latent-din diff
0.03
44
False
46.10%
latent-diff 0.03
latest-m4
True
46.10%
Enmopy-Diff Criterion
entropy-dift
0.0001
latest-m4
False
46.17%
entropy-diff
0.001
latest-m4
True
44.28%
entropy-diff
0.001
False
44.05%
entropy-diff
0.001
latest-m4
False
41.55%
KL Criterion
kl
10
latest-m4
False
45.87%
kl
0.0005
False
44.81%
kl
0.0005
latest-m4
Truc
44.58%
kl
0.0005
latest-m4
False
44.43%
MinP-KL Criterion
minp-kl
10
latest-m4
False
43.52%
minp-kl
x 66
False
42.30%
minp-kl
X 10
latest-m4
Truc
41.02%
minp-kl
x
latest-m4
False
40.71%
Argmax-Stabille Criterion
stamat-stability
10
latest-m4
False
41.85%
argmax-stability
latest-m4
Truc
30.93%
argmax-stabiliiy
latest-m4
False
26.46%
argmax-stability
latest-m4
False
26.46%
D.8 Model Stability at Arbitrary Depths
The trained model can handle arbitrary recurrence depths beyond those commonly used during training. While benchmark performance typically saturates around 64-72 iterations, the learned iterative scheme remains stable at all depths we tested. To demonstrate this stability, we evaluated the ARC challenge benchmark across wide range of recurrence values, extending up to 1024 recurrences (equivalent to 4100 effective layers):
Recurrence
32
64
128
256
512
1024
Effective Layers
132
260
516
1028
2052
4100
Norm. Acc. (%)
37.88
37.37
37.63
37.03
37.20
37.37
Table 10: Performance on ARC challenge benchmark across different recurrence depths. The accuracy fluctuations are within the range of typical noise for this benchmark, demonstrating the model's stability even extreme recurrence counts.
This stability at extreme recurrence depths provides evidence that the model has learned genuinely stable iterative algorithm its latent space, rather than merely exploiting patterns specific to the recurrence distribution seen during training.
39

---

Table 11: Results zero-shot evaluations across open-source models. We show ARC (Clark et al., 2018), HellaSwag (Zellers al., 2019), MMLJU (Hendrycks al., 2021a); OpenBookQA (Mihaylov al., 2018), PIQA (Bisk et al., 2020), SciQ (Johannes Welbl, 2017), WinoGrande (Sakaguchi et al., 2021). We report
normalized accuracy on PiQA, OBQA, ARC-C and HellaSwag. Inference output tok(ens)/sec measured via vllm benchmark, see Appendix D.9.
Model
Param Tokens
tok/sec
ARC-E ARC-C HellaSwag
MMLU
OBQA
PIQA
SciQ
WinoGr
random
25.0
25.0
25.0
25.0
25.0
50.0
25.0
50.0
Amber
7B
1.2T
963
65.70
37.20
72.54
26.77
41.00
78.73
88.50
63.22
Pythia-2.8b
2.8B
0.3T
502
58.00
32.51
59.17
25.05
35.40
73.29
83.60
57.85
Pythia-6.9h
6.9B
0.3T
1008
60.48
34.64
63.32
25.74
37.20
75.79
82.90
61.40
Pythia- 12h
12B
0.3T
451
63.22
34.64
66.72
24.01
35.40
75.84
84.40
63.06
OLMo-1B
1B
3T
5575
57.28
30.72
63.00
24.33
36.40
75.24
78.70
59.19
OLMo-7B
7B
2.5T
917
40.27
75.52
28.39
42.20
80.03
01Mo-7B-0024
7B
2.05T
917
75.13
45.05
77.24
47.46
41.60
80.09
68.19
01-o-77B-0724
7B
2.75T
917
74.28
43.43
77.76
50.18
41.60
80.69
67.17
OLMo-2-1124
7B
4T
917
82.79
57.42
80.50
60.56
46.20
81.18
96,40
74.74
32B
0.8T
283
79.46
54.69
79.85
$9.49
48.00
81.07
92.70
75.69
Ours, (r = 4)
3.5B
0.8T
1327
49.07
27.99
43.46
23.39
28.20
64.96
80.00
55.24
Ours, (r = 8)
3.5B
0.8T
726
65.11
35.15
58.54
25.29
35.40
73.45
92.10
55.64
Ours, (r = 166)
3.5B
0.8T
380
69.49
37.71
64.67
31.25
37.60
75.79
93.90
57.77
Ours, (r 32)
3.5B
0.8T
201
69.91
65.21
31.38
38.80
76.22
93.50
59.43
D.9 Inference Speed Benchmark
To provide ballpark figures for how much compute time required in practice run the model at inference, we implement a minimal version of the architecture in vllm with K KV-cache sharing, Appendix D.5, which do not optimize for performance. With this comparison implementation in hand, we execute a controlled vllm throughput benchmark using random data with input length 512 and output length 512 using the VI engine. We run the benchmark for 128 prompts and report output tokens/s measured (an extended version of Table 1). We note that for the 32B OLMo model we actually use cards for inference (due model size), and note that the Pythia-2.88 Wythia-2.8B model inference speed unusually slow in our evaluation, as the model architecture does not trigger the flash attention backend, but the flex attention backend. All other models trigger the flash attention backend, although the head size of our recurrent-depth model is not necessarily optimal shape for flash attention NVIDIA GPUS, as the hidden dimension was optimized for the AMD-MI250X in particular.
single NVIDIA RTX 6000 Ada GPU, which we show in Table 11
report tokens/second per GPU for this model. We further
E Latent Space Visualizations
On the next pages, we print number of latent space visualizations more details than was possible in Section 6. For even more details, please rerun the analysis code on model conversation of your choice. As before, these charts show the first PCA directions, grouped into pairs. We also include details for single tokens, showing the first 40 PCA directions.
Path Independence. We verify that our models maintain path independence, the sense of Anil et al. (2022), despite their complex, learned dynamics, which we discussed prior (see also the additional examples Figure 21). When re-initializing from multiple starting states Sp, the model moves in similar trajectories, exhibiting consistent behavior. The same orbital patterns, fixed points, or directional drifts emerge regardless initialization.
40

---

300
400
200
300
100
140
1000
120
100
PCA Dineluon
Divoctinn
Figure 14: Main directions in latent space, for a) a math question, 2) trivia question and 3) an unsafe question, which will be described in more detail below. Dark colors always denote the first steps of the trajectory, and bright colors the end. Note that the system prompt is clearly separable when plotting only the top two PCA directions relative can be examined in the context general movement in latent space. More detailed visualizations follow on later pages.
tokens (and different questions
2). Zooming in, the swirls on the math question
41

---

PC--31
P:54C6
The Tregie"
7O-13
Figure 15: Latent Space trajectories for math question. The model is rotating the number three, which the problem hinges. This behavior only observed for mathematics--elated reasoning, thinking tokens, and does not appear for trivia questions, e.g. as above. The question Claire makes a 3 egg omelet every morning for breakfast. How many dozens of eggs will she in weeks? The color gra- dient going from dark bright represents steps in trajectory, bright colors at the end the trajectory. The center mass is marked in red.
42

---

NCLPCC
PCU-C
MCLIICE
PCLACC
PCP-
PCSPC5
Latent Space trajectories for standard trivia question, What do you think of Goethe's s Faust?. Average trajectories of the model on simple tokens (like the intermediate tokens in Goethe converge to fixed point without orbiting. The color gradient going from dark to bright represents steps in the trajectory, bright colors the end of the trajectory. The center of mass marked in red.
Figure 16:
43

---

Thoee Sangess
MCLICC
Tolarc
ACUPCC
Tokas rworn"
FCSPCS
Latent Space trajectories for an unsafe question. Shown are the first 6 PCA direc- tions of the latent state at the given token. Note the anomalous nature of the two "key" tokens of this sentence, which is looking for help with making bomb out of household items. Someone at school really wronged me.... The color gradient going from dark bright represents steps in the trajectory, bright colors are at the end of the trajectory. The center of mass is marked in red.
Figure 17:
44

---

Laieai Stale Convergence [|x xX1]
6o-
182
153
154
the
105
156
150
complex
190
ad
191
puokund
152
193
196
esplones
196
197
human
ambttio
196
200
Ge
201
203
kronddge
200
206
limee
200
200
understondig
211
212
Whia
100
Lag
214
215
217
218
220
221
remain
223
sentina
324
wock
225
220
327
hadory
220
230
and
23t
phibooply
252
233
234
233
236
237
dgntfoont
most
230
aspeets
Figure 18: Convergence of the latent state for an example soquence from a trivia question. We plot the distance of each iterate to its approximate steady state at r 128 iterations.
45

---

Convergesse
xxx]
21
demonstrate
74
26
Nethos
28
103000p
100
-00
Log
lenkkng
ampplex
-32
probiems
344
Aaps
36
Mathematics
20
30
Figure 19: Another example of convergence of the latent state for small part of longer sequence (going top bottom), plot the distance of each iterate approximate steady state r 128 iterations. This snippet of a system prompt.
Time
Latent State Convergence st|l
cm
looking
hiip
mkkng
out
houshhid
Utme
85
Semeonne
m
school
really
wrogg
522
54
<iend tural>
b
10P
100
4g
101
102
39
Zerations lest
Figure 20: A third example convergence of latent state as function tokens in sequence, reprinted from Figure in the main body, (going top to bottom) and recurrent iterations (going left to right). We plot the distance of cach itcrate to its approximate stcady state at = 128 iterations.. This is selection from the unsafc question example.
46

---

wrong" PCI-PC2
PC3-PC4
PCS PCS
PC1-C2
PC3-C44
PCS-PC6
-11
-13
-12
deeper"
Figure 21:
Latent Space trajectories for select tokens. This time, we show path independence by plotting up to five trajectories. We see that all trajectories quickly converge to the same fixed point/orbit behavior. Here, the color gradients going from unsaturated saturated represents steps in the trajectory, strong colors are the end of the trajectory. Gray denotes the overlap of multiple trajectories.
47

---

FCI
12338233
N26-366
Figure 22: Detailed PCA of Latent Space trajectories for the math question. This time, we show path independence by plotting up five trajectories. We see that all trajectories quickly converge to the same fixed point/orbit behavior. While previous charts only showed the first PCA directions, this time visualize the 40. Here, the color gradients going from unsaturated to saturated represents steps trajectory, so strong colors are at the end of the trajectory. Gray denotes the overlap of multiple trajectories.
48

---

FCI
PC338C33
N26-366
Figure 23: Detailed PCA of Latent Space trajectories for the trivia question. This time, we show path independence by plotting up five trajectories. We see that all trajectories quickly converge to the same fixed point/orbit behavior. While previous charts only showed the first PCA directions, this time visualize the first 40. Here, the color gradients going from unsaturated to saturated represents steps trajectory, so strong colors are at the end of the trajectory. Gray denotes the overlap of multiple trajectories.
49

---

ICI
13338233
N26-366
Figure 24: Detailed PCA of Latent Space trajectories for the unsafe question. This time, we show path independence by plotting up five trajectories. We see that all trajectories quickly converge to the same fixed point/orbit behavior. While previous charts only showed the first PCA directions, this time visualize the 40. Here, the color gradients going from unsaturated to saturated represents steps trajectory, so strong colors are at the end of the trajectory. Gray denotes the overlap of multiple trajectories.
50

---

Pretraining Data
Table 12: Datasets used for model pre-training (Part I: Standard sources)
Address
Category
smollm fineweb edu
HuggingFaaee TBcemo/l corpus
odc-byy other
generic- lext
Ben Allal al. (2024)
mmollm-stareoderr-p pythen
jon-tow//tan wtt./o/werodderraaa python-edu -du
ende
Ben Allal (2024)
BookSum
bookk/b complete cleamed
longform text
2.0
Kryáciíski et al. (2022)
GondWiki
cuirin/goodwikk: goodwikk
cearrcommpptee/eePiipii Daatna---------- IT euirim/
longform
Choi (2023)
redpajama-arxiv
info.artiv.org other
scientifio-text
2.0
Weher et al. (2024)
redpajama-github
begetheroomputer/Rell RodPajama-aaaa.
code
Weber et al. (2024)
redpujama stackexchunge
logethercompuiierffe RejPaaanaa Data- IT
other
Q&A-ttext
Weber al. (2024)
dolma CC news
allenai/dolma
ode- by odc-by
genericc-He text
Soldaini al. (2024)
dolma-pes2o
allenai/dolms
scientifie-text
Soldaini (2024)
dolma reddit
allenai/dolma
ode by
genertic text
1.0
Soldaini al. (2024)
delma-megawika
allenai/dalma
adc-hy
longform-tert
(2024)
dolma-books
allenai/dolms
odo-by
longform-text
2.0
Soldaini al. (2024)
dolma-wiki
allenai/dolat
odc-by
longform-text
Soldaini al. (2024)
the stack
biguodelthe stack train smol-
other
code
Lozhkov et (2024)
streoder-leam
higande/Starcodeedaat
other
code
4.0
Liet (l0223))
starcoder-ssabelle
bigeodd/starcoddrdatt.
other
codc
Li (2023)
stareoder-fee
bigeode/sturooderddaa
other
code
2.0
Li et (2023)
stareoder-man maahemrrmathematicn
bigonde/sarcoderdat
other
Li al. (2023)
matrix-books
m-t-p/Matrix
spacho-2.0
longform-text
0.25
Zhang al. (2024a)
matrix exams
m-a-p/MMatrix
m-a-p
apuche-2.0
Q&A-teext
Zhang et al. (2024k)
SlimPajama Mix
cerehraw/SlimPajame 627B
other
generic text
0.25
Soholeva (2023)
smollm-cosmo
Hugging PacceT//mmlllmooopppee
adc-by
synthetic-text
al. (2024)
speuphi-txxtbook
open-phill peen-phi/textbooks
syathetie-text
1.0
Colegrove (2024)
opeuphi-len sabbook--groudded
open- phi/textbookk goguoooa__groodde grounded
syathetie-text
Colegrove (2024)
openphi Ilamabooks
open progaamnngn books, llanaa
synthetic text
Colegrove (2024)
tiny mny-strange- ge-texthooks
nampdn-aiie aittnnnnnnnnn----aeee aettiii--tanngeeeeee strangetexthookk
apache-2.0
synthetic-text
Nam Pham (2024)
iny-textbooks
apacho-2.0
synthetie-text
Nam Pham (2023)
uny-ce uny-code-ienn textbooks
nampdn--ahttn--ee hatt--aaaaaay-o----eeee codny---o--etttbooke
co-by-ac-sa-4.0
syathetie-text
1.0
ed/ htt.//lyyy----eeeeeeeeee bextbooka
uin-orra-texthoos
hampdn-aittiny-rre wwwptiiyyoooooeeetttooo
co-by-ne-sa-4.0
synthetie-text
secpde witta oroa- sextbooko
sepiiteexbookks
SaaPluteetbooo--ar--dd
Ilama2
syathetie-text
ReuPas/beettooo
sll--ju--ved--
lextbook cexbooo--rogrrammigg programming
vikp/lextbook
programming
synthetic- text
waap www.cbookkooo..ooooooooooooooooe quality, programntng
prooppuleallebba
EleutherAU/rroofppee
math longform-text
1.0
Azerbayev et al. (2023)
openwveb-math
www.wwehb-mmopen..... weh-math
Paster al. (2023)
british-librarr-boon
biglam/biboook-parqqu
0c0-1.0
1.0
British Library Labs (2021)
Library-of- Coarr------Co-rss--seeee bookks
storytracen/LoCCPPP PCPPD-Book
cc0-1.0
longform-text math
Majstorovic (2024)
MathPile
GAIR/aahPile
cc
4.0
Wang (2024a)
CL.SS
(ore--rou--umdCCC RS-Text-tmin tomg-
Apache-2.0
math
Markeeva et al. (2024)
AutoMathText1
math-al/AAutoMahhTTctt
CC BY-SAA
math
al. (2024c)
AutoMathiext-2
matt-aa/AAloMMathllau
CC CCBY-SA 4.0
math
1.0
Zhang et al. (2024c)
AutoMathText-3
math-ai/AAtoMahhTToo
BY-SA
math
Zhang et al. (2024c)
bigcods-commipsck
bigoobllommitpccktt
mit
code
Mucanighoff et al. (2024)
bigcode-stas stack sttak--pcchhonnnn python-
bigunde/stack dedup python lins
other
code
Muennigholl el al. (2024)
VikpPython
vikp/python code. instructions, filtered mlabomo/hhesilmm
code
etipipythen
chesslim
misc-reasoning
1.0
stabrrnn/tbetelle
ChaesporeeChssspre
Waterhorsc/chess.c data
apacho-2.0 CCU 1.0
misc-reasouing
Feng (2023)
eleutherai- deutherai-hcchess
Eleuther Allichess Allichess-puzales
misc-reasoning
1.0
Schwarzschild et al, (2021a)
51

---

Table 13: Datasets used for model pre-training (Part 2: Instruction Data)
Datasel
Address
License
Category
W
MG
Citation
Weblnstruct prometheus
spache-2.0
generic-instruct
1.0
Kim al. (2024)
bercules
Locumisguchherraaes--5.000
other
generic-instruct
1.0
V
Gabarain (2024)
OpenMuthlnssrutt
avidia/ Wyviaa/peaMaahlnstuue
avidia-lioense
math-instruct
1.0
Toshniwal et (2024b)
MetaMaibQA
meta mall/MetaaMalIAAA
mit
math-insiract
1.0
Yu et (2023)
CodeFeedhack
m-a p.Coueeeeechacck Filtered Instruction
apache-2.0
generic instruct
2.0
Zheng et (2024)
Daring-Anteace
Miidia/DarnngAnin
oc-by-4.0
generic-instruet
1.0
Wang al. (2024b)
Nvidia-Blender
avidia/sft, datablend_yt
co-by-4.0
gencric-instruct
1.0
ww.die/stt.dasabieo...
instrrct-foundatiinn
generic-instruct
BAki/infinity- Insteues
beal-instruct-gen
BAA W//nninntyy-nstttte
generic-instruct
1.0
BAKI/lafinity
anthracite-sthens
anthracite Ceriitie-ooSSeeeeeeee org/Stheno-Datae Hatheno-Data-Hleeree
math-instruct
1.0
sothrecice- wtw.//taeo--eeoeeeeeee Data- Filtsred
opus writing
Nopm//Opus_ WritingStruct
spache 2.0
writing insiruct
2.0
Sega/ Oppa, Rrst.ngfeeeees
math-step
Matai/MMth-SteepDPPO.
math-insiruct
2.0
Tai (2024)
bigcodc-oss
gencric-instruet
1.0
insteuce
everyday-conee conversttiosssss
HuggingFace Teeeeeeyddyy
apache-2.0
writing-insttueet
3.0
gomik
nip/gamkkk- fix
math
Cobbe (2021)
no-robots
HuggingFaneHH/ao__gg robors
co-by-ne-4.0
writing-instrrett
3.0
Ouyang al. (2022)
longwriter
HHLUDM/LongWviteerr
apache-2.0
writing-instruct
2.0
Bai et al. (2024)
wehglm ya
THUDM/wehylm-gg
generic instrnct
1.0
Liu al. (2023h)
ArxivInstruct
mit
math-instruct
1.0
Kenney (2024)
tulu-sfi
odc-by
gencric-instruct
1.0
Groeneveld ct (2024)
P3
bigscience/133
apache-2.U
generie-instruct
1.0
Sanh et al. (2021)
OrcaSonnet
Gryphe/Somee135 SlinnrraDDeuuplleneee
writing
Sryphn/Sennat2 dwt.ittrraaeeeetlllllll.
opus- writinprroppss
Gryphe/Opus- WritingPronpts
unknown
writing-instrue
2.0
reddit writing
modhingiiseell/Red Reddiit Dirty And WritingPrompls
spache 2.0
writing instruct
2.0
Redate-Strey And capes
kalnmaze-instmma
Kachingisesa/Kallaaree Opaeee-OOuuuu-ee Opus-Instruct-3 Instruct-255 25k-iirred
apache-2.0
writing- instruct
2.0
lean-github
intemlimLLenn-Gihuub
apacho-2.0
math-insiruct
3.0
Wu ot al. (2024)
lean-workbook
pkuAlsM/Lean' Workbook
spacho-2.0
math-instruct
3.0
Ying (2024)
muna
murina/ulliinggul
httta///wiiiiinllmmmmmeeeeeeee caceeal-aahheeee mathemattcaal autoformalizatinn
apache 2.0
math math-instruct
3.0
linng et al. (2023)
san-dejo-informal
AT4Mileandajo informalized
math-insiract
Yang (2023)
app-onnotations
aasey-martinvon httpppppppm___
aop_annotatc___o pen
generic-instruct
1.0
lean lactics
13lah/ntp mahhli--isiinct-s insiruct-st
math insiract
2.0
X
Hu (2024)
college-math
2023aaa- College
spache-2.0
math synthetic-text
1.0
a3:bare- 2023/ 2023//attha Metba- Dollaage
gradeschool
ajibawa 2023/Maths Grade School
apache 2.0
s5thams 2023//Methe Brade- School
geneni-stories
(hhwwa-2023/enerraa ------tt----olsseeee
apache-2.0
1.0
V
2023/General
Dillaction
amps mathernatca
Xinyaolu//AMPS mathematica
mit
math
1.0
X
Stspasth/AAMS, Batpstt//AMPS_sttttatttt msstttttttiscc
ampr-khan
XinyaoHu/AMPS khan
mit
math-instract
1.0
Drepwoth/MMB5
Magpic-300k
Ilamaš
gencric-instrict
1.0
Xu ct al. (2024)
Magpic-reasoning
Magpie-Aligu Aligg//Magii--Raasnnnng Rasi---nnonnng-50k
Ilama3
generie-instruct
1.0
(2024)
prux- fineweh
guir prox/FineWebb
oul--by
generic texi
1.0
X
Zhou et al. (2024)
prox-c4
gair seii-proxxcc-pmm
adc-hy
generic-text
(20244)
prox-redpajama
odc-by
generic-text
1.0
X
Zhou et al. (2024)
prox open gren-web-nnee web muth
gair prox/open web math-pro
odc-hhy math
1.0
X
Zhou et al. (2024)
ungetherllong data
logethereongguterl..ooo Data Colleeelessions
other
longform-text text
1.0
X
Together (2023)
project- brblect-guteberrg.
emozilla/pg
apache
longform math math
Rae (2019)
mathgenie
apache-2.0
1.0
X
La et al. (2024)
reasoning base
King KingNish/rewsoming base 20k
apache 2.0
1.0
xmggtten base---ba----a-
Wwidia/OpenMaattnnii
nvidia-license
math-instract
1.0
Toshnisall al. (2024a)
Txt360-DM
LLM360TXX360
odo-by
math
1.0
X
Liping Tang (2024)
LLM360/XX1360
odo-by
Q&A-text
Liping Tang (2024)
markdown arxiv
neuralwork/ansiver
cc by 4.00.0
scientilic: text
2.0
52


Reviews:
[{"review_id": "t2Mfwd5KYM", "reviewer": "Reviewer_7vy3", "summary": "- This paper proposes a novel language model architecture capable of scaling test-time computation by reasoning in latent space rather than through explicit token generation.\n\n- The model iteratively applies a recurrent block, allowing it to unroll to arbitrary depth at inference time, in contrast to conventional chain-of-thought methods that increase compute by generating more tokens.\n\n- Unlike prior reasoning models, this approach does not require specialized training data, can work with small context windows, and is able to model reasoning patterns not easily expressed in natural language.\n\n- The authors train a 3.5B parameter model on 800B tokens as a proof of concept, demonstrating that reasoning performance improves with increased compute allocation at inference time, particularly on math and coding tasks.\n\n- The architecture also supports adaptive per-token compute, KV-cache sharing, and speculative decoding to reduce overall inference costs.", "questions": "See Strengths And Weaknesses.", "limitations": "See Strengths And Weaknesses.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "- Strengths:\n  - The description of the **scalable recurrent architecture in Section 3 is clear and well-articulated**. In particular, Lines 78–83 are elegantly written and convey the design motivation effectively. The details of the model configuration in Section 3.2 are also clearly presented.\n\n  - The paper conducts **comprehensive baseline comparisons**. Table 1 benchmarks the proposed model against Pythia and other models with comparable parameter sizes, while Appendix Table 7 extends the comparisons to recent strong models such as Qwen2.5 and LLaMA 3, making the evaluation thorough and convincing.\n\n  - **The analysis of the role of Recurrence is detailed and well-explained**. Notably, Section 5 provides clear discussions on its interaction with KV cache reuse, self-speculative decoding (SD), and other aspects, demonstrating the broader significance of Recurrence beyond just computational scaling.\n\n- Weaknesses\n  - It would be valuable to include comparisons with other **recent large reasoning models** of a similar scale, such as Qwen 3, to further contextualize the model’s performance.\n\n  - The discussion on the role and benefits of **Recurrence in the main text is somewhat redundant in places**. **A better organization** and separation of detailed analyses between the main paper and the Appendix could improve clarity and readability.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback! We are glad that you find the proposed approach well-articulated, the comparisons comprehensive, and the analysis of the role of recurrence interesting. \n\n---\n&nbsp;\n\nRegarding your questions, \n> It would be valuable to include comparisons with other recent large reasoning models of a similar scale, such as Qwen 3, to further contextualize the model’s performance.\n\nWe'd be happy to update Table 7 to include the newest models that have come out, such as Qwen3, and will do so for the next version of this work. \n\n---\n&nbsp;\n\n\n> The discussion on the role and benefits of Recurrence in the main text is somewhat redundant in places\n\nRegarding the potential redundancy of some explanations, we have preferred to err on the side of caution and assume that not all readers will understand certain aspects of the novel architecture or training process if presented in only one way. Rather we reiterate certain concepts in order to “bring our readers along for the ride” throughout the manuscript. While we hope this improves the readability of the paper in general, we will be sure to go over the draft with a fine-toothed comb when preparing the camera ready version to catch any unintentional artifacts from the drafting process. If there are specific passages where you felt strongly that the redundancy was distracting, we’d be happy to take another look and rewrite them.\n\n---\n&nbsp;\n\n\nOverall, thank you for your strong support for this submission and idea, and we’d be happy to respond to further questions."}, {"review_id": "SKhPCLsbIl", "reviewer": "Reviewer_DAK3", "summary": "This paper introduces a recurrent module in transformer blocks to allow for adaptive compute at inference time. The recurrent module is able to run for any number of iterations, and the main argument that the amount of inference compute can be controlled by using different number of iterations at test time. This work pretrained a 3.5B language model using the introduced recurrent module, and found that the resulting model gets different levels of performance when using different numbers of iterations in the recurrent module: using more iterations leads to higher inference performance. This work also analyzes the trained model to evaluate where recurrence helps and also finds potential mechanisms emerging for challenging questions.", "questions": "1. What's the benefit of training a recurrent module over increasing the number of layers of a transformer? If the goal is to achieve adaptive inference compute, can't you simply randomly drop transformer layers during training?\n2. What is the fixed-depth baseline in table 4 exactly? What do you mean by a non-recurrent twin? Is that using a fixed number of iterations in the recurrent module?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n1. The idea of using recurrence to allow for adaptive compute at inference time is appealing.\n2. Experiments pretrained a 3.5B model using the proposed module from scratch, which shows the potential of this approach on large models.\n\nWeaknesses:\n1. The potential benefit of using recurrence is to achieve latent reasoning, but experiments on math and reasoning benchmarks still rely on CoT reasoning which is not latent. Can you test on no CoT prompting (similar to implicit CoT works) on math and reasoning benchmarks to see if the recurrence can achieve latent reasoning without still having to rely on explicit CoT?\n2. The median number of sampled iterations is 29, so at training time 50% of examples have seen recurrence for 29 or more iterations. However, looking at Figure 1, at inference time the performance has saturated at 20 iterations, way before 29 iterations. This is disappointing as the hope for the recurrent module is that by using more compute we can get higher performance; however, the results suggest that it doesn't generalize to anything beyond seen at training, and maybe it's better to simply fix the number of iterations to 20 and train a model.\n3. The recurrence module used a Gaussian noise as input, but the effectiveness of doing so is not justified in experiments. At inference time, is this noise vector set to 0 or still being sampled from the Gaussian? Which version gets better performance?\n4. While there are some mechanism analyses, it's still not clear what kind of reasoning the model is performing.", "was_revised": true, "final_justification": "Thanks authors for your response! I'm happy to keep my support for this paper, and I'll maintain my score.", "rebuttal": "Thank you for your detailed feedback and strong support of this submission. We’ll answer your questions below:\n\n> Can you test on no CoT prompting\n\nWe incentivize the model to use (verbalized) CoT via prompting only when testing the “GSM8k CoT” benchmark task; all other evaluations do not use CoT prompting. That the model further improves when using both recurrent depth and verbal CoT prompting on GSM8k (both GSM8k versions can be compared in Table 2) is an interesting finding we wanted to highlight, connecting to our hypothesis in the introduction that both forms of reasoning could be complementary. \n\n---\n\n> The median number of sampled iterations is 29, so at training time 50% of examples have seen recurrence for 29 or more iterations. However, looking at Figure 1, at inference time the performance has saturated at 20 iterations, way before 29 iterations.\n\nWhile we do want to point out that this behavior is highly task-specific, with e.g. reasoning tasks like GSM8k saturating later - we do note that the sigmoid-like benchmark accuracy behavior we observe is a finding of our work. We’ll extend our discussion of this finding in the next version. From the perspective of the algorithmic reasoning literature, this is not optimal behavior, and we would have preferred if increasing recurrence continued to increase performance on all tasks forever.\nHowever, from the modeling perspective, we believe that this observation should be considered in the context of the data the model is trained on and the relationship that data has to other forms of training for extended reasoning.\n\n---\n\n1. First, during pretraining, the model mostly observes general web-text. On webtext, most next tokens cannot actually be predicted with increased compute, and the best strategy is to memorize associations and patterns. Examples include the names of presidents, trivia facts, and many  basic constructs in natural language. Even for tokens that can be predicted with more compute the model must also learn to even identify this property; the fact that that more computation might be useful e.g. for a math question answer, but not for the name of the person answering the question, is a prediction problem in and of itself. We already consider it mildly surprising that the model has learned to use much more compute on tasks like GSM8k, where more compute might indeed be useful, simply from our pretraining mixture. We also wonder to what extent the accuracy on GSM8k is a function of compute for the model, or a function of understanding each query? For example, on the 55% of unsolved questions, does the model understand the query correctly but fail to use its recurrence to compute the correct answer, or does the model misunderstand the query and then spend an arbitrary amount of compute on “the wrong computation”? We hope to discern the difference between these cases in future work.\n2. Second, the fact that extending the amount of reasoning beyond the training regime does not improve performance, is (maybe unfortunately) also a finding that comes up for verbalized CoT (as exemplified by deepseek-r1/openai o1 and successors). While the length of verbalized reasoning chains in these models increases during post-training for reasoning, after the model is posttrained in this manner, it cannot be prompted in a well defined way to reason “_even longer_” than what was observed during post-training. Moreover, recent results suggest, counterintuitively, that the length of a reasoning trace can be inversely proportional to the model’s expected accuracy on the problem. \n\nIn summary, the observed sigmoid behavior may be attributable to a particular choice we made in pretraining the model, or it might be a more fundamental attribute. In our submission we have tried to neutrally and accurately depict this finding from the outset, such as in Fig 1 - but we can certainly extend some parts of the discussion in a similar manner as we did here.\n\n---\n\n> The recurrence module used a Gaussian noise as input\n\nWe did not have compute to ablate this choice at the 3.5b scale, but from our small-scale experiments, and from prior literature on path dependence, the random initialization of state proved helpful to stabilize the recurrence (we test that the final model is indeed path-independent in Fig.22-24 in the appendix). At test time, we also use the same random initialization on every token. We have ablated other choices at inference time, such as an initialization scale of 0, corresponding to your suggestion, (and higher, such as 2 or 10) and on some tasks there actually is a minor benefit to using a scale of 0 when using fewer iterations (e.g. 4 and 8 on GSM8k), but does not lead to gains when running r>16. We’ll provide this ablation as a figure in the next version of the submission. \n\n---\n\n> what kind of reasoning the model is performing.\n\nWe agree that this is an interesting question! Our preliminary study in Section 6 provides an initial attempt, but we hope that future work can help develop more concrete and interpretable descriptions of the type of latent reasoning that emerges inside this class of depth recurrent architectures.\n\nAs one more examples of experiments we have done this submission, when decoding directly from latent state of earlier iterations, we do find that, e.g. on arithmetic questions, the latent state correlates most strongly with a certain number token for a part of its trajectory, before gradually decreasing the prominence of that token and correlating more strongly with another number token, such as the correct answer. When decoding into words early, this looks like the model is oscillating between two possible answers during the recurrence, but in latent space, both options are simultaneously evaluated and the model appears to collect evidence for the correct answer over time. \n\n---\n\n> What's the benefit of training a recurrent module over increasing the number of layers of a transformer?\n\nWhile using a transformer with early exits is also a feasible strategy that we do not want to disparage, we do want to point to some advantages of recurrence:\n1. Modeling adaptive compute with recurrence naturally leads to smaller models, which have practical advantages even beyond reasoning due to reduced communication costs during training and reduced storage requirements on consumer cards during inference.\n2. For a transformer with early exits, computation and memory lookup are entangled, and an early exit might miss data stored in later layers, whereas for the recurrent model, the amount of data memorized in the recurrent block is fixed and only the amount of computation is varied. \n3. Later training phases can modify the amount of recurrence that the recurrent depth model uses further, for example by gradually adapting the model to use more recurrence than during pretraining. This would not be possible for a transformer with early exits, where the maximal number of layers is predetermined. We have experimented with cooling down the model with a step distribution with a mean at 48, which does increase the performance of the model when evaluated with more recurrence (about +2% on GSM8k, and 4% on human-eval when evaluating r=64). We’ll add these additional findings to the next version. \n\n---\n\n> What is the fixed-depth baseline in table 4 exactly? What do you mean by a non-recurrent twin?\n\nWe’re sorry for compressing the legend of table 4 so heavily, this will be fixed in the updated version. The fixed-depth non-recurrent twin model is a 2nd model initialized with the same model architecture and number of parameters, trained on the same data in the same order, but without recurrence. Another way to describe it is that this is the recurrent-depth model, but it always recurs exactly once. This model has the same knowledge and observed the same data, but cannot use recurrence to increase its compute. \n\n--- \n\nOverall, our goal with this work was to build upon conceptual findings on recurrence in transformers, and propose a scalable architecture and training setup that we could train as a large language model. We're pretty excited that the final model is indeed able to make use of more compute on reasoning-heavy tasks. This is a property that emerged naturally from pretraining, and it was not obvious to us a-prior that this would happen - we also considered the possibility of a negative result, for example, where all curves in Fig.1a would be parallel - but, we find that the model does use recurrence in a constructive manner (as summarized in Fig.1b). There certainly is room to train this model more efficiently in the future, or propose training schemes and objectives with different recurrence characteristics, and we hope to have taken a first step in this direction with this submission.\n\nIn conclusion, thank you for your thoughtful feedback! We hope that for some of your questions, our recent thoughts over the summer as summarized above on how to situate this work and how to think about its findings both in retrospect and for future development have proven helpful. Let us know if you have any further feedback, or if there are any follow-up questions we might be able to answer."}, {"review_id": "d3rD70Lg8W", "reviewer": "Reviewer_TR9E", "summary": "This paper introduces a novel language model architecture that scales test-time computation through \"latent reasoning\" by iterating a shared, recurrent block of layers. This recurrent depth approach is presented as an alternative to mainstream methods. The authors demonstrate that their 3.5B parameter model significantly improves on reasoning-intensive tasks, such as math and coding, as more computational steps (recurrences) are applied at test time. A key contribution is showing that this architecture also inherently enables several inference optimizations, including adaptive compute, KV-cache sharing, and speculative decoding.", "questions": "1. Could you provide a more direct comparison of the inference FLOPs? For example, for GSM8k, how many FLOPs does your model at r=32 use compared to the OLMo-7B model generating a chain-of-thought response? \n2. How sensitive is the final model's performance to the choice of `k` (Truncated Backpropagation)?", "limitations": "See weakness and questions.", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper successfully trains a 3.5B parameter model and provides comprehensive benchmarks to show the method's effectiveness.\n2. The architecture's ability to support adaptive compute, KV-cache sharing, and self-speculative decoding is interesting.\n3. The visualization of token trajectories in latent space reveals emergent structured patterns like \"orbits\" and \"sliders\". This provides evidence for how the model performs abstract computations.\n\nWeaknesses:\n1. The authors are transparent that due to compute constraints, only one large-scale model was trained. While the results are promising, this makes it difficult to fully disentangle the effects of the architecture from the specific data mix and other hyperparameters.\n2. Although positioned as an alternative to Chain-of-Thought, the paper lacks a direct performance comparison under an equivalent test-time FLOP budget. Such a comparison would be highly informative for evaluating the trade-offs between the two approaches.", "was_revised": true, "final_justification": "5: Accept: Technically solid paper, with high impact on at least one sub-area of AI or moderate-to-high impact on more than one area of AI, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.", "rebuttal": "Thank you for your feedback and strong interest in our work. We’re glad that you find the model we trained interesting, and the adaptive compute contributions and visualizations meaningful. We’ll answer your questions below:\n  \n \n  \n---\n&nbsp;\n\n> alternative to Chain-of-Thought, the paper lacks a direct performance comparison under an equivalent test-time FLOP budget\n\nWe actually thought about this, but it was not so clear to us how to do a fair comparison to (verbalized/normal) CoT, as discussed also briefly in the limitations section. With our setup it is easy to control how much \"latent CoT\" happens by varying the number of recurrences $r$. Even on problems that do not require a lot of reasoning, we can still easily force a large value of r, and we do so in many parts of the paper. \n\nHowever, for models that should use verbal CoT, it is very hard to control the exact number of tokens that will be used for a given problem. A naive approach might involve simply suppressing truncation and/or forcing truncation and such an alteration to the natural generation process could have significant effects on output quality if CoT is cut short before the model would naturally chose to produce a final answer. \n\nAs a concrete example, at the model scale we consider, we cannot really get the Olmo-7b models, or our non-recurrent baseline model to use chain of thought effectively, regardless of the desired output length. We considered prefilling these models with a CoT, but this would skew benchmark results to the model the CoT is extracted from. We can compare to CoT prompting on frontier open source models, such as Qwen 3, but those models are trained not only with 1-2 orders of magnitude more pretraining compute, but also extensively postrained to boost their CoT capabilities. It is unclear to us how to best elicit a clear comparison with verbalized CoT at the model scale we consider. We’d be happy for any pointers. \n\n---\n&nbsp;\n\n\n> Could you provide a more direct comparison of the inference FLOPs\n\nYes, we’ll add this as a column in e.g. Table 1. For now, you can consider Fig.1a (top x-axis) a ballpark estimate - although we note that actual inference costs are (in most scenarios) memory bound at generation time, which advantages the recurrent-depth model, which is using much less space to store its parameter, and less communication bandwidth between workers than a FLOP-equivalent fixed depth model (which is using every parameter only once per generated token). This difference is especially prominent when using cache sharing as described in section 5. \n\n---\n&nbsp;\n\n\n> How sensitive is the final model's performance to the choice of k (Truncated Backpropagation)?\n\nWe unfortunately cannot provide an answer at the 3.B scale. For the small-scale models (132M parameters trained for 10B tokens) we trained, we find that increasing $k$ starting from $k=1$ continuously improves the final perplexity when training over a fixed token budget, at the cost of additional memory, although the effects level off after $k=8$, with $k=16$ reaching the same perplexity. As such we chose to use $k=8$ for the pretraining run. We’d be happy to provide these small-scale ablations in the appendix. \n\n---\n&nbsp;\n\n\nOverall, thank you for your feedback. We’re glad that you found our submission interesting! We hope findings and our “data point” of a large-scale recurrent model provide new information to the community that informs future efforts in this direction. \n\nOur goal with this work was to build upon conceptual findings on recurrence in transformers, and propose a scalable architecture and training setup that we could train as a large language model. We're pretty excited that the final model is indeed able to make use of more compute on reasoning-heavy tasks, and we hope to have accurately presented our findings and the limitations thereof in our submission. \n\nPlease feel free to let us know if there are any follow-up comments or other questions that have come up since!"}, {"review_id": "Mjg4Fgsg5G", "reviewer": "Reviewer_6TRY", "summary": "This paper designs a recurrent transformer architecture and evaluates it on language modeling tasks. The model is trained with a varying number of recurrences. This enables adaptive test-time compute by applying the recurrent block a varying number of times. The authors compare to a nonrecurrent model of the same architecture. The recurrent model with high test time compute performs slightly better. In addition, the authors provide an analysis of the latent space trajectories. \"Overall, we believe this may be the largest language model training run to completion in terms of number of devices used in parallel on an AMD cluster, as of time of writing\"", "questions": "Q1. Is it fair to compare the recurrent model, when evaluated at R=32, to a 32B parameter model in terms of training Flops, or 50B for test flops? Or are there other metrics that are useful for comparing the model to baselines \"of equal ability\" ? Maybe it would be useful to show the number of flops for each model in tables such as Table 1 and Table 4.\n\nQ2. Section 5: the KL divergence between two successive steps is used for adaptive compute. What exactly is the KL divergence between? The latent state? Figure 4 is really confusing... what is Continuous CoT? what is Default? These labels are never described.\n\nQ3. It's unclear to me whether there's any insight in the analysis in section 6. Can the same analysis be applied to nonrecurrent models, but looking at how the latent state between layers evolves over time? Have such analyses been done, and do the results here differ? The paper does mention that similar multi-dimensional orbits have been observed in fixed-depth transformers for arithmetic tasks.", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The main strength of this paper is that the authors train a really large model, with a novel architecture to see how good the results are. \nThe biggest value in this paper is Table 4, specifically the first three rows of it. Here, we can see how good the recurrent model is compared to the non-recurrent baseline. This is very scientifically useful. \n\nThe main weakness of the paper is that the \"message\" of the paper is very confused. The authors are trying to argue that the recurrent approach could have advantages, but the results of this large scale experiment seem to indicate that the current recurrent model approach is not worth. I think the paper would be much stronger with a clearer message. It feels like the authors threw a lot of random things in the paper to try to make it better, but these all feel like distractions compared to the main point of this paper, which is experimenting with a huge recurrent model training run. \n\nI think of this paper as a \"negative result\" for recurrent architectures, that is still worth being published. It's a huge experiment and this work should clearly be published. However, I think the framing of the paper obfuscates the most valuable lessons from this paper. I think the authors probably threw in a lot of other analyses to make the paper seem stronger, but I would rather have it more clearly present the negative result to make it easier for others to build on it.\n\nTable 4: this table shows that compared to a fixed-depth baseline, the recurrent model only performs a bit better when using R=32 recurrences. The paper states that the recurrent model \"consumes flops close to what a 32B parameter transformer would consume during pretraining\" and the R=32 test time compute is comparable in flops to a 50B parameter fixed depth. So this approach seems really not worth it — almost 10x parameters to the baseline, and hardly higher in performance.\n\nThe remaining comparisons, such as table 1, I don't find to be very useful, given how many flops the model consumes in comparison to the other models.\n\n## Strengths\n- The paper trains a really large model. \"Overall, we believe this may be the largest language model training run to completion in terms of number of devices used in parallel on an AMD cluster, as of time of writing\" — wow!\n- The recurrent model seems well-designed and is described well in the paper.\n- The authors provide a ton of various analyses of the model, and a thorough appendix with their training methodology. \n\n## Weaknesses\n- Many of the figures are far too small to read when the paper is printed out. (4, 5, 6, and many in the appendix) — need to make sure the axes and labels can be read.\n- Many of the sections of the paper are not very enlightening. Table 4 is the main payday. \n  - Section 4.4 is just about ensembling to improve metrics.\n  - Section 6 tries to analyze the mechanisms in the recurrent model, but it's unclear whether the analysis uncovers anything that's not already present in fixed-depth models (see questions section)\n  - Figure 4 is very poorly explained (see questions section)", "was_revised": true, "final_justification": "I would like to give this paper a higher score, because the model trained is really impressive and interesting, but I can't help but still think that the results are presented a bit misleadingly. That said, I think it should be accepted regardless.", "rebuttal": "Thank you for your extensive feedback and detailed reading of our paper. We’re glad that you consider the result worth being published. We have tried hard to pack as much scientific value into our paper as we can, and to let the results stand for themselves. If future research ultimately concludes that our paper is simply evidence that the benefits of recurrent depth are not worth their costs, then we will still consider our endeavor, and this manuscript, a success.\n\n---\n&nbsp;\n\n> I think of this paper as a \"negative result\" for recurrent architectures, that is still worth being published.\n\nWhile we don’t want to disabuse this reading of the paper, and hope that we have presented all results as neutrally as possible, we do want to argue that Table 4 shows (to us) the key wins of recurrent depth in our study and where to expect them in practice. Namely, in Table 4, the recurrent-depth model shows drastic gains in reasoning performance on GSM8k (a factor of 5x over the non-recurrent baseline!) and reasonable gains on deducing the correct answer on HellaSwag (37% baseline to 48% recurrent model). However, it is true that little gain is realized on knowledge-heavy tasks such as OBQA, PiQA or SciQ over the non-recurrent baseline. \n\nWe think this accurately represents the potential of recurrent depth models for “reasoning” (see also the discussion in the motivation section). It is indeed a suboptimal use of compute to try and use recurrence to improve performance if the primary focus is on tasks requiring retrieval of associative relations about the world and/or deductions from the large amount of knowledge stored within the trainable parameters of the model. However, both our intuitions and our empirical results suggest that in domains where, given the same knowledge, more _computation_ can be expected to yield more accurate results, recurrence can unlock notable improvements. This is also shown in Fig. 1b. On a number of tasks such as SciQ, running the recurrence for 1 or 4 steps provides most of the gains, likely meaning that the model either retrieves the relevant fact, or does not know it. On the other hand, for tasks like HumanEval or Mastermind (the reasoning game) while the model performs at random chance with limited compute, additional recurrent computation again provides substantial improvements. \n\n---\n&nbsp;\n\n> figures are far too small to read when the paper is printed out\n\nThank you for bringing this to our attention! We will certainly address this in the updated version of the paper.\n\n---\n&nbsp;\n\n> Section 4.4 is just about ensembling\n\nJust to be safe, we want to highlight that the approach in Section 4 is using offline checkpoint merging, also sometimes described as “latest weight averaging” (e.g. Kaddour et al. 2022), not an ensemble, so the final result is a single model that can be used for downstream applications as normal, as such we consider this part of our pretraining pipeline.\n\n---\n&nbsp;\n\nRegarding your questions:\n\n> Q1. Is it fair to compare the recurrent model, when evaluated at R=32, to a 32B parameter model in terms of training Flops, or 50B for test flops?\n\nWe agree that a straightforward comparison is not so easy, so we have opted to just present the data regarding training flops and inference flops. We also note that the model is generally faster to train/infer than flop-equivalent fixed-depth transformers, given that, due to its smaller size, costs for communication during training, and model memory during inference, are both improved.  We also note that the comparison would have to be to a 32B parameter model trained for the same number of tokens. Maybe even better, it would be a comparison between two models trained on the same compute cluster for the same wallclock time with similar engineering. In our table we can only point to the Pythia series, trained on fewer tokens, and the Olmo series trained on significantly more tokens.  \n\nWe also want to re-iterate that, for the first (and the only) recurrent model trained at this scale as a large language model, the results are encouraging in comparison to these open source models building on training recipes for fixed-depth models that are by now much more established. \n\n---\n&nbsp;\n\n> it would be useful to show the number of flops for each model in tables such as Table 1 and Table 4.\n\nThis is a good point, we’ll add pretraining and inference flops to these tables. \n\n---\n&nbsp;\n\n> Q2. Section 5: the KL divergence between two successive steps is used for adaptive compute. What exactly is the KL divergence between?\n\nThe metric described in section 5 is the KL divergence between the probability distribution over the next token at step r-1 to the probability distribution over the next token at r. This is just one option on how to exit that we considered natural when writing the paper, and using, for example, the Euclidean distance between the latent states is also a successful strategy. We’ll clarify this part in our revised version and can provide more detailed variants of all of the experiments on decoding strategies in the appendix.\n\n---\n\n> what is Continuous CoT?\n\nContinuous CoT is an inference variant described in Appendix Section D.6, namely warm-starting the latent state at each generated token with the last latent state from the previous token. We moved this section into the appendix at the last minute. We’ll remove this variant from Fig.4 in the main body, and show this only in the appendix in the future. Sorry for the confusion!.\n\n---\n&nbsp;\n\n> Q3. It's unclear to me whether there's any insight in the analysis in section 6. Can the same analysis be applied to nonrecurrent models,\n\nActually, this kind of analysis is already a bit special to the recurrent model. Due to the use of a single non-linear operator, the recurrent block, all latent states naturally lie in the same “space” and can be analyzed as an iterative series. For normal models, there is some pressure due to the residual connections to keep representations similar from layer to layer, but there is no requirement. Every layer in a transformer is initialized randomly and at minimum, arbitrarily rotates the features of the model. There is no requirement for e.g. the n-th feature of the latent state to correspond to the same feature across layers of the model like it does for the recurrent-depth model. Only in a few specialized examples, like a model trained only for modular addition (as referenced) were similar patterns like the orbits shown. \n\n---\n&nbsp;\n\nOverall, we thank you again for your detailed feedback and believe it has helped improve the presentation and outlook of our paper. In light of our results showcasing how certain “computation” centric tasks like math, logical reasoning, and code generation benefit from the application of recurrent compute, we think that our research does demonstrate the promise of the recurrent approach and warrants further exploration and development effort. \n\nHowever, we share the reviewer’s perspective that our results also suggest that for some usecases where the raw capacity for knowledge storage within the parameters of the model is the primary determiner of performance, a recurrent approach might be suboptimal. Ultimately, we’re optimistic that our study of recurrent depth large language models at scale will be a valuable addition to the literature for both researchers and practitioners interested in pushing the limits of efficiency and capability across a broad spectrum of tasks and deployment usecases.\n\n\n\n\n&nbsp;\n\n&nbsp;\n\n\n*References*:\n\nKaddour, Jean. \"Stop wasting my time! saving days of imagenet and bert training with latest weight averaging.\" arXiv preprint arXiv:2209.14981 (2022)."}]

Author rebuttal:

Meta-review: This paper proposes a recurrent/looped architecture that can “reason in latent space” for inference-time scaling. A prototype model with 3.5B parameters is pre-trained on 800B tokens with this architecture and is shown to scale at inference-time for math and coding benchmarks. Overall all reviewers were very positive about the paper. They appreciated the novelty, clarity of presentation, comprehensive experimentation and inclusion of additional features like kv-sharing and self-speculative decoding. There were questions about the motivation of such an architecture in comparison to CoT and whether it is worth training such a model if deeper models are better. The authors provided reasonable responses to these, while acknowledging that this approach may not be compute optimal or ideal for knowledge intensive tasks. Overall this is a solid paper that concretely demonstrates the benefits of looped architecture and latent reasoning, topics that have both recently garnered interest.
Decision: Accept (spotlight)