PonderLM: Pretraining Language Models to Ponder in Continuous Space
Authors: Boyi Zeng
Shixiang Song
Siyuan Huang
Yixuan Wang
He Li
Ziwei He
Xinbing Wang
Zhiyu li
Zhouhan Lin
Venue: ICLR 2026 Poster (2026)
Keywords: language modeling
pondering language models
pretraining
continuous embedding space
Primary area: foundation or frontier models, including LLMs
TL;DR: We pretrain language models to ponder within a continuous embedding space.

Abstract: Humans ponder before articulating complex sentence elements, enabling deeper cognitive processing through focused effort.
In this work, we introduce this pondering process into language models by repeatedly invoking the forward process within a single token generation step. During pondering, instead of generating an actual token sampled from the prediction distribution, the model ponders by yielding a weighted sum of all token embeddings according to the predicted token distribution. The generated embedding is then fed back as input for another forward pass. We show that the model can learn to ponder in this way through self-supervised learning, without any human annotations.
Experiments across three widely used open-source architectures—GPT-2, Pythia, and LLaMA—and extensive downstream task evaluations demonstrate the effectiveness and generality of our method. On 9 downstream benchmarks, our pondering-enhanced Pythia models significantly outperform the official Pythia models. Notably, our PonderPythia models demonstrate remarkable effectiveness: PonderPythia-2.8B surpasses Pythia-6.9B and rivals Pythia-12B, while our PonderPythia-1B matches TinyLlama-1.1B, a model trained on 10 times more data.

Published as a conference paper at ICLR 2026
PONDERLM: PRETRAINING LANGUAGE MODELS TO PONDER IN CONTINUOUS SPACE
Boyi Zeng', Shixiang Song Siyuan Huang', Yixuan Wang**, He Li Ziwei He Xinbing Wang", Zhiyu Li Zhouhan Lin 1.23x
'LUMIA Lab, Shanghai Jiao Tong University,
Institute for Advanced Algorithms Research, Shanghai, "Shanghai Innovation Institute, 'Shanghai Jiao Tong University boyizenges jtu. enu.en "lin. Shouhannggmail com
ABSTRACT
Humans ponder before articulating complex sentence elements, enabling deeper cognitive processing through focused effort. In this work, we introduce this pon- dering process into language models by repeatedly invoking the forward process within single token generation step. During pondering, instead of generating an actual token sampled from the prediction distribution, the model ponders by yielding weighted sum of all token embeddings according to the predicted to- ken distribution. The generated embedding then fed back as input for another forward pass. We show that the model can learn to ponder in this way through self-supervised learning, without any human annotations. Experiments across three widely used open-source architecture--GPP3 Pythia, and LLaMA-and extensive downstream task evaluations demonstrate the effectiveness and gener- ality of our method. On 9 downstream benchmarks, our sonddering-ehhaneee Pythia models significantly outperform the official Pythia models. Notably, our PonderPythia models demonstrate remarkable effectiveness: PonderPythia-2..6 surpasses Pythia-6.9B and rivals Pythia-125 while PonderPythia- 1B matches TinyLlama- IB,,, a model trained on 10 times more data.'
Pythia
Ponde/Pycnia
2.25
Penderfythe-15 Pondurfydha-18
Vonkls
Pondee
Pylha
2.20
Ponde//ychia
222
0 2.10
2.65
2.1
2,05
2.0
vaining
2.60
1.9
200M
500M
70
608
1038
2038
3028
300
200
300
400
#Parameters Dog scale)
takers
(EFLOPs)
Figure 1: Scaling curves comparing PonderPythia with the official Pythia suite on the 300B Pile. Our 2.55B model matches the loss of Pythia-6.9B with 63% fewer parameters (left), while our PonderPythia-1I model reaches the baseline's final performance with 59% less training data (mid- dle). Furthermore, for the same computational budget (FLOPs), PonderPythia consistently achieves a lower loss than the baseline (right).
INTRODUCTION
In the pursuit improving model performance, scaling model parameters and data sizes has long been the most widely adopted and accepted approach (Kaplan et al., 2020; Brown et al.. 2020; Liu et al., 2024). However, this approach faces several bottlenecks, including the exhaustion of high- quality data (Villalobos et al., 2022: Muennighoff et al., 2023), the observed saturation in scaling laws (Hackenburg et al., 2025; Hoffmann et al., 2022a) and substantial communication overhead in
"Zhouhan Lin is the corresponding author.
'The code is available https:tpp..............tt
com/ LUMI httpb//.woo/..laaaaarrrrrroorrrr

---

Published as a conference paper at ICLR 2026
I1.7
Predicted Probibility
class PonderLanguageMooeel Module):
def _iinit_ (self, 1m, v, h, a):
self m 1m # language model velfvvocab.size
Pondering Language Model
self hidden dim
self londeron steps =
Pundesing Emnhedding
self selfeennndding mn. Parameter(torchh randn(v, h), requires.grad-Trus
0.11
0.2
00.5
Q.1
0.3
Word Embed
xN
def forvard input topent))
0.0
0.1
0.2
input_embedding
self www.eeddnggiinnntttteeeeee #Iterative pondering for
Pondering Language Model
range pondering.atepa))
?
Pondering Ersiodddng
predicted.prob aelf.1m(
imput_emboddinng)
pondering.embedding torch.matmul
0.1
0.3
0.3
Word Embed
Vocab Size
(predicted prob, self.
embedding)
0.2
0.2
0.1
0.2
0.3
geput.eemeddingg = input genbedinnn
Hidden
pondering.embedding
Pondering Language Model
Size
#Final forward pass
final prob self. 1m (input embedding)
Input
return final prob
Embedding
Figure 2: Overview of the PonderLM. Given input token embeddings, the base LM produces probability distribution over the vocabulary, which is used to compute continuous "pondering embedding" via a weighted sum token embeddings. This embedding then added residually to the original input embeddings and fed back into the LMM By repeating this process for steps within a single token prediction, the model iteratively refines its output distributions. The pscudocode on the right illustrates the implementation details.
distributed pre-training that grows super-linearly with model size (Narayanan et al., 2021; Pati al., 2023; Li et al., 2024).
On the other hand, we look at humans, the growth of human capabilities does not stem from simply increasing the number of neurons in the brain. Instead, when faced with complex problems, humans often enhance their problem-solving abilities by repeatedly pondering, engaging in deep cognitive processing before articulating their thoughts.
Analogously, large language models, the most relevant research direction test-time scaling. In particular, following the advancements in ol and RI (Jaech et al., 2024; DeepSeek-Al et al., 2025), generating long chains of thought (CoT) has emerged as the mainstream approach for scaling test- time computation. However, bosedd methods also exhibit several drawbacks: they often require curated human-annotated datasets (Allen-Zhu & Li, 2023) and carefully designed reinforcement learning algorithms (Pang et al., 2025). Moreover, small models rarely benefit from CoT (Li al., 2023), and the upper bound of performance remains constrained by the base pretrained model (Yue al.. 2025). Additionally, current language models employing CoT are still confined to discrete language spaces with fixed vocabularies, which, according to recent studies (Fedorenko et al., 2024; Hao al., 2024; Pfau et al., 2024), are primarily optimized for communication rather than for internal computational thinking.
To overcome these challenges and inspired by human pondering, introduce the Pondering Lan- guage Model (PonderL.M), which relies solely self-supervised learning. PonderL.Ms can be naturally learned through pretraining on large-scale general corpora, without the need for human- annotated datasets reinforcement learning.
During pondering, instead of generating a discrete token sampled from the prediction distribution, the model produces weighted sum of all token embeddings based on the predicted probabilities. This generated embedding is then fed back into the language model, allowing it to iteratively refine its predictions. As the weighted embedding is continuous, PonderLMs overcome the expressive lim- itations of discrete token vocabularies and enable fully differentiable, end-to-end pretraining via gra- dient descent. Furthermore, by performing computations per parameter, Ponderi.M achieve
2

---

Published as a conference paper at ICLR 2026
The
longest
river
in
the
world
is
the
Nile
Pondering,
Pondering;
Pondering;
Pondering,
Ponderings
Pondering.
Pondering;
Pondering
LM Output
LM Output
LM Output
LM Output
LM
river
0.60
LM
Amazon
0.41
LM
Amazon
0.30
LM
Nile 0.54
the
rivers
0.36
+
river
0.35
4
river
0.33
4
Amazon
0.23
Nile
0.04
Nile
0.24
Nile
0.27
Yang
0.23
pondering step1
pondering step2
pondering step3
residual connection
Figure 3: The inference process of pondering language model, illustrated with actual example from PonderPythia-2.88B shows how the model dynamically corrects its predictions over several pondering steps to arrive the correct answer. Crucially, the intermediate probability distributions and their top candidate tokens offer potentially interpretable view into the model's inference.
higher parameter knowledge density (Allen-Zhu & Li, 2024), potentially reducing communication costs at scale.
Experimentally, by introducing the pondering process during pretraining, our Ponder GPT-2, LLaMA, and Pythia models achieve pretraining perplexity comparable to that of vanilla models with twice as many parameters. Furthermore, PonderPythia models significantly outperform the official Pythia models across nine popular downstream tasks, PonderPythia-2..0 6.9B, and PonderPythia-18 data. Notably, increasing the number of pondering steps consistently enhances model performance, underscoring the substantial potential of this approach.
surpasses Pythia- which is trained on 10 times more
comparable to Jamaa_ Jla.amm.m..B.....
Moreover, our method is orthogonal traditional scaling strategies, including parameter scaling and inference-time scaling via CoT, and thus can complement existing techniques, potentially intro- ducing third scaling axis enhance model performance.
PONDERING LANGUAGE MODEL
In this section, we introduce our proposed PonderLM (Figure 2). which integrates pondering mechanism into language models via pretraining. Given that pretraining fundamentally constitutes a language modeling task, we briefly review this task before detailing our proposed model.
Language Modeling. Given sequence of tokens X [x Z2...., ......,,,ll, the primary objective of language modeling typically to maximize the likelihood predicting each token based on its preceding tokens. Formally, this is expressed through the joint probability factorization:
P2 2n.) = II P(x. xZe)
(1)
Current language models first map tokens to input embeddings E" jel, eg.... ed). where each embedding eạ E Rdd is selected from a vocabulary embedding matrix V ee, e2 e ejv with vocabulary size |V] and hidden dimension d. The language model then generates output probabilities P for predicting the next token at each position:
P= LM(EE),
PE PE R
The cross-entropy loss is computed directly from these predicted probabilities P to pretrain the language model."
Pondering Mechanism. In proposed method, instead of directly using the predicted output probabilities P to compute the cross-entropy loss, we utilize these probabilities as weights to sum embeddings of all candidate tokens, forming what we call "pondering embedding". Given the
3

---

Published as a conference paper at ICLR 2026
probability distribution p E RIV at each position, the pondering embedding t is:
P.e., Ps P R
(3)
For computational efficiency, pondering embeddings for all positions can be calculated simultane- ously via matrix multiplication::
T= TER"X
(4)
Through these pondering embeddings, we effectively map predicted probabilities back into the em- bedding space, preserving embedding information from all possible candidate tokens. To maintain the information from the original input embeddings, we integrate the pondering embeddings using residual connection:
E E + T = [ee tı, e + . e + tx]
(5)
We then feed the updated emheddings E back into the same language model obtain refined output probabilities:
P = LM(E'), Pl E R"X
(6)
After obtaining Pl, we can iteratively repeat the previous process to achieve multi-step pondering. Specifically, given predefined number of pondering sleps s², we iteratively compute new ponder- ing embeddings and integrate them with the original input embeddings using residual connections, feeding the result back into the same language model until s steps are reached:
E
jei. 2
p LM(E))
==="VV
n
E
E
T
[e
t E + tt.....,.,.
0% 1+ thh p = LM(E"), ==pp'v
E'= E
Σ T +
..... ea tt
t.
p-
= LM()))
This iterative pondering mechanism progressively refines the model's predictions. Finally, we can use the refined output probabilities P after pondering steps to compute the cross-entropy loss and optimize the language model to perform s-step pondering.
EXPERIMENTS
Our experiments consist of 3 parts. First, we validate the scaling curves of pondering models on widely used GPT-2 and LLaMA architectures. Second, we perform large-scale pretraining of Pon- derPythia models the Pile dataset and compare their scaling curves and language modeling pabilities with those of the official Pythia suite (Biderman et al., 2023). Third, we evaluate the downstream task performance of PonderPythia models, including popular general tasks and instruction-followimnn task, and compare the results with official Pythia, OPT (Zhang et al., 2022), Bloom (Le Scao al., 2023) and TinyLLaMA (Zhang al., 2024).
3.1 SMALL SCALE VALIDATION ON GPT-2 AND LLAMA
We apply our proposed method to two popular Transformer architectures, GPT-2 and LLaMA, to investigate its general applicability and effectiveness.
Experimental Settings. We trained both models from 405M to 1.4B parameters from scratch on subset of the Pile dataset using the same tokenizer. The amount of training data for each model follows the Chinchilla (Hoffmann et al., 2022b) scaling laws, with a fixed context length of 2048. Detailed configurations are specified Appendix B.
IIn practice, dering emhedding, redueing complexity from Onn|V]d) O(n With K 100 < [V], this does not degrade LM performance and makes the matrix multiplication overhead negligible within overall LM computations. We have done the ablation study in Section 4.4.
only the top-K K tokens with highest probabilities each position to compute the pon-
Unless otherwise specified, we 3 for subsequent experiments.
4

---

Published as a conference paper at ICLR 2026
Parplexity
Pythasd. Pyth Panderfythi
Perplexity
2.55
1.84
Perplexity
0,70
4.42
0.42
Perplexitt
20-
14.06
30-
0.49
9.0
OpenAl,
Figure 5: PonderPythia demonstrates substantial perplexity improvements over the official Pythia across all model sizes and datasets. Our PonderPythia-22 even surpasses the official Pythia-6.9B.
Results. As shown in Figure 4, the pondering mechanism significantly improves the perfor- mance of GPT-2 and LLaMA models across the 405M to 1.4B parameter range. For instance, an 834M pondering model achieves loss compa- rable to vanilla model trained with over 2x the parameter-token product.
2.6
fitted
GPT
2.5
PonderGPT
LLaMA
2.4
PonderLLaMA
9
014
2.26x
2.2
3.2 LARGE-SCALE PRETRAINING ON PILE
2.1
We further validate the effectiveness of our pondering method by conducting large-scale pretraining experiments on the entire Pile dataset (300B tokens) (Gao et al., 2020). We train a new model, named PonderPythia, from scratch using exactly the same architectural components (parallel attention and MLP layers, rotary embeddings with 1/4 head dimensions), same tokenizer and training hyperparameters (optimizer settings, learning rate schedule, batch size, and context length) as the original Pythia models. We then compare PonderPythia models' scaling curves and language modeling capabilities with the official Pythia model suite.
405M*78
834M115B
1.4B-26B
Parameters
Figure 4: Scaling curves of GPT3 LLaMA and their corresponding pondering models.
3.2. SCALING CURVES
We plot the scaling curves the PonderPythia and official Pythia models with respect parameter size, training tokens, and training FLOPs. As shown in Figure (left), the fitted curves indicate that a 2.55B- --5a-marreettrr PonderPythia model achieves validation loss comparable to that of the 6.9a-parrmeter official Pythia model, while requiring 63% fewer parameters. In Figure I (middle), fitted curves show that PonderPythia achieves performance comparable to the official Pythia model while using 59% fewer training tokens. In Figure (right), report the language modeling loss of vanilla Pythia-70MM 7 and PonderPythia- 70M under the same computational budget during pretraining. It can be observed that PonderPythia-70M consistently outperforms vanilla Pythia-70M when trained with the same number of FLOPs.
3.2.2 LANGUAGE MODELING ABILITY EVALUATION
We measure the perplexity several language modeling datasets to reflect general language model- ing capabilities. Specifically, we report perplexity scores on the Pile validation set, Wikitext, and the Lambada dataset (both OpenAl and standard versions), detailed in Figure 5. The results demonstrate significant perplexity improvements across datasets and model sizes. Notably, the perplexity achieved by the PonderPythia-2.8B model is even better than that of the official Pythia-6.9B model.
3.3 DOWNSTREAM TASKS EVALUATION
We previously pretrained PonderPythia models for downstream task evaluations.
"To match the training FLOPPs of vanilla Pythia-70M with PonderPythia- 70M, we trained the vanilla model
for epochs.

---

Published as a conference paper at ICLR 2026
Table 1: Five-shot and zero-shot evaluations on downstream NLP tasks. All pretrained model weights used for comparison are obtained from their official repositories. Aace is compared to the official Pythia models. Models in italics are excluded from bolding, as they use significantly larger training data or parameters.
Lambada ARC Lambada ARC Standard
Wino Grande
Hella Swag
Avg acc
Model (#training tokens)
PIQA
SciQ RACE
OpenAl
CC
Aacc
5-shot
Pythia-410M (300B)
43.9
54.7
32.8
22.3
53.4
68.0
33.8
88.9
30.4
47.6
OPT-350M (300B)
38.3
45.4
32.1
20.5
53.0
65.8
31.9
85.7
29.5
44.7
Bloom-560M (366B)
29.4
50.2
29.7
21.9
52.7
64.2
31.4
88.0
30.0
44.2
PonderPythia-4400 (3008)
48.9
58.7
43.7
26.1
54.0
70.5
37.3
91.0
32.4
/43.8
Pythia- (300B)
48.3
35.8
25.4
52.8
71.3
37.7
91.6
31.7
50.4
OPT-1.3B (300B)
54.0
60.4
49.0
26.9
56.9
72.4
38.5
91.8
35.4
52.7
Bloom- 1.IB (366B)
36.3
54.9
37.4
24.9
53.4
67.6
34.8
88.7
33.0
47.9
Tinyllama-I. 1.1B (31)
53.8
64.8
45.0
31.1
59.4
73.8
44.9
94.0
36.4
55.9
PonderPythia-I19 (300B)
57.7
63.2
52.5
28.6
58.6
73.3
41.9
93.4
36.3
56.2 / +5.8
Pythia- 1.4B (300B)
54.5
63.1
44.5
28.8
57.1
71.0
40.5
92.4
34.6
54.1
Bloom-1 1.7B (366B)
42.5
58.8
41.5
26.2
57.7
68.7
37.6
91.9
33.5
50,9
PonderPythia-i (300B)
59.2
67.5
49.9
32.4
60.4
73.5
44.2
94.3
37.1
57.6/
Pythia-2.81 (300B)
59.0
67.0
50.7
31.0
61.1
74.4
45.3
93.7
57.6
OPT-2.7B (300B)
60.2
64.7
55.0
29.8
62.2
75.1
46.1
93.0
37.5
58.2
Bloom 3B (366B)
46.2
63.8
47.1
31.7
57.8
70.8
41.4
93.4
34.6
54.1
Pythia- 6. 6.9B (300B)
62.5
69.6
54.8
35.6
62.9
76.6
48.0
94.6
36.7
60.1
Pythia 12B (300B)
66.5
71.0
57.1
36.0
64.8
76.5
50.7
94.9
37.4
61.7
PonderPythia-2.55 (300B)
64.2
70.6
58.7
35.8
65.3
76.7
49.0
94.3
61.5 +3.9
O-shot
Pythia-410M (3008)
51.4
52.2
36.4
21.4
53.8
66.9
33.7
81.5
30.9
47.6
OPT-350M (300B)
45.2
44.0
35.8
20.7
52.3
64.5
32.0
44.4
Bloom-560M (3668)
34.3
47.5
33.3
22.4
51.5
63.8
31.5
80.3
30.5
43.9
PonderPythia-410M (300B)
56,9
51.9
45.3
22.6
56.0
68.7
37.0
81.4
33.8
50.4/+2.8
Pythia- IB (300B)
55.9
56.8
42.0
24.2
52.5
70.5
37.7
83.3
32.7
50.6
OPT-1.3B (300B)
57.9
57.1
52.5
23.4
59.7
71.8
41.6
84.3
34.3
53.6
Bloom-1 1.IB (366B)
42.6
51.5
42.9
23.6
54.9
67.3
34.5
83.6
32.6
48.2
Tinyllama- I.IB (3T)
58.8
60.3
49.3
28.0
59.0
73.3
45.0
88.9
36.4
55.4
MonderPythia-le (300B)
62.3
60.5
51.9
27.0
56.5
72.2
41.8
87.4
35.4
55.0 +4.4
Pythia 1.4B (300B)
61.6
60.4
49.7
25.9
57.5
70.8
40.4
86.4
54.1
Bloom-I 1.7B (3668)
46.2
56.4
44.5
23.7
56.8
68.5
37.5
85.0
33.2
50,2
PonderPythia-11.4l (300B)
65.2
62.0
53.8
27.0
60.1
72.6
44.0
89.0
35.2
56.5 / 42.4
Pythia-2.88 (300B)
64.6
64.4
54.3
29.5
60.2
73.8
45.4
88.5
34.9
57.3
OPT-2.7B (300B)
63.5
60.8
56.0
26.8
61.2
73.8
45.9
85.8
36.2
56.7
Bloom-38 3B (3668)
51.7
59.4
50.9
28.0
58.7
70.8
41.4
88.8
35.2
53.9
Pythia-6. 9B (300B)
67.2
67.3
$5.9
31.4
61.0
75.2
48.1
89.3
36.9
59.1
Pythia 12B (300B)
70.4
70.6
58.9
31.7
61.0
75.2
50.4
89.3
36,9
60.5
PonderPy ManderPythia-22.. thia-2.8B (300B)
68.9
66.5
60.8
32.5
63.6
75.0
48.6
91.0
36.5
60.4 43.1
3.3. GENERAL DOWNSTREAM TASKS
We consider various widely-used henchmarks, including the tasks originally used by Pythia (L.AM- BADA (Paperno al., 2016). PIQA (Bisk et al., 2020), WinoGrande (Sakaguchi et al., 2021), ARC- and ARC-C (Clark et al., 2018), SeiQ (Welbl et al., 2017). We also include HellaSwag (Zellers al., 2019) for commonsense reasoning and RACE (Lai al., 2017) for reading comprehension.
We evaluate both 0-shot and 5-shot learning performance using the LM evaluation hamnsss (Gao al., 2023). Detailed results shown in Table 1. Across all evaluated model sizes, PonderPythia consistently and significantly outperforms the official Pythia models, as well as comparable OPT and Bloom models. Remarkably, with only 1/10 of the training data (300B tokens) and fewer parameters,

---

Published as a conference paper at ICLR 2026
Wrltin
Whiting
Humanties
Roleply
Humamtes
Roleplay
STEM
Reasoning
STEM
Reasoning
0.5 1/1.
L8225335
Extraction
Math
Exraction
Mash
Coding
Coding
Pythaa 18, avg scorem1.0p 109
Pytha--144 5oorem2.2
PondertyPiie- 185, screen2.22
Pondertyttee- 4B, so0re22.75
abilities evaluated on MT-Bench. PonderPythiar IB and 1.4B con- sistently outperform their corresponding official Pythia models across all subtasks.
Figure 6:
PonderPythia-11 IB achieves results comparable uses a more advanced I.LaMA architecture and 3T tokens. Furthermore, PonderPythia-22.5 not only surpasses Pythia-6.9B but also achieves performance nearly par with Pythia-12B.
surpassing. TinyLlama- 1.11 which
3.3.2 INSTRUCTTIN-FFOLLOWWY ABILITY EVALUATION
To assess the enstruction-oolloiing capability of our model, we further fine-tuned PonderPythia- B and 1.4B, as well as the corresponding official Pythia models, on the Alpaca dataset using the same settings (Taori et al., 2023). The fine-tuned models were evaluated with MT-Bench (Zheng al., 2023), a popular multi-turn question benchmark. The experimental results are shown in Figure As illustrated, both PonderPythia-11e and 1.4B consistently outperform their official Pythia counterparts across all subtasks, achieving average improvements of 0.33 and 0.55, respectively."
ABLATION STUDY
In this section, we conduct a series of ablation studies to dissect the components of our proposed pondering language model. We 70M parameter Pythia model trained 30B-token subset of the Pile dataset as our primary testbed to ensure controlled and efficient experimentation.
4.1 COMPARISON WITH RELATED BASELINES
First, we compare our pondering mechanism against several baselines designed to increase compu- tation per token. The goal is to verify that the performance gains come from our specific approach rather than merely from additional computation. The baselines include:
Replacing the pondering embedding with the last hidden state from the previous step. similar Coconut (Hao et al., 2024).
Last Hidden State as Embedding:
Projected Hidden State: A variant of the above where a linear projector (like LLaVA (Liu et al.,
2023)) maps the last hidden state to the embedding space.
Looped Transformer (Giannou et al., 2023; Saunshi et al.): Increases computation by iteratively
reusing the full stack of transformer layers.
Pause Token (Goyal al., 2023): A method that inserts special Iearnable "pause" token after each original token to encourage extra computation.
As shown in Table 2, our method significantly outperforms all baselines across every benchmark given the same number additional steps/loops/pauses. Notably, our model with only 3 pondering steps is more effective than baselines with 6 loops, steps pauses, confirming that the performance gains are attributable to our specific approach rather than just added computation.
'The marginal gains Coding and Math tasks may be attributed the limited coding and mathematical abilities of the Pythia models.

---

Published as a conference paper at ICLR 2026
Table Comparison on various benchmarks. PonderPythia-70M consistently outperforms all base- line methods, with the 3-step version already surpassing 6-step/loopppauses baselines.
Lambada OpenAI(4)
Lambada Standard({)
Avg Ace shot(1)
Avg Acc shou(1)
Model
Pile(L)
Wikitext(J)
PYthia-70M (baseline)
16.95
51.68
101.34
481.04
21.04
14.12
Models with additional steps/loops/pauses
Looped Pythia-70M (3 loops)
15.33
44.56
74.75
438.62
22.67
17.21
Pause Pythia-70M (3 pauses)
16.53
49.85
80.77
374.97
21.62
15.67
Last Hidden State steps)
15.64
45.60
78.11
479.85
22.11
15.44
+ Lincar Projector (3 steps)
15.64
46.10
81.94
453.78
21.62
17.69
PonderPythia (3 steps)
14.16
39.63
56.01
238.45
25.13
20.64
Models with additional steps/loopsppauses
Looped Pythia-70M (6 loops)
15.18
43.91
71.78
355.64
22.50
17.44
Pause Pythia-70M (6 pauses)
16.55
49.66
80.79
460.80
21.61
15,88
Last Hidden State steps)
15.30
44.48
78.87
393.03
22.46
18.18
Linear Projector (6 steps)
15.29
44.24
74.36
474.15
21.69
17.61
PonderPythia (6 steps)
13.56
37.57
49.15
196.67
25.58
21.71
4.2 IMPACT OF PONDERING STEPS
To further investigate the effect of pondering steps on model performance, we trained several 70M- parameter Pythia from scratch with different numbers of pondering steps: 0 (baseline), 1, 4, 5, and 10. The results in Figure (top) show that increasing pondering steps consistently reduces the language modeling loss on the Pile validation set, demonstrating the potential of our method.
4.3 TRAINING WITH RANDOMIZED PONDERING STEPS
In our main experiments, the number of ponder- ing steps is fixed during training and inference due source limit. To build more flexible model, we also experimented with randomiz- ing the number of pondering steps during train- ing. We trained a Pythia-700 model where the number of steps for each training batch was domly sampled from the range [1, 10].
2.80
9
2.70
2.60
10
Pondering Steps (0 Baseline)
2.76
2.70
2.64
This strategy yields single model that can operate with a variable number of steps at in- ference. As shown in Figure 7 (bottom), this model exhibits test-time scaling: its perfor- mance progressively improves as we increase the number of pondering steps, and thus the computational budget, at test time.
2.58
Inference Time Pondering Steps
Figure 7: (Top) Increasing the number of pon- dering steps consistently reduces the validation loss. (Bottom) Inference- cetrence-timeeeee scaling of a model trained with randomized pondering steps.
4.4 INFLUENCE OF TOP-K TOKEN SELECTION
To balance performance and efficiency, evaluated the effect of the hyperparameter K model perplexity. Increasing from 10 to 100 substantially reduced perplexity from 15.18 to 14.21. However, expanding the selection to the full vocabulary yielded only negligible improvement to 14.20. This indicates that tokens with ranks between 10 and 100 contribute substantially, while those beyond offer negligible benefit. We thus adopt K 100 in our main experiments.
RELATED WORK
Test- Time Compute Scaling, Scaling computation test time has proven effective for improving model performance without adding parameters (Snell et al., 2024), with existing methods mainly categorized into parallel and sequential scaling (Zeng et al., 2024; Muennighoff et al., 2025).
8

---

Published as a conference paper at ICLR 2026
Table 3: Comparison of PonderL.M .M with Related Methods.
Method
Training Data
Computation Space
Application Level
Training Method
CoT
CoT data
Explicit Token
Per question
RL/SFT
Lo
General corpus
Hidden State
Per token
Pretrain
Pause Tokens
General corpus
Fixed Token
Per token
Pretrain
Quiet-STaR
General corpus
Explicit Token
Per token
RL
Coconut
CoT data
Hidden State
Per question
SFT
PonderL.M
General corpus
Weight Sum of Embeddings
Per token
Pretrain
Parallel scaling generates multiple candidates simultaneously and selects the best via strategies like Best-of-N (BoN) (Cobbe et al., 2021; Sun et al., 2024; Gui et al., 2024; Amini et al., 2024; Sessa et al., 2024) or Majority Voting (Wang et al., 2022), but faces the challenge of reliably identifying the optimal candidate (Stroebl et al., 2024; Hassid et al., 2024).
Sequential scaling methods improve reasoning by iteratively refining a model's s output over multiple steps. This broad category includes foundational techniques like Chain of Thought (CoT) well as more recent iterative revision strategies (Wei et al., 2022; Nye et al., 2021; Huang et al., 2022; Min et al., 2024; Madaan et al., 2024; Wang et al., 2024b; Lee et al., 2025; Hou et al., 2025; Muennighoff et al., 2025; Li et al., 2025). State-of-the-art models also scale computation at test-time through extensive multi-step reasoning (OpenAI, 2024; DeepSeek-Al Al al., 2025; Comanici et al., 2025). However, these methods often depend on specialized datasets (Allen-Zil Milen-ZZu& & Li, 2023), require long context windows (Zhu al., 2025), or involve complex reinforcement learning (Pang et al., 2025). Our method is designed to overcome these limitations.
Latent Thinking in Language Models. Latent thinking refers to the intermediate computational processes that occur within language model's 's internal representations, separate from the explicit generation of text tokens (Yang et al., 2024; Biran et al., 2024). Prior work can be broadly catego- rized by the computational space in which this additional thinking occurs.
Discrete Token Space. One line of research elicits intermediate reasoning by manipulating discrete tokens. This includes predicting dedicated planning tokens (Wang et al., 2024a), using filler tokens to allocate more computation (Pfau et al., 2024), or inserting learnable "pause" tokens during train- ing encourage latent computation (Zhou et al., 2024; Goyal et al., 2023). More complex methods like Quiet-S use reinforcement learning to generate and then condense rationales at each token, embedding the reasoning process directly into generation (Zelikman et al., 2024).
Continuous Space. Another research direction explores reasoning directly within the model's con- tinuous hidden states. One prominent approach involves iteratively refining these states. This is achieved through recurrent structures that reuse hidden states over time (Dehghani et al.; Hutchins et al., 2022), by recycling model outputs back as inputs, a technique effective for reasoning tasks (Gi- annou et al., 2023; Yang et al., 2023; Saunshi et al.), or by iterating over model layers to refine in- termediate representations (Geiping et al., 2025). More recently, methods like Coconut (Hao al., 2024) train models to reason entirely in the latent space, while CoCoMix (Tack et al., 2025) extract the "concept" from the hidden states improve language modeling.
In contrast to these approaches that operate on hidden states, our method operates on pondering embeddings derived from the model's predictive prohability distributions. We further compare our method with the most relevant prior work in Table 3.
6 CONCLUSION
In this paper, we introduce the pondering process into language models through solely self- supervised learning- PonderL.M can be naturally learned through pretraining on large-scale general corpora. Our extensive experiments across three widely adopted architecture--GP1. Pythia, and LLaMA-highlige the effectiveness and generality of our proposed method. Notably, our Pon- derPythia consistently outperforms the official Pythia model on language modeling tasks, scaling curves, downstream tasks, and instruction- foollwiinn abilities when pretrained on the large-scale Pile dataset. As increasing the number of pondering steps further improves language model per- formance, we posit that our approach introduces a promising new dimension along which language model capabilities can be scaled.

---

Published as a conference paper at ICLR 2026
7
ACKNOWLEDGEMENTS
This work is sponsored by the National Natural Science Foundation of China (NSFC) grant (No. 62576211) and the National Key Research and Development Program of China (No. 2023ZD0121402).
8 ETHICS STATEMENT
This paper introduces novel pre-training architecture for language models that achieves significant performance improvements a given parameter scale. However, we acknowledge that, like other powerful language models, this technology is susceptible to misuse for malicious illegal pur- poses. Furthermore, as new architecture, our proposed method may contain unforeseen security vulnerabilities, posing potential risks to data privacy and system integrity.
REPRODUCIBILITY STATEMENT
To facilitate reproducibility, the source code for our core models and experiments is available in the supplementary material. We have documented all essential hyperparameters and implementation details in the appendix. We believe this provides sufficient information for the research community to verify and reproduce the results presented this work.
10 THE USE OF LARGE LANGUAGE MODELS
The core methodology and conceptual framework presented in this paper were developed without the assistance of Large Language Models (LLMs). The use of LLMs strictly limited to refining grammatical structure and enhancing the academic expression the text.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Zeyuan Allen-Zhu and Yuanzhi Li. Physics of language models: Part 3.2, knowledge manipulation. arXiv preprint arXiv:2309. /4402, 2023.
Zeyuan Allen-Zhu and Yuanzhi Li. Physics of language models: Part 3.3, knowledge capacity scaling laws. arXiv preprint urXiv:2404.05405, 2024.
Afra Amini, Tim Vieira, Elliott Ash, and Ryan Cotterell. Variational best-of-n alignment. arXiv preprint TrXiv:2407.06057, 2024.
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle OrBrin, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning. pp. 2397-2430. PMLR, 2023.
Eden Biran, Daniela Gottesman, Sohee Yang. Mor Geva, and Amir Globerson. Hopping too late: Exploring the limitations of large language models on multi-hop queries. arXiv preprint arXiv:2406. 12775, 2024.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, al. Piqa: Reasoning about physical com- monsense natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp.
2020.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared 1) Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc. the ai2 reasoning challenge. arXiv preprint arXiv: 054577 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems, 2021. URL https://arxiv. org/abs/2 1tg/abs/2110.141666 140.1488, 9, 2021.
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capa- bilities. arXiv preprint arXiv:2507.06261 2025.
DecpSeek-AL, Daya Guo. Dejian Yang, and Haowei Zhang et al. Deepseek-rl: Incentivizing reason- ing capability in Iims via reinforcement learning. 2025. URL https:: http///xxvvvvo/// abs/ 2501. 12948.
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. Universal transformers. In International Conference on Learning Representutions.
Evelina Fedorenko, Steven T Piantadosi, and Edward AF Gibson. Language primarily a tool for communication rather than thought. Nature, 630(8017):575-588) 2024.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang. Horace He, Anish Thite, Noa Nabeshima, al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.0002 2020.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Fos- ter, Laurence Golding, Jeffrey Hsu, Alain Le Noac h, Haonan Li, Kyle McDonell, Niklas Muen- nighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lin- tang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 12 2023. URL https: 10256836.
org/ htt//rdcord///////
Jonas Geiping, Sean McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Brian R Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, and Tom Goldstein. Scaling up test-time compute with latent reasoning: A recurrent depth approach. arXiv preprint arXiv=2502.05/71, 202..
11

---

Published as a conference paper at ICLR 2026
Angeliki Giannou, Shashank Rajput, Jy-yong Sohn, Kangwook Lee, Jason D Lee, and Dimitris Papailiopoulos. Looped transformers as programmable computers. International Conference Machine Learning, pp. 11398-11444 PMLR, 2023.
Sachin Goyal, Ziwei Ji, Ankit Singh Rawat, Aditya Krishna Menon, Sanjiv Kumar, and Vaishnavh Nagarajan. Think before you speak: Training language models with pause tokens, arXiv preprint arXiv:2310.02226, 2023.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752 2023.
Lin Gui, Cristina Gârbacea, and Victor Veitch. Bonbon alignment for large language models and the sweetness of best-of-n sampling. arXiv preprint arXiv:2406.000 00832, 2024.
Kobi Hackenburg, Ben M Tappin, Paul Röttger, Scott A Hale, Jonathan Bright, and Helen Margetts. Scaling language model size yields diminishing returns for single-message political persuasion. Proceedings of the National Academy of Sciences, 122(10):2443343312,, 2025.
Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason Weston, and Yuandong Tian. Training large language models to reason in continuous latent space. arXiv preprint arXiv:2412.06769, 2024.
Michael Hassid, Tal Remez, Jonas Gehring, Roy Schwartz, and Yossi Adi. The larger the better? improved Ilm gode-generation via budget reallocation. arXiv preprint 1xXiv:2404.00722. 2024.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Train- ing compute-optimal large language models. arXiv preprint arXiv:2203.15556 2022a.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, al. An empirical analysis of compute optimal large language model training. Advances in Neural Infor- mation Processing Systems, 55:30016-30033 2022b.
Zhenyu Hou, Xin Lv, Rui Lu, Jiajie Zhang, Yujiang Li, Zijun Yao, Juanzi Li, Jie Tang, and Yux- iao Dong. Advancing language model reasoning through reinforcement learning and inference scaling, arXiv preprint arXiv:2501. 11651, 2025.
Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. Large language models can self-improve. arXiv preprint 1xXiv:2210.116100 2022.
DeLesley Hutchins, Imanol Schlag, Yuhuai Wu, Ethan Dyer, and Behnam Neyshabur. Block- recurrent transformers. Advances in neural information processing systems, 35:33248-33261, 2022.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412.167100 2024.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0836 2020.
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. Race: Large-scale reading comprehension dataset from examinations. arXiv preprint arXiv:1704.0466. 2017.
Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilié. Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: 176b- parameter open-access multilingual language model. 2023.
Kuang-Huce Lee, Ian Fischer, Yuch-Hua Wu, Dave Marwood, Shumeet Baluja, Dale Schuurmans, and Xinyun Chen. Evolving deeper Ilm thinking. arXiv preprint (xXx:255110098) 2025.
12

---

Published as a conference paper at ICLR 2026
Dacheng Li, Shiyi Cao, Tyler Griggs, Shu Liu, Xiangxi Mo, Shishir G Patil, Matei Zaharia, Joseph E Gonzalez, and Ion Stoica. Lims can easily learn to reason from demonstrations structure, not content, is what matters! arXiv preprint arXiv:2502.07374, 2025.
Liunian Harold Li, Jack Hessel, Youngjae Yu, Xiang Ren, Kai- Kai-Wei Chang, and Yejin Choi. Symbolic ohain-of-thought distillation: Small models can also "think" step-by-step. In Proceedings of the 61st Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2666-2679, Toronto, Canada, jul 2023. Association for Computational Linguistics.
Wenxue Li. Xiangzhou Liu, Yuxuan Li, Yilun Jin, Han Tian, Zhizhen Zhong, Guyue Liu, Ying Zhang, and Kai Chen. Understanding communication characteristics of distributed training. In Proceedings of the 8th Asia- Asia-Pacific Workshop Networking, pp. 1-8, 2024.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892-34916, 2023.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36, 2024.
Yingqian Min, Zhipeng Chen, Jinhao Jiang, Jie Chen, Jia Deng. Yiwen Hu, Yiru Tang, Jiapeng Wang, Xiaoxue Cheng, Huatong Song, et al. Imitate, explore, and self self-improve: A reproduction report on slow-thinking reasoning systems. arXiv preprint arXiv:2412. 0294113 2024.
Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin A Raffel. Scaling data-constrained language models. Advances in Neural Information Processing Systems, 36:50358-500 503366, 2023.
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannanch Hajishirzi, Luke Zetllmoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling. arXiv preprint arXiv:2501. 19393, 2025.
Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, et al. Effi cient large-scale language model training on gpu clusters using megatron-Im. In Proceedings of the international conference for high performance computing, networking. storage and analysis, pp. 1-15, 2021.
Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, et al. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:21 2021.
OpenAl. New reasoning models: OpenAl ol ol-preveew and ol-mini. https://opena.: com/
research/ol welll-preeewwwe arreview-and--- ol mini, 2024.
Bo Pang, Hanze Dong, Jiacheng Xu, Silvio Savarese, Yingbo Zhou, and Caiming Xiong. Bolt: Bootstrap long chain-of-thought in language models without distillation. arXiv preprint arXiv:2502.03860, 2025.
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoe Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring a broad discourse context. arXiv preprint arXiv:1606 1606606033. 2016.
Suchita Pati, Shaizeen Aga, Mahzabeen Islam, Nuwan Jayasena, and Matthew D Sinclair. Com- putation vs. communication scaling for future transformers on future hardware. arXiv preprint arXiv:2302. 02825, 2023.
13

---

Published as a conference paper at ICLR 2026
Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung. Matteo Grella, et al. Rwkv: Reinventing mns for the transformer era. arXiv preprint arXiv:2305.13048, 2023.
Jacob Pfau, William Merrill, and Samuel R Bowman. Let's think dot by dot: Hidden computation in transformer language models. arXiv preprini arXiv:2404. 15758, 2024.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021.
Nikunj Saunshi, Nishanth Dikkala, Zhiyuan Li, Sanjiv Kumar, and Sashank J Reddi. Reasoning with latent thoughts: On the power of looped transformers. In The Thirteenth International Conference on Learning Representations.
Pier Giuseppe Sessa, Robert Dadashi, Léonard Hussenot, Johan Ferret, Nino Vieillard, Alexandre Ramé, Bobak Shariari, Sarah Perrin, Abe Friesen, Geoffrey Cideron, et al. Bond: Aligning Ilms with best-of-n distillation. arXiv preprint arXiv:2407. 14622, 2024.
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute optimally can more effective than scaling model parameters. arXiv preprint arXiv:2408.03:: 03314, 2024.
Benedikt Stroebl, Sayash Kapoor, and Arvind Narayanan, Inference scaling flaws: The limits of Ilm resampling with imperfect verifiers. arXiv preprint arXiv:2411.17500 2024.
Hanshi Sun, Momin Haider, Ruiqi Zhang, Huitao Yang. Jiahao Qiu, Ming Yin, Mengdi Wang. Peter Bartlett, and Andrea Zanette. Fast best-of-n decoding via speculative rejection. arXiv preprint arXiv:2410.20290. 2024.
Jihoon Tack, Jack Lanchantin, Jane Yu, Andrew Cohen, Ilia Kulikov, Janice Lan, Shibo Hao, Yuan- dong Tian, Jason Weston, and Xian Li. Llm pretraining with continuous concepts. arXiv preprint arXiv:2502.08524 2025.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-folloingg llama model. https: githhb. com/ tat www.uuaaaaaaaaaaaaaeaaaaaaaaaaaa stanford_alpaca 2023.
Pablo Villalobos, Jaime Sevilla, Lennart Heim, Tamay Besiroglu, Marius Hobbhahn, and Anson Ho. Will we out of data? an analysis of the limits of scaling datasets in machine learning. arXiv preprint #:Xiv:2211.043222 1, 2022.
Xinyi Wang, Lucas Caccia, Oleksiy Ostapenko, Xingdi Yuan, William Yang Wang, and Alessandro Sordoni. Guiding language model reasoning with planning tokens, 2024a. URL https:// arxiv. org/ abs /
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- ery, and Denny Zhou, ceff-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv: 11171, 2022.
Yifei Wang, Yuyang Wu, Zeming Wei, Stefanie Jegelka, and Yisen Wang. A theoretical understand- ing of self-correction through in-context alignment. arXiv preprint arXiv:2405. 18634, 2024b.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 25:24824-24837, 2022.
Johannes Welbl, Nelson F Liu, and Matt Gardner. Crowdsourcing multiple choice science questions.
arXiv preprint arXiv: 1707.06209, 2017.
Liu Yang, Kangwook Lee, Robert Nowak. and Dimitris Papailiopoulos. Looped transformers are better at learning learning algorithms. arXiv preprint arXiv:2311.12444 2023.
Sohee Yang, Elena Gribovskaya, Nora Kassner, Mor Geva, and Sebastian Riedel. Do large language models latently perform multi-hop reasoning? arXiv preprint 1683777, 2024.
14

---

Published as a conference paper at ICLR 2026
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does inforcement learning really incentivize reasoning capacity in Ilms beyond the base model? arXiv preprint arXiv:2504.13837 2025.
Eric Zelikman, Georges Harik, Yijia Shao, Varuna Jayasiri, Nick Haber, and Noah D. Goodman. Quiet-star: Language models can teach themselves to think before speaking, 2024. URL https: / //aaxxii..oo///////e.e......o.aa org. /2003 09629.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- chine really finish your sentence? arXiv preprint 1rXiv:1905.07830 2019.
Zhiyuan Zeng, Qinyuan Cheng, Zhangyue Yin, Bo Wang, Shimin Li, Yunhua Zhou, Qipeng Guo, Xuanjing Huang, and Xipeng Qiu. Scaling of search and learning: A roadmap to reproduce from reinforcement learning perspective. arXiv preprint arXIn:2412.14133,, 2024.
Peiyuan Zhang, Guangtao Zeng, Tianduo Wang. and Wei Lu. Tinyllama: An open-source small language model. arXiv preprint arXiv:2401.088 04.00.00885,, 2024.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Opt: Open pre-trained transformer language models. arXiv preprint arXiv:205500166 2022.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging ilm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595 46623,, 2023.
Junkai Zhou, Liang Pang, Huawei Shen, and Xueqi Cheng. Think before you speak: Cultivating communication skills of large language models via inner monologue, 2024. URL https://
arxiv. org/ abs 2211 02111.7445.
Dawei Zhu, Xiyu Wei, Guangxiang Zhao, Wenhao Wu, Haosheng Zou, Junfeng Ran, Xun Wang, Lin Sun, Xiangzheng Zhang, and Sujian Li. Chain-of- thtuught matters: Improving long-context language models with reasoning path supervision. arXiv preprint arXiv:2502.20990, 2025.
15

---

Published as a conference paper at ICLR 2026
A TRAINING DETAILS OF THE MAIN RESULT
The main computational cost incurred during the pretraining of PonderPythia- 1,4B, and 2.8B on the 300B-token Pile dataset. We pretrained these models on a cluster of pigh-perfornance GPUs with 64GB memory, which required 19,886, 31,680, and 109,570 GPU hours, respectively.
B TRAINING DETAILS OF THE SCALING LAW
In Section 3.1, have discussed the basic scaling law of pondering models. These hyperpa- rameters primarily follow the GPT-3 specifications (Brown et al., 2020). However, unlike GPT-3, we untie the input and output embedding matrices. We specify the parameters of our models and the training hyperparameters in Table 4.
Table 4: Model sizes and hyperparameters for scaling experiments.
learning rale
batch size (in tokens)
params
Blayers
dmoodl
tokens
405M
24
1024
16
3e-4
0.5M
7B
834M
24
1536
2.5e-4
0.5M
15B
1.4B
24
2048
32
2e-4
0.5M
26B
C LIMITATIONS AND FUTURE WORK
C.I LIMITATIONS
There are two limitations to our work. Firstly, due to computational constraints, we only scaled our method up 2.8B-parameter model trained on 300B tokens from the Pile dataset. It would be interesting to extend our approach to larger models and larger-scale datasets in the future. Secondly, although our results demonstrate that the proposed method scales better than vanilla models under the same training FLOPs (Figure 1), it also introduces additional inference overhead (increasing roughly linearly with the number of pondering steps), similar to test-time scaling methods.
C.2 FUTURE WORK
There are several promising directions for future work. Firstly, our proposed method is not limited to decoder-only architectures or language modeling; it has the potential to be applied to a wide range of model types and domains. For example, it could be adapted state-space models such as Mamba (Gu & Dao, 2023), encoder models, or RWKV (Peng et al., 2023), as well as extended other areas. Another promising direction is the introduction of token-adaptive pondering, which may significantly reduce computation and further enhance our method. It would also be interesting investigate the interpretability of the pondering process, such as how the model "thinks" during pondering, the semantics of the pondering embedding, and whether the model learns to reflect on its predictions through pondering. Finally, exploring the combination our method with orthogonal approaches such as CoT and other test-time scaling methods could also be an interesting direction.
D INTEGRATING PONDERING VIA CONTINUAL PRETRAINING
To investigate if our pondering mechanism can be integrated into existing large language models, conducted a continual pretraining experiment. We started with the pre-trained Pythia- IB model and continued its training on 30-billion-token subset of The Pile. We compare approaches: stan- dard continual pretraining the vanilla model (Pythia- 1B- Vanilla-CPT) and continual pretraining with our pondering mechanism (PonderPythia- 1ettyttaaa---PPP11 IB-CPT).
Our results show that this integration is effective. As seen in the training loss curves in Figure 8, PonderPythia consistently achieves lower loss, demonstrating superior learning efficiency. This
16

---

Published as a conference paper at ICLR 2026
Continue
Vanilla Loss
Training Loss
2.20
PonderLM Loss
2.15
2.10
2.05
x10
Learning Rate
1.0
Vanilla LR PonderL.M
0.5
0.0
5
10
15
20
25
30
Training Tokens
Figure 8: Training loss during continual pretraining. PonderPythia- IB achieves a lower loss than the vanilla baseline, suggesting greater learning efficiency.
efficiency translates improved downstream performance, detailed in Table The Ponder- Pythia model achieves higher average accuracy in both 5-shot and O-shot settings across suite of benchmarks.
Collectively, these findings confirm that continual pretraining is viable and effective strategy for equipping pre-trained models with our pondering mechanism, enhancing both adaptation efficiency and downstream capabilities.
Table Downstream task performance after continual pretraining on 30B tokens. PonderPythia- shows improved average accuracy in both 5-shot and O-shot evaluations.
Lambada ARC OpenAI
Lambada ARC Wino Standard
Hella
Avg / Aace t
Model (Hiraining tokens)
PIQA
SeiQ RACE
-E
-C Grande
Swag
5-shot
Pythia- 1Biia--B-Maa Vanilla-CPP PonderPythia- IB-CPT
49.4
60.0
36.5
27.1
51.9
71.4
37.9
90.4
31.9
50.7
49.5
58.8
42.6
25.4
54.3
69.2
37.9
91.2
34.7
51.5 (+0.8)
0-shot
Pythia- 1B- Vanilla-CPT
56.7
56.3
42.7
24.6
52.5
70.6
37.7
83.6
32.6
50.8
PonderPythia-IB-CC IB-CPT
56.7
56.3
45.4
23.8
55.6
68.3
37.8
86.3
33.0
51.5(+0.7)
E SCALING LAWS WITH DIFFERENT PONDERING STEPS
To further validate the robustness of our method, extended analysis LLaMA models trained with different numbers of pondering steps. Specifically, we tested configurations with 2 and 4 steps while keeping all other experimental settings constant. As illustrated Figure 9, our method mains consistently effective, with model performance improving further as the number of pondering steps increases.
F ANALYSIS ON INPUT EMBEDDINGS AND OUTPUT DISTRIBUTIONS
To better understand the dynamics of the pondering process, analyze the properties of input embeddings and output distributions using the Pythia-7OM 70M model trained with pondering steps (as described in Section 4.2). We conduct the evaluation batch of 64 validation sequences with a context length of 2048 tokens.
F.1 INPUT EMBEDDINGS
We first examine the evolution of the input embeddings F across pondering steps.
17

---

Published as a conference paper at ICLR 2026
2.6
fitted
LLaMA
PonderLLaMA (2 steps)
2.5
PonderLLaMA (3 steps)
PonderLLaMA (4 steps)
2.4
0
2.3
2.2
2,44
2.1
405M*78
834M-15B
1.4B+268
Parameters
Figure 9: Scaling curves of LLaMA with different pondering steps.
Cosine Similarity. To measure the convergence of the embeddings, we calculate the cosine sim- ilarity between consecutive steps, Cosine(E*~1 E*). Here we define s the original input embedding (before pondering). As shown in Figure 10, the similarity starts at approximately 0.88 and gradually converges 1.0, indicating that the input representations stabilize over time.
Cosine pondering step
KL vs pondering step
1.00
cosicurrntt 1, currente)
0.98
0.3
1p
0.96
0.94
0.2
KLip
0.92
0.90
0.88
4
8
10
2
4
10
pondering step (s)
pondering step
Figure 10: The cosine similarity between con- secutive embedding states E=-1 and E across pondering steps.
Figure 11: KL divergence between consecutive output distributions, averaged over tokens.
Spectral Properties. We further analyze the geometry of the embeddings. Figure 12 visualizes the spectral energy distribution (explained variance ratio) of the top singular values. Consistent with this distribution, we observe steady decrease in Effective Rank and simultaneous increase in Cumulative Variance (Figure 12). This collective evidence indicates that the embedding energy progressively concentrates into dominant components during the pondering process.
F.2 OUTPUT DISTRIBUTIONS
We also monitor the changes in the model's predictions by calculating the Kullback-Leibler (KL) divergence between the output probability distributions Pa of consecutive steps, Dxz(P==1||PP
As illustrated in Figure 11, the KL divergence exhibits large variations in the initial steps (starting around 0.35) and decreases monotonically thereafter, suggesting that the model's predictions refine quickly and then settle into stable state.
G CASE STUDIES
To provide a deeper insight into the internal pondering process of MonderLM, we present case stud- ies derived from the pre-trained PonderPythia-22 model. The following tables visualize the evolu- tion of the top-3 candidate tokens and their corresponding probabilities across sequential pondering steps.
18

---

Published as a conference paper at ICLR 2026
Effective rank cumulative variance
Spectral energy (top-8)
saplained vartance rablo (log scale)
effective
27
cum var (tap-8) affective
10"-
26
6x 10-1
0.50
0.48
4x 10
10
singular
pondering step
Figure 12: Left: Spectral energy (explained variance ratio, log scale) of the top-8 singular values of the token-embedding matrix at each pondering step Right: Effective rank (1 more spiky) and cumulative variance of the top-8 components (t more spiky) versus step 8, indicating progressive concentration of energy into few dominant directions.
The speed of light in vacuum is approxinately times ten the power of
Output Probs
Pondering steps
Pondering steps 2
Pondering steps
Final predicted
Rank 1
logarithm (0.45)
(0.38)
_8 (0.48)
(0.55)
Rank 2
(0.38)
aggreg (0.32)
(0.36)
eight (0.23)
Rank 3
ten (0. 18)
approximately (0.31)
tion (0.17)
(0.22)
I'he chemi cal
symbol 1 for vee
Output Probs
Pondering steps
Pondering steps 2
Pondering steps
Final predicted
Rank
symbol (0.47)
atoms (0.39)
Ag (0.40)
Ag (0.94)
Rank 2
symbols (0.32)
atomic (0.32)
symbol (0.33)
(0.04)
Rank 3
nickname (0.20)
elemental (0.30)
elements (0.27)
(0.02)
The largest ocean
Earth is the
Output Probs
Pondering steps
Pondering steps 2
Pondering steps
Final predicted
Rank 1
oceans (0.66)
Pacific (0.40)
Pacific (0.59)
Pacific (0.77)
Rank 2
ocean (0.32)
oceans (0.32)
oceans (0.21)
Atlantic (0.19)
Rank 3
(0.02)
Antarctic (0.28)
ocean (0.20)
Indian (0.03)
The derivative of sin is
Output Probs
Pondering steps
Pondering steps 2
Pondering steps
Final predicted
Rank 1
differentiable (0.55)
homework (0.61)
approximately (0.37)
cos (0.36)
Rank 2
the (0.23)
differentiable (0.2 (0.21)
cos (0.32)
x (0.34)
Rank 3
derivatives (0.21)
basics (0.17)
differentiable (0.31)
(0.30)
The opposite of north is
Output Probs
Pondering steps
Pondcring steps 2
Pondering steps
Final predicted
Rank
directions (0.41)
directions (0.40)
south (0.92)
south (0.95)
Rank 2
compass (0.36)
noun (0.32)
unclear (0.04)
east (0.02)
Rank 3
the (0.22)
idiot (0.28)
South (0.03)
not (0.02)
Thee chemi cal symbol gold is
19

---

Published as a conference paper at ICLR 2026
Output Probs
Pondering steps
Pondering steps 2
Pondering steps
Final predicted
Rank 1
symbol (0.48)
atoms (0.45)
Au (0.78)
Au (0.87)
Rank 2
symbols (0.40)
elemental (0.28)
Au (0.17)
"(0.09)
Rank 3
approximately (0.12)
metals (0.27)
element (0.05)
the (0.04)
H GRADIENT NORMS COMPARISON
In this section, we take a closer look at training stability by comparing the gradient norms of the PonderLM against the vamilla GPT-2 1.4B model (Section 3.1). Both models were trained from scratch on the Pile using the exact same data and hyperparameters. As shown in Figure 13, while the pondering model shows few minor spikes early on, quickly recovers and maintains stable training trajectory similar the baseline.
0.40
Vanilla Norm
PonderLM Grad Norm
0.35
Grad Norm
0.30
0.20
2500
5000
7500
10000 12500 15000 17500 20000 Training Steps
Figure 13: Gradient norms during pre-training for vanilla PonderL.M 1.4B.
20


Reviews:
[{"review_id": "DmjWFJ8Sjg", "reviewer": "Reviewer_Zzsq", "summary": "This paper proposes a pondering process to replace the standard single forward pass in transformers. Instead of producing a predictive distribution in one pass, the model refines it iteratively. At each iteration, the model computes the pondering embeddings by using the current predicted distribution to take a weighted sum over all token embeddings. This pondering embedding is then added back to the original token embeddings through a residual connection, and the updated embeddings are fed back into the model to produce a refined output distribution. \n\nThe pondering method shows strong empirical performance across extensive experiments. The pondering-trained Pythia models reach the same performance with significantly fewer parameters and training data, and they consistently outperform their counterparts on 9 downstream benchmarks. Ablation studies on alternative embedding strategies and the number of pondering steps further demonstrate the effectiveness of the method.", "questions": "1. In Eq. (5) you add the pondering embedding via a residual connection. What happens if this residual connection is removed, i.e., if you simply set $E^1=T$. Did you run this ablation? It would help isolate the contribution of the residual pathway itself.\n\n2. Have you examined how the predicted distribution evolves across pondering steps? It would be interesting to see whether each step moves the prediction closer to the target distribution, as this could provide additional interpretability into how the refinement process actually works.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The idea is simple yet effective, and it does not rely on any external supervision. The proposed mechanism is easy to implement and can be plugged into standard Transformer architectures with minimal changes.\n\nThe empirical results are solid and sufficiently demonstrate the effectiveness of the proposed approach. The observation that performance improves monotonically with more pondering steps suggests that the method provides a controllable way to trade compute for performance.", "weaknesses": "Since the method introduces additional iterative passes beyond the standard forward pass, it incurs non-trivial training and inference cost, which may become particularly expensive for larger models and longer sequences.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DZthJ68cnE", "reviewer": "Reviewer_Nkod", "summary": "This paper takes the standard output token probabilities and feeds them back into the model a set number of time with gradients attached. The intuition is that this allows the model to \"think\" more before outputting a token.", "questions": "**Q1.** What is meant by: “potentially reducing communication costs at scale”?\n\n**Q2.** It is unusual for results to be quite this consistent. Are the authors sure there were no other changes that contributed to this?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Strong consistent results, good that it only needs general corpus data, comprehensive set of experiments.", "weaknesses": "**W1.** Limited novelty - a very simple change and very similar to prior methods. But perhaps this is not a weakness as the results seem good.\n\n**W2.** 4x compute at inference time. With LLMs actually being used now, inference cost is important. I think they should perhaps therefore be compared to 4x larger models which will have the same inference cost. In this case the performance is less strong.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ra5lKSnPI7", "reviewer": "Reviewer_w4mh", "summary": "The paper introduces PonderLM, a self-supervised, architecture-agnostic mechanism that lets a language model “ponder” within a single token step by feeding back a probability-weighted sum of token embeddings for additional forward passes, creating a continuous, fully differentiable inner loop that increases compute per parameter without RL or curated CoT data. Implemented across GPT-2, Pythia, and LLaMA, the approach yields lower pretraining perplexity and strong downstream gains—e.g., PonderPythia-2.8B surpasses Pythia-6.9B and rivals Pythia-12B, while PonderPythia-1B matches TinyLlama-1.1B trained on 10× more data—with performance improving as pondering steps increase. \n\nThe key contributions are: (i) a simple pondering loop that replaces discrete token emission with continuous weighted embeddings; (ii) proof that such behavior emerges via standard next-token pretraining alone; (iii) consistent benefits across model families and scales, especially for smaller models; and (iv) a framing of pondering as a third, orthogonal scaling axis (complementary to parameter and CoT test-time scaling) that may improve parameter knowledge density and reduce communication costs at scale.", "questions": "None", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper introduces a differentiable, self-supervised pondering loop that feeds a probability-weighted embedding back into the model within a single token generation step. This elegant mechanism eliminates the discrete bottleneck imposed by vocabulary spaces during internal computation. By conceptualizing pondering as a third scaling axis, orthogonal to both parameter scaling and test-time CoT scaling, the work offers a novel perspective on model scaling dynamics. Moreover, demonstrating that this behavior can emerge without reinforcement learning or curated CoT supervision substantially relaxes the conventional prerequisites of test-time scaling approaches.\n\n2. The proposed inner loop is straightforward, easily integrable into standard language model architectures, and fully differentiable for end-to-end training using conventional next-token prediction. Empirical evaluations across three architectures (GPT-2, Pythia, and LLaMA) and nine downstream tasks reveal consistent and substantial performance gains—most notably, size-for-size improvements where a 2.8B model outperforms a 6.9B counterpart and approaches the performance of a 12B model. Moreover, the results exhibit monotonic improvements as the number of pondering steps increases.", "weaknesses": "1. The paper’s motivation and theoretical foundation appear insufficiently developed. It remains unclear why repeating the forward pass within a single token-generation step should improve performance. The current justification—an analogy to human “slow thinking”—is conceptually interesting but lacks a mechanistic explanation or connection to established findings in neural or cognitive science. Providing a clearer rationale, ideally supported by formal analysis of how the proposed weighted-embedding feedback influences model expressivity, optimization dynamics, or the compute–performance trade-off, would considerably strengthen both the motivation and the overall argument.\n\n2. To substantiate the claim of a latent thinking process, it would be valuable to analyze the model’s internal states and compare them with those from explicit (e.g., CoT) and implicit thinking methods.\n\n3. Several evaluated tasks appear susceptible to data contamination [1]. It would strengthen the paper to quantify the extent of contamination and disentangle its contribution to the observed gains—for example, by applying contamination checks, re-running on decontaminated splits, or reporting performance deltas with/without potentially contaminated items.\n\n\nReferences\n\n1. Koala: An Index for Quantifying Overlaps with Pre-training Corpora. Vu et al. 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1GUHBdwZ6L", "reviewer": "Reviewer_We2v", "summary": "The paper introduces a novel technique (called \"pondering\") to iteratively call the forward pass of an LLM for a single token generation. Typically, a forward pass with an LLM over an input sequence results in a probability distribution over the vocabulary, which is then used for decoding the output token. The pondering approach in this paper utilizes the probability distribution at each token position to compute a weighted sum of the vocab embeddings. The main idea is to leverage such refined embeddings in subsequent forward passes (\"s\" times), followed by the calculation of the CE loss. The results show that the pondering approach improves the FLOPs and data usage efficiency during pre-training of various model architectures and sizes.", "questions": "1. Did the authors analyze any properties of the embeddings after every pondering step? For ex: spectral properties, cosine similarities etc\n\n2. How do the gradient norms change when incorporating pondering vs the vanilla pretraining? \n\n3. Were there any training stability challenges when training from scratch, since I see some spikes in Figure 8 for continued pre-training? How did you address them?\n\n4. An analysis of how the output distributions evolve with iterative pondering steps would be a great addition to the paper. For ex, does the KL divergence of the output distributions change drastically in early pondering steps or later?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The methodology is well presented along with clear writing, code, and results.\n\n2. The language modeling and downstream task evaluation results are promising and can influence research on such iterative embedding refinements as a new frontier to explore the scaling of compute.", "weaknesses": "1. A major concern I have is about the validity of the scaling law shown in Figure 4 if one were to change the number of pondering steps from 3 to a lower or higher value. Or even if the randomized pondering step approach was employed for that matter. If the scaling laws were similar to figure 4 with a randomized approach of pondering step selection, then that is an even better approach to scale since it is currently unclear how the value of (s=3 steps) was chosen. I believe the message of the paper can be further strengthened if this weakness is addressed.\n\n2. This is a minor concern, but can a model trained to ponder for (s=3) steps be effective when pondering for (s>3) steps during inference? Some early results (either positive/negative) can influence a lot of future work in this direction.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)