Latent Thought Models with Variational Bayes Inference-Time Computation
Authors: Deqian Kong
Minglu Zhao
Dehong Xu
Bo Pang
Shu Wang
Edouardo Honig
Zhangzhang Si
Chuan Li
Jianwen Xie
Sirui Xie
Ying Nian Wu
Venue: ICML 2025 poster (2025)
Keywords: Latent thought models
Inference-time computation
Language models
Variational Bayes
Latent variable model
Primary area: deep_learning->generative_models_and_autoencoders
TL;DR: We propose a novel class of language models with explicit latent thought vectors that follow an explicit prior model in latent space.

Abstract: We propose a novel class of language models, Latent Thought Models (LTMs), which incorporate explicit latent thought vectors that follow an explicit prior model in latent space. These latent thought vectors guide the autoregressive generation of ground tokens through a Transformer decoder. Training employs a dual-rate optimization process within the classical variational Bayes framework: fast learning of local variational parameters for the posterior distribution of latent vectors (inference-time computation), and slow learning of global decoder parameters. Empirical studies reveal that LTMs possess additional scaling dimensions beyond traditional Large Language Models (LLMs), such as the number of iterations in inference-time computation and number of latent thought vectors. Higher sample efficiency can be achieved by increasing training compute per token, with further gains possible by trading model size for more inference steps. Designed based on these scaling properties, LTMs demonstrate superior sample and parameter efficiency compared to autoregressive models and discrete diffusion models. They significantly outperform these counterparts in validation perplexity and zero-shot language modeling tasks. Additionally, LTMs exhibit emergent few-shot in-context reasoning capabilities that scale with model size, and achieve competitive performance in conditional and unconditional text generation. The project page is available at [https://deqiankong.github.io/blogs/ltm](https://deqiankong.github.io/blogs/ltm).

Latent Thought Models with Variational Bayes Inference-Tine Computation
Deqian Kong" 12f Minglu Zhao
Dehong Xu Bo Pang Shu Wang' Edouardo Honig'
Zhangzhang Si
Chuan Li Jianwen Xie 21
Sirui Xie
Ying Nian Wu !
Abstract
MDLM
70M, N2 96
76M. N2
Validation Perplexity
LNM-5
MD4-Medium
5IM, M2 24
We propose a novel class of language models, La- tent Thought Models (LTMs), which incorporate explicit latent thought vectors that follow ex- plicit prior model in latent space. These latent thought vectors guide the autoregressive gener- ation of ground tokens through a Transformer decoder. Training employs a dual-rate optimiza- tion process within the classical variational Bayes framework: fast learning of local variational pa- rameters for the posterior distribution of latent vectors (inference-timg computation), and slow learning of global decoder parameters. Empiri- cal studies reveal that LTMs possess additional scaling dimensions beyond traditional Large Lan- guage Models (LLMMs), such as the number of iterations inference-time computation and num- ber of latent thought vectors. Higher sample efficiency can be achieved by increasing train- ing compute per token, with further gains pos- sible by trading model size for more inference steps. Designed based on these scaling prop- erties, LTMs demonstrate superior sample and parameter efficiency compared to autoregressive models and discrete diffusion models. They sig- nificantly outperform these counterparts in val- idation perplexity and zero-shot language mod- cling tasks. Additionally, LTMs exhibit emer- gent few-shot in-context reasoning capabilities that scale with model size, and achieve competi- tive performance in conditional and unconditional text generation. The project page available https://deqiankkngg.o wwt.:iowoonnnng.oiiiiioe gitthu.io/bllos/ltm..
2
LTM-M
GPT-2-Large
36M. Na
2
LTM-L
2
24
2
2
2
Inference Steps
Figure 1: Analysis model scaling behavior of valida- tion perplexity across model size, inference steps, and the number of latent thought vectors Ng. Autoregressive and diffusion baselines are plotted dashed lines.
1. Introduction
Recent years have witnessed remarkable advancements in the field of natural language processing, primarily driven by the development of large language models (LLMs).. These models, exemplified by GPT-3 (Brown al., 2020), PaLM (Chowdhery al., 2022), and their successors, have demon- strated impressive capabilities across wide range of lan- guage tasks, from text generation and translation to question answering and complex reasoning. Their performance has often approached, and in some cases even surpassed, human- level competence in specific domains.
The remarkable success of LLMs is underpinned by well- established scaling laws (Kaplan et al., 2020; Hoffmann et al., 2022), which predict performance improvements with increased model and data size. The induced equations re- veal that larger models achieve significantly higher sample efficiency (evaluated by the number of training tokens for achieving certain performance). making it computationally optimal to train very large models and stop before conver- gence. However, as model sizes grow rapidly, data availabil- ity has emerged This limitation motivates our exploration of novel class of language models that introduces new scaling dimensions to unlock further improvements in sample efficiency.
'Equal contribution *Equal advising 'UCLA "Lambda, Inc. "Saeesforce Research 'KUNGFU.AL. This work was partially con- ducted while D. was an intern at Lambda, Correspondence to: Deqian Kong <deqiankong @ cdu>..cdu>.
critical bottleneck for continued scaling.
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
We propose Latent Thought Models (LTMss), which incor-

---

Latent Thought Models
porate explicit latent thought vectors that follow explicit prior model in the latent space. These latent vectors control an autoregressive Transformer decoder's (Vaswani et al., 2017) generation of each token throughout the sequence, effectively creating an abstract representation of the entire sequence, LTMs are trained within the classical variational Bayes framework (Jordan et al., 1999; Blei et al., 2017; Murphy, 2012), with a dual-rate optimization process: fast learning or inference-time computation of local variational parameters for the posterior distribution of latent vectors, and slow learning of global decoder parameters. This ap- proach enables rapid adaptation to specific inputs while gradually accumulating general linguistic knowledge.
Pretraining Perplexity: Given fixed training compute, LTM-Medium achieves perplexity comparable to GPT- 2-Large (10.95 vs. 11.5) with equivalent trFLOPs/tok but only 6.7% of GPT-2-Large parameters. ITMll achieves 1 11.85 perplexity with 26% less trF1.OPs/tok and 5.0% of GPT-2-Large parameters. LTM-Large, chosen
for its favorable tradeoff between sample efficiency and inference speed, reaches validation perplexity of 3.05 using only 76M parameters trained on 3B tokens.
Language Modeling: LTMs' superior pretraining per- plexity translates to zero-shot language modeling perfor- mance, with LTM-Medium and LTMe-Large achieving 52.2% and 91.7% reductions in perplexity compared to state-of-the-art results at GPT-2 scale.
The architecture and learning scheme of L.TMs draw inspi- ration from established cognitive models. Within the frame- work of the declarative- proceuuur model (Ullman, 2004), the latent thought vectors and local variational parameters parallel the declarative or episodic memory, while the global decoder parameters correspond to procedural memory. The dual-rate learning scheme reflects the interplay between fast episodic learning and slow schematic learning in hu- man cognition (Kumaran et al., 2016). Moreover, under the language of thought hypothesis (Fodor, 1975), the latent thought vectors can be interpreted as "words" of an internal cognitive language.
Arithmetic Reasoning: LTMs demonstrate emergent few- shot in-context learning at scales that are significantly smaller than GPTs. This is significant even in our smallest model, LTalS This capability scales further with increased model size. We also find scaling the number of latent thought vectors appears to be helpful.
Text Generation: LTM-Large outperform both autore- gressive and diffusion counterparts in conditional sentence completion when measured with MAUVE score (Pillutla et al., 2021). In unconditional generation, LTM-Large achieves generative perplexity (Dieleman al., 2022) and token-level entropy (Zheng et al., 2024) comparable to GPT-2-Large, while being significantly faster.
LTMs introduce novel dimensions for investigating scaling behaviors: the number of iterations in inference-time com- putation (inference steps), and the number of latent thought vectors (latent size). To empirically study the scaling be- haviors of LTMs, GPT-2 scale (Radford et al., 2019) using the Open Web'eeebTer Text dataset (Gokaslan & Cohen, 2019). The perplexity of LTMs scales with data size, model size, inference steps and latent size. While traditional L.MS primarily trade off between data size and model size, LTMs introduce a higher-level trade-off between data size and compute per token (training FLOPs per token (tFFLOPs/tok)). At fixed tFFLOP//tok budget, LTMs can be optimized across multiple dimensions: inference steps, model size, and latent size. While scaling any of these dimensions improves performance, as shown in Fig. 1, increasing inference steps enhances both sample and compute efficiency, with larger latent sizes providing addi- tional headroom for improvement (Fig. 4). These relation- ships provide preliminary guidance for sample-efficient and compute-optimal training of LTMs, revealing that inference- time computation represents fundamentally new axis that complements traditional model parameter and data scaling,
conducted extensive experiments at
Contributions. Language models with explicit latent thought vectors that follow prior model in latent space are much under-explored in recent years. Compared to ground tokens, the latent thought vectors provide a highly compact, abstract and structured representation in a lifted latent space. This paper constitutes a systematic exploration of this model class with the following contributions:
1. Introduction of language models incorporating explicit latent thought vectors and prior models in latent space.
2. Development of dual-rate optimization algorithm that effectively combines learning and posterior inference.
3. Comprehensive analysis of scaling properties, especially along the dimensions of inference steps and model size.
4. Demonstration of superior pretraining perplexity and zero-shot performance compared to existing approaches.
In comparison with traditional autoregressive models (Rad- ford et al., 2019) and more recent diffusion- ap- proaches (Lou et al., 2024; Shi et al., 2024; Sahoo et al., 2024), LTMs demonstrate superior efficiency in data and parameters, and excel in several key language tasks:
5. Evidence that our models achieve in-context learning capabilities for arithmetic reasoning with significantly fewer parameters than GPTs.
6. Demonstration of competitive performance in both con- ditional and unconditional text generation tasks.
2

---

Latent Thought Models
(M)
in future work.
Thought-Guided Generator. The key component of our model is thought conditioned autoregressive gen- erator px( It can be realized by Transformer de- coder (Vaswani et al., 2017) with parameter B. Unlike stan- dard autoregressive models that only condition on previous elements (Radford al., 2019), our model incorporates the thought vector z at each generation step:
cross-atlention
B,.
Layer
self-attention
:
z Z~,0
Layer
cross-allention
N
Z1
Layer
(x(x = II (a(x
z, z, x
(I)
self-attention
where x x(<)) denotes previous tokens before (x)) Each Transformer decoder layer incorporates its corresponding vectors Zι through cross-attention, where 2₁ provides the keys and values while the input x offers the queries. The be considered instance-specific local
(0)
Figure 2: Illustration of the LTM. The latent thought vec- tors are sampled from a standard normal distribution (0, I). For each layer in the autoregressive genera- tor (x(x) the corresponding vectors 2; are incorporated through aross-attention. z represents restance-specifee lo- cal parameters, while denotes global parameters shared across all samples.
thought vectors parameters, while / represents the global parameters shared across all samples.
Short Context Window. We are particularly interested in models with a short context window of size k: pa(x(z) I (z, (x==k111) where (" denotes the k previous elements. This short context forces z to serve as a information carrier, integrating information across tem- poral segments that would otherwise be disconnected due to the short context window. 256 in our experiments.
2. Method
2.1. Latent Thought Models (LTMs)
Let denote the latent thought vectors and x (e(0)),(()) (,("),...,,x()))) represent the sequence of ground to- kens of natural language. Our model assumes that z follows a prior model p(z) and generates x via Transformer de- this setup, z controls the generation of
2.2. Learning and Posterior Inference
We present three approaches for learning and posterior in- ference of I.TMs, each offering different trade-offs between computational efficiency and modeling flexibility.
coder p(x|z). each token, making our model a conditional autoregressive model where z cross-attends each layer of the decoder.
Maximum Likelihood Learning with Langevin Sampling. This baseline approach directly maximizes the marginal ligelikelihood L(B) E log ps(;). The marginal distribution given by:
We formulate framework as a structured probabilistic model that captures the relationship between latent thought vectors and observed sequences as shown in Fig. 2.
Layered Thought Vectors. We assume z = (Z),... Zt), where z consists of thought vectors roos-attending to layer I of the Transformer decoder. N₂ denotes the total number of latent vectors, except in Section 2.4 where it represents alternative the number per layer. While we explored design using a single set of thought vectors attending to all layers simultaneously, empirical evidence strongly favors the layered approach. The layered structure, where distinct sets thought vectors attend to different layers, appears to capture multiple levels of abstraction more effectively.
(x(x)= = ps(x(2)p(z)dz,
(2)
where p(z) (O, I). The learning gradient is:
V loggpsp() = Ep,( x) [V log2s(x|z)].
(3)
The expectation can be estimated with Monte Carlo sam- ples from the posterior distribution (2 using Langevin dynamics:
Prior Model. For the prior model p(z), we assume isotropic Gaussian prior over the latent thought vectors z = (Zi,..., Z ~ (0, I). This prior model is proper starting point due to its simplicity.
z +++ = + + logps(z" + V2se",
(4)
where indexes the time step, 8 is the step size, and 6" 10,
already a structured prior model with multiple layers latent thought vectors. We shall explore more sophisticated learnable prior model Pa (z)
Classical Variational Bayes Learning. This approach,
which we adopt, introduces a sequence-specific variational
3

---

Latent Thought Models
posterior g(z]x) = (N, σ with variational parameters (1, 2 (Jordan et al., 1999; Blei al., 2017; Murphy, 2012). 1 is the posterior mean vector and Ø² is the poste- rior variance-covarianee matrix, assumed to be diagonal for computational efficiency. We maximize the evidence lower bound (ELBO) (Hoffman et al., 2013; Murphy, 2012):
Algorithm Fast-Slow Learning of LTM
1: Training data {x+)i=>> generator ps learning
rates Wast and f/soow, fast learning steps Tast-
while not converged do
3: Sample mini-batch {x1}
4: for each x; in the mini-batch do
[(9, M4, x²) = By(axx) [log (x)p)))())) (xL((()))))ll()) (p(z))))
5:
// fast learning
Inference-time computation Initialize μ4: p4:07 o²
(5)
6:
where 2 ~ ((z)x) x) is sampled using re-parametrization trick (Kingma & Welling, 2013).
7:
for t = to Tisst do
Sample z q,. (z)x,)
Compute
It is crucial to emphasize that (², are local parameters, specific to each training testing sequence x. This is in contrast to the parameters in the decoder generator, which
L:
zz] DeL (g(z)x))ll(x))
10:
Update ps; of using Adam with Wast. end for
11:
are shared by all the training sequences and thus are global parameters. As detailed in Algorithm I, we employ dual-
12:
end for
13:
// slow learning
rate learning algorithm: fast inference of local parameters using a gradient descent algorithm, Adam (Kingma & Ba, 2014; Loshchilov & Hutter, 2019), with high learning rates (e.g., 0.3) and few steps (e.g.. 16), alternating with slow updates of global decoder parameters (e.g., learning rate 0.0004). This enables rapid per-instance adaptation while gradually building general linguistic knowledge.
14: Compute batch loss Chatch = ΣE fii
15: Update 3 using AdamW with 7]slow.
16:
end while
Compared to Langevin sampling, it provides more efficient optimization. Compared to VAE, avoids learning large inference model and mitigates posterior collapse by avoid- ing the initial mismatch between the inference model and the true posterior. More importantly, the classical variational method allows us to explore gradient descent for inference, connecting our approach to fast-slow learning and inference- time or test-time computation paradigms (Ba et al., 2016; Krause et al., 2018).
In our work, we use finite number of steps (e-g.. Tixe = 16) for fast learning inference-time computation for the poste- rior distribution of latent thought vectors. Such finite-step inference-time computation is usually affordable on mod- ern GPUs, especially for a relatively small decoder model with short context window. While finite-step fast learning may introduce a bias relative to maximum likelihood if lo- cal variational inference does not converge (Hoffman et al., 2013), we empirically study how scaling the number of steps influences this bias under I.TMs' architectural conditions.
2.3. Conditional and Unconditional Generation
To generate samples from a trained LTMs, we need to first sample latent thoughts z. For conditional generation, the principled distribution for completion y given prefix or prompt x is:
Variational Autoencoder with Amortized Inference. As another baseline, the VAE approach (Kingma & Welling, 2013) introduces an inference model (z x) with global parameters é to amortize the iterative inference computa- tion in classical variational learning. In our experiments on VAE, we observe severe posterior collapse (Lucas al., 2019; Pang et al., 2021), even with careful annealing on the KL-divergence term in ELBO (Eq. (5)). Note that the infer- ence model only has a fixed number of parameters, which are shared by all data points, while the classical variational Bayes inference has local parameters whose size is propor- tional to the number of training examples. As a result, the inference model is more likely than the classical variational Bayes to take the easy route and only minimize the KL term in ELBO. A simple fix is to infer the local parameters in the traditional variational Bayes framework, and then distill the inferred local parameters to the inference model.
ps(y|x) - f (x)x = Ep(x/x) (p 2)]
(6)
We sample the posterior distribution p(z|x) p(z)p using classical variational inference, following the same mechanism as the fast learning of g(z)x) in Eq. (5) during training. The actual sampling distribution becomes:
Pa ( Ro Ex(()) (y) z)]
Zelikman et al. (2022); Hu et al. (2023); Phan et al. (2023) also sample posterior latent (chain-of- -fhougghstt for condi- tional generation from p(y(x), but their approaches differ fundamentally from LTMs since they work on post-training of traditional autoregressive models on finetuning sets, while L.M's' posterior inference naturally optimized during
Comparisons. We adopt classical variational Bayes, leav- ing Langevin- learning and VAE as ablation baselines.
4

---

Latent Thought Models
LTM-5 (38M)
LTM-M (51M)
LTM-L (76M)
of trFLOPs/tok by drawing analogies with the chain-of- thought tokens in ARMs (Guo al., 2025). Among all three factors -Ng. L,, and Tist- N₂ has minimal impacts on trFLOPs/tok because we use far fewer latent vectors than input tokens (N₂ < N). We anticipate it to play a different role than scaling the number of chain-of- -ftouughttthttt tokens in ARMs even though these two number appear to be quite rel- evant. The contribution of will not become dominant until the computation in attention layers exceeds the offset of em- bedding layers, as illustrated in Fig. 3. We anticipate mod- erately significant scaling when L. is comparable to V/N, which is the regime we explore. Tia is the most influential factor for trFLOPs/tok. When Tima > 1, the compute for fast learning dominates slow learning, and the trFL.Os/too of O ( + NN,H + N H22 + sents both the training compute (with negligible slow learn- ing step) and the inference-tine compute (pure Tist itera- tions). We anticipate Tims to be the primary scaling factor, potentially playing similar role to the number of chain-of- thought tokens in ARMs.
4.0%
64.4%
5.9%
5%
31.2%
15.7%
23.2%
30.5%
15.9%
23.4%
Cross Attention
Seff Attontion
Food Forword
Embedöing Layer
Figure 3: Distribution of compute in different model sizes.
pre-training. Sampling from ps z) follows standard autoregressive sampling techniques (Freitag & Al-Onaizan, 2017; Holtzman et al., 2019). For unconditional generation, we sample from:
Ti repre-
(x (x) p [x[22]
An alternative sampling scheme is to incorporate each newly generated token into the prefix and then updating through variational inference. We leave exploration of this more computationally intensive approach to future work.
During testing, N varies by task: it represents the token sequence length for latent vector inference in likelihood estimation and generation tasks. As detailed in Section 2.3, generation tasks' inference-time compute can further vary by sampling scheme. For our adopted sampling scheme, the tnFLOP//tok derived above provides a worst-case estimate of inference-timg compute across all tasks.
2.4. Inference- Time Computation
Compared to language models operating in the token space (e.g., ARMs and DDMs), LTMs introduce distinct com- putational cost in the form of inference-time compute requirement stemming from the fast learning of latent thought vectors. This inference-time computation occurs in both model training and testing. Let's start from analyzing it within the context of total training compute.
3. Empirical Study
3.1. Experimental Setup
Open Web dataset (OWT) (Gokaslan & Cohen, 2019), which is an open-source replication of the WebText dataset used in GPT- 2 (Radford et al., 2019) training. OWT includes around 8B web-crawled text tokens and is a standard choice to compare against GPT-2 and other language models. Following Lou et al. (2024), we reserve the last 100K documents as valida- tion set. For zero-shot perplexity evaluation, we include the validation splits of Penn Tree Bank (PTB) (Marcus et al., 1993), Wikitext (Merity et al., 2016), One billion word benchmark (LMIB) (Chelba et al., 2013), Lambada (Pa- pemoo et al., 2016), AG News (Zhang et al., 2015), PubMed and Arxiv subsets (Cohan et al., 2018).
Datasets. For model pre-training,
For one single iteration of LTM's dual-rate learning with Tast inference steps cabulary size V), we consider a model with L. attention layers, N₂ latent thought vectors per layer, and hidden di- mension H. The forward pass computational complexity is approximately (LL + NN + NH2 + HV), comprising OLL*2 for self-attention, O O(LN for cross-attention with latent vectors, O(LN H2) for feed- forward layers, and O(NVH) for embedding layers. The backward pass doubles this cost due to gradient computa- tion and activation storage (Chowdhery et al., 2023). With Tiss backward passes in fast learning, and 1 stw additional backward pass in slow learning, the training compute per token (trFL.OPstoo)) is ( slww I L(N22 H N N, + N H2 (Tisst + ) NV HV. Thus, while both LTMs and ARMs involve gradient prok-propagttion for training, LTMs distribute compute differently: they trade ARMs' compute in slow learning of global parameters for fast learning of local parameters.
imput sequence of N tokens (vo-
Baselines. We evaluate LTMs against both autoregres- sive models and discrete diffusion models. For autore- gressive baselines, we include GPT-2-Medium and GPT- 2-Large (Radford et al., 2019), as well as variants trained by Sahoo et al. (2024) and by ourselves. For text diffu- sion models, we compare against three diffusion models: SEDD (Lou et al., 2024), MDLM (Sahoo et al., 2024), and MD4 (Shi et al., 2024).
To anticipate the scaling behavior of LTMs, we analyze how the three key scaling factors influence the profile
5

---

Latent Thought Models
Training Perplexiity (log srale)
2-
Training Perplexity (log scale)
2
38M, Twn 16, 24
30M. Tue 16, -4
22
22
76M. Tux 16, =
76M. Tea 16
38M. Tau:
38M. Tru
S3M, Tee 32, 66 24
5IM. Teu 32, 24
70M, Taxe 32,
21
70M, Yeu =4
Te M,-
30M.
51M, Tow 64, N6 24
51M. Two 64,
Tr
76M. Yeu
2c
70M. Tia 64. 02
70M. Tre 54
78M, Taee
76M. The
0.0R
0.18
1.0R
0.18
1.0R
10.08
100.0R
Training Tokens
Training ning GRLOPS
Figure 4: Scaling behaviors over training tokens and compute. We plot the performance of LTM training runs across inference steps aais =16-64), latent size (N₂ =24-96) and model sizes (38M-76M). Models with more inference steps demonstrate improved sample efficiency and become compute-efficien beyond certain training compute thresholds.
Architectures and Training. All LTMs share similar ar- chitectures, with small, medium, and large variants using 3, 6, and 12 layers respectively. Our training was conducted on 8 H100 GPUs with epoch batch size of 512. We employed two learning rate schedulers for dual-rate learn- ing: fast learning schedules linearly increasing from 0.3 to 0.34, and slow learning schedules beginning at x 10 10-4 with cosine decay. Other training details are provided in Appendix A.2.
(2020), training larger models without reaching convergence proves more compute-cfficient than training smaller mod- els convergence. Fig. 4 shows that LTMs possess similar properties: models with more inference steps achieve greater sample efficiency and become more compute-efficient be- yond certain thresholds of training compute. Additionally, larger latent sizes (N₂ = 48, 96) further enhance both sam- ple and compute efficiency when combined with more infer- ence steps. The minimal influence of model size on these curves likely stems from embedding layers' computation remaining comparable to attention layers this scale.
3.2. Scaling Behaviors
Scaling model size, inference steps, and latent size. LTMs extend traditional autoregressive models with two additional design axes: inference steps and latent size. Fig. shows validation perplexity across configuration sweep.
3.3. Comparison with Existing Language Models
Our scaling study yields three representative models with varying uFFOOP/tok, for which controlled the latent size to highlight the comparison between scaling model sizes and scaling inference steps. LTM-Small, our most lightweight model, uses only 38M parameters with minimal inference steps. ITM-Mediim matches GPT-2-Large' 's trFLOPs/tok while using only 6.7% of GPT-2-Large parameters. LTM- Large is selected for its favorable tradeoff between inference speed and sample efficiency. When consuming compute that is equivalent training other LTMs, it is far from conver- gence on OWT. Detailed configurations of them are reported in Table Variations in latent size will be discussed sepa- rately where relevant.
Latent size: More latent thought vectors improve perfor- mance across all model sizes and inference step configura- tions. The 76M parameter models show clear performance gains when increasing from N₂- 24 to N₂ 96, in- dicating that latent dimensionality serves as effective scaling dimension for LTMs.
Inference steps vs model size: Performance improvements from inference steps become apparent starting from 16 steps 128 steps. For larger steps, we find that scheduling the fast learning rate helps for stable training, In particular, we adopt cosine decay scheduler. Conversely, at fixed latent size and inference steps, model size has minimal impact, likely because attention layers' contribution has not yet overtaken that of embedding layers at this scale.
Pretraining Perplexity. LTMs' perplexities on OWT vali- dation set marked in Fig. 1. The inference-time compute for this evaluation close to fFFLOP//tok, except that there is slow learning. Trained with equivalent trFLOP/took as GPT-2-Large, LTM-Medium performs slightly better, with only 10% parameters. The model size can be further re- duced to 38M, as LTM-Small, without compromising much performance. LTM--Large achieves state-of state-of-the-art validation perplexity: 3.05 even if is only trained with 3B
Inference steps drive sample and compute efficiency. When extrapolating scaling properties to larger training compute regimes, converged performance becomes less rel- evant for model selection. As demonstrated by Kaplan al.
6

---

Latent Thought Models
Table I: Zero-shot unconditional perplexity (4) across datasets. LTMs are trained with N₂ - 24 and evaluated at checkpoints with equivalent total training compute. The total compute used is less than other listed models. Both diffusion models and LTMs report perplexity upper bounds. Results without citations are from our reproductions or evaluations.
Model Family
Model Size
trFLOPs/ok
# Tokens
PTB
WikiText
LMIB
LAMBADA
AG News
PubMed
Arxiv
GPI-2-Medium
345M
2.42G
130.04
$2.14
44.03
36.09
44.53
23.33
23.82
GPT-2-Large
762M
5.32G
161.33
30.09
45.61
34.26
39.33
68.15
21.01
AR (Sahoo 2024)
110M
0.85G
524B
82.05
25.75
51.25
51.28
52.09
49.01
41.73
AR RR-Reerainee
76M
0.46G
105B
258.95
52.49
107.37
61.55
110.31
60.61
55.35
SEDD (Saboo 2024)
L10M
0.85G
524B
< 100.09
< 34.28
68,20
49.86
62.09
44.53
38.48
SEDD (Lou al,, 2024)
345M
2.42G
87.12
S 29.98
61.19
42.66
MDLM (Sahoo et 2024)
110M
0.85G
524B
95.26
S 32.83
67.01
47.52
61.15
41.89
37.37
MD4 (Shi 2024)
345M
2.42C
66.07
S 25.84
51.45
44.12
LTM-Small (The 16)
38M
4.07G
7B
34.71
< 18.87
23.59
19.31
34.76
73
21.67
LTM-Medium (Tima
51M
5.52G
5.2B
32,06
S 17.39
25.16
17.32
27.80
20.45
19.22
I.TM- arge (The 64)
76M
32.2Gi
0.98
X 4.43
3.66
S 3.92
S 3.48
S 4.56
S 3.87
S 3.54
tokens. While more inference steps could yield higher sam- ple efficiency, and better perplexity we choose L-rgee as it provides a favorable tradcoff between inference speed and sample efficiency.
30
0-shot
25-
10-shot
GPT-2-Medium
test-time sparch)
Pass@8 Accuracy
20
GFT-2-Meding
Language Modeling. LTMs' pretraining perplexity trans- lates to zero-shot language modeling performance. Different evaluation schemes exist for this task, which mainly differ in using sliding windows quences. We pick the non-overlapping blocks following Lou et al. (2024) and subsequent work Sahoo et al. (2024); Shi et al. (2024) as sliding windows may favor autoregressive models. Table summarizes these results. For fair compari- son, we evaluate all LTMs at checkpoints with equivalent training compute. LTMs consistently outperform existing baselines across all benchmarks.
15
(finetuned)
10-
non-overlapping blocks text
GPT-2 Medium
GPT:2-Large
LTM-S
LTM-M
(TML-
LTM-L (N2 192)
Figure 5: Evaluation of arithmetic reasoning (GSM8K). LTMs with few-shot demonstrations outperform GPT-2s across various settings. Dashed lines indicate baselines reported by Li et al. (2022): GPT-2-Medium finetuned on GSM8K, and GPT-2-Medium with test-time search.
Arithmetic Reasoning on GSM8K. LTMs significantly out- perform GPT-2 counterparts in zero-shot testing on GSM8K (Cobbe et al., 2021). The evaluation metric at this scale pass 5 metric (pass rate given trials of conditional generation), following Li et al. (2022).
from related approaches. Unlike autoregressive models (Brown et al., 2020), LTMs use gradient-based inference for latent thought vectors, enabling few-shot learning at much smaller model scales. This suggests more efficient pattern discovery at abstract levels. The emergent nature of this capability contrasts with meta-learning via livll optimization downstream tasks (Finn et al., 2017; Yoon et al., 2018) LTMs achieve few-shot learning directly within the context window without specialized training.
We then explore LTMs few-shot in-context learning capabil- ity, which traditionally emerges only at GPT-3 scale (Brown et al., 2020). Using randomly sampled training examples as in-context demonstrations, we find that LTMS exhibit this capability even in our most lightweight configuration (38M parameters). As shown in Fig. 5, LTM-Small with 5-shot demonstrations surpasses the baselines from Li et al. (2022) that incorporates finetuning or test-time search. Increased model size further improves both zero-shot and few-shot performance. Motivated by the hypothesis that more ex- pressive latent space enables stronger abstract reasoning, we tested an LTM-Large variant with 192 latent thought vectors, which achieves the best performance. Additional experiment details are included in Appendix A.3.
Conditional Generation. We evaluate LTM's conditional generation capabilities by generating fixed-length comple- tions for 50-token prompts from the OWT validation set, following Lou et al. (2024). We assess generation quality using MAUVE scores (Pillutla et al., 2021), which measure the distributional similarity between generated and ground- truth text, following Lou et al. (2024) and Han et al. (2022).
While GPT-2 requires nucleus sampling to achieve compa- rable performance with diffusion models, LTMs outperform
LTMs' few-shot learning capability differs fundamentally
7

---

Latent Thought Models
Table 2: Evaluation of conditional generation. LTM achieves better performance in text completion than autore- gressive model and diffusion model counterparts. Baselines are obtained from Lou et al. (2024).
Table 4: Ablation results on inference strategies. LTM with Langevin sampling and variational Bayes learning mit- igates posterior collapse, while the variational Bayes ap- proach enables more efficient optimization.
Model
Sampling method
MAUVE(1)
Inference type
Model Size
Val. PPL
Gen PPL
Entrupy
GPT-2-Medium
Nucleus-0.95
0.955
Langevin
76M
148.9
5.1
Multinomial
0.802
VAE
114M
29.96
1.83
SEDD Standard
None
0.957
SEDD Infill
None
0.942
Large
Multinomial Greedy
0.974
0.972
encoder with full attention, suffers from posterior collapse, resulting in repetitive prior samples and low entropy dis- tributions. While implementing Langevin sampling with LTMs using the same decoder helps mitigate posterior col- lapse, it produces lower quality generations compared to the variational Bayes learning approach.
Table 3: Evaluation of unconditional generation. LTMs achieve comparable performance on Gen PPL and Entropy while offering substantially faster generation speed.
Model
Gen PPL(L)
Entropy(1)
Samples/<(t)
GPT-2-Medium
229.7
6.02
0.053
G-rg2-Large
60.4
5.71
0.014
3.5. Probing Results on Latent Thought Vectors
LTM-Small
178.7
5.67
0.23
LTM-Medium
104.5
5.62
0.14
We investigate how semantic information distributes hier- archically across LTMs' layers through progressive recon- struction experiments, where we evaluate reconstruction accuracy by progressively including layers of latent thought vectors from bottom to top.
LTM-Large 87.1
5.61
0.08
both approaches using standard multinomial sampling. As shown in Table 2, LTMs maintain nearly equivalent per- formance even with greedy decoding, suggesting that the per-token distribution conditioned on latent thought vee- tors, ( z x x<<))), is highly concentrated. We include additional samples Appendix A.5.
The study in Fig. 10 reveals that LTMs process information in layered fashion, with different model sizes showing distinct hierarchical patterns. For the 12-layer L.TM model with 96 latent thought vectors, we observe distributed infor- mation processing with steady increases in reconstruction accuracy through bottom and middle layers (1-8), reaching approximately 65% accuracy. This is followed by crucial synthesis at top layers (9-10), where accuracy jumps dramat- ically to over 95%. The case study in Fig. 11 demonstrates this clear semantic progression. Bottom layers produce scat- tered, disconnected terms, middle layers develop structural coherence with emerging phrases and descriptive elements, while top layers achieve complete semantic integration and perfect reconstruction. This hierarchical organization re- veals distinctive "synthesis layers" in the top of the network that integrate information from earlier layers, showing how LTMs encode and process semantic information through the layered thought vectors. See Appendix B for more details.
Unconditional Generation. One principled metric eval- uate unconditional generation is
(ps (xx(()))))))))))())))) (P)ata (x)) pp) log Pdats - (pg).
As both terms are intractable, alternative metries have been proposed: Dieleman et al. (2022) introduce generative per- plexity (Gen PPL), which approximates Pdata in the first term using a larger language model, while Zheng et al. (2024) propose token-level entropy to approximate the sec- ond term and detect mode collapse. We use GPT-2-XL as the proxy for Pdata to calculate the Gen PPL.
Table 3 presents the results. While SEDD-M achieves Gen PPL of 32.63 with 1024 sampling steps and an entropy of 5.27, we follow Zheng et al. (2024)'s 's recommendation to consider only baselines with entropy exceeding 5.6. Under these criteria, LTM-Large achieves performance compara- ble to GPT-2-Large on both metrics while providing 5x faster sampling speed. Experiment details can be found in Appendix A.3, with additional samples in Appendix A.4.
4. Limitations: Prior and Reward
Learnable Structured Prior Models. Our current work as- sumes simple Gaussian prior model for the latent thought vectors. The only struetural design we employ is to assume separate sets of thought vectors that cross-attend to different layers of Transformer decoder. While such a simple prior model is a suitable starting point for initial systematic inves- tigation, much can be gained by imposing a more structured and learnable prior model with more interpretable latents,
3.4. Ablation Studies
We explore inference strategies for LTMs. Our VAE base- line, which employs an identical decoder and 12-layer
8

---

Latent Thought Models
Pa(z). For instance, language of thoughts (Fodor, 1975) may be modeled by a latent reasoning model that gener- ates chain of latent thought vectors in the latent space, transforming posterior inference into process of parsing, formalization, compression, and understanding.
man, 2001). Procedural memory: Involved in learning and executing cognitive and motor skills. In language, it is linked to grammar rules, regular morphology, and syn- tax (Ullman, 2004). In our model, z parallels declarative or episodic memory, representing explicit facts and events. The decoder generator corresponds to procedural memory, embodying the implicit rules and patterns for language gen- eration and comprehension.
Reward or Verifier Models in Latent Space. Our model currently lacks a reward model or verifier model defined in the latent space, Py (rlz), which can be used guide the optimization of z as a form of inference-time computation for reasoning. In our recent work on latent plan transformer models, we have applied such models to offline reinforce- ment learning (Kong et al., 2024b) and online optimization for molecule design (Kong et al., 2024a).
Language of Thought (LOT) Hypothesis. Proposed by Fodor (Fodor, 1975), the LOT hypothesis posits that think- ing occurs in a mental language with its own syntax and semantics. This "mentalese" is theorized to underlie our ability to learn and use natural languages. Recent work has explored computational implementations of LOT-like structures in cognitive modeling (Piantadosi et al., 2011) and program induction (Lake al., 2015).
5. Related Work
Autoregressive and Diffusion Language Modeling, LLMs based on autoregressive modeling, like GPT-3 (Brown et al., 2020), PaLM (Chowdhery et al., 2022) and their succes- sors, have achieved tremendous successes across a wide range of language tasks. On the other hand, discrete diffu- sion (Austin al., 2021) arises as an alternative for language modeling (Lou al., 2024; Shi al., 2024; Sahoo al., 2024) recently. A popular version is masked diffusion that iterative transits tokens into masked state in the forward process. It is closely related to any-order autoregressive models (Uria et al., 2014; Hoogeboom et al., 2022).
Complementary Learning: Fast and Slow. The dual-rate learning can be connected to the theory of complementary learning systems (McClelland et al., 1995), which suggests that the hippocampus supports rapid learning of specific experiences, while the neocortex facilitates slower learning of general knowledge.
Test-Time Computation. The field of language modeling has seen growing interest in adaptive computation also known as dynamic evaluation - a method to enhance test- time performance. Graves (2016) pioneered this approach to introduce the Adaptive Computation Time mechanism for recurrent neural networks, enabling dynamic adjustment of per-step computation. The concept evolved with Krause et al. (2018), who developed dynamic evaluation to adapt model parameters at test time based on recent context. A recent advancement came from Kasai et al. (2022), who in- troduced a parammeeettic cache mechanism that efficiently adapts to local context during test time without modifying model parameters.
Variational Bayes Language Modeling. Bowman et al. (2016) introduce a variational autoencoder for text genera- tion. Building on this, Xu & Durrett (2018) propose the use of von Mises-Fisher distribution in VABs. Li et al. (2020) present OPTIMUS, large-scale pretrained deep latent vari- able model for natural language. Pang & Wu (2021); Yu et al. (2022); Xu et al. (2023) study language modeling with learnable prior model.
Large Language Models with Explicit Latent Space. Ze-
likman et al. (2022); Hu et al. (2023); Phan et al. (2023) repurpose token-level LLMs to generate latent chains of thought. Hao et al. (2024) repurpose the hidden state of Transformers as continuous latent space. They are all post- training methods that demonstrate the advantages of explicit latent learning. Concurrent to our work, The et al. (2024) train generative models for the latent embedding of pre- trained auto-encoder.
6. Conclusion
In this paper, we introduce Latent Thought Models (LTMs), which incorporate explicit latent thought vectors that fol- low explicit prior models in latent space. We develop a novel dual-rate optimization algorithm for training these models and conduct extensive empirical investigations on their properties, with particular focus on scaling behaviors along inference steps and latent dimensionality. Our ap- proach draws inspiration from cognitive science theories, including aeclaaaaive-pooceee guage of thought hypothesis, and complementary learning systems. Our work lays the groundwork for further develop- ment of more structured and interpretable prior models and reward-verifier models in the latent space for the purpose of reasoning and planning.
Declarative- Model in Cognitive Science. The declarative- procedural model, primarily developed by Ull- man (Ullman, 2004), offers a cognitive framework for un- derstanding language processing and memory. This model posits two distinct but interacting systems: Declarative mem- ory: Responsible for storing and recalling facts, events, and arbitrary associations. In language, it is associated with vocabulary, irregular forms, and idiomatic expressions (UIl-
memory systems, the lan-
9

---

Latent Thought Models
Acknowledgment
with pathways. Journal of Machine Learning Research, 24(240):1-113, 2023.
thank Ruiqi Gao and Kevin Murphy for insightful dis- cussions and valuable suggestions. Y. W. was partially sup- ported by NSF DMS-2015577, NSF DMS-2415226, and gift fund from Amazon. We gratefully acknowledge the support of Lambda, Inc. for providing the compute for this project.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L.,, Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 14/68, 2021.
Cohan, A., Dernoncourt, F., Kim, D. S., Bui, T., Kim, S., Chang, W., and Goharian, N. A discourse-ave aware attention model for abstractive summarization of long documents. arXiv preprint urXiv::804,000
Impact Statement
Our paper investigates new model class for language mod- eling with explicit latent thought vectors and inference-time computation. This model class has the potential to learn more explicit internal representations and enable more ex- plicit reasoning and planning based on such representations.
2018.
Dao, T., Fu, D., Ermon, S., Rudra, A., and Ré, C. Flashat- tention: Fast and memory-efficient exact attention with io-awareness. Advances Neural Information Process- ing Systems, 35:16344-16359, 2022.
References
Dieleman, S., Sartran, I.,, Roshannai, A., Savinov, N., Ganin, Y., Richemond, P. H., Doucet, A., Strudel, R., Dyer, C., Durkan, C., et Continuous diffusion for categorical data. arXiv preprint arXiv:2211.1508) 2022.
Austin, J., Johnson, D. D., Ho, J., Tarlow, D., and Van Den Berg, R. Structured denoising diffusion models in discrete state-spaces. Advances in Neural Information 2021.
Processing Systems,
Finn, C., Abbeel, P., and Levine, Model-agnostic meta- learning for fast adaptation of deep networks. Interna- tional conference machine learning, pp. 1135- PMLR, 2017.
Ba, J., Hinton, G. E., Mnih, V., Leibo, J. Z. and Ionescu, C. Using fast weights to attend to the recent past. In Advances in Neural Information Processing Systems, vol- ume 29, pp. 4331-4339, 2016.
Fodor, J. A. The Language Thought. Harvard University Press, 1975.
Blei, D. M. Kucukelbir, A., and McAuliffe, J. D. Varia- tional inference: A review for statisticians. Journal of the American Statistical Association, 1 12((518):559-87) 2017.
Freitag, M. and Al-Onaizan, Y. Beam search strate- gies for neural machine translation. arXiv preprint arXiv:1702.01806, 2017.
Bowman, S. R., Vilnis, L., Vinyals, O., Dai, A. M., Joze- fowicz, R., and Bengio, S. Generating sentences from continuous space. In Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learn- ing. pp. 10-21, 2016.
Gokaslan, and Cohen, V. Openwebtext cor- pus.
http://SkylionD0// maa.00..itthb.io.. OpenteeTeextCorpuse 2019.
Graves, A. Adaptive computation time for recurrent neural networks. arXiv preprint arXiv:1603.0800 089983, 2016.
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. Dhariwal, P., et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901, 2020.
Guo, D., Yang, D., Zhang, H., Song, Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi. X., et al. Deepseck-rl: In- centivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501. 2025.
Chelba, C., Mikolov, T., Schuster, M., Ge, Q., Brants, T., Koehn, P., and Robinson, T. One billion word benchmark for measuring progress in statistical language modeling. arXiv preprint arXiv: 13/2.3003 2013.
Han, X., Kumar, S., and Tsvetkov, Y. Ssd-Im: Semi- autoregressive simplex- based diffusion language model for text generation and modular control. arXiv preprint arXiv:2210.17433 2022.
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., et al. Palm: Scaling language modeling with pathways. arXiv preprint 22Xiv:2204.02311 2022.
Hao, S., Sukhbaatar, S., Su, Li, Hu, Z., Weston, J., and Tian, Training large language models to reason in a continuous latent space. arXiv preprint arXiv:2412.06769 2024.
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. Palm: Scaling language modeling
10

---

Latent Thought Models
Hoffman, M. D., Blei, D. M., Wang, C., and Paisley, Stochastic variational inference. Journal of Machine Learning Research, 2013.
inference. Advances in Neural Information Processing Systems, 37:123379-1124001 2024b.
Krause, B., Kahembwe, E., Murray, I., and Renals, Dy-
Hoffmannn, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E.,, Casas, D. d. L., Hendricks, L. A., Welbl, J.. Clark, A., al. Training compute-optimal large language models. arXiv preprint arXiv:2203. 2022.
namic evaluation of neural sequence models. In Proceed- ings of the 35th International Conference Machine Learning, pp. 2766-2775 2018.
Kumaran, D., Hassabis, D., and McClelland, J. L. What learning systems do intelligent agents need? complemen- tary learning systems theory updated. Trends in Cognitive Sciences, 20(7):512-534, 2016.
Holtzman, A., Buys, J., Du, L., Forbes, M., and Choi, Y. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09775 09751, 2019.
Lake, B. M., Salakhutdinov, R., and Tenenbaum, J. B. Human-level concept learning through probabilistic pro- gram induction. Science, 350(6266); (1022266):333222 1332-1338, 2015.
Hoogeboom, E., Gritsenko, A. A., Bastings, J., Poole, B., van den Berg, R., and Salimans, T. Autoregressive dif- fusion models. In International Conference on Learning Representations, 2022.
Li, C., Gao, X., Li, Y., Li, X., Peng, B., Zhang, Y., and Gao, J. Optimus: Organizing sentences via pre-trained modeling of a latent space. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Prucessing (EMNLP), pp. 4678-4699, 2020.
Hsu, P.-..., Dai, Y., Kothapalli, V., Song, Q. Tang, S., Zhu, S., Shimizu, S., Sahni, S., Ning, H., and Chen, Y. Liger kernel: Efficient triton kernels for Ilm training. arXiv preprint arXiv:2410.1090 2024.
Li, S., Du, Y., Tenenbaum, J. B., Torralba, A., and Mor- datch, I. Composing ensembles of pre-trained models via iterative consensus. arXiv preprint arXiv:2210.11522. 2022.
Hu, E. J., Jain, M., Elmoznino, E., Kaddar, Y., Lajoie, G., Bengio, Y., and Malkin, N. Amortizing intractable infer- ence in large language models. In The Twelfth Interna- tional Conference on Learning Representations, 2023.
Loshchilov, I. Decoupled weight decay regularization. arXiv preprint 1rXiv:1711.05101 2017.
Jordan, M. I., Ghahramani, Z., Jaakkola, T. S., and Saul, L. K. An introduction to variational methods for graphical models. Machine learning, 37(2):183-233 1999.
Loshchilov, I. and Hutter, F. Decoupled weight decay reg-
ularization. In International Conference on Learning Representations, 2019. URL https: http::: /ttp://ttttt:://eeeeeeeeia /ttp////ppeeee.eee//eee/ net/ / fow.mmiddraiii 6
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint arXiv:2001.08300 2020.
Lou, A., Meng, C., and Ermon, Discrete diffusion mod- eling by estimating the ratios of the data distribution. In Forty-first International Conference Machine Learn- ing, 2024.
Kasai, J., Pappas, N., Peng. H., Cross, J., and Smith, N. A. Deep encoder, shallow decoder: Reevaluating non- aon-autorrerrssive machine translation. In International Conference on Learning Representations, 2022.
Lucas, J., Tucker, G., Grosse, R., and Norouzi, M. Don't blame the elbo! linear vae perspective on posterior collapse. In Advances in Neural Information Processing Systems, volume 32, 2019.
Kingma, D. P. and Ba. J. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980, 2014.
Marcus, M.. Santorini, B., and Marcinkiewicz, M. A. Build- ing a large annotated corpus of english: The penn tree- bank. Computational linguistics, 19eissi19900 19(2):313-330, 1993.
Kingma, D. and Welling, M. Auto-encoding variational bayes. arXiv preprint
2013.
Kong, D., Huang, Y., Xie, J., Honig, E., Xu, M., Xue, S., Lin, P., Zhou, S., Zhong, S., Zheng, N., et al. Molecule design by latent prompt transformer. Advances in Neu- ral Information Processing Systems, 37:89069-89097, 2024a.
McClelland, J. L., McNaughton, B. L., and O' ReReily, R. C. Why there are complementary learning systems in the hip- pocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. Psychological Review, 102(3)419,, 1995.
Kong, D., Xu, D., Zhao, M., Pang, B., Xie, J., Lizarraga, A., Huang, Y., Xie, S., and Wu, Y. N. Latent plan trans- former for trajectory abstraction: Planning as latent space
Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models. arXiv preprint arXiv: 0609.008733. 2016.

---

Latent Thought Models
Murphy, K. P. Machine Learning: A Probabilistic Perspec- five. Adaptive Computation and Machine Learning series. MIT Press, Cambridge, MA, 2012.
Ullman, M. T. The neural basis of lexicon and grammar in first and second language: The declarative/proccddraal model. Bilingualism: Language and cognition, 4(2): 105- 122, 2001.
Pang. B. and Wu, N. Latent space energy-based model of symbol- vector coupling for text generation and classifica- tion. In International Conference on Machine Learning, pp. 8359-8370. PMLR, 2021.
Ullman, M. T. Contributions of memory circuits to language: The declarativeppoocedural model. Cognition, 92(1-2): 231-270, 2004.
Pang, B., Nijkamp, E., Han, T., and Wu, Y. N. Generative text modeling through short run inference. In Merlo, P., Tiedemann, J., and Tsarfaty, R. (eds.), Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, 2021.
Uria, B., Murray, I., and Larochelle, H. A deep and tractable density estimator. International Conference on Ma- chine Learning, pp. 467-475. PMLR, 2014.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L.,, Gomez, A. N., Kaiser, E.., and Polosukhin, I. Atten- tion is all you need. In Advances in Neural Information Processing Systems, volume 30, pp. 5998-6008, 2017.
pp.
Paperno, D., Kruszewski, G., Lazaridou, A., Pham, Q. N., Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G., and Fernández, R. The lambada dataset: Word prediction requiring a broad discourse context. arXiv preprint arXiv:/606.000 06031, 2016.
Xu, J. and Durrett, G. Spherical latent spaces for stable variational autoencoders. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 4503-45015 2018.
Phan, D., Hoffman, M. D., Dohan, D., Douglas, S., Le, T. A., Parisi, A., Sountsov, P., Sutton, C., Vikram, S., and A Saurous, R. Training chain-of- variable inference. Advances in Neural Information Pro- cessing Systems, 36, 2023.
Xu, Y., Kong, D., Xu, D., Ji, Z., Pang, B., Fung, P., and Wu, Y. N. Diverse and faithful wwwledge-groundde dia- logue generation via sequential posterior inference. arXiv preprint arXiv:2306.0115 011513 2023. https: //arxiv. org/ 2 01153.
via latent-
Piantadosi, S. T., Tenenbaum, J. B., and Goodman, N. D. Bootstrapping in language of thought: A formal model of numerical concept learning. Cognition, 123(2):199- 217, 2011.
Yoon, J., Kim, T., Dia, O., Kim, S., Bengio, Y., and Ahn, S. Bayesian model-agnostic meta-learning. Advances in neural information processing systems, 31, 2018.
Yu, P., Xie, S., Ma, X., Jia, B., Pang, B., Gao, R., Zhu, Y.,
Pillutla, K., Swayamdipta, S., Zellers, R., Thickstun, J., Welleck, S., Choi, Y., and Harchaoui, Z. Mauve: Mea- suring the gap between neural text and human text using divergence frontiers. Advances in Neural Information Processing Systems, 34:4816 4828, 2021.
Zhu, S.-C., and Wu, Y. N. Latent diffusion energy-base model for interpretable text modeling. arXiv preprint Stti:2206.05899) 2022.
Zelikman, E., Wu, Y., Mu, J., and Goodman, N. Star: Boot- strapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35:15476-15488 2022.
Radford, A., WU, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Zhang, B. and Sennrich, R. Root mean square layer nor- malization. Advances in Neural Information Processing Systems, 32, 2019.
Sahoo, S., Arriola, M., Schiff, Y., Gokaslan, A., Marro- quin, E., Chiu, J. T., Rush, A., and Kuleshov, V. Simple and effective masked diffusion language models. arXiv preprint arXiv:2406.00 075544 2024.
Zhang, X., Zhao, J., and LeCun, Y. Character-level convolu- tional networks for text classification. Advances in neural information processing systems, 28, 2015.
Shi, J., Han, K., Wang. Z., Doucet, A., and Titsias, M. K. Simplified and generalized masked diffusion for discrete data. arXiv preprint arXiv:2406.00 04329, 2024.
Zheng, K., Chen, Y., Mao, H., Liu, M.-Y., Zhu, J., and Zhang. Q. Masked diffusion models are secretly time- agnostic masked models and exploit inaccurate categori- cal sampling. arXiv preprint arXiv:2409.02008, 2024.
The, L.,, Barrault, I.., Duquenne, P.-A., Elbayad, M., Kozhevnikov, A., Alastruey, B., Andrews, P., Coria, M., Couairon, G., Costa-jussà M. R., et al. Large concept models: Language modeling in a sentence representation space. arXiv preprint Xiiv:2412.0882. 2024.
12

---

Latent Thought Models
A. Appendix
A.I. Model Details
We adopt flash attention (Dao et al., 2022) and the Liger kernel (Hsu et al., 2024) to accelcrate training and posterior inference. For the attention layers, we apply RMS layer normalization (Zhang Sennrich, 2019) and use SwiGLU as the activation function.
All LTMs have 512 hidden dimensions, 8 attention heads, and maximum sequence length of 1024. The latent thought vector z shares the same dimensionality as the hidden vectors, Our autoregressive generator uses sliding window size of 256. We employ rotary position embedding for both ground tokens and latent thought vectors in each layer.
We use the GPT-2 tokenizer for Wpeneee adding a single [EOS] token. We do not pad or truncate sequences. Instead, we concatenate documents and wrap them to a maximum length of 1024, inserting the [BOS] token between wrapped segments. Because Open WebText does not include predefined validation split, we follow Sahoo al. (2024) and reserve the last 100K documents for validation.
A.2. Training Details
We train all models using a "slow" learning rate of 4 10-4 followed by cosine decay schedule to 4 10-5, linear warmup schedule to the first 1000 iterations, and clip the gradient norm to during training. For the "fast" learning rate, we start from 0.3 and linearly increases to 0.34.
55 We also apply
We use AdamW optimizer (Loshchilov, 2017) with 31 = 0.9, and 32 = 0.95 to update the global parameters. We use Adam to update the latent thought vectors without introducing additional inductive bias in the optimization.
A.3. Experiment Details
Zero-shot Perplexity Following prior works in language modeling (Radford et al., 2019; Lou et al., 2024; Sahoo et al., 2024), we evaluate the zero-shot capabilities of LTMs by taking our models trained on Web and measuring perplexity on standard benchmarks. Specifically, we use the validation splits of Penn Tree Bank (PTB) (Marcus et al., 1993), Wikitext (Merity et al., 2016), One billion word benchmark (LMIB) (Chelba et al., 2013), Lambada (Paperno et al., 2016), AG News (Zhang et al., 2015), PubMed and Arxiv subsets (Cohan et al., 2018). We adopt the detokenizers used by Sahoo al. (2024) and insert an [EOS] token in between sequences in the dataset.
Arithmetic Reasoning GSM8K Each GSM8K problem consists of question, intermediate reasoning steps, and final solution. We evaluate both baseline models and LTMs the IK test set, using pass muss@5 accuracy in Li et al. (2022), For each problem, we generate five candidate solutions (each up to 50 new tokens) and consider the problem solved if any candidate matches the final solution.
For GPT-2 baselines, we use beam search with a beam size of 5. contrast, LTMs infer z five times per prompt, and then draw multinomial sample for each inference. In few-shot scenarios, we concatenate examples as prompts and generate responses accordingly.
Conditional Generation Following Lou et al. (2024) and Han et al. (2022), we evaluate conditional generation on 1,000 samples from the OWT validation set. For each ground sample, we generate five new sequences by conditioning on the first 50 tokens and then generating 50 additional tokens. We then compute MAUVE on these generated samples. All baseline results in Table 2 are taken from Lou et al. (2024).
Unconditional Generation We evaluate the unconditional generation capability of LTMs using the generative perplexity metric proposed by Dieleman et al. (2022). Specifically, we prompt LTMs with a single [BOS] token to produce 64 sampled sequences of length 1024 with greedy decoding (top-k = 1, temperature= 1). We then measure the perplexity of these sequences using GPT-2-XL as the evaluation model. While Lou et al. (2024) and Sahoo al. (2024) use GPT-2-Large for evaluation, we opt for GPT-2-XL ensure a fair calculation on the Gen PPL of GPT-2-Large. All evaluations are performed with batch size of 8.
13

---

show is always Latent Thought Models
A.4. Samples for Unconditional Generation
What is this more like an angry person's life?
From this year's season, the most recent episode of comic season has come out of nowhere. But it's year of serious drama. It's still fun to watch. But it's not year of story.
Dead Future: A True Story, like any other medium, is just an adaptation of the story of television show. It's story about story that relives years of story, and the story itself has big degree in intelligence. The series was never good story. But, as the series grew popular and with interest and relives as much as anybody else, the characters are a lot smaller.
It's not that the series has any particular focus on what it's like to an actor, and even if it's something you might be interested in doing something that might foster a deeper understanding the story.
But it's hard say if the story could be an adaptation for another long time. It's series that focuses on a story that has gone beyond the story of the past, and it doesn't 't have any distinctive characteristics to be seen.
Dan Abrams is a fan and fan of writing and voice in series of comics and television shows, and he also created a very original series about the story of The Wire. He was born in Sydney in 1991 and grew up in Sydney, the family home of a well-known Melbourne businessman.
So he's 's been a regular on a television show since 2003 - and he's also very regular character. But he hasn't always been much invested in storytelling. His first TV show is about exploring relationships and co-created stories with people in the community.
So far, the stories about people who work in the comics and don't end up being familiar with the comics, Dan Abrams is a much more relaxed character. He's not just a "fun" character that's been given yet another new set of episodes.
"I'm just a masterful man," said. "I can't say I'm happy with my life. I'm happy with my life."
The second half the show frequently appears somewhere between Jon and Dana. He's playing with Brian O' Malley in the first season, but he isn't 't shy about making deal with that guy.
"I can't t say that's going to be funny." he said. "The best part is that when you get know him and you't going to get know him, and I' m happy with him and I'm happy with his life."
The show ended in some awkward scenes, but there was little I could tell about the past. There was no line of dialogue that led to the end of the episode, but there was no line of dialogue that left Jon unanswered for the second season's are to end.
Perhaps the 'fun' series had been set in motion over the last two seasons, but wasn't 't entirely self-aware. As Jon hobbled with the plot and has become angrier about whether or not he's going to be fired, he was quickly moved forward and out of power.
"I was not comfortable with that," Jon said. "The question of whether not I'm willing matter of time."
But it wasn't it easy to come up with kind of self- seff-perpetuating character. But Jon and Jon's relationship grew increasingly strained, and many fans felt the show was more stable than ever before.
"We're playing very young guy who can't even play his character anymore," Jon said. "But that's not what I'm saying, but it's not what I'm saying, I'm not saying I'm giving that."
It's hard imagine how the show would work if Jon's 's character had been found.As we go through some of the most popular anime series, I've found myself constantly being uninterested in anime content. There is way say that, because it is series that does pretty well, and suspect that one of the most popular series is based on anime. There's 's need to worry about that at all.
Figure 6: Unconditional sample for LTM-Small.
14

---

Latent Thought Models
One of the most notorious Patriots litigators, Ted Gronkowski mixed up with Chris now openly taking an in-kind tirade the offense. Angry over the performance of Ted Gronkowski, Patriots' running back for the Super Bowl win over Cincinnati Bengals, jostly, we rate him woefully above than was Opher by Rich Fisenblick in this week's 's roundup. Even though Gronkowski sparked an even more fury with criticism, he continued to rant off the opportunities created by the Dallas Cowboys. Gronkowski allowed 277 yards or less to tag as wide receiver, but his fans only showed up when Cowboys fans broadcast to the Spots at city hall to mark the Aggies' feast of Oxnard. Brady fans should respect the Brady matchup line for Gronay when that against the Dallas Cowboys, the ones which dominated the day. When Ahmad themselves exploded in CBS's Morning Report this week, it was a glitch in the statistics that it could only be mentioned by a 2 to 1 person bracket. Ahmad was at his best the Browns so far with a head coaching job that included J.J. Watt, Drew Brees, Hunter Henry, Charles Hasson, Earl Thomas, and Malcolm McDaniels. However, the Browns got a surprise offensive breakdown when the Falcons stepped up from within the five-headed dominance that did little to an elite offense like USC. In contrast Brady's 73 wins showing in his next game, Ahmad was both able tank under a one-point situation in which he turned 500+ calls and never showed too much during his coverage. Ahmad came off as late-stage, catalysts in the scheme of his 49-y rush for a 44-yard touchdown with one touch, and appeared to do so to celebrate with a game like that during the game. You count that game, and there's 's many un un-becacious numbers to fall in the end zone against Brinson and edge-cut-keepen like the production figures of many exploring zone led by Ahmad and the no-hards. Other garbage-pr players are even more pricyies for Fort Worth veteran Boogie Miller.
With the offensive lows but, at the very least, Ahmad helped build truly dynamic offense teams that were all serving the same demands, being put in the same building at high rate. Newton, running back, wide receiver and wide receiver, led Newton in the third most important mark of his career. Four interceptions, including, quite simply, reverses pretty sloppy bob defense, was shorted. There was a shot by Garrett Gardner to show off his exceptional ability to harass and duck from there. Through 4'12 and over, Gardner unleashed a barrage of ringing seconds during a 10-yard burst, and, eventually, abandoned one of the then-prize quarterback pressures Newton had given him. Needless to say, these screams never really occurred to passers Burge. Every touch injury created fumble return that might explain Burge these days.
Aside from truly dynamic passing linebackers- like the legendary lefty Michael Snterreo-JJabeet calm and, yes, slow motion, leading gas canister. One of the Texans is simply making the fourth-sevent ranked defensive line all over the league from outside of theide, Kevin Kynellish- the now potent blue six. You can't generate a quarterback from nowhere that's too much of a prodigious speed to win a game. His speed also tells you just how far this can go for greater leverage. On top of all the crying over the cigarette, this line is one of the sweet places both Xavier and his defense have earned, where every game was run together.
The truth is defense is particularly important. Aaron Rodgers never moaned anything for anything over before beating Carolina's 's Joey Robinson in 2013. He's the best player on the football Eiel though it's top-10 football team that needs to cry out for going over and fighting like nothing, in the end they were in luck when the first benefit was paid off. Every team laughs their awful quarterback antics every lumps quarters that separates teams around them. If these tiny mistakes somehow make you even seem like a kind of mark on the past, soon you'll '1l see defense does. CHARGE CANNAPS THAN WHAT WELL THAN the Panthers could be proud of in today's pictures.
When anyone does an offense pressureily putting toward the line, you need an excuse to drop back and conduct miracle. Top of the line is Aaron Brooks, who is huge leap forward next to Seattle's next post-reception swing, Heisman Trophy-winning and career career. Giants? The team knows this?! That's manda-good-but-ba--ttee excuse to say.
Figure Unconditional sample for LTM-Medium.
15

---

Latent Thought Models
(2) Affirm a hospital leave. all may feel *better* that the intervention is there. However has been taken choke off the baby. It painful and painless. It can render you "less-attractive" if you assume your situation is there. Just we can point out any imperfections with which you have stuck, hoping for recommendation later.
After all the beating, forgetting more than you know, adore My Baby turns out to be good for her life but the patient who caused it manages to cause it. And as a pediatric practitioner, she needs to get at least a sniff at what know about her baby. To start, the patient need to appreciate the fact that early sometime has not happened and home-cooked bread is missing. Brian Carr, BCCI Bournemouth. The United States has total dependence on most fossil fuels, including natural gas of every form, and continues to hold on nature's greatest fossil energy addiction, by killing as many 3,000 Americans, scientists say. Alina Minerva Venable's colleague, Stefan Megaläke of the University of Götecschmid in Munich, Germany, says that using renewable sources such as renewable energy, technology based on bad weather, to help cut CO2 by 39 percent, is mistaken. When exercise supplies turn on CO2 gas it releases methane and halts the CO by up to 75 percent. But the emissions holds up as a by-product - using just enough gas to cool down meteorologists and crooks are far from 100 percent. Almost everything, through every storm, has been exceeded only by CO 2.1 has tripled or tripling worldwide on weather systems on hundreds of billions of miles of land. The United States an exception. In fact, scientists sometimes wonder if climate change will benefit just well.
Some of the countries rich in green fossil fuels have buckled under government environmental regulation, seeking even more than 15 percent of our active fossil-fuel use. Ideally, they could support continued progress in clean energy policies so that fossil sources keep to rock and that energy can produce far more than their attempts to supply new fossil fuels. But the two proposals that raise goals for humanity are a continuing thorn in the side of scientists alarmed by rapidly increasing federal programs for hundreds billions of dollars research and development. Elsewhere, heads of countries have become increasingly hospitable, faithful users. And in France, where 80,000 individuals lay their loved ones the base of cannon, cannon, Fortunately for our care and privacy, recent environmental studies widely discovered some of the worst abuses that the United States has been experiencing the grave levels of growing carbon emissions from below.
Most Canadians are disgusted with warming land. But should they let the huge quantity of 20,000 barrels per year carry on, society won't get to living on the one thing the United States led the global megadunnel, which can all but mandate its own unimaginable task. We don't 't need to keep creeping the self-infficted Mephistophe-Berttee Bertrand Aristide stomach the degree which he has been behaving consistent with reality.
"The ideal application of science and natural science right now involves reporting people practices that deviate from reality into the confines of evolutionary evolution," " Dr. Ann Paxton, director of the Natural Resources Defense Council's Bureau of Meteorology, or Bioethics, makes explicit this assertion. "For even a object has its staying power," she says.
That said, so-called green asteroids hit record levels 2005 or atmospheric cloudbursts and CME- bursts when they're 're amphitonic: they have hit rock basins with unusually light atmosphere that air droplets in the crust were understung in 1952 couple of years carlier when the asteroids eventually crossed the atmosphere and reached cesarean limit. A scale back to 66 years in 2002 and memory of when its too light days to challenge Scanlan- Tri Garin to see chutes meant it acted entirely in line with reality? How could scientists determine that such a feat possible?
Rather than waiting for Creation to pay more attention to scientific questions so big, there are at least a handful of families the Greenpeace science director Terence Benton who have the capacity to sidestep science to express its genetic data.
For many families who simply not knowing because only they know the Creation Museum is there, they wonder if their loved ones have had something akin to science wrong - rather faith in the nature of nature's 's intricate organization, their cognitive beings leading them into the galaxy. But they need and that's 's why, each couple has had to make their displeasure public. Tyrannosaurus rex's wing on the other hand struck off with a shattering force.
Figure 8: Unconditional sample for LTM-Large.
16

---

Latent Thought Models
A.5. Samples for Conditional Generation
The man accused of plowing into group of people at the South By Southwest festival has been charged. "A man suspected of drunken driving is charged with capital murder in the deaths of two people the South by Southwest conference in Austin, Texas Dec. 22. He faces "capital murder" charges, plus capital murder. The 15-year-old victim was strangely drunk when he drove into the Austin district building in big accident. He is married to the
Figure 9: Conditional sample for Lagee Generated tokens in blue.
B. Probing the Latent Thought Vectors
To understand how LTMs hicrarchically encode information, we evaluate reconstruction accuracy by progressively including layers of latent thought vectors from bottom top across 200 samples from the Open WebText validation set. We test two model configurations shown in Fig. 10: LTM-Medium (6-layer, 24 latent vectors with 4 per layer) and LTM-Large (12-layer, 96 latent vectors with per layer), measuring how reconstruction accuracy improves more layers during text generation. Additionally, present detailed case study in Fig. 11 that demonstrates the specific reconstruction patterns emerging at each layer of latent thought vectors.
incrementally include
B.1. Progressive Layer Inclusion
Distribution of Reconstruction Accuracy with Progressive Layer Inclusion
Distribution of Reconstruction Accuracy with Progressive Layer inclusion
Reconsruction Accuraty
Reconsiruction Accuracy
Number Layers Included
Number of Leyers Included
Figure 10: Left: 6-layer LTM-Medium with with 24 latent vectors (4 per layer). Right: 12-layer LTM-Large with 96 latent vectors (8 per layer). Distribution of Reconstruction Accuracy with Progressive Layer Inclusion for LTM models. The plots show how reconstruction accuracy improves as layers are progressively included from bottom to top, measured across 200 sequences from Web validation set. (a) 6-layer LTM-Medium shows gradual improvement through layers 1-5 (~55% accuracy) followed by sharp jump layer complete reconstruction. (b) 12-layer L.M--Large demonstrates more distributed information processing with steady increases through layers 1-8 (~65%), followed by crucial synthesis layers 9-10, reaching >95% accuracy. This reveals the hierarchical nature of LTMs' latent representations, with deeper models distributing information more gradually across layers and featuring distinctive "synthesis layers" that integrate information from earlier representations.
17

---

Latent Thought Models
B.2. Case Study
Progressive Inclusion of Latent Thought Vectors (z)
Using Layers 1-3 Only (22% Accuracy):
The to its The ... of
of The to its surface,
sea
and sea in the
in
with
and
the
as ... a.. of the sea ... of ... sea of be....... of with the ... and ... the
the surface.
and the of
Using Layers 1-6 Only (30% Accuracy):
The through way... the along each
to while by blue
beneath sea and ... into
sea of carried tide. A ... waters had ... of secluded
of Wild
and
with of light across ... ... surface of the occan. the boat as more than a..... the horizon, procession of ... seabirds in its wake. The to exist in perpetual of by tide, and season...
... between
the began to
to and the ... of...
Using Layers 1-9 Only (65% Accuracy):
its way along the cliffs, revealing new with turn mist 10 into a gradient of silvery ..... air carried... ... distant of gulls and in ... of purple, interrupted by ... yellow gorse flowers. where the trail ... toward ... cove, its crescent
between sea and of ... against stone. heather... ...
visible only at low tide. The ... waters had ... a ... of tide each a ... seawater water, while tiny- ... scuttled between ...
with life.
Overhead, the began to ... through the casting ... of light across ... undulating surface of the ocean. ... the ... ... ... appeared as little more than against the horizon trailing procession of opportunistic seabirds in its wake. The tide, and season... somehow unchanged, indifferent to concerns and the
exist
Using Layers 1-10 Only (99% Accuracy);
coastal path wound its way along the rugged cliffs, revealing new vistas with each turn. Morning mist clung to the landscape, softening the boundary between sea and sky into a gradient of silvery blues. Salt-laden air carried the distant cries of gulls and the rhythmic percussion of waves against stone. Wild heather painted the hillsides in swathes of purple, occasionally interrupted by the defiant yellow of gorse flowers. I paused where the trail dipped toward secluded cove, its crescent of golden sand visible only at low tide. The receding waters had revealed tapestry of tide pools, each miniature universe teeming with life. Emerald seaweed swayed in crystalline water, while tiny crabs scuttled between cernacle-encrutee rocks. Overhead, the sun began to burn through the haze, casting diamonds of light across the undulating surface of the ocean. In the distance, a fishing boat appeared as little more than a silhouette against the brightening horizon, trailing a procession of opportunistic seabirds in its wake. The landscape seemed to exist in a perpetual state of change-shaped by wind, tide, and season- -et somehow timeless, indifferent human concerns and the passage of years.
Using All Layers (1-12) (100% Accuracy):
The coastal path wound its way along the rugged cliffs, revealing new vistas with each turn. Morning mist clung the landscape, softening the boundary between and sky into gradient of silvery blues. Salt-laden air carried the distant cries of gulls and the rhythmic percussion of waves against stone...
Figure 11: Progressive reconstruction of text using latent thought vectors from a 12-layer LTM. This figure displays only the correctly reconstructed words at each layer, showing how text accuracy improves as more layers are included. Dots (...) represent incorrect or missing words. Color coding: purple for partial reconstructions and orange for near-complete or complete reconstructions. At layer 0-3 (22% accuracy), only scattered words match the original. By layer 0-6 (30%), more structural elements emerge, including some phrases about the ocean and landscape. Layer 0-9 (65%) shows substantial improvement with coherent phrases and key descriptive elements. Complete accuracy (100%) is achieved with all 12 layers. This progression demonstrates how semantic information is hierarchically distributed across the model's 's latent space.
18


Reviews:
[{"review_id": "iO2b8Zx8mK", "reviewer": "Reviewer_FqJd", "summary": "This work presents novel Latent-Thought Language (LTM) class of models, where the explicit latent vector is introduced to guide the generation of tokens. The model is optimized withing the variational bayes framework, using faster learning rate for latent vector distribution parameters and slower rate for learning decoder parameters. Empirical evaluation against autoregressive and diffusion models highlighted few results: increased parameter efficiency, improved validation perplexity, and showing the few shots in-context reasoning capability.\n\n## update after rebuttal\nAfter reading other reviews and author responses, I decided to keep my initial grade.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*Thank you for your insightful comments and for recognizing the novelty and strong performance of our work. We address your concerns point-by-point as follows.*\n\n**1. Scalability.**\n\nOur Latent Thought Models (LTMs) scale along two primary dimensions: model size and the number of inference steps ($T_{fast}$). This parallels scaling model size and scaling the length of CoTs in Auto-Regressive (AR) models. Crucially, as discussed in Sec. 2.5, the FLOPs/token cost from $T_{fast}$​ scales linearly, similar to CoT length cost. Therefore, LTMs pose no fundamental asymptotic barriers to scaling beyond those of current LLMs.\n\nIn practice, we do acknowledge the engineering challenge of building robust infrastructure for large-scale gradient-based Variational Inference (for pre-training and deployment) and designate this as future work.\n\n**2. Inference expense and latency compare to other LLMs**\n\nOur analysis in Sec. 2.5 indicates that the inference expense (FLOPs/token) scales linearly with the number of inference steps, $T_{fast}$​. To offer a concrete comparison: performing 16 inference steps with our LTM-M model has latency comparable to generating ~8.7 CoT tokens in the iso-trainFLOPs/tok AR baseline, GPT2-L. \n\nIt is also necessary to highlight that practical inference expenses vary significantly depending on the task. While likelihood estimation (Table 1) processes the full sequence length, conditional generation (Fig 6 & Table 2) benefits from typically shorter prefill contexts. Furthermore, for unconditional generation (Table 3), there is no gradient backpropagation in LTMs at inference time, which may lead to considerable speedups compared to other LLM baselines whose decoder networks are significantly larger. \n\n**3. Were all compared models using the same GPT2 tokenizer?**\n\nYes. (line 612 in Appendix)\n\n**4. Explanation of \"lifted latent space\"**\n\nThank you for the feedback. \"Lifted\" is synonymous to \"latent\" in our context, which is meant to emphasize abstraction from \"ground\" tokens. We shall make this clear in revision. \n\n**5. Typo on page 2: \"is not as good as have separate sets\"**\n\nThanks. We will fix it in revision. \n\n**6. Discussion of other language models like Mixture of Expert(MoE), State Space Model(Mamba) etc.**\n\nWe will cite these LLM architectures in revision and discuss building our LTMs based on these architectures in future work. Our cross-attention layers that incorporate $z$ can be naturally inserted into these architectures. Since at each cross attention layer, $z_l$ is compact (consisting of only a small number of vectors), the cross-attention is very light, and this is actually quite consistent with the attention-less state vector in Mamba. We shall explore this direction in future work. \n\n*We will revise our paper based on your deeply insightful comments. Thank you for your positive evaluation!*", "claims_and_evidence": "The claims of increased parameter efficiency, improved validation perplexity, and few shots in-context reasoning capabilities are in general well supported by empirical evidence.", "methods_and_evaluation_criteria": "Yes, the proposed methods and evaluation criteria are appropriate for the problem. Well know benchmark sets are used: OpenWebText for pretraining,  seven open datasets for zero shot unconditional tasks and GSM8k for few shots evaluation of arithmetic reasoning.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Experiment design and analysis appear sound. Ablation is performed. Appropriate metrics like perplexity and MAUVE scores are used.", "relation_to_broader_scientific_literature": "Main idea is inspired by declarative procedural model (Ullman, M.T. Contributions of memory circuits to language: The declarative/procedural model. Cognition,92(1-2):\n 231–270,2004.) and language of though (Fodor, J.A. The Language of Thought. Harvard University Press,1975.). Most technical details are related to variational Bayes literature.", "essential_references_not_discussed": "Other flavors of Large language Models like Mixture of Experts Models as well as State Space Models were not considered or discussed.\nJiang, Albert Q., Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot et al. \"Mixtral of experts.\"\nGu, Albert, and Tri Dao. \"Mamba: Linear-time sequence modeling with selective state spaces.\"", "other_strengths_and_weaknesses": "Strong points:\n* The idea seems quite novel\n* Strong performance against GPT2, SEDD, MDLM and MD4.\n\nWeak points\n* Scalability might be difficult\n* While overall clear, the paper is \"dense\" at some moments. And certain phrases require more explanation, for example what is referred by \"lifted latent space\"", "other_comments_or_suggestions": "Typos:\nPage 2, right column, line 109: \"is not as good as have separate sets\"", "questions_for_authors": "1. How expensive is the inference, and how does the inference latency compares to other LLM approaches?\n2. Were all compared models using the same GPT2 tokenizer?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "P6y9P2Ksii", "reviewer": "Reviewer_YgR1", "summary": "This paper proposes a probabilistic language models called Latent-Thought Language Models (LTMs), which introduce explicit latent vectors to layers of transformers.  The authors claim that this setup yields new “scaling dimensions” beyond traditional LLMs, allowing more efficient use of training compute per token. They report gains in both sample efficiency and parameter efficiency, as well as improved validation perplexity, zero-shot language modeling, and conditional/unconditional text generation.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*Thank you for your constructive feedbacks. Below are our responses.*\n\n**1. Comparison to amortized VI and clarification on multi-step inference**\n\nWe wish to humbly clarify a possible misunderstanding in terms of the Variational Inference (VI) in our Latent Thought Models (LTMs). LTMs employ the classical variational Bayes (VB) framework rather than amortized inference (as in variational autoencoders (VAEs)).\n\nAs detailed in Sec. 2.2 and Algorithm 1 (line 165-183), for each training sequence $x$, the posterior $p(z|x)$ is approximated by a variational distribution $N(\\mu, \\sigma^2)$. Note that $(\\mu, \\sigma^2)$ are local parameters specific to each sequence $x$, and are optimized by maximizing the evidence lower bound (ELBO) for each $x$. To clarify the reviewer's comment about \"multi-step inference\": it refers to the iterative update of $(\\mu, \\sigma^2)$, where the number of steps is the number of gradient ascent steps for maximizing the ELBO for each $x$. \n\nIn contrast to our VB approach, in amortized inference all the training sequences share an encoder (or inference model) with global parameters $\\phi$. The encoder maps each $x$ to the corresponding $(\\mu, \\sigma^2)$ directly. There is no iterative update in typical amortized encoders. \n\nOur ablation study in Sec.3.4 confirmed that classical VB achieves better ELBO and avoids VAE's potential posterior collapse problem.\n\n**2. Implementation and hyperparameters**\n\nWe described our inference algorithm on page 4, Algorithm 1. We discussed the key hyperparameters (`inference steps` and `latent size`) in Sec. 3.2. Other hyperparameters can be found in Appx. 1 & 2. To further address your concern, we have built anonymous links for both [pseudocode (link)](https://anonymous.4open.science/r/icml25_submission3398-F84E/) and [executable code (link)](https://anonymous.4open.science/r/icml25_submission3398_code-4F9D). \n\n**3. About novelty of our work regarding variational inference** \n\nTo our knowledge, our work on incorporating classical VB in language models is novel, as confirmed by Reviewer FqJd and Reviewer ReTi. We are unaware of prior work using classical VI for fine-tuning LLMs. If we are provided with any related references, we would be happy to cite them in revision. The explicit latent abstraction in our model can be an important direction to explore for language modeling. \n\n**4. Empirical backing of cognitive analogy of \"Language of Thought\"**\n\nThanks for your comment. Our model was conceptually inspired by the \"Language of Thought\" model in cognitive science. However, there has not yet been a machine learning model for it. Our work may be considered the first step towards such a model. To further address your concern, we conduct in-depth analysis of our latent thought vectors through progressive latent inclusion experiments as [probing results (link)](https://anonymous.4open.science/r/icml25_submission3398_probing-F2BB/), where our findings reveal:(1) Clear hierarchical latent representation in LTMs, with higher layers integrating information from lower ones. LTM-M shows a significant accuracy jump at the final layer(55% to 100%), while LTM-L demonstrates more gradual information distribution with a notable improvement at layers 9-10(65% to 95%). (2)Case study reveals that lower layers capture grammatical elements and higher layers encode increasingly complex semantic content.\n\n**5. Direct analysis confirming latent vectors capture global information**\n\nIn lines 149-164, the latent vectors $z$ controls the generation of each token in $p(x_{t}|z, x_{<t})$ via cross attention, and $z$ is inferred from the whole sequence $x$: $p(z|x) \\propto p(z) \\prod_t p_\\beta(x_t|z, x_{<t})$, where $\\prod_t$ is over all $t$. \n\nEmpirically, we validate this by reconstruction experiments where we first infer $z$ from the sequence $x$ and then generate the sequence using only these inferred $z$ (without conditioning on ground truth context). This yields 100% reconstruction accuracy on the OpenWebText validation set, confirming that $z$ successfully captures the global information necessary for generation. \n\nThis finding is further supported by our probing results across different models (link is mentioned by response to point 4).\n\n**6. Empirical support for conceptual claims (short windows forcing global information into latent variables)**\n\nIn our response to your point 5 above, $z$ controls the generation of each $x_t$, and is inferred from the whole sequence $x$. With the short context window, the generation of $x_t$ must rely on $z$ to provide information beyond the short window. Empirically, our inferred $z$ can reconstruct the whole sequence accurately as mentioned above. Our probing results (link is mentioned by reponse to point 4) further confirm that the inferred $z$ captures the global information. \n\n*We will improve our paper based on your helpful feedbacks. Given our responses above, we humbly request you to reconsider your rating. Thanks!*", "claims_and_evidence": "- Supported Claims:\n\nThe paper’s experiments compare LTMs with standard autoregressive models and discrete diffusion models, showing quantifiable improvements in perplexity and sample efficiency—these results appear well-supported by the reported metrics.\n- Potentially Under‐Supported Claims:\n\nLatent Thought Vectors as a “Language of Thought”\nThe cognitive analogy is intriguing but not strongly backed by empirical studies.\nMultiple Inference Steps\n\nThe statement that “increasing inference steps” yields better performance seems reasonable, but the exact algorithm (and how it differs from typical amortized VI) is not fully explained. Why is it necessary to infer for multiple steps?", "methods_and_evaluation_criteria": "- Methods:\n\nThe authors’ methodology of adding latent vectors to a Transformer decoder, learned via a variational approach, makes sense.\n\nEvaluation Criteria:\n\nValidation Perplexity and zero-shot performance are standard metrics for assessing language models, so these are appropriate.\nAdditional benchmarks (e.g., text generation tasks, in-context reasoning tasks) improve the assessment.", "theoretical_claims": "N.A.", "experimental_designs_or_analyses": "- Experimental Soundness:\nThe overall design—comparing against standard LLMs and a discrete diffusion baseline—makes sense. Reporting perplexities and a few-shot test is appropriate.\n- Potential Weakness:\nThe paper does not sufficiently detail hyperparameter choices or the exact inference procedure (especially the “multiple inference steps”), potentially leaving some designs unclear.\nThe authors claim that latent vectors store global context, but there is no direct analysis (like an attention map or latent dimension probe) confirming that they indeed capture such information.", "relation_to_broader_scientific_literature": "Original research seems to incorporate VI with pre-trained models for fine-tuning, while this work investigates training starting from scratch. The significance of this work becomes the engineering part. However, the code and implementation are not provided.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "-- Key Strengths:\n\n- empirical gains in perplexity and few-shot reasoning.\n- Offers new “scaling dimensions” and suggests paths for more efficient training or inference.\n\n-- Key Weaknesses:\n\n- Vague details about the practical inference procedure and reproducibility.\n- Some conceptual leaps (e.g., short windows forcing global information into latent variables) without robust empirical backing; the relation between the method design of this work and cognitive concepts )", "other_comments_or_suggestions": "N.A.", "questions_for_authors": "See above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wv9pqKiKE9", "reviewer": "Reviewer_ReTi", "summary": "This paper presents a novel method called latent-thought language model (VTM) for autoregressive language modeling.\nVTM introduces an additional family of scaling dimensions, latent thought vectors, to implicitly learn a sequence representation and guide the generation.\nTraining VTM requires sampling from the variational posterior of the latent vector, which is done by MCMC or variational Bayes. \nThe resulting method enjoys a good scaling property by achieving a better perplexity score or sample accuracy with a smaller number of tokens and model sizes.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*Thank you for your thoughtful review acknowledging the novelty of our work. We appreciate your recognition that our claims are supported by clear and convincing evidence. We will address your concerns as follows.*\n\n**1. Clarification on the cross-attention mechanism.**\n\nWe wish to humbly clarify a potential misunderstanding regarding the reviewer's comment that incorporating $z$ makes \"mask attention in classical autoregressive models not applicable.\" Our Latent Thought Models (LTMs) adopt the decoder Transformer in the original transformer paper of (Vaswani et al. 2017.), where each layer $l$ of the decoder transformer consists of (1) self-attention sub-layer with attention of $x_t$ to $x_{\\leq t}$ (masked), (2) cross-attention sub-layer with attention of $x_t$ to $z_l$, and (3) feed-forward sub-layer. The cross-attention sub-layer (2) for each layer $l$ is ${\\rm softmax}\\left(\\frac{Q_{h_l}K_{z_l}^\\top} {\\sqrt{d}}\\right)V_{z_l},$ where $h_l$ is the hidden embedding of the input $x$ and $z_l$ consists of the latent thought vectors at layer $l$. Crucially, the latent vectors $z$ are inferred by gradient-based variational inference. This means there is no network that maps from $x$ to $z$. \n\nIt might be a slight misunderstanding that LTMs \"will require $L$ NFEs to evaluate the probability\", if $L$ means the number of layers. Given a latent $z$, evaluating $p(x|z)$ only needs 1 NFE. The reviewer correctly notes additional computation is needed, but this is for approximating the posterior $p(z∣x)$ or evaluating the ELBO. This variational inference requires $T_{fast}$ NFEs, scaling with the inference steps $T_{fast}$, not layers $L$. \n\n**2. Memory requirements of additional variational posterior model (possibly 2x)?**\n\nWe wish to respectfully clarify that the 2x memory requirement regarding our additional variational model may be a slight misunderstanding. The classical variational Bayes of our LTMs does not require a separate \"variational encoder model\" - it only needs the local parameters $(\\mu, \\sigma)$. The memory overhead for these parameters is almost negligible, with our LTM-L incurring just 1.5% additional memory cost on H100 compared to same-sized AR models with the same batch size. This is more efficient than the encoder (inference model) used in amortized variational inference (VAEs), which typically requires the same memory as the decoder.\n\n**3. Demonstration that model learns informative latent vectors (e.g., posterior distribution analysis)**\n\nFollowing your advice, we conduct in-depth analysis of our latent thought vectors through progressive latent inclusion experiments as [probing results (link)](https://anonymous.4open.science/r/icml25_submission3398_probing-F2BB/). Our findings reveal: (1) Clear hierarchical latent representation in LTMs, with higher layers integrating information from lower ones. LTM-M shows a significant accuracy jump at the final layer (55% to 100%), while LTM-L demonstrates more gradual information distribution with a notable improvement at layers 9-10 (65% to 95%). (2) Qualitative text reconstruction reveals progression from basic grammatical elements (22% at layers 1-3) to structural patterns (30% at layers 1-6) to coherent semantic content (65% at layers 1-9) to complete sentence (99% at layers 1-10). \n\n\n**4. Computational cost reporting (training time, memory usage)**\n\nWe reported the trainFLOPs/tok and the number of training tokens in Table 1 of our submission. Here we provide other costs such as training time and memory usage.\n\n| Model | Parameters | Memory Usage | Time per Epoch | Gradient Accumulation |\n|-------|------------|--------------|----------------|------------------------|\n| GPT-2-L | 762M | 52214 MiB | 924.63 ms | 4 |\n| GPT-2-M | 345M | 62706 MiB | 453.71 ms | 2 |\n| LTM-M (steps=16) | 51M | 59040 MiB | 2209.76 ms | 1 |\n| LTM-L (steps=64) | 76M | 68158 MiB | 13190.58 ms | 1 |\n\nAll benchmarks were conducted on a single node with 8 H100 GPUs with batch size 512. In our approach, we trade compute for slow learning of global parameters for fast learning of local parameters (line 242-246), which is essential to LTM's data efficiency. \n\n*We will revise our paper based on your valuable feedbacks. Thank you!*", "claims_and_evidence": "The claims of this paper are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "There are some potential issues with the implementation of the method.\n- The authors incorporate a latent variable $z$ in each layer of the transformer, as illustrated by Figure 2. This makes the mask attention in classical autoregressive models not applicable, and will require $L$ NFEs to evaluate the probability.\n- An additional variational posterior model $q_{\\theta}(z|x)$ is employed to train the VTM model, which may introduce at least 2x memory. By the way, the authors do not explicitly explain how they design $q_{\\theta}(z|x)$.\n- The authors state that they use cross-attention to incorporate $z$, which is still vague for me. I suggest the authors use mathematical expressions or illustrations to explain how the generative model and the variaitonal posterior model are designed.", "theoretical_claims": "There are no the theoretical claims in this paper.", "experimental_designs_or_analyses": "- It is not sure that the model indeed learns informative latent vectors. The authors could demonstrate the posterior distribution of $z$ given a sequence $x$; For example, $z$s may have a large correlation if $x$s have the similar semantic meaning.\n- No computational cost in terms of training times or memory are reported.", "relation_to_broader_scientific_literature": "I have no other comments.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "I have no other comments.", "other_comments_or_suggestions": "I have no other comments.", "questions_for_authors": "I have no other comments.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: I recommend accepting this paper on Latent-Thought Language Models. This novel approach incorporates explicit latent thought vectors that guide token generation through a Transformer decoder, utilizing classical variational Bayes for optimization. The key strengths include superior sample and parameter efficiency, demonstrated improvements in perplexity and few-shot reasoning capabilities. This work offers a promising new direction for language modeling with compelling scalability properties along multiple dimensions. I think the authors did a good job addressing the reviewers' concern including the one who gave weak reject rating.
Decision: Accept (poster)