Stop-Think-AutoRegress: Language Modeling with Latent Diffusion Planning
Authors: Justin Lovelace
Christian K Belardi
Sofian Zalouk
Adhitya Polavaram
Srivatsa R Kundurthy
Kilian Q Weinberger
Venue: COLM 2025 (2025)
Keywords: diffusion
latent diffusion
language generation
TL;DR: We introduce a unified architecture that pauses autoregressive text generation for latent diffusion planning, enabling higher quality and more controllable text generation with improved language understanding.

Abstract: The Stop-Think-AutoRegress Language Diffusion Model (STAR-LDM) integrates latent diffusion planning with autoregressive generation. Unlike conventional autoregressive language models limited to token-by-token decisions, STAR-LDM incorporates a ``thinking'' phase that pauses generation to refine a semantic plan through diffusion before continuing. This enables global planning in continuous space prior to committing to discrete tokens. Evaluations show STAR-LDM significantly outperforms similar-sized models on language understanding benchmarks and achieves >70% win rates in LLM-as-judge comparisons for narrative coherence and commonsense reasoning. The architecture also allows straightforward control through lightweight classifiers, enabling fine-grained steering of attributes without model retraining while maintaining better fluency-control trade-offs than specialized approaches.

Published as a conference paper at COLM 2025
Stop-Think-AAtoReggrss: Language Modeling with Latent Diffusion Planning
Justin Lovelace, Christian Belardi, Sofian Zalouk, Adhitya Polavaram, Srivatsa Kundurthy,
Kilian Q Weinberger
Department of Computer Science, Cornell University
Abstract
The Heap-Thinnk-AtoRggreee Language Diffusion Model (STAR-LDM) in- tegrates latent diffusion planning with autoregressive generation. Un- like conventional autoregressive language models limited to token-by- token decisions, STAR-LDM incorporates a "thinking" phase that pauses generation to refine a semantic plan through diffusion before continuing. This enables global planning in continuous space prior to committing to discrete tokens. Evaluations show STAR-LDM significantly outperforms similar-sized models language understanding benchmarks and achieves > 70% win rates in LLM-as-judge comparisons for narrative coherence and commonsense reasoning. The architecture also allows straightfor- ward control through lightweight classifiers, enabling fine-grained steering of attributes without model retraining while maintaining better fluency- control trade-offs than specialized approaches. Our code is available at https://glt thub. httpm///ssniiiloooeee//////a/aa
1 Introduction
Large Language Models (LLMs), typically based on the Transformer architecture (Vaswani et al., 2017), have demonstrated remarkable capabilities across a vast range of natural language tasks (Kojima et al., 2022; Brown et al., 2020). Their success largely stems from the autoregressive (AR) generation paradigm, where text is produced sequentially, predicting one token at a time conditioned on the preceding context. However, this strictly left-to-right token-by-token generation process fundamentally differs from human writing practices. Human authors frequently pause, reflect, plan ahead, and revise their text to ensure global coherence, stylistic consistency, and adherence to overarching goals Hayes & Flower (1996). In contrast, standard autoregressive models make irrevocable, local decisions at each step, which inherently limits their capacity for long-range planning or dynamic control over the generation process (Liu et al., 2021b). Consequently, ensuring that generated text globally satisfies complex constraints, maintains consistent persona, or avoids undesirable content purely through local token predictions remains a significant challenge, often necessitating computationally expensive fine-tuning reinforcement learning procedures (Ziegler et al., 2019; Ouyang et al., 2022).
Recent advancements in generative modeling, particularly score-based diffusion models (Ho et al., 2020; Song et al., 2020), offer fundamentally different paradigm. These models generate data through an iterative denoising process, beginning with pure noise and guided by a learned score function. Crucially, this score function can be readily modified during generation to incorporate external guidance signals from lightweight classifiers (Dhariwal & Nichol, 2021; Ho & Salimans, 2021), enabling flexible control over various attributes of the generated output (c.g., style, content). This inherent controllability which can be effectively applied in continuous spaces such as image latents (Rombach et al., 2021) or text embeddings (Li et al., 2022; Lovelace al., 2023), presents a particularly promising direction for enhancing control mechanisms in large language models.
Corresponding author: @l335e@comnelleduu

---

Published as a conference paper at COLM 2025
Inspired by the planning capabilities suggested by human writing and the controllability of diffusion models, we propose TeeTTThiiuutteeggnne Language Diffusion Model (STAR- LDM). STAR-LDM is novel, unified architecture that integrates latent diffusion planning directly into the autoregressive generation framework. It introduces a "thinking" phase where standard token generation is paused. During this phase, the model utilizes a diffusion process operating in a continuous sentence embedding space to refine a latent representation (a soft prompt) that encodes a plan for the subsequent text. This refined latent representation then guides the resumption of autoregressive token generation. By jointly training the autoregressive and diffusion components, STAR-LDM learns to perform global semantic planning in continuous space before committing to discrete token choices, enabling more coherent and controllable generation.
Our contributions are as follows: (1) We propose STAR-LDM, unified architecture that integrates latent diffusion planning within an autoregressive language model. (2) We demonstrate that STAR-LDM achieves improved language generation quality. (3) We show that our unified architecture leads to substantial improvements on standard language understanding benchmarks, indicating that the planning mechanism enhances, rather than hinders, core language capabilities. (4) We validate that STAR-LDM's integrated diffusion component enables efficient, plug-and-play control over text attributes (like sentiment and toxicity) using lightweight classifiers during inference, without requiring model retraining. STAR-LDM represents step towards language models that can "stop and think" entirely in latent space, leading to higher-quality, more controllable, and potentially more reliable text generation.
Background
Diffusion models (Sohl-Dickstein al., 2015; Ho et al., 2020; Song al., 2020; Kingma & Gao, 2023), are a class of generative models that iteratively refine a sample of noise to a sample from some data distribution in a coarse-to-fine manner. Given clean data Xdata from q( data),, diffusion models learn a model (x approximating (Xdata). Here, Xdata could be images, audio, or continuous latent representations relevant to NLP (Li et al., 2022; Rombach et al., 2021; Lovelace et al., 2024), such as the sentence embeddings used in this work (Ni et al., 2022).
Forward process. Diffusion models define a forward noising process that transitions clean data Xdata to noise over continuous time E [0,1]. The noisy latent z₁ at time is given by (z, Xdata) = N (z1; a; Xataa of which can be sampled as:
z 0;Xdata + oie, where e~ c0,)
(1)
Using the pariance-preservinn formulation (Ho et al., 2020) (0? = 1 a²), the noise level is parameterized by the log Signal-to-Noise Ratio (SNR), A log (27 /o?). The schedule a decreases such that z Xdata and the distribution of zι approaches a standard normal, i.c., q (Z) (data) ~ N (0,1).
Generative model. The generative process reverses the forward process, starting from noise 21 10, and iteratively denoising it to produce final sample zo that approximates the data distribution q (N This requires the score function Vz, log pz(2), the gradient of the log-density of noisy data at time t. Diffusion models introduce a neural network se f) to approximate this score. Sampling involves iteratively applying the learned model Sp (or an equivalent parameterization, like noise prediction) to reverse the diffusion from f 1 to
t 0, often using solvers like DDPM (Ho et al., 2020).
Training objective. The score network sy can be learned via denoising score matching (Song & Ermon, 2019). An equivalent and common objective (Ho et al., 2020) trains network (z( to predict the noise e added to xdata to get Zı, using a loss like:
[()) = ( (~-aaaaee-i....
20 (At))
||e E 1) 0|]
(2)
where z, = 0,Xaata + dhe and (AA)) is an SNR- weighting term (Kingma et al., 2021). The score and noise predictions are related by sø !) -Eo (z)
2

---

Published as a conference paper at COLM 2025
Plug-and-play control. A key advantage of diffusion, opposed to autoregressive, models is their controllability via plug-and-pll play guidance (Song et al., 2020; Dhariwal & Nichol, 2021). To generate samples conditioned on y, the sampling process is guided by the condi- tional score Vn, log Pt (y))). Using Bayes' rule, this conditional score can be decomposed:
22, log (z(l)) = V Pz))) + V. log (y)lz))
Unconditional Score
Guidance Term
The first term is the standard score approximated by the unconditional diffusion model sp(21, The second term guides the sampling towards latents 2, that are likely to produce the condition y. This guidance term be approximated using gradients from a separate classifier trained on noisy data, which is often called classifier guidance (Dhariwal & Nichol, 2021).
In practice, a guidance scale s 2 0 is introduced to modulate the strength of the condition:
$ 8(z, f, = Z (: + s. 22 log (y|z:))
(4)
Setting = 0 yields unconditional generation, while >> 1 increases the influence of y. This provides flexible "plug-and-play" control without retraining the base model for each condition y. This inherent controllability offers a promising avenue to address the control challenges of autoregressive language generation, which motivates our work.
3 Stop-Think-AutoRggree Language Diffusion Model
STAR-LDM Training
STAR-LDM Generation
Diffusion Loss CDM
Diffusion
Language Modeling Loss LLM
Autoregressive Generation
Sampling
Denaised Sentance
datt
Generated Continuation Sccos
Embodding Reace
dook soked
relendesott
old diock ticked
elerteessy
Autoregressive Decoder
Autoregressive
DIT
DIT
relentessty
relent
Prefix X xpeef
Prefix X
Notey Sentence) Emboditing
Geussian
Sentance Embedding
Continuation
Gaussian Noise
Xyont
Figure 1: Unified diffusion-guided language model architecture illustrating the training (left) and generation (right) processes. See text for details.
We propose tecture that augments autoregressive language modeling with latent diffusion planning. STAR-LDM can pause generation, plan in a continuous latent space, and resume autoregres- sive token generation with improved coherence.
Language Diffusion Model (STAR-LDM), a unified archi-
Figure 1 depicts our proposed architecture, which consists of three major components, each with a primary role in enabling our "stop-think-autoorgress" approach: (1) The Au- toregressive Decoder (in purple) serves as the foundation of our model, handling token generation in a sequential manner similar traditional language models. This component maintains the local fluency and grammatical structure necessary for coherent text. (2) The continuous Sentence Embedding Space (blue vectors) provides latent planning space that captures semantic meaning. Using 768-dimensional Sentence-T5 XL embeddings (Ni et al., 2022), this space represents text at a higher level of abstraction than tokens, with representations that capture core semantic content while being invariant to shallow. (3) Two Diffusion Transformers (DiTs) (Peebles Xie, 2022) (in green) facilitate the translation between the continuous embedding space and the discrete token representations used by the decoder. These transformer encoders are also conditioned on the noise level through adaptive normalization layers following Peebles & Xie (2022).
3

---

Published as a conference paper at COLM 2025
In our approach, text generation is conceptualized two-part process: first processing a prefix (the initial portion of text that has already been generated), then planning and generating coherent continuation. The prefix might be prompt provided by user or previously generated text, while the continuation represents new text the model will generate with guidance from semantic plan.
During training, these components work together as follows. First, the autoregressive de- coder processes the prefix text Xpref ("The old clock ticked") like standard autoregressive language model. The model also processes noisy representation of the continuation' 's semantic meaning. This representation is created by encoding the ground truth continuation Xcont ("relentlessly as ") into a clean embedding dataa using Sentence- then adding noise according to the diffusion schedule to obtain 2 = d,taata + d1c.
We linearly project this noisy embedding into sequence of 8 soft vectors, which the first DiT processes (conditioning on the current noise level) to form soft prompt. This prompt will both be used to guide diffusion denoising and subsequent token generation. For diffusion denoising, the autoregressive decoder naturally incorporates information from the prefix as it processes this soft prompt. The decoder's 's outputs then flow to the second DiT, which generates the final representations for the diffusion prediction. We apply a learned linear projection to obtain the final diffusion prediction Ždata, completing a step of the denoising process.
After processing this soft prompt, the decoder reverts to standard next-token prediction. The therefore conditioned on both the textual prefix and
predictions for the text continuation the (noisy) sentence embedding of that text continuation. The sentence embedding provides valuable signal for the token prediction objective, teaching the decoder to generate text aligned with the sentence embedding.
This integrated design creates model that can simultaneously handle token-level predic- tion and semantic-level planning. As noise levels decrease during training or sampling, the semantic guidance becomes increasingly precise, enabling the model to generate continua- tions that follow the semantic content in the denoised embedding while maintaining both local fluency and global coherence.
3.1 Training Procedure
Our training approach combines autoregressive language modeling with diffusion-based planning in a multi-task learning framework. This training strategy will enable STAR- LDM learn both token generation and semantic planning simultaneously. During self- supervised pre-training (left side of Figure 1), we sample text segments and randomly split each into a prefix Xpref and continuation Xcont- This simulates stopping points where the model would pause to plan. We jointly optimize with two objectives:
Language Modeling Loss (LLM). The loss combines standard prefix modeling and noise- conditioned continuation modeling:
LIM = Σ log P(xpret. i (ppet,<c;
Σ log P(Xcont,) Xpretr Xeont, Z1j Zi;0),
55)
Prefix modeling
Continuation modeling
where p is the prefix length, и the total length, and 2, the noisy embedding of Xcont-
Diffusion Loss (CDM). We encode the continuation Xçont using Sentence-T5 to obtain the clean embedding zp. Following standard diffusion training, we then sample a timestep and noise vector / (0, compute the noisy embedding z; = 2,20 + dje, and train the model to predict the added noise conditioned on the prefix and timestep:
LPM = Exxennte (W(()) . ||e -E( t, Xpret)
(6)
4

---

Published as a conference paper at COLM 2025
This objective is weighted by w(A)), a function of the log(SNRR) that balances the im- portance of different noise levels! The final training objective combines both losses, = + PLDM, where the scalar & balances the two components. STAR-LDM is therefore trained for autoregression and diffusion in multi-task manner.
Noise Conditioning. The diffusion training process exposes the autoregressive decoder to embeddings z; with varying noise levels, from ai 0 (pure noise) to Ag R (clean). Like prior work that has utilized noise-conditioning to improve robustness (Ho et al., 2022; Lovelace al., 2024), our decoder learns adapt based on noise level: relying heavily on the semantic plan at low noise while falling back to standard prefix conditioning at high noise. This creates a natural control parameter during inference- the noise level-that adjusts the balance between diffusion planning and autoregressive generation.
3.2 Generation Process
While our training process optimizes STAR-LDM to both generate coherent text and denoise semantic embeddings, the inference process leverages these capabilities in a sequential Saap-TThinnAAAoRRggeeee
workflow. During inference, we assume prompted setting where the user provides a prefix, and the model plans and generates a continuation.
During inference (Figure right), our approach unfolds in three distinct phases: 1) Stop: The model processes the input prompt Xpref (provided by the user) through the autoregressive decoder. 2) Think:
The model then "thinks" by sampling initial noise 2 = é ~ (0, and performing iterative denoising obtain semantic plan. This iterative denoising process, guided by the prefix Xpref/ gradually transforms random noise into a coherent semantic embedding, Zg.
Starting from pure noise 2,, the model applies its trained noise prediction network ca, consisting of pass through the network, over multiple timesteps (e.g. 50 steps). At each step I, the model uses the predicted noise component to reduce the noise level of the embedding following predetermined schedule. This process iteratively refines coherent plan consistent with the given prefix. The generation process concludes with the third phase: 3) AutoRegress. Here, the generated semantic plan žσ guides the autoregressive decoder to produce a fluent continuation &cont. This achieved by processing 2, through the first DiT to create soft prompt that conditions the decoder alongside xpref- Once conditioned this combined prompt, the decoder generates the continuation via standard autoregressive sampling.
Prompt guidance. During the thinking phase, can optionally strengthen the influence of the prefix on the generated plan using classifier-free guidance (Ho & Salimans, 2021). This technique combines an unconditional prediction (ignoring the prefix) with a conditional prediction (using the prefix) to amplify the prefix's influence:
ē = Ea (z + w (Eo(z), Xpret) (,(,,,,
where is the guidance scale and eω (, 1) represents unconditional prediction ob- tained by masking the prefix embeddings from the diffusion model. Higher values of w increase the prompt's influence on the semantic plan.
To enable efficient classifier guidance (Dhariwal & Nichol, 2021), we train MLP that directly operates on noisy sentence embeddings:
Plug-and-Play Control with Classifier Guidance.
(8) where TimeEmb(t) encodes the noise level via sinusoidal embeddings. To make the MLP robust to varying noise levels, we follow Dhariwal Nichol (2021) and train on labeled data that we perturb with randomly chosen noise levels. During inference, we compute the guidance gradient directly:
(y(y)) Ri (z, MLP(z, TimeEmb(tt),
logp:(y|z:) ~ (z,f((z;,t)
(9)
We adopt the sigmoid weighting and cosine noise schedule used by Hoogeboom al. (2024).
5

---

Published as a conference paper at COLM 2025
This lightweight approach enables attribute control with negligible computational overhead.
Implementation Details. We implement our model using GPTg-Large as the autoregressive backbone which has 770M parameters. Our unified architecture, with the two small DiT encoders, has 956M trainable parameters. During training, we use the sigmoid loss weight- ing and cosine noise schedule following Hoogeboom et al. (2024). For diffusion generation, we use a cosine noise schedule (Dhariwal & Nichol, 2021) with 50 diffusion steps and add small noise (0? = 0.1) to the final generated embedding Zo to ensure robustness to minor errors introduced by diffusion sampling. We train our model on approximately 16B tokens from the FineWeb dataset (Penedo et al., 2024). We report full architecture details and hyperparameters in Appendix F.
4 Impact of Diffusion Process Language Modeling
Next token prediction in traditional autoregressive models focuses primarily on local coher- ence, making these models inherently myopic in their generation decisions. In contrast, our diffusion-based planning approach introduces form of non-local semantic guidance. To un- derstand how this semantic guidance influences token prediction, we conducted controlled experiment visualized in Figure We begin with simple prefix ("I went to the carnival and") and two distinct continuations describing different experiences. For each continuation, we compute token probabilities under two conditions: (1) when conditioned on a clean semantic embedding of the entire continuation (zo) and (2) when conditioned on pure Gaus- sian noise (e). The difference in log likelihoods, log (Xcontt pref, zo) - log p (Xcontl prref,e), E), reveals which tokens are most influenced by semantic planning with the ground-truth embedding.
As shown in Figure 2, the semantic plan primarily increases the likelihood of content-bearing words that capture the essence of each narrative. In the first continuation, words like "ring toss," "gold fish," "named," and "Flash" show the largest positive shifts. These precisely correspond to the key narrative elements describing winning a pet fish at carnival game. Similarly, in the second continuation, words like "lost," "mirrors," "eaten," and "funnel" experience the greatest positive influence-all central different carnival experience involving hall of mirrors and food. This analysis demonstrates that our model's diffusion- based planning mechanism effectively captures and emphasizes semantically meaningful content, rather than merely influencing surface-level linguistic patterns. By explicitly integrating a latent semantic plan, STAR-LDM generates coherent language that maintain global semantic consistency. We present additional experiments exploring this semantic influence across varying noise levels in Appendix A.
Prompt (Xpret): went to the carnival and
Forced Continuation (Xcent
tried luck the toss, somehow) winning gold fish that survived exactly three days apartment.) named him Flash due to his brief but memorable tenure in my: life
Forced Continuation (Xeee)
got
mirrors,, spending nearly an trying my way out my) waited
impatient ent y When finally emerged they had already eaten most of my funnell cake
66.77
4.06
-1.44
0,17 log poxc de) log pla
2.28
4.40
6,51
Figure 2: Visualization of the change in log likelihood due to conditioning on the clean con- tinuation embedding versus pure noise. Tokens highlighted red represent the information provided by the semantic plan.

---

Published as a conference paper at COLM 2025
Zero-Shot NLU (LM)
Zero-Shot NLU (STAR-LDM)
LDM Q)
L [Ag
Z,,0, Q
LLM [ Q old person
old person
DIT
Autoregressive Decoder
Autoregressive Decoder
What hyiically contracts Answer:
What type of person typicallyy contracts llness? Answer:
DiT
Ziet
old person
A
person
Gaussion
Notse
Serfience Embedding
Sontanco
T5
A
Figure 3: Diagram comparing zero-shot NLU evaluation for standard LMs (left) and STAR- LDM (right). Both models score candidate answers (A,) given a question (Q). STAR-LDM utilizes both the answer text A, and its latent embedding 2₁,0 from Sentence-15 for scoring.
5 Natural Language Understanding
Standard Natural Language Understanding (NLU) benchmarks, particularly multiple- choice question answering (QA) datasets (e.g., CSQA, SIQA, ARC), assess model's s ability to identify the most plausible answer or completion (A) from a set of candidates, given context or question (Q). Conventional autoregressive models (Figure 3, Left Panel) typically score each candidate answer A; by its negative log-likelihoode LLM (Ai(Q) = log p (A, Q)) This likelihood is calculated using teacher forcing, and the candidate with the lowest score (highest probability) is selected as the answer.
We adapt this protocol for STAR-LDM to select the most likely candidate answer given the question Q (Figure 3, Right Panel). We assess candidate answers by approximating log (AA,IQ) using the variational lower bound on the likelihood (Kingma al., 2021), as direct calculation is intractable. We encode each candidate A; using Sentence T5 to produce a semantic latent plan Z;,0, add Gaussian noise to generate Z at specific timestep I, and compute our scoring function derived from the Evidence Lower Bound (ELBO):
log (A/()) S (A, Q) + (z,,
(10)
A
Σ
log p AAA/14 Zi,0. Q) + E,,,- ("EBBO())))) - ||ee c cz(z,,,4, 2)1
where the weighting (A)) for the regression objective follows from the derivation of the lower bound (Kingma et al., 2021). We present a detailed derivation of this scoring function in Appendix C. The LM term measures how well the model reconstructs A given Q and Z,,, while the diffusion term measures how plausible the latent plan Zı, is for question Q. We select the candidate with the minimum overall score, choosing the answer that is both linguisticallyy well-formed and has plausible semantic meaning. We compute the expectation in Equation 10 with an unbiased Monte Carlo estimate of the weighted reconstruction error. In practice, autoregressin models often normalize the log-likelihood. LLM, by the length of the candidate A, to improve performance. We also adopt this approach as we observe it similarly improves performance.
Results. STAR-LDM outperforms similarly sized autoregressive models on standard zero- shot NLU benchmarks as shown in Table 1. The model excels particularly on commonsense reasoning tasks (CSQA, SIQA), demonstrating the effectiveness of latent semantic plan- ning. Notably, the diffusion component alone performs strongly, indicating that semantic plausibility contributes significantly to reasoning capabilities. While the autoregressive component underperforms independently, combining it with diffusion yields the best re- sults, suggesting complementary strengths. This demonstrates that our diffusion planning enhances the NLU capabilities of comparable autoregressive models.
2The benchmarks include CommonsenseQA (CSQA), Social TQA (SIQA), HellaSwag (HS), Wino- grande (WG), Physical IQA (PIQA), OpenBookQA (OBQA), and ARC (Easy and Challenge).
7

---

Published as a conference paper at COLM 2025
Model
Params
Scoring
CSQA
SIQA
H5
WG
PIQA
OBQA
ARCIE
ARC:C
Avg.
Random Baselime
20.0
33.3
25.0
50.0
50.0
25.0
25.0
25.0
31.7
LarzeLarge
.7B
LM Loss
36.6
42.1
42.9
51,9
69.2
34.2
46.6
25.1
43.6
GPT2-XL
1.58
LM Loes
36.3
42.3
47.9
53.1
70.5
34.4
51.1
28.5
45.3
Pytha-1b
1B
LM Loss
35,3
42.4
45.6
52.6
69.2
32.4
49.1
27.0
44.2
Pytha-1.4b
54.2
54.0
LM Lees
38.20.
42.2
44.0a.1
50.5p.
69.3p.
27.103
37.60.5
26.3n3
41.5ml
STAR-LDM
Diffusion
48.7p.3
46.805
37.9pu
50,402
63.5g2
39.30.1
55.0g.
32.1a5
47.0p.1
LM + Diff Loss
49.833
46.80.
39.0p.
51.401
66.10.1
41.302
55.00.3
31.408
47.6p1
Table 1: Zero Shot NLU Evaluation Results. monte-carlo evaluations for STAR-LDM.
We report mean and standard error over three
6 StoryCloze Generation
Experimental Setup. To evaluate generation quality, we assess narrative continuations the StoryCloze dataset (Mostafazadeh et al., 2016), which requires coherent fifth-sentence completions for four-sentence stories. We employ Claude 3.7 Sonnet (Anthropic, 2025) as a judge to compare STAR-LDM against baseline models (GPT-2 Large/ XL and Pythia models (Biderman et al., 2023)) blind head-to-head evaluations across 200 contexts, using four criteria: Narrative Coherence, Commonsense Reasoning, Language Quality, and Emotional/ Pasthalooyyycccccccca Plausibility, Full evaluation details, including the prompt, are in Appendix D.
Results. STAR-LDM significantly outperforms similar-sized and larger models across all criteria ( Figure 4), with win rates exceeding 70% against models like GPT-2 XL Coherence/ /eeeseninasnngnne (p < 0.01). These substantial gains suggest our latent diffusion planning effectively improves global structure and logical consistency compared to pure autoregression. While competitive against much larger models (Pythia 6.9/12B), STAR- LDM's improvements in Language Quality are less pronounced, indicating the planning mechanism primarily enhances semantic coherence rather than surface fluency.
Coherence
Plausibllity
STARSLDN
STARLOO 10 9681
9601
Pythia 128
Pj010
Pythia 8.98
Pythi 2.88
Pythia 6998
CPT-2 XL Pytia
GPT2 XL Pythiw 1.4m
GFT22 Large
parameters) Reasoning
Quality
SLAR-LUM
SIAILLUM
Pytriu 128
Pythia 6.98
TP50 12B
Pythia
We noce us
Pythw 2.0
GPT-2
Pythia 2.00
GPT2 Large
Pychia 1.48
GPT2
Wodel Size parametersi
Model Size (B parsmeters)
Larg
Pythia
Figure 4: LLM-Judge evaluation. Results presented with 95% confidence intervals.
7 Language Generation Evaluation
Experimental Setup. We evaluate generation on 5000 C4 validation samples (Raffel et al., 2019), generating 64-token continuations from 32-token prefixes. We evaluate using three complementary metrics: (1) Llama Ppl measures fluency using perplexity computed by Llama-3.2-3B (Grattafiori et al., 2024); (2) MAUVE (Pillutla et al., 2021) assesses distributional similarity to human-written text using GPT-2 Large embeddings; and (3) Div quantifies
8

---

Published as a conference paper at COLM 2025
C4 Validation
Params
CFG (w)
MAUVE 1
Generative Perplexity
Diversity
GPT-2 Large
.77B
85.2
30.50.3
41.8
GPT-2 XL
1.5B
86.60.4
27.10.6
43.3p.1
Pythia-lb
1B
85.5p.1
36.50.7
45.6p.1
Pythia 1.4b
1.4B
84.8
33.1,
47.10.0
STAR-LDM
.96
94.602
36.
45.3p.0
STAR-LDM
.96
1.5
94.102
31.
45.4p.0
STAR-LDM
.96B
2.0
94.00.2
29.80.6
45.40.0
Table 2: Language generation evaluation on C4 validation examples. We report mean and standard error over three sets of generations.
lexical diversity as l
Junique n-gramsl
[total n-grams] (Su et al., 2022). For STAR-LDM, we report metrics
across different classifier-free guidance strengths (w), which increase the influence of the prefix, during diffusion planning.
Results. As shown in Table 2, STAR-LDM consistently outperforms baselines in MAUVE score, while maintaining competitive perplexity and diversity. Increasing prefix guidance improves fluency without sacrificing diversity. We present results regarding inference latency in Appendix
8 Plug-and-Play Control
Experimental Setup. STAR-LDM enables attribute steering during generation without model retraining. We evaluate sentiment control and toxicity mitigation and compare against DeXperts (Liu al., 2021a)-a strong baseline that fine-tunes specialized language models to guide generation. In contrast, our approach needs only requires training lightweight MLP MLP classifier on the target attribute to guide diffusion planning.
For toxicity control, trained our MLP on the Jigsaw Unintended Bias dataset (cjadams et al., 2019). The toxicity guidance was evaluated using 1,000 randomly selected neutral prompts from RealToxicityPrompe Amazon Polarity and Stanford Sentiment Treebank (SST-2) (Socher et al., 2013) to train our sentiment MLP classifier. We tested sentiment steering using 1,000 randomly selected neutral prompts from OeenWebText (Gokaslan Cohen, 2019; Liu al., 2021a).
(Gehman et al.). For sentiment control, we used both
For evaluation in both settings, we generated 10 continuations per prompt. We measure the alignment with the control condition, language fluency, and language diversity. For toxicity, we follow prior work Deng & Raffel (2023); Liu et al. (2021a) 2021a)aand use the Perspective API. For sentiment we utilize RoBERTa-Large (Liu et al., 2020) fine-tuned on sentiment classification across diverse domains. For language fluency, we report the generative perplexity with 1lama-3.2-3B. To quantify generation diversity, we follow prior work (Lu et al., 2022b) and measure the average number of unique 3-grams in each set of continuations.
Results. Figure 5 shows STAR-LDM achieves better control-flunccy trade-offs than DeXperts across guidance scales for both negative sentiment and toxicity reduction. Our model generates near-zero positive sentiment text at lower perplexity than GPT-2 Large, while maintaining lexical diversity even under strong guidance (additional results presented in Appendix E).

---

Published as a conference paper at COLM 2025
GPT
GPT (Delpersst
STAR (Delsett))
Proportion Posatiu Senkimank
Average Maximum Toxicity
Uama-3.2-30 Generalive Perplexity
30
Uama-3.2-30 Generative Perplexily
(a) Negative Sentiment (Lower is better)
(b) Toxicity Mitigation (Lower is better)
Figure Relationship between perplexity and content attributes across guidance scales (s).
9 Related Work
Prior work explores both discrete diffusion (applying noise to token sequences (Hoogeboom
et al., 2021; Austin et al., 2021; Hoogeboom et al., 2022; Lou et al., 2024; Sahoo et al., 2024)) and continuous approaches (in embedding space (Li et al., 2022; Gong et al., 2022; Strudel et al., 2022; Gulrajaní & Hashimoto, 2023) latent spaces (Lovelace et al., 2023; Zhang et al., 2023; Lovelace et al., 2024)). While discrete models lag behind autoregressive transformers (Zheng al., 2024), continuous models show promise for controllability but face integration challenges that our unified architecture addresses. Controlled generation methods include model Lie-ttunin (domain-adaptive pretraining (Gururangan et al., 2020), control tokens (Lu et al., 2022b), RLHF (Wu et al., 2023; Ouyang et al., 2022; Jang et al., 2023)) and guided decoding with lightweight components (classifier guidance (Dathathri et al., 2019; Yang Klein, 2021; Deng & Raffel, 2023) or specialized LMs (Krause et al., 2021; Liu et al., 2021a)). Our approach performs guidance during diffusion planning for semantic control.
Recent work enhances LM capabilities through natural language reasoning (chain-of- thought prompting (Wei et al., 2022; Zelikman et al., 2022; 2024)), specialized tokens for paused thinking (Goyal et al.), recurrent computation over hidden states (Hao et al., 2024; Geiping et al., 2025)), or by learning discrete latent actions via inverse dynamics models (Jia et al., 2025). Our work introduces "thinking" through diffusion in semantic space, comple- menting discrete token generation. Hybrid generative architectures have been explored in prior work primarily for multimodal settings rather than enhancing language generation alone (Zhou et al., 2024; Wang et al., 2024; Team, 2024; Liang et al., 2024).
10 Conclusion
We presented STAR-LDM, a unified architecture that integrates autoregressive language modeling with latent diffusion planning. Our experiments demonstrate three key advan- tages: (1) improved natural language understanding capabilities, (2) higher quality narrative generation with better coherence and reasoning, and (3) effective plug-and-play control without model fine-tuning. By enabling models to "stop and think" in continuous seman- tic space before generating text, STAR-LDM. combines the global planning capabilities of diffusion models with the fluency of autoregressive generation. This approach not only improves performance on standard tasks but also enables precise control of generated text attributes through lightweight guidance mechanisms. Our work represents step toward language models that more closely mirror human writing processes, pausing to plan before committing to specific wording.
10

---

Published as a conference paper at COLM 2025
Acknowledgments
CB is supported by the National Science Foundation (NSH) through the NSF Research Traineeship (NRT) program under Grant No. 2345579. SZ is supported by the Defense Advanced Research Projects Agency (DARPA) under Grant No. D24AP00259-00. This research is supported by grants from the National Science Foundation NSF 2118-00 OAC-1934714, IIS-2107161, and IIS-172428) HDR-2118310) the Cornell Center for Materials Research with funding from the NSF MRSEC program (DMR-1719875), DARPA, arXiv, and the New York-Presbyters authors acknowledge the National Artificial Intelligence Research Resource Pilot (NAIRR- 240157) and TACC Vista for contributing to this research result.
for the NYP-Cornel Cardiovascular AT Collaboration. The
References
Anthropic. Claude 3.7 Sonnet and Claude Code, February 2025. URL https://www.
cww/news/ hotbroppi.ccom/ewws//eee anthropic.
catsss/lauus claude- 1oaodd---7-so0n 3-7-sonnet.
Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Structured denoising diffusion models in discrete state-spaces. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL https://openreviiet www.:net/ooummiid-h---aaao
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pp. 2397-2430.
PMLR, 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
cjadams, Daniel Borkan, inversion, Jeffrey Sorensen, Lucas Dixon, Lucy Vasserman, and nithum. Jigsaw unintended bias in toxicity classification, 2019. URL https: //kags//kaggle, httpccomptiiiiossjjjigeeeess wtw.r/ooooiiiiiaaannnnnnneneeeee
unintended-biate -ootiiiitttiiiiitclllll-e-ee toxscitty
Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. Plug and play language models: A simple approach to controlled text generation. In International Conference on Learning Representations, 2019.
Languagg
Haikang Deng and Colin Raffel. Reward-augmented decoding: Efficient controlled text generation with unidirectional reward model. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. Realtox- icityprompts: Evaluating neural toxic degeneration in language models.
Jonas Geiping, Sean McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Brian R. Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, and Tom Goldstein. Scaling up test- time compute with latent reasoning: A recurrent depth approach, 2025. URL https: //arxiv. www.iv..or//bb//552..o 05171.
Aaron Gokaslan and Vanya Cohen. Openwebtext corpus. http://Skylionnenn www.iiooon7..ggggi github. io/ OpenNebTeettCopps,, 2019.
Shansan Gong, Mukai Li, Jiangtao Feng, Zhiyong Wu, and Lingpeng Kong. Diffuseq: Sequence to sequence text generation with diffusion models. The Eleventh International Conference on Learning Representations, 2022.
11

---

Published as a conference paper at COLM 2025
Sachin Goyal, Ziwei Ji, Ankit Singh Rawat, Aditya Krishna Menon, Sanjiv Kumar, and Vaishnavh Nagarajan. Think before you speak: Training language models with pause tokens. In The Twelfth International Conference Learning Representations.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXi::2407.211 217783, 2024.
Ishaan Gulrajani and Tatsunori B Hashimoto. Likelihood diffusion language models. arxiv preprint ar XrXto22305.86619 2023.
Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. Don't stop pretraining- Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 1342--3301 2020.
Shibo Hao, Sainbayar Sukhbaatar, Dijia Su, Xian Li, Zhiting Hu, Jason Weston, and Yuan- dong Tian. Training large language models to reason in a continuous latent space. arviv preprint xrii::4412.06669 2024.
John R Hayes and Linda Flower. The science of writing:
Theories, methods, individual differences, and applications. Lawrence Erlbaum Associales, Inc, 1996.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXie:2207.12598, 2021.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models.
Advances in neural information processing systems, 33:6840-6851,, 2020.
Jonathan Ho, Chitwan Saharia, William Chan, David Fleet, Mohammad Norouzi, and Tim
Salimans, Cascaded diffusion models for high fidelity image generation. The Journal of Machine Learning Research, 23(1):2249-22 2281, 2022.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference on Learning Representations.
Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling. Argmax flows and multinomial diffusion: Towards non-autoregressive language models. CoRR, abs/2102.05379 2021. URL https://artiv.org/bb///// 05379.
Emiel Hoogeboom, Alexey A. Gritsenko, Jasmijn Bastings, Ben Poole, Rianne van den Berg, and Tim Salimans. Autoregressive diffusson models. In International Conference Learning Representations, 2022. URL https: net hett
Emiel Hoogeboom, Thomas Mensink, Jonathan Heek, Kay Lamerigts, Ruiqi Gao, and Tim Salimans. Simpler diffusion (sid2): 1.5 fid on imagenet512 with pixel-space diffusion. Xiv preprint ar Xir::4441. 19324, 2024.
Joel Jang, Seungone Kim, Bill Yuchen Lin, Yizhong Wang, Jack Hessel, Luke Zettlemoyer, Hannaneh Hajishirzi, Yejin Choi, and Prithviraj Ammanabrolu. Personalized soups: Personalized large language model alignment via post-hoc parameter merging. arXiv preprint ar www::22100.156666 2023.
Chengxing Jia, Ziniu Li, Pengyuan Wang, Yi-Chen Li, Zhenyu Hou, Yuxiao Dong, and Yang Yu. Controlling large language model with latent actions. arvio preprint ar arXio:2503.21383, 2025.
Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. Advances in neural information processing systems, 34:21696-21707, 2021.
Diederik P Kingma and Ruiqi Gao. Understanding diffusion objectives as the ELBO with simple data augmentation. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https:// //tppr/v///eeeeieee/ wtwprrrewwweeeeeoooouriiiieeeeee wwtt:oorumi.d--MMMaaaaa..
12

---

Published as a conference paper at COLM 2025
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systemss, 35:22199-222 22213,, 2022.
Ben Krause, Akhilesh Deepak Gotmare, Bryan McCann, Nitish Shirish Keskar, Shafiq Joty, Richard Socher, and Nazneen Fatema Rajani. Gedi: Generative discriminator guided sequence generation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 4929-4952, 2021.
Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori Hashimoto. Diffusion-LM improves controllable text generation. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL
Weixin Liang, Lili Yu, Liang Luo, Srinivasan lyer, Ning Dong, Chunting Zhou, Gargi Ghosh, Mike Lewis, Wen-tau Yih, Luke Zettlemoyer, et al. Mixture-offtaanstormess A sparse and scalable architecture for multi-modal foundation models. arXiv preprint arXiv:2411.0499 2024.
Alisa Liu, Maarten Sap, Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A Smith, and Yejin Choi. Dexperts: Decoding-time controlled text generation with experts and anti-experts. In Proceedings of the 59th Annual Meeting of the Association for Computa- tional Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 6691-6706, 2021a.
Alisa Liu, Swabha Swayamdipta, Noah A. Smith, and Yejin Choi. Dexperts: Decoding- time controlled text generation with experts and anti-experts. In Proceedings of the 59th Annual Meeting of the Association for Compulational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 6691-6706. Association for Computational Linguistics, 2021b.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. [brbert A robustly optimized [bert) pretraining approach, 2020. URL https: //tttnrve///ppnriiieee... nett/forummldd-SyyS0Atttt
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL httpss/s//pppneeeii.ee////ooonss
Bkg6RiCqY7.
Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion language modeling by estimating the ratios of the data distribution, 2024. URL https::/ ntw./e/wwwe/ffrmnn id=7ImqtQdKB9.
Justin Lovelace, Varsha Kishore, Chao Wan, Eliot Seo Shekhtman, and Kilian Q Weinberger. Latent diffusion for language generation. In Thirty-scoenth Conference on Neural Information Processing Systems, 2023. URL https: /ttps////////eeeee/eeeee nwt/forum? dit.mmmdddddddtttaddt
Justin Lovelace, Varsha Kishore, Yiwei Chen, and Kilian Weinberger. Diffusion guided language modeling. In Findings of the Association for Computational Linguittss ACL 2024, pp. 14936-14952, 2024.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm- solver++: Fast solver for guided sampling of diffusion probabilistic models. Xrv preprint arXiv:2211.010995 2022a.
Ximing Lu, Sean Welleck, Jack Hessel, Liwei Jiang, Lianhui Qin, Peter West, Prithviraj Ammanabrolu, and Yejin Choi. Quark: Controllable text generation with reinforced unlearning. Advances in neural information processing systems, (ystems,35:27591-115 35:27591-27609 2022b.
Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. A corpus and cloze evaluation for deeper understanding of commonsense stories. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 839-849, 2016.
13

---

Published as a conference paper at COLM 2025
Jianmo Ni, Gustavo Hernandez Abrego, Noah Constant, Ji Ma, Keith Hall, Daniel Cer, and Yinfei Yang. Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models. In Findings of the Association for Computational Linguistics: ACL 2022, pp. 1864-1874, 2022.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances Neural Information Processing Systems, 35:27730-777777 2022.
William Peebles and Saining Xie. Scalable diffusion models with transformers. arXiv preprint ar arXi::2212.09748, 2022.
Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data at scale. In The Thirty-eight Conference on Neural Information Processing Systems Dalasets and Benchmarks Track, 2024. URL Forum?id=n6SCke
net//
Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. Mauve: Measuring the gap between neural text and human text using divergence frontiers. Advances in Neural Information Processing Systems, 34: 4816-4828, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. arXiv e-prints, 2019.
2019.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models, 2021.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for
biomedical image segmentation. In International Conference on Medical image computing and computer-assiste intervention, pp. 234-241. Springer, 2015.
Subham Sekhar Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin T Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and effective masked diffusion language models, 2024.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In International Conference Learning Representations, 2022. URL https://openrevii net//forummdd=TTddMpppphbb
Noam Shazeer. Glu variants improve transformer. ar preprint ar Xiie:20002.05202, 2020.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 conference empirical methods in natural language processing, pp. 1631-1642, 2013.
Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics, 2015. URL https:t/aaxiii org/abs /100.. 03585.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 2019.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2020.
Robin Strudel, Corentin Tallec, Florent Altché, Yilun Du, Yaroslav Ganin, Arthur Mensch, Will Grathwohl, Nikolay Savinov, Sander Dieleman, Laurent Sifre, et al. Self-conditioned embedding diffusion for text generation. arXiv preprint ar XrXi::2211.0423 2022.
14

---

Published as a conference paper at COLM 2025
Yixuan Su, Tian Lan, Yan Wang, Dani Yogatama, Lingpeng Kong, and Nigel Collier. A contrastive framework for neural text generation. arXiv preprint (xXi::2022.6417,, 2022.
Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arxiv preprint arXiv:2405.09881, 2024. doi: 1044850/arXiv.2205.008525 URL https://githue com//facebookkreseacch//cc chw.ereen.cccameeeee...oo
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https://procceding.nelle www..ps.cc/paape.eeppe httprr/pp..cc/ppppee..e////////p files//aper/2117/ilee 3f5ee243547dee91 (lo002441 Flde53c1144845aa-aaee
Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need. arx preprint ar XXiv:2409.18866 2024.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Mhain-of-thhuugh prompting elicits reasoning in large language models. Adoances in neural information processing systems, 35:24824- 24837, 2022.
Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A Smith, Mari Ostendorf, and Hannaneh Hajishirzi. Fine-grained human feedback gives better rewards for language model training, arXiv preprint ar #rXio:2306.01699 2023.
Kevin Yang and Dan Klein. Fudge: Controlled text generation with future discriminators. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 3511-3535, 2021.
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35:15476-15488, 2022.
Eric Zelikman, Georges Raif Harik, Yijia Shao, Varuna Jayasiri, Nick Haber, and Noah Goodman. Quiet-STar: Language models can teach themselves to think before speaking. In First Conference on Language Modeling, 2024. URL httpp:://ppetiii http:///pppeeeeee.//eee/e. wttp:/wwwwn/fooumiiiette oRXPiSOGH9.
Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in Neural Information Processing Systems, 32, 2019.
Yizhe Zhang, Jiatao Gu, Zhuofeng Wu, Shuangfei Zhai, Josh Susskind, and Navdeep Jaitly. Planner: Generating diversified paragraph via latent language diffusion model. arviv preprint (iXii:23366025331 2023.
Kaiwen Zheng, Yongxin Chen, Hanzi Mao, Ming-Ya Yu Liu, Jun Zhu, and Qinsheng Zhang. Masked diffusion models are secretly time-agnostic masked models and exploit inaccurate categorical sampling. ar Xiv preprini or XrXi::2409.0290 2024.
Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Predict the next token and diffuse images with one multi-modal model. ar preprint ar Xix::2408.11033 2024.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human prefer- ences. ar preprint ar Xia::1999008533, 2019.
15

---

Published as a conference paper at COLM 2025
A
Additional Visualizations of the Impact of Diffusion Process on Language Modeling
We provide a number of additional results in this section help build intuition for the contribution of the diffusion model to language generation. Both Figure 2 and Figure show the change in log likelihood at a token level when conditioned on a semantic plan. Figure 2 shows this for arbitrary text continuations which embed as semantic plans with Sentence T5, while Figure shows this for latent plans sampled from STAR-LDM.
Prompt (Xpr)): went for run
Generated Continuation (Xcont
Saturday moming it was lovely day and
felt: great
am luckyy that weather isn too cold (1)
usually freezing it's
Generated Continuation (Kcont)
yesterday started slow
body
just In mood hard
butt
hurt
like hell
that
-9.18
-6.91
-4.64
-2.37
-0.09
2.18
4.45
log Mpenn Zo) -lgg Sog.pRRanlPaee..
Figure 6: Visualization of the change in log likelihood due to conditioning on a semantic plan. Shown for two different generations. Positive values (red) indicate tokens that become more likely when conditioned on the plan, while negative values (blue) indicate tokens that become less likely.
In order to quantitatively assess the impact of the semantic plans, we measure the perplexity of STAR-LDM function of the log(SNR) in Figure We do this on subset of the C4 validation set, well as set of generations nucleus sampled from STAR-LDM. We see that at high log(SNR), where the semantic plan is very clean, the perplexity is much lower. However at low log(SNR), where the semantic plan is very noisy, the perplexity is much higher. This demonstrates that the semantic plan reduces the uncertainty in the language generation; helping the language model focus specific semantic direction.
Perplexity Comparison by logiSNR) on Subset of Dralnation Set
Sampling
15
15
Astorgmmien
Astergmisio
Astorepnoterr
Artrrppation
123
24
150
log Signaltto Nuise Ralio (2)
log Signal to Noise Rolio (hy)
Figure 7: Perplexity at various noise levels on a subset of 100 examples from the C4 evaluation set (left), or 100 sampled generations (right).
Additionally we show the same perplexity comparison but at the maximum log(SNR), corresponding to f = 0, in Figure 8.
Finally to complement our quantitative evaluation of STAR-LDM across noise levels, we visualize how the token wise log likelihoods change with the log(SNR) in Figure 9.
16

---

Published as a conference paper at COLM 2025
Perplexity Comparison on Subset of C4 Evaluation Set
Validation
Sampling
Parplesity
Astorogression ([tcaee)ppre)
Ditlusion + Autoregrestion PPReaed Rq)
Autoregresson P(AnsttXpree)
Diflusion Autoregression PlAcaet/Kgren, En)
Figure 8: Perplexity on subset of 100 examples from the C4 evaluation set (left), or 100 sampled generations (right).
Teacher Forced Continuation Token Log Likelihoods
-10.00
-7.78
Log Likelihood
-5.36
log(SNR)
-3.33
-1.11
-10
7.78
10.00
ried
RV
at
the rlag
samnow wimslog
gold f2s8 thet srrvived eactly
days
da
a
named
kie Flase
td
Ads
brite
Ammorabl fantira
but
ay Life
Teacher Forced Continuation Token Log Likelihood Deltas
-10.00
-7.78
10
log (SNR)
Log Likelihood
-1.11
1.11
3.33
5.56
7.78
ried
RV
the
risg
tomeiiv
viering
gold fish
sarslved
eeactly
three
apar taann
narred
Flazh
die
his
brief
burt
add
tanure
By Life
Continuation Tokens
Figure 9: Visualization of the token wise log likelihood by log(SNR) (top). Visualization of the token wise change in log likelihood by log(SNR) (bottom).
B Inference Latency
To provide comprehensive view of the practical trade-offs of our method, we analyze the inference latency of STAR-LDM. As the primary source of computational overhead is the iterative "thinking" phase, we evaluate how generation quality varies with the number of diffusion sampling steps. For this analysis, we adopt the DPM-Solver++<2M SDE sampler (Lu et al., 2022a), which is highly effective for few-step sampling, and we sweep the number of steps from 5 to 50. We set the prefix guidance to w 2.0 for this study.
We generate 96-token continuations for 32-token prompts, using set of 250 examples randomly drawn from the C4 validation set. We report generative perplexity and diversity as a function of the wall-clock time per generation in Figure 10. For reference, we also benchmark the inference speed and quality of the standard GPT-2 Large and GPT-2 XL autoregressive models.
It is important to note that current STAR-LDM implementation is not optimized for inference speed. Specifically, key-value (KV) caching, a standard technique for accelerating autoregressive generation, is not yet implemented for the diffusion planning phase, during
17

---

Published as a conference paper at COLM 2025
which the prefix is re-processed at each denoising step. Our reported wall-clock times for STAR-LDM are therefore loose upper bound on its inference speed.
As shown in Figure 10, STAR-LDM introduces tunable trade-off between computation time and generative perplexity. We observe diminishing returns as the number of sampling steps increases; perplexity improves up to around 15-20 steps and then largely plateaus, while latency continues to increase. that sampling step range, the latency similar to GPT-2 XL, which STAR-LDM consistently outperforms in NLU evaluations and generative metrics such as MAUVE score.
Quality-Speed Tradeoff in Language Generation
50
Numbers indicate diffusion timesteps
STAR-LDM
GPT-2 Large
Generative Perplexity
40
GPT-2 XL
10
50
30
15
20
10
0
2
5
Time per Sample (seconds)
Figure 10: Quality-Speed Tradeoff in Language Generation. We plot generative perplexity (lower is better) against the wall-clock time to generate a 96-token continuation. The STAR-LDM curve shows performance as the number of diffusion sampling steps is varied. Baselines are standard autoregressive models.
C NLU Derivation
In this appendix, we derive the Evidence Lower Bound (ELBO) for our STAR-LDM model, which serves as the foundation for both training and evaluation. This derivation illustrates how standard diffusion ELBO principles are adapted when the reconstruction term parameterized with an autoregressive decoder.
C.1 Standard Diffusion ELBO
We first recall the standard diffusion ELBO for continuous diffusion models (Kingma et al., 2021). For a diffusion model that generates data starting from pure noise, the log- likelhoodd of data point be bounded from below
18

---

Published as a conference paper at COLM 2025
log (xx) 2 (xIBO(x) Eg (2 (x) pox )
(11)
Reconstruction lerm
DKL. Dg (Z (x) |l (222min)
(12)
Prior matching term
Amax
||e -
cp (0(
(13)
Amin
Diffusion loss term
Where: zA represents the noisy latent variable at noise level parameterized by À (log-SNR); ((za |x) is the forward process that adds noise to the data; pxx[ZAmax) is the reconstruction term (generating data from noisy latent); DKL (9(2) xx) p=ZAmp)) is the prior matching term; and the integral term represents the diffusion denoising objective.
C.2 ELBO for STAR-LDM
For STAR-LDM, we adapt this framework to the language modeling context where we have prefix xrref and continuation Xcont- The key innovation is that the reconstruction term is parameterized using an autoregressive decoder. We derive the ELBO for pXXcont (xret)) as follows:
First, we introduce the latent sentence embedding z that represents the semantic plan for the continuation. The joint probability can be factorized as:
p(Ncont, z| prett = p (x X 2)p (reet)
(14)
The marginal probability of the continuation is:
/ p(xoont) preer
[ x(ree)
(15)
Since this integral intractable, we derive a lower bound using variational inference. We introduce variational posterior (z) Xcont) which in our case is implemented by the Sentence-T5s encoder which produces clean embedding, which then serves as the basis for the forward diffusion process.
Applying the standard variational inference procedure and Jensen's 's inequality, we arrive at the Evidence Lower Bound. Here, represents the full hierarchy of latent variables defined by the diffusion process::
log p(x (x x
E Eg(e)xcom)))
[log pX | z)
DKL (9()
p(z)xpret)
(16)
For the diffusion model part, we expand the KL term using its standard parameterization. The reconstruction term p( Xprei 2) is evaluated at specific low-noise level A which
corresponds to the latent variable Zymax The KL divergence over the full diffusion path from the variational posterior ((z)xcont) to the prefix-conditioned prior ((z)x)tr)) can then be expressed as:
DKL (x( ((z))xpret)) = DKL (q(ZAmml (xcont) p p(ZAmm))
(17)
+ f
Amaa
1
B (xcont)
[lle co(zx, 2, xpret) (ppeer)] 12|dx
(18)
Amnn
2
19

---

Published as a conference paper at COLM 2025
The first term is the KL divergence between the prior and the most noisy distribution. Substituting this back into (16), our final, complete ELBO is:
log (X
2 E EF(Ammav (Xcomt) [log Po (xcont) prrefs 2 ZAmax)
-
-DLL (q (ZAmii (xcont)
|| p pzZAmin)
(19)
-M
Amax
E g(2x) Koont)
[||e 2, |da
2,
(20)
Arnin
The prior matching term (19) ensures that the distribution of the most-noised latents matches a standard Gaussian prior. In many practical applications, this term either assumed to be approximately zero (if the noise schedule is such that (3 Acont) collapses to a standard normal) or omitted from the training objective as it is often small and constant with respect to the model parameters 0.
C.3 Parameterization with Autoregressive Decoder
For STAR-LDM, we parameterize the reconstruction term with an autoregressive decoder. Specifically:
[xcent Σ
log (Xcontl ( 2)
log Po (Xcont) xpreff Xcont, 2)
(21)
This allows the model to generate text token-by-token while being guided by the semantic plan represented by the latent variable z.
C.4 ELBO for Zero-Shot Evaluation
For zero-shot NLU evaluation, we use the negative of the ELBO derived above to score candidate answers A; given question Q. For the purpose of creating a practical scoring function, we drop the prior matching term (DxL (..) in (19)). This is a common practice as the term constant with respect to the model parameters being evaluated and does not affect the relative ranking of candidate answers. Its non-negativity ensures that we are still optimizing a valid (though slightly looser) bound.
A key aspect of our evaluation protocol is that the reconstruction noise level, Amax, is tunable hyperparameter. This allows us to precisely control the trade-off between the influ- ence of the semantic plan (the diffusion component) and the fluency of the autoregressive decoder.
The resulting scoring function is based on the two main components of the ELBO bound: the reconstruction loss and the diffusion loss.
A Σ
Score(A/)Q::mal
-E Fy(tttmax (A))
log po (0(A/11,5) 7 Q)
(22)
+ IE ((..)
WELBO(Ar) 116--- t, 2)1]]
(23)
In practice, we compute an unbiased estimate of this score by approximating both expecta- tions with Monte Carlo sampling
In this formulation, Zi,Ampx represents the candidate answer's clean embedding noised to the level Amax, while Zi is the same embedding noised to random level r. By varying Amax, we control how much the autoregressive decoder relies on the semantic plan versus its own
We use 16 Monte Carlo samples for the reconstruction term and 128 for the diffusion term, which found sufficient for stable estimates.
20

---

Published as a conference paper at COLM 2025
internal knowledge. We fix Amax -2 across all datasets, though we observed that tuning this on per-dataset basis would lead to further improvements. The candidate answer with the lowest score is selected as the most probable answer.
This derivation provides the theoretical foundation for the zero-shot evaluation protocol for STAR-LDM.
D StoryCloze Evaluation Details
For Claude judged StoryCloze evaluation randomly sample subset of 200 stories from the StoryCloze validation split. For each story in our sample, we generate 32-token continuation from each baseline well STAR-LDM. We then compare for each story each baseline generation against that of STAR-LDM by inserting the two continuations into the prompt show in Figure 11. The ordering of the continuations (i.e. which is given option A/B) is randomized for every comparison. The prompt gives explicit instruction on the criteria to use to evaluate the continuations well how denote the winner. The winner is indicated with specific formatting that is easily parsed with regular expression. We present qualitative example in Figure 12.
E Additional Control Experiments
In this appendix, we provide additional results for our controlled text generation experi- ments that complement the findings presented in Section 7.
E.1 Certext-yerplexity Trade-offs
While the main paper shows negative sentiment control and the average maximum toxicity metric, here we present additional perplexty trade-offs for positive sentiment and the average toxicity metric.
Figure 13 demonstrates that STAR-LDM effectively controls positive sentiment and reduces average toxicity scores while maintaining competitive perplexity. For positive sentiment control, our model achieves high scores (> 0,95) with guidance scale s = 2.0, showing minimal fluency degradation compared to the base model. Similarly, for toxicity reduction, STAR-LDM achieves substantially lower average toxicity scores than both GPT-2 Large and DeXperts at comparable perplexity levels.
E.2 Control-Diversity Trade-offs
We also evaluate the relationship between attribute control and lexical diversity, measured by the ratio of unique trigrams in the generated output. These metrics complement the perplexity evaluations by assessing whether controlled generation maintains vocabulary richness and avoids repetitive patterns.
Figures 14 and 15 show that STAR-LDM consistently outperforms DeXperts by achieving better attribute control while maintaining higher lexical diversity. For both positive and neg- ative sentiment control, our approach preserves trigram diversity even at strong guidance scales. Similarly, for toxicity mitigation, STAR-LDM with guidance scale 5.0 dramati- cally reduces both average and maximum toxicity while maintaining diversity comparable to the base model.
These results highlight a key advantage of our diffusion planning approach: by guiding generation at the semantic level before committing to specific tokens, STAR-LDM achieves natural attribute control that preserves linguistic richness without resorting to repetitive or restricted vocabulary choices.
21

---

Published as a conference paper at COLM 2025
LLM-as-a-Judge Prompt
Task Overview
You evaliaating two competing language models generate story continuations Given foun-sentence story context, each model will generate continuation. Your task to judge which continuation is better based on a set of defined criteria.
Evaluation Criteria
contest? Does it maintain consistencyy with charactens, events, and situations established in the first four sentennes? Does it advance the parrative in natural and satislying way?
Narrative Coherence well does the continuatonn logicilly follow
Cummonsense Reasoning Does the continuation demonstrate understanding of causal and temporal relationships? real-world know ledge about how events typically unfold? Does it aveid contradicting established facts about the world?
corsistent with
Quality the continuation grammaticaaly correct and well-formed?? Does it match the style, tone, and vocabulary level of the context! Does it flow naturally from the preceding text?
Pnautiona//syccologiie Plausibility Does the continuation reflect plausible human reactions, motivations, or emotions? Does it approprii stely address characters' guals, feelings, intentions set up the context?
Scoring Process Read the four-sentence context carefully. Read both generated continuations (labeled A and B) without knowing which system produced which text. Rate each continuation on four eriteria using 1-5 scale. For each criteria select the continuation with higher score as the winner. Provide brief justification (2:3 sentences) explaining your decision.
Clearly indicate your final judgment each criteria placing the criteria (either "Coberence" or "Reasuning" "Quality" "Plausibility ") and winning contimuation (either "A" or "B") within double brackets, like: [[C [CCeherence Al]. Make sure to provide winner for each criteria. Massing judgements for Criteria will be considered ti.
Example Evaluations
Example 1:
Context: Bab was bored at his job school teacher. He had been working so hard this past month. He decided to treat himself with something special. He ordered tickets for weekend snowboarding trip.
Continuation A: He was looking forward to getting away.
Continuation
boss told him
work weekend.
Judgement: Narrative Coherence: Continuation A follows the established natrative of Rob's self-reward better Continuation (3/5), naturally builds
anticipation while introduces cooflccting element that shifts away from focus on Bob's well deserved break. [[Caherence All Commonsense Reasoning: Continuation A (5/5) demonstrates better understanding of the natural progression from planning, trip to looking forward to it, while B (4/5) introduces plausible hut less alighed obstaale that beeaks the causal flow of treating oneself. [[Ressoning
Language Quality: Both continuations focus Bob's perspective rather than shifting Plausibility: Contimuation A (4/5) hetter addresses Boh' is established intention to reward himsell with positive anticipation, while B (4/5), though realistic, introduces disappolntment that contradicts the emotional trajectory set up in the contect. [|Plausibllity All
grammatically sound and match the context's lone, with A boss's actions.
(4/5) fowwng slightly more naturally than B (4/5) as it maintains
[[Quality AI] Emootional/syychooggi
Example 2:
Harry a date, Harry thought the date was going well. Olivia thinks he a complete jerk and never wants to see him again, Harry keeps raving about their chemistry.
Contex:: went
Continuation Olivia about to leave.
Continuation B: Olivia had her friend call her, to tell her that her mother was ill so as not offend him.
Judgement: Narrative Coherence: Continuation B (5/5) provides a more specific action that builds upon Olivia's 's established discomfort and desire to end the date without confrontation, while A (4/5) logical but less developed. [[Coberence Bi] Commonsense Reasoning: Continu- ation (5/5) demonstrales better understanding complex social dynamics and face-saving strategies commenly used uncoefortable dating situations, compared grammatiaally correct, but B (4/5) provides more detail matching the context's complexity, while A (3/5) is much simpler than the established narrative style. [[Quality Bi] Emottona//Psycologee
(4/5) which direct lacks social nuance. [|eessoiing B]] Language Quality: Both continua tions
Plausibility Continuation B (5/5) shows deeper understanding of Olivia's is conflicting desires escape while avoiding direct confrontation consistent with her awareness Harry's obliviouseess, while (4/5) captures her desire leave but not her likely approach. [|Plausibdityy Bi]
Example 3:
Cuntext: Jack and Ferris always fought for headphones. One day Jack broke Ferris' headphones jogging Ferris was furious at Jack. Their parents yelled at them.
Continuation A: Jack promised Ferris to buy him new headphones. Continuation Rt Jack promised to take Ferris jogging.
Judgement: Narzative Coherence: Continuation A (5/5) directly addresses the central conficct (broken headphones) with an resolution, while B (2/5) ignares the primary
established in the context. [[Coherence All Commonsense Reasoning: Continutiion A (5/5) offers logical resolution consisient with world wooblem-sokving. while (2/5) fails by suggesting the very activily caused problem, which would likely weesen the situation. [[Ressoning All Language Quality: Both contmnations are gr grammatically correct, but A (4/5) fows more naturally from the content's 's focus on the headphone conflict while B (1/5) creates jarring shift in focus. I[Quality All Emational/sycholle
Plausibility: Continuation (5/5) acknowledges responsibility and offers appropriate compensation addressing anger, while (3/3) shows poor understanding of emotional situation and would likely increase Ferrs's frustration. [[Plausibility All
Ferris's
Implementation Notes Judge contimuations primarily on how well they follow from the established context. Prioritize narrative coherence and commonsenise reasoning over creativity or surprise, Be consistent applying the evaluation criteria actoss different story types. Corsider the natural prognession of events and chatacter motivations when evaluating plausibility. Avoid biss toward specific narrative styles genne preferences.
Figure 11: Prompt template used for L.M-as-Judge evaluations with Claude 3.7 Sonnet on StoryCloze generations.
22

---

Published as a conference paper at COLM 2025
Context: Rick grew up in a troubled household. He never found good support in family, and turned to gangs. It wasn't long before Rick got shot in robbery. The incident caused him to turn a new leaf.
STAR-LDM (A): Ricks attempt to change his life was long and hard. However, the process has been worth it. The way he lived changed when he started new chapter...
GPT2-XL (B): Rick now looks after his younger brother, who is disabled.
The two are the heart of what is perhaps the most important project this summer: a massive...
Evaluation of Story Continuations
After analyzing the four-sentence context about Rick and the two competing continua- tions, here is my assessment based on the specified criteria:
Narrative Coherence
Continuation follows the context's redemption are naturally, describing Rick's diffi- cult but rewarding journey to change his life after the shooting incident. Continuation introduces Rick caring for disabled younger brother (new information) and then abruptly shifts to discussing "project" without clear connection to the established narrative. A maintains consistent focus on Rick's transformation while B introduces disconnected elements.
[[Coherence All
Commonsense Reasoning
Continuation A demonstrates understanding of how personal transformation is typi- cally difficult but rewarding, showing appropriate causal relationships. Continuation B presents a plausible caregiving scenario but fails to connect this to the "important project" mentioned afterward, breaking logical flow. The second sentence in B feels disconnected from both the context and the first sentence of the continuation.
[[Reasoning All
Language Quality
Continuation has minor issues (e.g. "Ricks" missing apostrophe) but maintains the narrative's s tone and flows naturally from the context. Continuation B starts cohesively but abruptly transitions to an incomplete thought about a "project," creating a jarring stylistic break that doesn't 't match the established narrative voice. A feels more like complete thought while seems fragmented. I|Quality All
Emootioal//syycollgiie Plausibility
Continuation A acknowledges the psychological reality that changing one's life after trauma is challenging but worthwhile, reflecting plausible emotional growth. Con- tinuation B presents plausible caring relationship but fails connect this to Rick's emotional journey established in the context, and the sudden mention of a "project" lacks psychological grounding in the character's established motivations. [[Plausibility A]]
Figure 12: Qualitative example of story completions from the StoryCloze dataset. The LLM judge evaluated both continuations, where Continuation A is STAR-LDM and Continuation GPT2-XL. The judge preferred STAR-LDM's continuation across all evaluation criteria.
F
Implementation Details
E1 STAR-LDM
We report additional architecture details and hyperparemeters for STAR-LDM in Table Our architecture processes the noisy latent embedding through a sequence of two DiT blocks, with the main autoregressive decoder situated in between (see Figure 1). To enable classifier-free guidance for the prefix, must simultaneously train our model to perform unconditional prediction, iee. without information from the prefix. We achieve this with two modifications. First, to remove the prefix conditioning, replace the output representations
23

---

Published as a conference paper at COLM 2025
GPT-2 Large
CPT Largu (Datperis)
STAR-LDR DMR-IDM
STAR-LNN 1x-1.
Propertion of Posttvv Sentinam
STAR LOW 15-2..0
Average Teatcity Score
STAR: LDW 15-3..0
STAR-LDR DM 1s-4
"'RR LDW 12-3.00
(Delsent))
STA be100
608
Lama-332.38 Generative Perplexity
Lama-332238 Generative Perplexity
(a) Perplexity vs. Positive Sentiment
(b) Perplexity vs. Average Toxicity
Figure 13: Additional control trade-offs beyond those shown in the main paper.
(Dciperis! STARLLEM
Proportion Poottue Sentiment
STANLLEM (1-2.11
Proportton Posttiv Sentiment
STA 5.-..1
STAR EM (1-4.))
6.10
GPT2 Lange
GP52 Lange (DaKcerts)
STARLEE
STAR4IDM (se1
STAR4 DM (x-2.00
180
101
680
104 600
Cal Unique Trigrams
000
Uniquue Trigrams
Diversity Negative Sentiment
(b) Diversity Positive Sentiment
Figure 14: Relationship between n-gram diversity and sentiment control.
Reeragge Maximum Teatcity
Average Score
Ratio of Unique Trigrms
Ratio of Unique Trigrams
(a) Diversity vs. Average Toxicity
(b) Diversity vs. Maximum Toxicity
Figure 15: Relationship between n-gram diversity and toxicity control.
24

---

Published as a conference paper at COLM 2025
Table 3: Implementation details for STAR-LDM training.
Architecture
Base Language Model
GPT-- Large (774M parameters)
Sentence Embedding Model
Sentence-13 XL (768 dimensions)
Total Trainable Parameters
956M
DiT Architecture - Prompt Encoder and Diffusion Prediction
DiT Layers per Module
6
DiT Hidden Dimension
1024
DiT Attention Heads
16
DiT Head Dimension
64
Activation Function
SwiGLU (Shazeer, 2020)
Normalization Layer
Adaptive RMSNorm (Zhang & Sennrich, 2019; Peebles & Xie, 2022)
Soft Prompt Sequence Length
8
Diffusion Configuration
Output Parameterization
v-prediction (Salimans & Ho, 2022)
Diffusion Steps (Inference)
50
Diffusion Sampler (Inference)
DDPM (Ho et al., 2020)
Loss Weighting
Sigmoid (Hoogeboom et al., 2024)
Noise Schedule
Cosine (Dhariwal & Nichol, 2021)
Training Details
Dataset
FineWeb
Training Tokens
16B
Max Sequence Length
128
Optimizer
A (Loshchilov & Hutter, 2019)
Learning Rate
5e-4
Training Steps
250000
Batch Size
512
Weight Decay
.01
Gradient Clipping
1.0
Diffusion Loss Weight (B)
5.0
from the autoregressive decoder with single, learnable null-context embedding. However, this embedding contains no information about the noisy sentence embedding processed by the first DiT block. To preserve this initial processing, we therefore introduce a UNet- style skip connection (Ronneberger et al., 2015) that concatenates the output of the first DiT directly to the null-context embedding before it is passed to the second DIT. This design ensures the unconditional path still benefits from the initial latent processing while remaining independent of the prefix. For the conditional path, the output of the first DiT is concatenated with the output of the autoregressive backbone, integrating information from the prefix.
F.2
Decoding Parameters
For all of our generation experiments, we utilize nucleus sampling (p=0.95) (Holtzman al.) with repetition penalty of 1.2 for the STAR-LDM decoder and the autoregressive baselines.
E3
Baseline NLU Results
All baseline results are computed with the lighteval evaluation harness using the Fine (Penedo et al., 2024) evaluation prompts?.
tasks.py
25

---

Published as a conference paper at COLM 2025
Table Implementation details for the Noise-Conditioned MLP classifier.
MLP Architecture
Input Dimension
768
Hidden Dimension
1536
Number MLP Blocks
Number of Hidden Layers
hidden layer
Normalization
Pre-norm with adaptive RMSNorm
Residual Connections
Applied after each MLP
Activation Function
SwiGLU
Time Embedding
Sinusoidal positional encoding
Diffusion Configuration
Noise Schedule
Cosine (Dhariwal & Nichol, 2021)
Loss Weighting
Sigmoid (Hoogeboom et al., 2024)
Training Details
Optimizer
A le-4
Learning Rate
Batch Size
256
Weight Decay
.01
Gradient Clipping
1.0
E4
Noise-Conditioned MLP
We present results of the noise conditioned classifier for sentiment and toxicity detection on held-out sets for our dataset in Figure 16. We observe that they smoothly improve in performance as noise levels decrease, outperforming logistic regression baseline for even moderately noisy data.
(a) Sentiment Classification
(b) Toxicity Classification
1.0
1.0
0.9
0.8
0.8
0.6
Score
Score
0.7
0.6
0.5
0.0
-3
3
-6
Log Signal-to- Noise Ratio (A)
Signal-to-Noise
Log hignal-to-Noiss. Ratio (A)
F1 (Naise-conditioned MLP)
(Logistic Regression)
AUROC (Noise-canditioned MLP)
AUROC (Logistic Regression)
Figure 16: Classifier performance across noise levels for sentiment (left), and toxicity (right)
classification.
26


Reviews:
[{"review_id": "oBRJqpl2GK", "reviewer": "Reviewer_iRQd", "summary": "This work violates the double-blind policy. L68 says \"our work Lovelace et al. (2024)\"", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "N/A", "reasons_to_reject": "N/A", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FB1MPCiviH", "reviewer": "Reviewer_UtyY", "summary": "The paper presents a means of incorporating a diffusion model over an embedding space of text into the auto-regressive generation process of a decoder LM. The authors make the claim that this is good based on analogies to how humans produce text. This is plausible but is only speculation. The empirical results however of their method do suggest it does have benefits, albeit compared to baselines it is not always better, but is on average.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Results are presented on public datasets against a few baselines**. Overall these show that for similar sized models, the proposed approach does improve on the story generation task consistently. It has mixed performance compared to others on the NLU task, although is stronger on average. \n\n**I believe there are other papers which have included some form of diffusion into the discrete text-token generation problem though, but can't cite them from memory as I'm not particularly familiar with the literature. I think it's worth checking this aspect, and what other reviewers comment.", "reasons_to_reject": "What are the results of the largest/best LM's on these tasks? There is an implied argument here that the \"reasoning\" of the diffusion process introduced is necessary, however I do wonder what the results of the largest and best LMs are on these benchmarks? I acknowledge it's a difficult comparison due to not controlling for parameter size/model capacity. \n\nThe arguments in section 4 that the diffusion produced \"semantic plan\" is important are weak imho. It's subjective whether these highlighted words are more or less important. What happens with less plausible forced continuations? What happens if the \"semantic steering\" is rather different to the prompt, does the model follow the prompt or the steering? There are several aspects here which need more analysis. \n\nThe proposed model has mixed results in table 1 and 2 compared to the Pythia models.", "questions_to_authors": "- What is it about the HS, WG and PIQA tasks that meant the model performed worse than Pytha-1.4b?\n- How often is the token produced by the diffusion process different (or significantly less likely) compared to the base LM's next token (or distribution)?\n\nnit: there's an incorrect reference on line 247.", "extra_scores": {}}, {"review_id": "RGq54nCCJu", "reviewer": "Reviewer_2MBE", "summary": "This paper proposes STAR-LDM, a unified architecture that integrates a latent diffusion “thinking” phase into autoregressive language modeling. The model pauses token generation to perform semantic planning in a continuous embedding space, then resumes generation guided by the refined latent. This design improves coherence, controllability, and performance on language understanding tasks.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Innovative architecture: The integration of diffusion-based semantic planning introduces non-local guidance that complements the myopic nature of standard autoregressive generation, enabling the model to produce globally coherent and semantically consistent text.\nEmpirical performance: The model achieves strong results on multiple NLU benchmarks, especially commonsense reasoning tasks, outperforming comparable autoregressive baselines.\nTheoretical grounding: The use of a variational lower bound (ELBO) provides a principled scoring function for candidate answers in multiple-choice QA, offering theoretical justification for improved accuracy on NLU tasks.", "reasons_to_reject": "Time complexity analysis is missing: Diffusion models require iterative denoising steps, which, even when applied in latent space, can significantly slow down training and inference compared to standard autoregressive generation.\nPlanning tied to prefix quality: The quality and effectiveness of the generated latent plan are still inherently dependent on the prefix. If the initial context is ambiguous or incoherent, the planning phase may struggle to provide meaningful long-range guidance.\nLack of explicit control in planning: The latent diffusion-based semantic plan is continuous and implicit. Relying on the latent representation may exacerbate hallucination issues, as any inaccuracy in the denoising process could propagate and mislead the downstream generation.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qZId2iqNrx", "reviewer": "Reviewer_fPq9", "summary": "This paper proposes Stop-Think-AutoRegress Language Diffusion Model (STAR-LDM), which integrates latent diffusion planning with autoregressive generation.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The writing is clear and easy to follow.", "reasons_to_reject": "1. To train the diffusion model, we need embeddings from another model (e.g., the used Sentence-T5). But authors do not discuss the impact of this embedding model.\n\n2. Lacking experiments on larger models.", "questions_to_authors": "1. The biggest model used in experiments is 1.5B. How about scaling model size to 7B?\n\n2. The compared model does not inherently have thinking ability. You can consider comparing with thinking Models like Qwen3.\n\n3. A related work, \"Controlling Large Language Model with Latent Actions\", can be discussed.\n\n4. The incorporated diffusion module learns to model the embedding distribution from Sentence-T5. However, the autoregressive model backbone is GPT2-Large. Why does the embedding of Sentence-T5 can effectively manipulate the generation process of GPT2-Large? Moreover, Sentence-T5 may not be strong enough if we use a larger autoregressive backbone.", "extra_scores": {}}]

Author rebuttal:

Meta-review: STAR-LDM proposes a novel integration of a latent diffusion “thinking” phase into standard autoregressive text generation, pausing token production to perform semantic planning in a continuous embedding space before resuming guided decoding. 

Empirically, it achieves consistent gains over same-sized autoregressive baselines on story generation tasks and shows mixed but generally positive improvements on commonsense and multiple-choice QA benchmarks, supported by a principled ELBO‐based scoring framework.

However, it also has notable shortcomings. As highlighted by R2, the human-analogy motivation for the diffusion phase remains speculative and the presented “semantic plans” are subjective, with no robustness analysis under divergent or adversarial steering. R3 emphasizes the absence of any detailed time-complexity or latency evaluation—iterative denoising in latent space could significantly slow both training and inference. Finally, R4 points out that the model’s reliance on external Sentence-T5 embeddings is never scrutinized, leaving open questions about how embedding quality or choice affects planning effectiveness and downstream generation.

During rebuttal, R4 raised the score and acknowledged the efforts of the authors. Overall, I think this paper presents interesting idea which would be very useful for the community to combine diffusion-style thinking in LLM generation.

[Automatically added comment] At least one review was discounted during the decision process due to quality]
Decision: Accept