Large Language Models to Diffusion Finetuning
Authors: Edoardo Cetin
Tianyu Zhao
Yujin Tang
Venue: ICML 2025 poster (2025)
Keywords: language models
large language models
diffusion
finetuning
test-time scaling
Primary area: deep_learning->large_language_models
TL;DR: New finetuning method powering large language models with some of the key properties of diffusion models and the ability to scale test-time compute.

Abstract: We propose a new finetuning method to provide pre-trained large language models (LMs) the ability to scale test-time compute through the diffusion framework. By increasing the number of diffusion steps, we show our finetuned models achieve monotonically increasing accuracy, directly translating to improved performance across downstream tasks. Furthermore, our finetuned models can expertly answer questions on specific topics by integrating powerful guidance techniques, and autonomously determine the compute required for a given problem by leveraging adaptive ODE solvers. Our method is applicable to any foundation model pre-trained with cross-entropy and does not modify any of its original weights, fully preserving its strong single-step generation capabilities. We show our method can be more effective and is fully compatible with traditional finetuning and search approaches, introducing an orthogonal new direction to unify the strengths of the autoregressive and diffusion frameworks.

Large Language Models to Diffusion Finetuning
Edoardo Cetin
Tianyu Zhao
Yujin Tang'
Abstract
Base language madel
36.06
36.24
We propose new finetuning method to provide pre-trained large language models (I.Ms) the abil- ity to scale test-time compute through the diffu- sion framework. By increasing the number of diffusion steps, we show our finetuned models achieve monotonically increasing accuracy, di- rectly translating to improved performance across downstream tasks. Furthermore, our finetuned models can expertly answer questions on spe- cific topics by integrating powerful guidance tech- niques, and autonomously determine the compute required for a given problem by leveraging adap- tive ODE solvers. Our method is applicable any foundation model pre-trained with cross-entropy and does not modify any of its original weights, fully preserving its strong single-step generation capabilities. We show our method can be more effective and fully compatible with traditional finetuning and search approaches, introducing an orthogonal new direction unify the strengths of the autoregressive and diffusion frameworks.
36
35.18
35.50
35.53
Average Score
34
32.10
32
30
28.54
28
26
2
2
L2D Inference Steps
Figure I. Test-time compute scaling with L2D. Our framework empowers LMs the scaling properties of diffusion, yielding increasingly higher inference performance with additional steps.
Established the predominant approach in visual domains, the diffusion framework offers properties that appear partic- ularly complementary to the LM paradigm (Sohl-Dickstein et al., 2015; Song & Ermon, 2019; Ho et al., 2020; Dhariwal & Nichol, 2021; Peluchetti, 2023; Esser et al., 2024). For in- stance, the iterative nature diffusion allows to adaptively scale compute to the difficulty of a specific task or any level of accuracy demanded by the user, regardless of the gener- ated output's 's length. However, despite these useful prop- erties, diffusion models trained for language currently lag significantly behind their autoregressive counterparts (Lou et al., 2024; Gat et al., 2024; Guirajani & Hashimoto, 2024) putting into question their inductive bias and scalability when applied to this highly relevant domain.
1. Introduction
The scalability of autoregressive large language models (LMs) is a pivotal component the current generation of foundation models (Team al., 2023; Achiam al., 2023; Dubey et al., 2024). However, despite their unprecedented capabilities, LMs inherently lack many valuable properties that could be expected of an "artificial general intelligence," such as the ability to scale computation for their most criti- cal decisions (Sutton, 2019). Efforts to address this limita- tion primarily focused on eliciting more nuanced responses through prompting and targeted searches over the space of possible completions (Feng et al., 2023; Kumar al., 2024; Trinh et al., 2024; Jaech et al., 2024), anchoring the reasoning process in the space of generated tokens.
In this work, aim to unite the strengths of these frame- works by introducing LM to Diffusion (L2D): new finetun- ing method powering rre-trained LMs with the scaling prop- erties and potential of diffusion (Karras et al., 2022; Uchara et al., 2025). Rather than learning a diffusion model from scratch, our method harnesses the large amount of "system 1" understanding efficiently acquired during autoregressive pre- training by casting LMs as single-step diffusions. Then, by introducing small fraction of new parameters comparable to modern parameter-efficien approaches (Hu et al., 2021)- we imbue the model with a new set of multi-step "reasoning" skills, the ability to scale computation on-demand, and the potential to incorporate powerful guidance techniques (Ho & Salimans, 2022), all without compromising its original
'Sakana AI, Tokyo, Japan. Correspondence to: Edoardo Cetin <edo@sakana.ais Tianyu Zhao <tianyu @ sikana.ai>, Yujin Tang <yujintang @ si>.
Proceedings of the 4nnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Large Language Models to Diffussin Finetuning
single-step capabilities.
Neural networks (NNs) in single-step generative modeling solely rely on an external source of pure randomness to generate new samples from scratch. In contrast, the goal of diffusion is to learn neural network fo conditioned on samples from each Pt and tasked with solving the sim- pler problem of generating new samples from lower nearby noise levels Pt+ Thus, effectively splitting the challenge of learning and generating new samples in multiple steps, which can be scaled based on computational availability
In summary, our technical contributions are the following:
We introduce L2D, a new finetuning method to power LMs with the scaling properties of diffusion, combin- ing key strengths from these two frameworks.
We show that L2D significantly improves four differ- ent LMs on math, coding, and a variety of reasoning tasks; and that its benefits can be both superior and complementary to traditional finetuning and search.
2.2. L2D Parametrization and Training Formulation
We demonstrate that L.2D allows to scale performance with additional compute, while opening the door to LMs equipped with autonomous per-token scaling and powerful diffusion guidance techniques.
An effective choice of loss to train diffusion models is sim- ply to predict the values in the uncorrupted target datapoints from pt (i.e., p²) given the partial information contained at each corruption level ã = fo (Z,, t). When Pi is distribu- tion over a continuous domain, this is commonly done by using a simple mean squared regression loss on all timesteps L, as popularized by the DDPM algorithm (Ho et al., 2020);
We provide our full code' to facilitate future advances in developing new scalable foundation models with diffusion.
2. Gaussian Diffusion for LM Finetuning
I 22) = Ee [[|=] - fo 1)]
(2)
Another key design decision for diffusion is the choice of schedules a and 3r, which define the denoising process that fo will be learning. This is one of the most significant choices for continuous diffusion models, affecting all as- pects of both training and inference dynamics (Nichol & Dhariwal, 2021; Esser et al., 2024). In our work, we employ the schedules a = and 3 (1 - -t)o,, where a is hyper- parameter linearly scaling the signal-to-noise ratio for all timesteps between and po within the samples dt ~ Pt- This choice is closely tied to the rectified flow matching schedules (Liu et al., 2022), which have been shown to possess particularly desirable "straightening" properties for diffusion (Lee et al., 2024; Lipman et al., 2024) and have been widely adopted in the recent diffusion literature (Esser et al., 2024). To ease our notation and make this connection explicit, we absorb the hyper- parameter a in the standard deviation of our base distribution po = NO0 T²T), which simplifies our schedules to a = t and Br = (1 -)..
In this section, we describe the key components of our L2D framework. In partieular, we provide details about the con- sidered diffusion formulation, together with our designed training and inference approaches. Although each of the following subsections offers a concise introduction to the concepts and modern practices of diffusion and language modeling, we refer to recent work (Nakkiran et al., 2024; Lipman et al., 2024) and Section 5 for more comprehen- sive resources. We conclude the section explaining how our design decisions make L2D natural extension to mod- ern language modeling aimed to complement rather than supersede the autoregressive framework.
2.1. Gaussian Diffusion
Gaussian diffusion decomposes the problem of generating new samples from a target unknown distribution p" from a source distribution q == [(0, I) over multiple "simpler" steps. The Gaussian diffusion decomposition effectively reuses the intermediate information computed in the model's attempts in each previous step. These subsequent diffusion steps can be seen as discretization of a continuous "denois- ing" process from /. = 0 to time / = 1, over which the model is tasked to transform samples from q to p". All interme- diate distributions along the denoising process are defined by a corresponding corruption process, mixing target data points æ1 p" with noise from q to produce 24 ~ Pe!
Unlike for the continuous case, language modeling operates over target distribution p1 defined ble V. where to each index y E 1, ...... |V| there corresponds token embedding 2 E Rd. This key difference is one of the main reasons that diffusion in language modeling is yet to have a predominant recipe with several recent approaches even exploring alternative diffusion formulations over the discrete space of vocabulary indices y (Austin al., a:; Lou et al., 2024; Gat al., 2024). In this work, we choose to still diffuse over the token embeddings x, as in standard continuous diffusion, but do not employ an MSE loss as done by Li et al. (2022). Instead, we learn our diffusion model with a simple cross-entropy loss, establishing a di- rect connection to traditional single-step language modeling. In particular, given a token æq indexed by label y sampled
finite vocabulary ta-
Zt 0 + B,ro, where Io N I). (1)
Here, the schedules a₂ and 3g are defined as monotonic functions with ag- = 31 = 0 and 01 = 90 = 1, satisfying the constraints such that pa == q and p1 := p².
httpp:://gi thub. com/ SakanaAl /112
2

---

Large Language Models to Diffusson Finetuning
Algorithm 1 Diffusion language modeling predictions
these probabilities together with the vocabulary embeddings stored in V estimate é for any valid velocity (in our case, defined in Equation 4). While Dieleman et al. (2022) takes ② as the weighted average over the embeddings, we instead use the probabilities predicted by fo I, c) to sample an individual & E V at each diffusion step t. Although the ex- pectation of these two estimates matches, we note our choice reintroduces some stochasticity into the denoising trajectory traced by the ODE. In practice, we find this stochasticity beneficial to better harness some of the self-correcting prop- erties of the diffusion framework, which Karras et al. (2022) showed might be limited in fully deterministic inference formulations. We summarize this next-token prediction procedure with our sampling approach (lines 5-6), Fuler integration, and a budget of T total steps in Algorithm 1.
1: Input diffusion model fa. context e, budget T
2: Initialize t + A. f- 1/ 11)
3: Sample 24, .N(0,
4: for
do
5: Sample 1/1 Ja(we,by lc))
6: Set à
+ Vie
7: Compute dx. = 1-1
E-ze
8: Update t + 1 + D., It x Te + De x dra
9: end for
10: Return y ~ 1,)
along with a context of preceding tokens e from the target data distribution p1, our diffusion loss is formulated as:
(f) Fx.. at [log (Fo 1 (,)), where
2.4. LMs as Single-step Diffusion Models
To ~ 5(0,021) x1 = Vi= ~ ~P:
Our choices in designing L2D establish a clear connection with the traditional LM framework. As detailed above, train- ing a diffusion model with Equation 3 can be interpreted as standard next-token prediction where the model is pro- vided with an additional "diffusion token" 2t containing some amount of knowledge about the target y, ranging from no information (t = 0) to perfect information (t = 1). Therefore, LMs are essentially trained with an equivalent prediction objective to L.2D's when / = O, where x, is en- tirely uncorrelated with the target y. Similarly, inference following Algorithm involves iteratively sampling increas- ingly accurate next tokens å from the model's logits up to a sampling budget T. Thus, traditional LM inference can be again viewed as a special case of this procedure with T = 1, where only the model's 's first sample is used to predict y.
t 1[0 and T₁ tz + (1 t)
This formulation allows our diffusion network fo to still predict [V] logits over the vocabulary tokens, just like a standard language model, while leveraging partial informa- tion about the next sequence token provided by xt. Despite its simplicity, this choice still enables our diffusion process to draw continuous trajectory during inference, similar to traditional diffusion models with continuous outputs explained by Dieleman et al. (2022) and detailed below.
2.3. L2D Inference Formulation
To generate new samples with a traditional continuous dif- fusion model, an effective approach is to use the predic- tions 2 from fe E) to construct an ODE that preserves the marginal distribution pt at each timestep t (Song et al., 2020a;b). While many such valid ODEs exist for a sin- gle diffusion process, we adopt the formulation from Liu et al. (2022), which is designed to yield a constant expected velocity along the denoising trajectory at each timestep t:
The purpose of these design choices is that L2D aims to ex- tend pre-trained LMs via finetuning approach, rather than learning new models from scratch. While fully adopting dif- fusion training from the start might appear more general, we argue this risks losing some of the training scalability and powerful inductive biases inherent to traditional autoregres- sive modeling which led to their wide establishment in the language domain (Allen-Zhu & Li, 2023a:b). Furthermore, L.2D directly enables leveraging the extensive "system 1" understanding (Kahneman, 2013) already encoded in open foundation models. In fact, by building their existing capabilities we avoid the prohibitive costs required in past attempts to match their performance with diffusion.
8 Zt
dr.
(4)
1 t
The denoising process can then start at t 0 by drawing xt from pure noise and be performed over a sequence of steps where previous predictions are reused to bring 24 to a lower noise level at t + A, toward the direction dr,, In the sim- plest case, this process amounts to Euler integration where + A, x dæ,, However, any ODE solver can be employed with constant or adaptive costs given by fixed discretization levels A, or adaptive accuracy requirements.
3. L2D Implementation
We design our L2D implementation a modular exten- sion for pre-trained transformers to efficiently harness the multi-step scaling capabilities of diffusion while preserving their original single-step generative power. To achieve this, L2D introduces a parallel "diffusion" path to their archi-
Given our parameterization of fe. outputting categorical probabilities over the vocabulary, its predictions cannot be directly used to obtain dr, as with continuous diffusion. However, as shown by Dieleman al. (2022), we can use
3

---

Large Language Models to Diffusson Finetuning
logits of g
architecture. This choice allows us optimize only a subset of the model's parameters with no risk of losing its original ability to process the "uncorrupted" tokens in the context c. We implement the diffusion path, denoted fe.. with a transformer architecture and the same number of blocks as the main path fo. each comprising a subset of its layers (from the MLP blocks and the query layers self-attention). Moreover, to make the most of the pre-trained LM's knowl- edge, all layers in the diffusion path are also initialized with the weights from O,, similarly to Zhang et al. (2023). In practice, we find this initialization enables fast and inex- pensive training, allowing us to optimize the diffusion path with simple low-rank adaptation (LoRA, Hu et al. 2021). Furthermore, this approach greatly minimizes L,2''s mem- ory overhead, as requires us only to store the small LoRA modules by reusing the LM's original weights in both 0a and 0,-
LM perameters 8, L2D parameters 2g
LM Head
RMS Norm
Wa
MLP
f
f
emod
Decoder Layer X
MLP
MIPP
RMS Norm
Ada. RMS Norm
fe
Self Amention
Cross Attention
Diffusion path components. The transformer blocks in the diffusion path comprise a sequence of residual MLP and cross- pross-attention modules. While the MLP modules follow the same structure as the corresponding modules in fa., the cross-attention modules exclusively parameterize query and output linear layers. In particular, during cross-attention, the diffusion token 2 for target token y attends over all previous keys and values already computed from the cor- responding self-attention module in Se,- We only integrate the information processed in fo back to the main path after all blocks, right before the LM's linear head. Specifically, we merge the two paths with an element-wise weighted sum So, + wafe, where the rescaled latents of diffusion token of are added to the latents of the previous token ==1
foz
fe
RMS Norm
Ada. RMS Norm
faf
as Pe.
az B2.Y2
MLP
fer
Seg
Emod
Embedding
MLP
/Embedder
y0
x
Training-time time inputs
[-0[0,1]
Properties and advantages. Our design choices have sev- eral key advantages over prior diffusion architectures tar- geted for multi-token generation (Li et al., 2022; Dieleman et al., 2022). During inference, by saving the latent represen- tation from fe, together with the KV cache, we only need to compute the output of the main path once for each generated token, no matter the number of diffusion steps. Furthermore, as the diffusion token for the k-th target only affects the main path at the previous position, we can fully parallelize training across the sequence batch dimension, sampling timesteps 4 1....4 Lx and diffusion tokens =7, " inde- pendently. By doing this, we greatly mitigate the variance of the diffusion optimization objective, efficiently obtaining independent diffusion losses for all K sequence positions for each sampled input context . .....KK K-1 in the data batches.
L2 Norm
Embeddings
y
x =1)
Figure 2. L2D LMS overview. Training-time sampling of diffu- sion tokens !:k (bottom) and architecture diagram for L2D (top).
tecture, where the hidden representation of the diffusion token 24 is propagated, affecting the frozen "main" LM path only the final layer. In this section, we provide details about each specific I.2D) component, highlighting how our choices ensure scalability and efficiency advantages over prior designs. To accompany our explanations, we show an overview of the L2D pipeline illustrating transformer architectures augmented with our framework in Figure 2.
3.2. L2D Conditioning
Diffusion space vocabulary. To condition fe.. construct the vocabulary containing the discrete set of token embed- dings for the diffusion path e V from the pre-trained token vocabulary of the base LM, denoted V In particu-
3.1. Diffusion Path Parametrization
Structure and initialization. We process the diffusion to- kens Fe within a separate parallel path the LM's original
4

---

Large Language Models to Diffusson Finetuning
lar, we learn a linear mapping We Rdxd to convert each pre-trained embedding Vý to an efficient lower-dimensional embedding in Rd, later rescaled to fixed norm vd:
then construct "guided" target prediction ig for Eqn. 4:
fg - wi x fo 99. By, c) -1-()) x fo L, go. c)2 (7)
where Wig 2 1 is the guidance strength parameter. This method effectively provides diffusion models with targeted generation capabilities and plays a key role in their state-of- the-art computer vision performance (Dhariwal & Nichol, 2021). Moreover, it allows users to trade off general pur- pose with task-specific expertise, potentially allowing to overcome the impractical need for prompt engincering LMs.
V = V WoV. v [W, Vlll'
for all y 1, 1,... [V]
(5)
This normalization step is required to avoid the magnitude of the tokens in V growing unboundedly to minimize the corruption effects from the sampled noises zi ~ [N 1) while training with Equation 3. Instead, as proposed by Dieleman et al. (2022), this approach will make the token embeddings in V naturally spread out, which will lead to their distribution possessing unit variance in each compo- nent across the data manifold. Lastly, we use a small 2-layer "translation module" at the beginning of the diffusion path, mapping back the diffusion tokens embeddings to IR for compatibility with the transformer blocks in soa-
4. Experimental Results
In this section, we provide descriptions for the implemen- tation specifics, training, and evaluation of our new L2D method. Then, we present comprehensive quantitative re- sults, evaluating the benefits of L.2D) across ofate-of-the-art LMs of different sizes from the Llama 3 (Dubey et al., 2024) and Qwen 2.5 families (Hui et al., 2024). Lastly, we focus on Llama 3.2 1B Instruct to study the properties of L.2D in greater depth showing its complementarity to traditional finetuning and search approaches, and also pushing perfor- mance with further advances from the diffusion literature, such as adaptive ODE solvers and mussiffier-rre guidance.
Timestep conditioning. We condition the diffusion path on the current timestep & [0, 1] in three distinct ways. First, based on established practices from the modern diffusion literature, we extract sinusoidal features from and pro- cess them with a small network to output shift and scale parameters for all layer normalizations in feg. Second, fol- lowing Peebles& & Xie (2023), we parametrize additional time-conditioned element-wist rescalings which we apply before summing back the residuals from each transformer block. Third, we make final use of the timestep embeddings to condition the last element-wiss weighting term wa used to scale the outputs of the diffusion path Sa.- However, rather than making this weight the output of a network Way (t), like in the first two cases, we shift wa with the value of wo, (0):
To complement this section, we refer to Appendices A and B for a full set of hyper- parameters, further implementation details, and comprehensive descriptions of our datasets and tasks. Furthermore, we refer to Appendix C for thorough ab- lations of I.2D and our baselines, together with Appendix D for results on additional benchmarks, analyses of additional extensions, detailed per-task performance tables, and com- parisons with additional scaling approaches including the concurrent R1 -ssslee reasoning framework (Guo et al., 2025; Muennighoff et al., 2025).
(t) Wo, (t) wo, wo,( (0).
(6)
The main direct consequence of this parametrization is that the diffusion path will always be multiplied with zeros at / = 0, leaving the original output of fe, unchanged. Thus, this practice ensures that L2D will never trade off the powerful single-step capabilities of the pre-trained LM when 2, is pure noise, and provides a strong inductive bias for the diffusion path to increasingly affect predictions as t grows to and 2, contains more past compute and knowledge.
4.1. Implementing, Training, and Evaluating L2D
As described in Section 2, our main L2D implementa- tion adapts the frozen pre-trained model parameters with LoRA (Hu et al., 2021), efficiently reusing them in the dif- fusion path. Thanks to this design choice, training L.2D) is also relatively inexpensive and scalable, as the number of optimized parameters needed for backpropagation is domi- nated by the weights for the vocabulary of our new diffusion path, which do not grow with more layers. We employ a = 64 for the standard deviation of the base distribution Po. as the discrete nature of language makes token classifi- cation trivial for low noise levels and we want to regularize against the model's most influential diffusion steps being concentrated early on during inference. Similarly to re- lated work (Dieleman et al., 2022: Gulrajani & Hashimoto, 2024), we employ a small diffusion dimension d = 256 and rescale the inputs for fo, such that the standard deviation of each component of xe has expectedly unit variance at
Classifier-free guidance. Finally, we can effectively condi- tion L2D models on additional contextual information about a task or a dataset through classifier-free guidance (Ho& & Sal- imans, 2022). During training, this is done by simply adding to the sinusoidal timestep embeddings an additional learned class embedding from set of J + 1 options yo.... .J-- Here, option go is used as the "null" class embedding ap- plied when no additional contextual information is provided and trained with a given "class-dae dropoutt"" probability. During inference, given access to a task label j E (1,.., J), we can
5

---

Large Language Models to Diffusson Finetuning
all timesteps L. While we did not consider it, we note that further decreasing d = 16 could be option to explore to make our method's s optimized parameter count closer to LoRA and further reduce its cost, which has been shown viable by Dieleman et al. (2022). In all main results, we perform multi-step inference with a midpoint solver and 8 discretization levels, resulting in only 15 evaluations of Soa-
Base language model
36
35.50
34.46
Average Score
34
32-
31.83
30.23
30.51
30
29.84
Typical applications of modern LMs involve processing large fixed context of tokens before tackling the target task, such as user-provided prompts or fetched background resources. We note that this first step does not involve any active generation which could make use of improved reasoning skills. Thus, in contrast to prior diffusion LMs trained with unmasked pre-tnnining language data, we fine- tune L2D on an inslroction-followiing dataset targeted for tasks requiring non-trivial cognitive abilities, such math and coding (Allal et al., 2024). As a consequence, I.2D's learning signal is focused on powering the LM's conditional generation capabilities in complex problems reflecting the conditions that would potentially benefit most from test- time scaling. We train each method for epoch with the AdamW optimizer (Loshchilov, 2017), 100 warmup steps up to tuned learning rate, and a linear decay afterward.
29.15
28.54
28
26
0
0.14
0.29
0.71
0.86
L2D Inference
Figure 3. Diffusion performance evolution. Performance with the progression of the timestep I within L.2''s s diffusion process.
the focus of our targeted training dataset, while optimiz- ing a small fraction of the original weights (less than 6% for Llama 1B and 3.5% for Llama 8B). Although expect- edly more limited, we still find some benefits in general knowledge tasks, indicating that the inductive bias from multi-step inference might also allow the model to better extract pre-acquired knowledge even beyond the finetuning corpus. Overall, we believe these results highlight the gener- ality and effectiveness of L2D, allowing LMs to go beyond pure autoregression and harness some of the scaling prop- erties of the diffusion framework, in line with this work's s primary goal.
We evaluate L.2D on challenging generation tasks broadly focused on math, coding, and general knowledge in a 5- shot setting. We choose to keep our evaluation consistent across all our tasks, without task-specific system prompts, sampling parameters, or involved answer extractions. Since L2D scaling does not provide a direct way for logits manip- ulation and due to the stochasticity requirements of pass @k evaluation for coding, we employ a simple untempered sam- pling strategy for generation. In Appendix D, we compare the effects of our evaluation setup with a close replication of the one from Dubey et al. (2024) on a sample task for all our main baselines, and provide further discussion about our scaling approach's 's current sampling constraints. We con- sider the following tasks: GSM8K (Cobbe al., 2021) and competition MATH (Hendrycks et al., 2021b) to evaluate mathematical reasoning: HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021b) for coding skills; together with MMLU (Hendrycks et al., 2021a) and MMLU-Pro (Wang et al., 2024) to assess knowledge retention. However, due to its targeted design, we note that our training dataset is not meant to provide our models with new real-world know]- edge that would be directly relevant to this last general knowledge category.
To disentangle the benefits of our method from our choice of data, we compare L2D with both LoRA and full weight fine- tuning baselines. As shown in our results, these traditional strategies appear to yield lower overall benefits with even frequent performance drops for the Llama instruct models on the coding problems. This is consistent across our mod- els/task combination with the sole exception of the MBPP task with Qwen 7B, where the performance of L2D (76.79) comes at a close second behind the LoRA baseline (79.60). In Appendix D, we show that finetuning the base versions of Llama does not experience similar drops on coding tasks but fails to achieve competitive performance, suggesting that the private datasets employed in the instruction finetun- ing phases of these models might be superior to our public sources for certain problems. Nonetheless, L2D empiri- cally shows consistent performance gains for all models, even in coding, indicating that its empirical properties are qualitatively different from traditional weight optimization: augmenting the model to leverage past computation and improve future predictions, without suffering the potential downsides of trying to alter its capabilities and knowledge.
4.2. L2D Across Modern Large Language Models
In Table 1, provide quantitative results after training L.2D on top of four different LMs spanning different model families and scales. L2D yields consistent improvements that are particularly evident in the math and coding tasks,
4.3. Analysis and Extensions
Inference-time diffusson scaling. In Figure 1, we show the performance of L.2D while simply scaling the number
6

---

Large Language Models to Diffusion Finetuning
Table Quantitative L2D evaluation Performance aggregated statisties considered math, coding, general knowledge problems. All tasks are evaluated in a consistent 5-shot setting, and coding performance is measured under the pass 10 metric (Chen al., 2021).
Mathematics
Coding
General Knowledge
Overall
Method/Task
GSM8K
MATH
HumanEval
MBPP
MMLU
MMLU-Pro
Average Score
Parameters
Llama 3.2 IB Instruct
13.86
10.00
45.26
50.00
38,46
13.63
28.54
LoRA finetuning
26.29
11.06
42.45
47.20
38.24
14.56
29.97
3M
full finetuning
33.48
12.40
32.08
30.00
39.57
14.70
27.04
1235M
L.D (Ours)
38.86
17.18
47.80
51.80
41.99
15.35
35.50
73M
Qwen 2.5 1.5B Instruct
13.56
16.04
69.18
58.80
57.18
25.54
40.05
LoRA finetuning
45.68
21.79
61.00
63.80
54.47
23.62
45.06
3M
full finetuning
50.45
23.34
65.41
51.80
52.63
22.59
44.37
1543M
L2D (Ours)
53.03
31.91
69.81
66.60
58.53
26.16
51.01
103M
Llama 3.1 8B Instruct
51.97
23.05
83.65
70.20
63.83
31.85
54.09
LoRA finetuning
69.70
27.21
78.62
70.40
60.37
29.38
55.95
13M
full finetuning
65.53
22.59
68.54
56.60
49.28
20.37
47.15
8030M
L2D (Ours)
75.61
35.69
83.65
71.03
66.69
35.28
61.33
281M
Qwen 2.5 7B Instruct
5.61
18.34
87.42
58.60
71.41
38,51
46.65
LoRA finetuning
70.08
33.82
88.05
79.60
69.39
39.12
63.34
10M
full finetuning
69.55
33.67
84.91
69.60
59.47
28.32
57.59
7615M
1.2DD (Ours)
82.80
43.62
91.20
76.79
71.11
39.96
67.58
233M
of diffusion steps performed during inference. Moreover, in Figure 3, we show how performance varies within the L2D diffusion process we expectedly observe a monotonic increase in overall LM performance, clearly analogous to the scaling properties of the diffusion framework for image modeling. Further- more, comparing the scores of the highest and our default choice of 15 evaluations, in Figure Table shows that over 90% of the performance boost be retained without excessive overhead costs. These results evidence that the ef- ficiency benefits of diffusion formulations based rectified flows empirically transfer to the language domain, allowing effective generation in a handful of steps (Liu et al., 2022).
70
Performance
156.12
180
Steps
function of t. In both cases,
60
49.05
50.00
144.34
160
42.77
50
42.50
140
109.83
83
227.43
208.00
40
120
30
80.27
100
18,0x
15.62
20
80
10
60
GSMBK
MATH
HumanEval
MBPP
MMLU
MMLU-Pro
daaptiv diffusion process. In the first section of Table 2, we evaluate scaling compute using L2D with an adaptive second-order Runge- Kutta ODE solver (Fehlberg, 1969), running inference for 1 118.33 steps average. Remarkably, this extension allows the Llama IB model to exceed the highest previous results obtained with the midpoint solver and fixed number 127 steps - notably showing the effec- tiveness of adaptively tuning compute based on the diffusion errors for each generated token. In line with these observa- tions, illustrated in Figure to visibly vary between different tasks. For instance. when dealing with the challenging MATH and coding benchmarks (whose performance is provided in the pass @ 10 regime) the adaptive solver intuitively takes a larger number of steps than for GSM8K. Furthermore, we find that the tasks requir-
Figure 4, Adaptive LM scaling Performance (left) and average steps (right) across tasks using L2D with an adaptive ODE solver.
ing the model to provide an answer in single token without allowing an initial reasoning trace (MMLU and MMLU- Pro) are distinctively the ones where the solver takes the most steps. These findings appear to suggest that integrating advanced solvers can provide L2D the ability to dynamically adapt compute to compensate for increasingly challenging settings and go beyond the current dependence of LMs on heuristic chain-of-thougee traces (Wei et al., 2022).
find the number of steps
Full fa, optimization and weight finetuning. In the sec- ond section Table 2, we show the effects of extending 1.2D with additional trained components. First, we examine

---

Large Language Models to Diffusson Finetuning
29.5
50.57
28.97
51.0
pare and integrate L2D with traditional ways of increasing compute by searching over the space of generated tokens. In particular, using domain knowledge, we combine different effective heuristics to evaluate partial generations, such as the token sequence's likelihoods, lengths, and repetitions which we tune by task category. We then use the resulting scores by performing a beam search over the generated se- quences, keeping a set of 15 hypotheses to match the default number of L.2 steps. Although the benefits of token search with the instruct model remarkably appear beyond tradi- tional weight finetuning, even nearing the ones of T.2DD on coding, we note its cost and complexity are notably superior to our method: each L2D step only executes the far cheaper Se.. while each searched hypothesis even requires its own separate KV cache. Yet, by combining beam search with our method, each with half the original budget, we obtain the highest performance recorded by our extensions. We believe these results show how I.21) makes diffusion highly complementary with traditional approaches for test-time scaling, and its future potential to accelerate progress to- ward advancing its current bounds (Liu et al., 2023; Brown et al., 2024; Snell al., 2024).
Math
29.0
Coding
28
28.45
50.5
28
28.5
50.0
28.
28.0
49.5
39.06
27.22
48.67
2.6
27.5
49.0
27.0
78.07
48.5
26.5
48.0
26.0
47.5
We 0
Wy = 0.5
Wg 1
Wo 1.5
Wg 3
Wg
Classifier-free Guidance Strength
Figure 5. Classifier-free guidance Performance the math (left)
and coding tasks (right) varying 1.2D's 's guidance strength Wig.
Table L2D extensions. Summarized performance statistics.
Method/Metric
Math Coding All tasks Params.
Llama 3.2 IB Instruct
11.93 47.63
28.54
+ L2D
28.02 49.80
35.50
73M
+ I.2D (127 steps)
28.39 51.90
36.24
73M
Classifier free guidance. In the last section of Table 2, we illustrate the effects of integrating classifier-free guid- ance into L2D. As detailed in Appendix B, we partition the training data into the subsets most relevant for math, coding, and general knowledge to reflect the nature of the examined tasks. Then, by simply conditioning foa on the resulting labels during test time, our results demonstrate visible performance gains, further amplified by raising the guidance strength Wg. Yet, shown in Figure 5, we find the optimal value for Wy varies greatly across task categories, with single-answer math tasks benefiting from much higher guidance strengths than the pass@ 10 coding setting. This dichotomy mirrors the well-known trade-off between IS and FID metrics with traditional guided diffusion models (Sali- mans et al., 2016; Heusel al., 2017). In fact, exploiting this property with per-domain tuning of Wy even attains gains similar to running the unguided L2D for 127 steps with only 15. We believe these results further demonstrate the potential of the L.D framework to advance language modeling and bring to LMs some of the key advances that played a crucial role in establishing diffusion as state-of-the- art in computer vision (Dhariwal & Nichol, 2021).
+ L2D (adaptive solver)
30.26
49.53
36.34
73M
+ L2D (full for ft.)
27.60
50.52
35.63
992M
LoRA finetuning
18.68
44.82
29.97
3M
+ L,2 (from LoRA ft.)
29.19
48.45
35.51
76M
+ full finetuning
22.94
31.04
27.04
1235M
+ L.2 (from full ft.)
33.37
43.37
35.84
1309M
+ tuned token search
27.76
49.35
33.83
+ L2D and token scarch
35.95
49.79
38.57
73M
+ L2D (guidance, wy = 1) 28.01
50.57
35.55
73M
+ 1.2D (guidance, Wig = 1.5) 28.65
49.46
35.62
73M
+ 1.2D (guidance, tuned wig) 29.14
50.57
36.26
73M
going beyond LoRA and optimizing the full set of param- eters of fa, (still initialized from the LM's frozen blocks). We find this simple change leads to improvements in L2D's overall performance, especially visible in the coding tasks. However, we note these benefits come with non-negligible additional resource cost, comparable trade-off to the one between traditional LoRA and full weight finetunings of LMs. Second, we study the effects of training L.2D from already finetuned model checkpoints with these same tradi- tional approaches. Our results confirm that L.D is fully com- patible with direct parameter optimization, achieving some of our highest results math where both methods were individually beneficial. Moreover, L2D also largely fills the performance drop observed when directly altering the weights of the Llama model on coding, further evidencing its synergy with traditional weight finetuning approaches.
5. Related Work
There have been several proposed generalizations of the diffusion process for discrete token spaces. Many works in this area focused on sineence-to-squenne tasks and multi- step generation (Reid et al., 2022; Zheng et al., 2023; Sahoo et al., 2024) by extending the seminal D3PM (Austin et al., 2021a). Other discretizations have seen success even for im-
L.2D and search. In the third section of Table 2, we com-
8

---

Large Language Models to Diffusson Finetuning
age and biological data (Hoogeboom et al., 2021; Campbell al., 2024). Of particular relevance, the recent SEDD (Lou et al., 2024) and discrete flow matching (Gat et al., 2024) demonstrated the early potential of this direction, making concrete strides in approaching small-scale traditional LMs.
inspiration from recent work in RL finetuning of diffusion models in computer vision (Black et al., 2023; Wallace et al., 2024). We hope this first work provides new inspiration for unifying the strengths of the foundational autoregressive and diffusion paradigms, which power some of the greatest milestones yet seen in AL.
Most related to our work, continuous diffusion LMs in- stead adapt the Gaussian diffusion framework to the lan- guage domain (Savinov al., 2021; Li et al.. 2022). This area has seen rapid evolution with techniques such self- conditioning (Chen et al., 2022), new approaches to em- bed tokens in continuous spaces (Strudel et al., 2022; Ma- habadi et al., 2023), and extensions to encoder-decoder do- mains (Yuan et al., 2022). In particular, CDCD (Dieleman et al., 2022) brought key advances also employed in this work, such as cross-entropy optimization and token normal- ization. Attempting to scale this line of work, PLAID (Gul- rajani& & Hashimoto, 2024) managed to train IB model outperforming 124M GPT2 (Radford et al., 2019).
Acknowledgements
The authors would like to thank Stefano Peluchetti for pro- viding important discussion and feedback to earlier versions of our work. Furthermore, we would like to thank the anony- mous ICML reviewers and area chair for providing valuable feedback and suggestions to improve our work.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Similar in purpose but diverging from L2D's ' design, other works also aimed at combining the properties of LMs and diffusion. For instance, DiffusionBERT (He et al., 2022) proposed to use a pre-trained BERT model (Devlin, 2018) to accelerate masked diffusion training (Austin et al., 2021a). In addition, the SSD framework (Han et al., 2022; 2023) trained autoregressive and diffusion models together to act on different hierarchical language levels. Lastly, DGLM (Lovelace et al., 2024), proposed to learn a dif- fusion model on the latent space of an encoder-decoder LM to introduce classifier-fre guidance support.
References
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint 23Xh::230300877 2023.
Allal, L. B., Lozhkov, A., Bakouch, E., Blázquez, G. M..
Tunstall, I.., Piqueres, A., Marafioti, A., Zakka, C., von Werra, L., and Wolf, T. Smollm2 with great data, comes great performance, 2024.
6. Discussion and Future Work
In this work, we provide concrete steps toward a new gen- eration of autoregressivell--raiee LMs with the scaling capabilities of diffusion. We show how, after a small fine- tuning phase, L2D enables trading test-time compute for performance, providing higher and highly complementary benefits to further training and search- Additionally, we demonstrate how our new method provides LMs with the key properties of diffusion models, enabling effective adaptive computation and domain guidance exper- tise specific to user demands. However, the L2D frame- work still faces limitations left to be addressed by future work, as by scaling compute using continuous diffusion LM (Dieleman et al., 2022) the model loses direct access to its ground-truth confidence scores, and mechanism for simple logit manipulation - factors we further discuss and analyze in Appendix D. Concurrently with our work, scal- ing LMs with RL finetuning (Jaech et al., 2024; Guo et al., 2025) has emerged had another popular option, yet, still fully grounded in the space of tokens. To this end, com- bining the two approaches by harnessing RL training for L2D is another interesting future research direction, drawing
Allen-Zhu, Z. and Li, Y. Physics of language models: Part 1, context-free grammar. arXiv preprint arXiv:2305.13663, 2023a.
Allen-Zhu, Z. and Li, Y. Physics of language models: Part 1.1. knowledge storage and extraction. arXiv preprint arXiv:2309. 2023b.
optimizations.
Austin, J., Johnson, D. D., Ho, J., Tarlow, D., and Van Den Berg, R. Structured denoising diffusion models in discrete state-spaces. Advances in Neural Information Processing Systems, 34: 19911-1991999, 2021a.
Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732. 2021b.
Ben Allal, I.., Muennighoff, N., Kumar Umapathi, L., Lipkin, B., and von Werra, L. A framework for the evaluation of code generation models. bigcode-evallutionn-ee
2022.

---

Fehlberg, E. Low-order classical Runge-Kutta formulas Large Language Models to Diffusion Finetuning
Bisk, Y., Zellers, R., Bras, R. L.,, Gao, and Choi, Y. Piqa: Reasoning about physical commonsense in natural language. In Thorth-Fourth AAAI Conference Artificial Intelligence, 2020.
Esser, P., Kulal, S., Blattmann, A., Entezari, R., Müller, J., Saini, H., Levi, Y., Lorenz, D., Sauer, A., Boesel, F,, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learning, 2024.
Black, K., Janner, M., Du, Y., Kostrikov, L,, and Levine, S. Training diffusion models with reinforcement learning. arXiv preprint 2..Xiv:2305.3300 2023.
with stepsize control and their application to some heat transfer problems, volume 315. National aeronautics and space administration, 1969.
Brown, B., Juravsky, J., Ehrlich, R., Clark, R., Le, Q. V., Ré, C., and Mirhoseini, A. Large language monkeys: Scaling inference compute with repeated sampling, arXiv preprint arXiv:2407.21787 2024.
Feng. X.,
Wan, Z., Wen, M., MeAleer, S. M., Wen, Y., Zhang, W., and Wang, J. Alphazero-like tree-search can guide large language model decoding and training. arXiv preprint arXiv:2309.17177 2023.
Campbell, A., Yim, J., Barzilay, R., Rainforth, T., and Jaakkola, T. Generative flows on discrete state-spaces: Enabling multimodal flows with applications to protein co-design. arXiv preprint arXiv:2402. 049977 2024.
Fried, D., Aghajanyan, A., Lin, J., Wang, S., Wallace, E., Shi, F, Zhong, R., Yih, S., Zettlemoyer, L., and Lewis, M. Incoder: A generative model for code infilling and synthesis. In The Eleventh International Confer- ence on Learning Representations, 2023. URL https: / //openrevis ew net/ / forum? d=hQwb- 16MEEI.
Chen, M., Tworek, J., Jun, H., Yuan, Q. Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et Evaluating large language models trained on code. arXiv preprint 1xXiv:2107.03374, 2021.
Chen, R. T. Q. torchdiffeq, 2018. URL https:// gi github. www./o///////eeeeeeee
Gat, I., Remez, T., Shaul, N., Kreuk, F., Chen, R. T., Syn- naeve, G., Adi, Y., and Lipman, Y. Discrete flow match- ing. arXiv preprint arXiv:2407.15595 2024.
Chen, T., Zhang. R., and Hinton, G. Analog bits: Gen- erating discrete data using diffusion models with self- conditioning. XrXiv preprint
htl.:2208.04202.. 2022.
Gulrajani, and Hashimoto, T. Likelihood basedd diffu- sion language models. Advances in Neural Information Processing Systems, 36, 2024.
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try are, the ai2 reasoning challenge. 1oXx:://003.0455vv1, 2018.
Guo, D., Yang, D., Zhang, H., Song. J., Zhang, R., Xu, R., Zhu, Q., Ma. S., Wang, P., Bi, X., et al. Deepseek-rl: In- centivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501.1294 2025.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers solve math word problems. arXiv preprint arXiv:21100 10.22110.11166 14168, 2021.
Han, X., Kumar, S., and Tsvetkov, Y. Ssd-Im: Semi- autoregressive simplex-base based diffusion language model for text generation and modular control. arXiv preprint arXiv: 20..:2220.1.333 2022.
Devlin, J. Bert: Pre-training of deep bidirectional trans- formers for language understanding. arXiv preprint arXiv:1810.04805, 2018
Han, X., Kumar, S., Tsvetkov, Y., and Ghazvininejad, M. David helps goliath: Inference-time collaboration be- tween small specialized and large general diffusion Ims. arXiv preprint 1rXxi:2305.17711, 2023.
Dhariwal, P. and Nichol, A. Diffuusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
He, Z., Sun, T., Wang, K., Huang, X., and Qiu, X. Diffu- sionbert: Improving generative masked language models with diffusion models. arXiv preprint arXiv:221.115029, 2022.
Dieleman, S., Sartran, L., Roshannai, A., Savinov, N., Ganin, Y., Richemond, P. H., Doucet, A., Strudel, R., Dyer, C., Durkan, C., et al. Continuous diffusion for categorical data. arXiv preprint arXiv:2211.1508) 2022.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The Ilama herd of models. arXiv preprint 2xXiv2447.21788 2024.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021 la
10

---

Large Language Models to Diffusion Finetuning
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E.,, Song, D., and Steinhardt, J. Measuring math- ematical problem solving with the math dataset. NeurIPS, 2021b.
Lipman, Y., Havasi, M., Holderrieth, P., Shaul, N., Le, M., Karrer, B., Chen, R. T., Lopez-Paz, D., Ben-Hamu, H., and Gat, I. Flow matching guide and code, arXiv preprint arXiv:2412.06266 2024.
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. Gans trained by two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Liu, J., Cohen, A., Pasunuru, R., Choi, Y., Hajishirzi, H., and Celikyilmaz, A. Don't throw away your value model! generating more preferable text with value- guided monte-carlo tree search decoding, arXiv preprint 23X9::2309. /5028, 2023.
Ho, J. and Salimans, T. Classifier-free diffusion guidance. arXiv preprint arXiv:2207. 125998, 2022.
Liu, X., Gong, C.,, and Liu, Q. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint WvXiv:2209.03003 2022.
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion proba- bilistic models. Advances in neural information process- ing systems, 33:6840-6851, 2020.
Loshchilov, I. Decoupled weight decay regularization. arXiv preprint arXiv: 2017.
Hoogeboom, E., Nielsen, D., Jaini, P., Forré, P., and Welling, M. Argmax flows and multinomial diffusion: Learning categorical distributions. Advances in Neural Information Processing Systems, 34:12454-12465 2021.
Lou, A., Meng. C., and Ermon, S. Discrete diffusion model- ing by estimating the ratios of the data distribution. arXiv preprint arXiv:2310. 16834, 2024.
Hu, E. J., Shen, Y., Wallis, P., Allen- Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2 100.0956,,, 2021.
Lovelace, J., Kishore, V., Chen, Y., and Weinberger, K. Q. Diffusion guided language modeling, arXiv preprint arXiv:2408.04220 2024.
Hui, B., Yang, J., Cui, Z., Yang. J., Liu, D., Zhang. L., Liu, T., Zhang, J., Yu, B., Lu, K., et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409. 12186, 2024.
Mahabadi, R. K., Ivison, H., Tae, J., Henderson, J., Belt- agy, I., Peters, M. E., and Cohan, Tess: Text-to- text self-conditioned simplex diffusion. arXiv preprint arXiv:2305.08879, 2023.
Jaech, A., Kalai, A., Lerer, A., Richardson, A., El-Kishky, A., Low, A., Helyar, A., Madry, A., Beutel, A., Car- ney, A., al. Openai ol system card. arXiv preprint arXiv:2412. 1672000. 2024.
Muennighoff, N.. Yang. Z., Shi, W., Li, L.. Fei-Fei, L., Hajishirzi, H., Zettlemoyer, L.,, Liang, P., Candès, E., and Hashimoto, T. sl: Simple test-time scaling, arXiv preprint urXiv:2501.19300 2025.
Kahneman, D. A perspective on judgment and choice: Map- ping bounded rationality. Progress Psychological Sci- ence around the World. Volume I Neural, Cognitive and Developmental Issues., pp. 1-47, 2013.
Nakkiran, P., Bradley, A., Zhou, H., and Advani, M. Step- by-step diffusion: elementary tutorial. arXiv preprint arXiv:2406.0880 08929, 2024.
Karras, T., Aittala, M., Aila, T., and Laine, S. Elucidating the design space of iiffusion-based generative models. Advances in neural information processing systems, 35: 26565-26577, 2022.
Nichol, A. Q. and Dhariwal, P. Improved denoising diffu- sion probabilistic models. In International conference on machine learning, pp. 8162-811 8171. PMLR, 2021.
Peebles, W. and Xie, S. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 4195 4205, 2023.
Kumar, A., Zhuang, V., Agarwal, R., Su, Y., Co-Reyes, J. D., Singh, A., Baumli, K., Iqbal, S., Bishop, C., Roelofs, R., et al. Training language models to self-correct via reinforcement learning. arXiv preprint arXiv:2409. 2024.
Peluchetti, S. Diffusion bridge mixture transports, schrödinger bridge problems and generative modeling. Journal of Machine Learning Research, 24(374):1-51, 2023.
Lee, S., Lin, Z.,, and Fanti, G. Improving the training of rectified flows. arXiv preprint arXiv:2405.20320 2024.
Li, X., Thickstun, J., Gulrajani, I. Liang, P. S., and Hashimoto, T. B. Diffusion-Im improves controllable text generation. Advances in Neural Information Process- ing Systems, 35:4328 4343, 2022.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAl blog, 188)99 2019.

---

Large Language Models to Diffusion Finetuning
Reid, M., Hellendoom, V. J., and Neubig, G. Diffuser: Discrete diffusion via edit-based reconstruction. arXiv preprint arXiv:2210. 16886, 2022.
Uchara, M., Zhao, Y., Wang, C., Li, X., Regev, A., Levine, S., and Biancalani, T. Inference-time alignment in diffu- sion models with reward-guider generation: Tutorial and review. arXiv preprint arXiv:2501.0960 2025.
Sahoo, S., Arriola, M., Schiff, Y., Gokaslan, A., Marro- quin, E., Chiu, J. T., Rush, A., and Kuleshov, V. Simple and effective masked diffusion language models. arXiv preprint arXiv:2406.07522 2024.
Wallace, B., Dang, M., Rafailov, R., Zhou, L.,, Lou, A., Purushwalkam, S., Ermon, S., Xiong, C., Joty, S., and Naik, N. Diffusion model alignment using direct pref- erence optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., and Chen, X. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
2024.
Wang. B., Min, S., Deng, X., Shen, J., Wu, Y., Zettlemoyer, L., and Sun, H. Towards understanding chain-of- -f-thughhht prompting: An empirical study of what matters. arXiv preprint arXiv:2212.1000 2022.
Savinov, N., Chung, J., Binkowski, M., Elsen, E., and Oord, A. v. d. Step-unrolled denoising autoencoders for text generation. arXiv preprint arXiv:2112.0677 2021.
Wang, Y., Ma, X... Zhang, G., Ni, Y., Chandra, A., Guo, S., Ren, W., Arulraj, A., He, X., Jiang, Z., Li, T., Ku, M., Wang, K., Zhuang, A., Fan, R., Yue, X., and Chen, W. MMLU-pro: A more robust and challenging multi-task language understanding benchmark. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Truck, 2024. URL https: // /oopnneeveew.. net forum? coorrmmiidyy 0DM6R2r3.
Snell, C., Lee, J., Xu, K., and Kumar, A. Scaling Ilm test- time compute optimally can be more effective than seal- ing model parameters. arXiv preprint urXiv:2408.03314 2024.
Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, S. Deep unsupervised learning using nonequi- librium thermodynamics. In International conference on machine learning, pp. 2256-2265, PMLR, 2015.
Wei, J., Wang. X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824- 24837, 2022.
Song, J. Meng, C., and Ermon, S. Denoising diffusion im- plicit models. arXiv preprint arXiv:2010.02502. 2020a. 2020a.
Song, Y. and Ermon, S. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Yuan, H., Yuan, Z., Tan, C., Huang, F., and Huang, Seqdif- fuseq: Text diffusion with encoder-decoder transformers. arXiv preprint arXiv:2212. 10325, 2022.
Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Er- mon, S., and Poole, B. Score- generative modeling through stochastic differential equations. arXiv preprinl arXiv:2011.13445 2020b.
Zhang, L., Rao, A., and Agrawala, M. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1836-3847) 2023.
Strudel, R., Tallec, C., Altché, F,, Du, Y., Ganin, Y. Men- sch, A., Grathwohl, W., Savinov, N., Dieleman, S., Sifre, L.. et al. Self-conditioned embedding diffusion for text generation. arXiv preprint 1eXiv:2211.042555 2022.
Zheng, L., Yuan, J., Yu, L.,, and Kong, L. A reparameter- ized discrete diffusion model for text generation. arXiv preprint 200.0773. 2023.
Sutton, R. The bitter lesson. Incomplete Ideas (blog), 13(1): 38, 2019.
Team, G., Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312. 1805, 2023.
Trinh, T. H., WU, Y., Le, Q. V., He, H., and Luong, T. Solv- ing olympiad geometry without human demonstrations. Nature, 625(7995):476 2024.
12

---

Large Language Models to Diffusion Finetuning
Table 3. Implementation hyper-parameteers of the weight finetuning baselines and L.2D.
Hyper-parameter
Weight finetuning
L2D
Flow hidden dimensionality d
256
Timestep embedding dimensionality
256
Diffusion path conditioning hidden dimensionality
256
Noise scaling ratio の
64
Optimizer
AdamW
A
Warmup steps
100
100
Maximum learning rate
1 x 10-5
1 x 10-4
Final learning rate
1 x 10-6
x 10-6
Decay
Linear
Linear
LoRA alpha
64
32
Batch size
32
32
Training epochs
Maximum sequence length
2048
2048 Uniform
Timestep training sampling &
ODE solver
Midpoint
Total diffusion budget T
15
ODE velocity
Constant (Liu al., 2022)
A. Implementation Details
A.1. Language modeling hyper-parameters
We provide full set of the default saraa--araeeters for our baseline approaches and L2D in Table 3, including details about the training. inference, and modeling design of our new approach. In particular, we note that training is performed using the AdamW (Loshchilov, 2017) optimizer with a simple linear decay after a brief warmup phase, as we did not find any significant benefit from integrating more complex cosine schedules. As exemplified and detailed in Appendix C, we swept the learning rate and the other key hyper-parameters of each approach to ensure their efficacy. Our maximum sequence length, however, was selected for efficiency considerations from the quadratically scaling costs of transformer architectures, as we found monotonic performance improvements when increasing its value in preliminary experiments. For our beam search strategy, we score each partial completion based on the model's total loglikelihood divided by LP where L is the current length and PL is syper-parameter to bias toward longer or shorter generations. We sample completions after each steps and, to further improve diversity, also at the end of the full procedure, by treating the resulting final scores as logits, which we divide by a per-task tuned temperature. We found this sampling approach particularly helpful for the pass @ k coding class, which otherwise would be hurt by the lower resulting diversity.
A.2. Inference L2D specifics
As described in Sections 2 and 3, we perform inference by starting the diffusion process from noise To ~ N(0, o²T), and iteratively update I, at each diffusion step using predictions å sampled from the logits. We can perform this process with any ODE solver by discretizing the timestep interval [0, 1] into a set of subintervals and integrating each segment with n-th order approximation. In our default implementation, we integrate (0, 1] with eight endpoints, i.e., S (0, 2, 7 7. 7 7: 1). Thus, with the second-order midpoint method, we perform two forward passes with the diffusion path to integrate each of the seven resulting subintervals [S,, S:+a]: once to compute the initial slope drs, at / = S; with diffusion token input æs,; and a second time at t = S, + h with diffusion token input Is.+h = Is. + hdxs, yielding d where h S St The output of the subinterval integration is then used to compute the value of its endpoint TS..1 = -ss + 2hdrs, th, and the process is repeated for the next subinterval. One final forward pass is then done through the model to obtain and sample from the logits at the final step, resulting in total diffusion budget of T = 15.
2.
a 5
To avoid our proposed sampling procedure with higher-order solvers affecting the final diffusion token prediction by providing an embedding unseen from training averaged from different tokens, we found two implementation details useful
13

---

Large Language Models to Diffusion Finetuning
Table 4. Overview of evaluation datasets for the considered tasks and their characteristies.
Dataset (subset)
Huggingface Repository
Split
Few-shot split
Size
JnstrvctHumanEvll
codeaazzoo/nnstrncchhuaaaaae
test
test prompt
159
MBPP (full)
goagles-rasarrcc--aaattttttt
test
499
GSM8K (main)
openai/gsm8k
test
train
1,319
MATH
Lighteval/MATH
test train
4,347
MMLU (all)
cais/mmlu
test dev
13,666
MMLU-Pro
TIGER-Lab//MLU-Prr
test validation
11,955
PIQA
ybisk/piqa
validation train test
1,838
ARC-Easy
allenai/ai2 are
validation
2,376
ARC-Challenge
allenai/ai2.arc
test validation
1,172
in practice. First, we linearly anneal the sampling temperature for the diffusion velocity toward zero with the progression of t or simply take the most likely velocity. Second, we end the diffusion procedure slightly earlier at t = 1 as similarly done by Dieleman al. (2022). However, we note that we did not find this last implementation detail always necessary with fixed-step first and second-order solvers, and only employed it for the adaptive and RK4 solvers (Fehlberg, 1969) part of extensions evaluated in Section 4.3 and Appendix D). Furthermore, for the analyzed adaptive solver, we employed both absolute and relative thresholds to regulate the step size with values of 3 x 10 each.
As explained in the main text our efficient design allows us to only compute the output of the model's pre-trained main path once during generation by simply storing it together with the KV cache. Then, by exploiting the fact that the main path is independent the diffusion path until the final layer, we simply collect the updated residuals from the smaller- sized fej which take as input the latest diffusion token d, containing the compute and information gathered during all previous diffusion steps. Lastly, we want to acknowledge the torchdiffes (Chen, 2018) library, which we use in our implementation to compute the diffusion path with L.D.
B. Datasets
B.1. Training Dataset Composition
Our targeted training and validation data used for L.2D and our baselines a carefully extracted combination of different subsets the recent large open-source SmolTalk dataset (Allal et al., 2024). In particular, its specific composition was devised for the best performance with traditional weight finetuning approaches and for correlation downstream reasoning tasks such mathematics and coding. The adopted SmolTalk components include the subsets corresponding and openhermes-
to se self-oss-instruce met amat amathga-50k, numina-cot-100 also extract and include part of the examples from the smol-magpie-ulk subset only considering data points "informa: sertiiioonnsninn"
Furthermore,
"math" or "reasonine Lastly, we also note that we discard examples whose length exceeds 2048 tokens, matching the maximum considered sequence length employed during training. In total, the produced training and validation datasets contain 892,283 and 46,848 examples, respectively.
with a category belonging to either "coding", "data-analysii
B.2. Evaluation datasets
As described in Section 4 and line with the training data, our evaluation suite comprises popular and challenging coding, math, and general knowledge tasks. Together with the sample from each of the tasks problems, we provide the model with a fixed 5-shot context from the task's data with either the first equally spaced-out indexes (in case the task data is not i.i.d.) not included in the evaluation. We format the few-shot context past conversation adhering the instruct LMs default tokenizers. In Table 4, we provide summary of the data sources used for our evaluation, including for the additional tasks evaluated in Appendix D. We also provide high-level descriptions of our integrations and answer extraction procedures below:
InstrultHumanEval is a coding dataset designed to assess instruction finetuned models. extends the original Hu- manEyal (Chen et al., 2021) and prepends each prompt with natural language instruction that describes the coding problem.
14

---

Large Language Models to Diffusson Finetuning
The tasks typically involve writing Python functions that meet specific requirements. We compute pass@ @ 1, pass 0, and pass @ 10 by executing model generations provided unit tests.
MBPP (Multiple Basic Programming Problems, Austin et al. 2021b) contains programming problems written in natural language along with their solutions in Python. Following InCoder (Fried et al., 2023) and BigCode Evaluation Har- ness (Ben Allal et al., 2022), we include one unit test case in each prompt. Similarly, pass I, pass 5, and pass@ /0 are calculated by verifying model generations on unit tests.
GSM8K (Grade School Math 8K, Cobbe al. 2021) a dataset of grade school math word problems. Each problem requires breaking down the solution into several steps and applying basic arithmetic operations. A response has the format " "'multistep reasoning) ### (final answer)", ". We extract the final answer and compare it against the ground truth to compute exact match accuracy.
MATH (Mathematics Aptitude Test of Heuristics, Hendrycks et al. 2021b) consists of problems from mathematics competitions, including the AMC 10, AMC 12, AIME, and more. Each MATH response describes a full step-by-step solution and the final answer wrapped in \boxed }. We match and parse the content in \boxed ( ), then compute accuracy by comparing with the ground truth. In case, no \boxed } answer is found, we simply take the final generated number the model's 's response.
MMLU (Massive Multitask Language Understanding, Hendrycks et al. 2021a) is a broad evaluation benchmark testing knowledge across 57 different subjects, including humanities, STEM, social sciences, and more. The questions are in a multiple-choice format and require both general knowledge and specialized understanding. Options in question are marked by letters from "A" to "D", and an answer is a single option letter. We report the accuracy of predicted option letters.
MMLU-Pro (Wang et al., 2024) presents more challenging multiple-choice questions that focus on professional knowledge. It extends options in MMLU to 10 options (i.e. "A" to "J").
B.3. Classifier-free Guidance Conditioning
As described in Sections 3 and 4, in our classifier-free guidance extension, L2D conditions on explicitly provided labels that reflect the nature of the examined tasks. Matching the task categories from our tables, "general knowledge" labels to partition both the training and evaluation dataset for the considered tasks, shown in Table 5, We believe that more fine-grained partitionings might allow L.2D to develop even more nuanced capabilities. To this end, we believe our approach might have future untapped potential for the personalization of LMs, where different labels could provide the model contextual information to target behavior toward individual users through diffusion.
the "math", "coding", and
C. Parameter Studies and Ablations
C.1. Learning Rate
At the beginning of this work, we performed thorough LR sweeps for both L2D and the finetuning baselines on our training data, In practice, we found L2D benefits from much higher LR than direct weight finetuming, which we believe to be in line with our observation that traditional optimization can much more easily incur unwarranted knowledge loss than our new method. In Table 6, we provide summarized results locally modifying this parameter within (1 x 10 .3 x 10 We note that going lower than 1 x 10s makes the performance of the finetuning baselines regress rapidly to the base model, defeating the very purpose of these approaches.
10*).
C.2. Diffusion Schedule
As described in Sections 2 and 4, the choice of the standard deviation a for the base distribution po is critical, implicitly defining the process that our deffiusion-augmeent the most relevant steps at either end of the diffusion interval, wasting both training and inference compute. In Table 7, we provide results with alternative values for a around our choice of a = 64. As suggested by Dieleman et al. (2022), we note that the optimal diffusion schedule might evolve throughout training, with recent diffusion advances like wirp-ng being immediate directions for potential future improvements of our framework.
LM will be learning. Too small or too large of a choice might concentrate
15

---

Large Language Models to Diffusion Finetuning
Table 5. Classifier-free guidance categories of the training and evaluation task datasets.
Dataset
Category
Guidance Category
Smol'Tal
met.amathaa-500 100k
math
SmolTalk
math general knowledge coding
SmolTalk
openhermes-100X
SmolTalk
sel sess-
SmolTalk
se self--oss-iasrrrtt/tttt---aaall
general knowledge
SmolTalk
se sell--oss--ssttccccc//ffmoottoos
general knowledge
SmolTalk
self-ose--nnstruutt/mttt
math
SmolTTal
self salffooss-hnstruct/rreessst
general knowledge
HumanEval
default
coding coding math
MBPP
default
GSM8K
default
MATH
default
math
MMLU
default
general knowledge
MMLU
abstract_algette
math math
MMLU
co college.mathematiie
MMLU
e leementtarryyymmhhmatiic
math
MMLU
higghsschhollmatheemticcc.
math
MMLU
highhscchool.stttissies.
math coding general knowledge math general knowledge
MMLU
high.school.ccmmptte..cciicc
MMLU-Pro
default
MMLU-Pro
math
PIQA
default
ARC-Easy
default
general knowledge
ARC-Challenge
default
general knowledge
C.3. Initialization
As detailed in Section 3, we initialize the weights of the diffusion path from the corresponding layers in the main path. The main goal behind this choice is to incentivize the model to learn representation of the diffusion tokens close to one of the main path tokens and try to reuse the computation ability already present in the main path from pretraining. Our key hypothesis is that learning such a solution would be easier and provide a better inductive bias than learning the diffusion path from scratch. In Table 8, provide this explicit ablation validate our choice, showing comparison with the full-finetuned version of L.2D equate the number of optimized parameters. However, note that the performance of the randomly initialized L2D appears even lower than the less-costly LoRA version of our method corroborating the usefulness and reusability of the parameters of open foundation models.
C.4. Velocity Computation
As detailed in Section 2, compute the target velocity, simply sample ye from the output distribution of our L2D model fo (x f, c). Then, we set è = Vg,- In contrast, Dieleman et al. (2022) opt to take the expectation over fo(xs, t, c) directly as a weighted sum:
f Σ fol x Vjr
(8)
We provide results in Table 9, empirically comparing these choices. While in principle Dieleman et al. (2022)'s choice has the same expected value but lower variance than our sampling approach, we hypothesize the empirical advantage of our method when using deterministic ODE solvers comes from reinjecting some structured stochasticity, which Karras et al. (2022) showed might allow to better harness some of the self-correcting properties of the diffusion framework.
16

---

Large Language Models to Diffusion Finetuning
Table 6. Performance and aggregated statistics for different learning rates with L.2D and traditional weight finetuning.
Mathematics
Coding
General knowledge
Overall
Method/Metric
GSM8K MATH HumanEval
MBPP
MMLU
MMLU-Pro
Average Score Parameters
Llama 3.2 IB Instruct
13.86
10.00
45.26
50.00
38.46
13.63
28.54
+ L2D (LR 10-5)
38.41
17.39
44.65
43.25
43.47
15.57
33.79
73M
+ L2D (LR=
10
39.70
17.90
45.91
51.19
42.29
15.32
35.38
73M
L.2 (LR
10 10"*)
38.86
17.18
47.80
51.80
41.99
15.35
35.50
73M
+ full ft. (LR= = 10
33.48
12.40
32.08
30.00
39.57
14.70
27.04
1235M
+ full ft. = 10
26.74
10.28
29.56
22.20
33.71
13.05
22.59
1235M
full ft. x 1
15.91
7.54
20.75
7.40
25.91
11.37
14.81
1235M
Table 7. Performance and aggregated statistics for L,2D trained and evaluated with different standard deviation a of the base distribution Po (1, o r
Mathematics
Coding
General knowledge
Overall
Method/Metric
GSMSK
MATH
HvmanEval
MBPP
MMLU
MML.U-Pr
Average Score
Parameters
Llama 3.2 IB Instruct
13.86
10.00
45.26
50.00
38.46
13.63
28.54
+ L2D (a = 64)
38.86
17.18
47.80
51.80
41.99
15.35
35.50
73M
L2D (a 32)
37.50
16.82
45.28
52.38
42.22
15.54
34.96
73M
L2D (a 128)
41.06
18.45
44.03
46.83
42.09
16.06
34.75
73M
D. Extended Results
D.1. Inference ODE Solvers
Our main experiments in Section were collected with second-order midpoint solver, an empirically robust choice the traditional diffusion framework for different computational budgets (Lipman al., 2024). When evaluating our framework with adaptive solver, we also employed second-order adaptive Runge- Kungee-Kutaa (RK) solver (Fehlberg. 1969). Here, extend these results, analyzing additional fixed-sized solvers with different properties, understand their behavior with L2D and our relatively small default diffusion budget. In Table 10, we provide results with the first-order Euler and fourth-order RK methods, evaluated for 15 and 17 steps (the lowest number that allows fourth-order integration above our default budget). In particular, we find that simpler solvers seem to work best, with Euler integration even slightly outperforming our midpoint method. These results appear consistent with the literature on fast diffusion methods (Liu et al., 2022). However, we note they might not necessarily hold for higher diffusion budgets as well (Karras et al., 2022).
D.2. Timestep Schedules
For simplicity, in this work, we opted to sample timesteps E [0, 1] uniformly during training. However, we note that there exist other choices recently developed that have been shown to provide empirical benefits for diffusions based on rectified flows (Esser al., 2024). Thus, we validate the potential of these recent contributions for L2D and evaluate our method with the "cosmap" timestep schedule from Nichol & Dhariwal (2021). As shown in Table 11, this extension appears to yield consistent improvements over uniform sampling in all but one task, confirming how complementary advances from the diffuuion literature can provide further improvements toward improving test-time LM scaling through our new framework.
D.3. L2D Performance on Additional Tasks
In Table 12, we provide the performance of L.2D and traditional weight finetuning strategies on additional evaluation settings and tasks from the language modeling literature. In particular, we report the pass@ @ and pass @5 metrics for the HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021b) coding benchmarks, together with performance on the PIQA (Bisk et al., 2020), ARC-Easy, and ARC-Challenge (Clark et al., 2018) question-ange answering tasks. We note that these last three tasks are less relevant than the ones considered in Section 4, given our data curation strategy targeted toward math
17

---

Large Language Models to Diffusion Finetuning
Table Performance aggregated statisties L2D ablating our reuse of the main pretrained path's 's weights 0 to initialize the weights of the diffusion path Oa.
Mathematics
Coding
General knowledge
Overall
Method/Metric
GSM8K
MATH
HumanEval
MBPP
MMLU
MMLU-Pro
Average Score
Parameters
Llama 3.2 IB Instruct
13.86
10.00
45.26
50.00
38.46
13.63
28.54
+ L2D (full for ft.)
37.50
17.71
49.05
52.00
41.98
15.52
35.63
992M
+ L2D (full fe, ft. from scratch)
38.03
17.46
42.77
51.19
41.71
14.71
34.31
992M
Table Performance and aggregated statisties for L2D evaluated with the å estimate proposed by Dieleman et al. (2022) compute the velocity.
Mathematics
Coding
General knowledge
Overall Average Score Parameters
Method/Metric
GSM8K MATH HumanEval MBPP
MMLU MMLU-Pro
Llama 3.2 IB Instruct
13.86
10.00
45.26
50.00
38.46
13.63
28.54
+ L2D
38.86
17.18
47.80
51.80
41.99
15.35
35.50
73M
+ L2D (velocity from expectation)
37.12
18.33
46.23
51.60
41.31
14.96
34.92
73M
and coding problems. Remarkably, however, while weight finetuning appears to deteriorate performance across several model-task combinations, L2D once again provides much more consistent benefits throughout. These results are in line with our observations in the main text that by focusing on augmenting rather than altering the original model, L.D does not seem to suffer the potential pitfalls of traditional weight finetuning atop powerful instruct models.
D.4. L2D and Best-of-N Scaling
In Section 4, we evaluate two approaches for "best-of-N" scaling. First, the token search baseline is itself an advanced version of best-of-N scaling, where the tuned beam-search scores are used as a heuristic metric to assess which the best response. Second, we also consider best-of-N using ground-truit truth correctness, which assumes access to an oracle verifier and is typically only considered for coding, where the oracle could come in the form of compiler and a set of test cases to solve. In fact, this is precisely what the pass@ @ K metric used for Humaneval/MBBP considers. In Table 13, we also provide the @ performance of L.2D) and traditional weight on the remaining set of math and general knowledge tasks using the Llama IB model, which could be viewed as an upper bound for any critic-based inference-scaling approaches. As shown, with access to an oracle verifier, simple repeated sampling is likely the preferred scaling approach. However, consistently with prior results, L.2D is expectedly complementary to this scaling approach and remains an effective, viable strategy to push performance beyond best-of-N's inevitable saturation.
D.5. L,D Performance with Base Models
As exemplified for the coding tasks in Section and further evidenced in the above subsection, some of the private data involved in the instruction-tume -tuning phases of state-of-the-art models seems to be more effective than publicly available sources. However, to validate our curated reasoning dataset, we trained and evaluated both our weight finetuning baselines starting from the base Llama 3.2 1B model. As shown in Table 14, without previous instruction tuning, both strategies seem to provide remarkable benefits across all considered tasks, with full weight finetuning achieving the highest overall scores, in clear contrast to the results atop the Llama 3.2 IB Instruct model.
D.6. Current Sampling Constraints and Evaluation Differences
As described in Section 6, the new effective test time scaling framework of I.2D still faces limitations, in terms of its fiexibility and interpretability, left to be addressed by future work. In particular, scaling compute using a continuous diffusion LM (Dieleman et al., 2022) always introduces stochasticity into generation and makes the model's log likelihood directly dependent on the diffusion timestep t. Thus, this makes LMs evaluated with multi-step L2D scaling lose direct access to their ground-truth confidence scores and removes their inherent mechanisms for simple logit manipulation. However, unlike
18

---

Large Language Models to Diffusion Finetuning
Table 10. Performance and aggregated statisties for 1.2D evaluated with fixed-step ODE solvers of different order.
Mathematics
Coding
General knowledge
Overall
Method/Metric
GSM8K
MATH
HumanEval
MBPP
MMLU
MMLU-Pro
Average Score
Parameters
Llama 3.2 IB Instruct
13.86
10.00
45.26
50.00
38,46
13.63
28.54
+ L2D (midpoint, 15 steps)
38.86
17.18
47.80
51.80
41.99
15.35
35.50
73M
L2D (Euler, 15 steps)
39.77
17.30
48.42
50.20
42.19
15.30
35.53
73M
L2D (RK4, 17 steps)
39.70
17.28
45.91
51.19
42.41
14.95
35.24
73M
Table 11. Performance and aggregated statistics for I.2D trained with the sampling schedule from (Nichol & Dhariwal, 2021) for the diffusion timestep t.
Mathematics
Coding
General knowledge
Overall
Method/Metric
GSMSK
MATH
HumanEval
MBPP
MMLU
MML.U-Pro
Average Score Parameters
Llama 3.2 IB Instruct
13.86
10.00
45.26
50.00
38.46
13.63
28.54
+ L2D
38.86
17.18
47.80
51.80
41.99
15.35
35.50
73M
+ L2D (cosmap schedule)
39.92
18.38
48.43
51.60
42.06
15.36
35.96
73M
previous work trying to learn language diffusion models from scratch (Li al., 2022; Dieleman et al., 2022; Gulrajani & Hashimoto, 2024), we note that when requiring access to the model's probabilities or for tasks that might benefit on particular sampling strategies, L2D's design always offers the possibility of running the model for a single step fully preserving the original capabilities and behavior of autoregressive I.Ms.
As detailed in Section 3, we choose to keep our evaluation consistent across all our tasks, without any task-specific system prompts, sampling parameters, or involved answer extractions. In Table 15, we compare the effects of our evaluation setup with close replication of the one from Dubey al. (2024) the GSMSK task for all our main baselines. In particular, this implementation uses a much relaxed answer extraction that first looks for specific answer patterns even beyond the chain-of -f of-thhuugt examples, and if no answer is properly formatted, it still attempts to match the solution with any numerical value present in the response. Furthermore, this implementation also uses eight particular chain-of-thought examples and a greedy sampling generation strategy, even though L2D's behavior is not affected by logit manipulations. As shown in our results, all baselines appear to consistently benefit from our considered constraint relaxations with the additional taain-of-thoure samples and the different generation strategy, especially the base Llama models, whose performance becomes very close to the one reported by Dubey et al. (2024) on this task implementation. We note that, the small 1B models, also LoRA and full finetuning become significantly more effective, and only by combining them with L2D we were able to improve upon their performance for Llama. However, on the larger 8B Illama model, we find their inclusion comes at a detriment to the original model's performance, while L2D scaling leaves it relatively close to the original. Overall, we believe these results highlight once again that L2D should be viewed not just to replace but also to be used in combination with traditional weight finetuning, bringing to traditional LMs new compounding benefits that can be enabled on demand with extra compute.
D.7. Reasoning and Other Test Time Scaling Approaches
In Table 16, we evaluate L.2 with the concurrent RL-induced LM reasoning framework for test time scaling (Jaech et al., 2024; Guo al., 2025). Since RL training requires expensive multi-node settings, far beyond L2D training, and appears mainly effeective on very large LMs, we added results with the pre-trained DeepSeck RI Distill Qwen 1.5B reasoning model (Guo et al., 2025). We used this model both as an additional baseline and an extra base model from which train L2D. As highlighted in our results, since the DeepSeek R model is trained on recent private dataset, heavily focused on math, we find its performance exceeds the original Qwen 1.5B Instruct model on this task category. However, we find this comes at an expected actual loss in performance on coding and general knowledge, which our L2D approach avoids. Moreover, further fine-tume tuning this baseline with L2D achieves the highest results on math, even surpassing the much larger 7B and 8B non-RL, models - as well as recovering a large part of the performance loss on the other tasks. In line with the
19

---

Large Language Models to Diffusion Finetuning
Table 12. Performance and aggregated statisties for 1.2D and our main ablations all Ilama and Qwen models for additional pass @
settings and tasks.
Coding extended results
Additional fasks
Overall
Method/Task
Eva@ HumanEval@" @1 MBPP@5 MBPP@1
ARC-Easy ARC-Challenge PIQA Parameters
Llama 3.2 IB Instruct
38.54
20.94
43.88
23.16
63.68
44.20
55.98
LoRA finetuning
33.03
16.64
40.24
20.28
64.56
45.48
57.56
3M
full finetuning
27.47
16.42
22.39
7.66
67.59
43.60
58.11
1235M
L.2 (Ours)
41.14
25.09
45.83
28.40
67.68
47.95
56.03
73M
Qwen 2.5 1.5B Instruct
59.42
32.58
50.41
25.66
89.23
75.09
76.44
LoRA finetuning
54.13
30.60
54.25
29.14
86.20
70.99
74.43
3M
full finetuning
55.63
30.50
42.50
17.72
86.70
70.90
74.05
1543M
L2D (Ours)
62.41
39.21
59.99
38.40
89.60
75.68
76.79
103M
Llama 3.1 8B Instruct
78.32
55.47
65.04
47.70
92.59
80.20
81.23
LoRA finetuning
71.66
44.37
64.08
41.22
90.61
78.84
78.94
13M
full finetuning
60.00
33.24
48.81
25.00
81.57
67.41
71.49
8030M
L2D (Ours)
77.10
53.96
66.08
48.12
92.97
82.85
83.64
281M
Qwen 2.5 7B Instruct
83,83
67.30
54.60
39.88
96.04
89.59
86.51
LoRA finetuning
81.17
53.02
72.76
47.42
95.33
87.63
85.31
10M
full finetuning
77.22
49.28
61.75
33.52
91.88
80.80
77.75
7615M
1.2D (Ours)
86.27
68.58
70.53
48.43
96.04
88.65
86.74
233M
other results, we believe these findings confirm that our new method should viewed potentially complementary to this recent reasoning framework. However, note that evaluating these reasoning models distilled from RL was over 10x more expensive than vanilla L2D and did not work out-of-the-box, requiring us modify the prompts and relax the answer style responses. Furthermore, while this has not been explored
extraction code for compatibility with "<think>//< in this work, scaling training data and its quality, or even including an additional RL phase, could potentially allow L2D to achieve similar "latent" reasoning abilities on its own, and remains an open research direction.
In Table 17, we also compare L.2D with higher quality chain-of-thought scaling (CoT) (Wei et al., 2022). For this comparison, we made versions our tasks with new chain-of-thoughe few-shot examples designed to elicit better and longer reasoning. In particular, these examples were obtained by prompting Claude Sonnet 3.7 to provide more effective and longer chain-of- thought based on the heuristics recommended in Wang et al. (2022). We note this change significantly increased inference time, especially for our multiple-choice tasks, going from the models generating single letter answer directly producing lengthy reasoning traces beforehand (averaging 84 new tokens). As shown by our results, this tuned chain-of-thouge prompting strategy indeed achieves improvements for both the base Llama model and our other finetuning baselines, albeit lower than our previous baseline results and I.2D. Furthermore, in line with our other findings, using L.2D) models together with chain-of-thought prompting yields compounding test-time benefits, which believe again evidences the synergy between our method and orthogonal scaling approaches that work by increasing generation length.
D.8. Full L2D Extensions Results
In Tables 18 and 19, we provide the full of results for the extensions L2D analyzed in Section As discussed in the main text, we find the effects of adaptive solvers and test-time advances like classifier-fres guidance to be of remarkable importance, considerably beyond simply scaling the number of training parameters. We find these results quite analogous to similar findings from the diffusion literature (Karras et al., 2022), showing how I.2D has the potential to open doors beyond the current language modeling framework, where data and training compute are the current predominant approaches for scaling.
20

---

Large Language Models to Diffusion Finetuning
Table 13. Performance L.2D and our main baselines additional pass@ settings Math General Knowledge tasks, to provide strict performance upper bound for any best-of-N approach.
Mathematics
Coding
Method/Metric
GSM8K
MATH
MMLU
MMLU-Pro
Llama 3.2 Instruct pass@1
13.86
10.00
38.46
13.63
LoRA finetuning pass@]
26.29
11.06
38.24
14.56
+ full finetuning pass @
33.48
12.40
39.57
14.70
+
L2D pass @
38.86
17.18
41.99
15.35
Llama 3.2 Instruct p
40.83
27.96
75.24
42.97
LoRA finetuning p
58.64
30.68
73.60
43.39
+ full finetuning pass@5
63.33
32.35
76.59
42.94
+
L2D 5
67.35
40.05
77.34
43.74
Llama 3.2 IB Instruct pass
52.65
37.20
87.87
60.80
LoRA finetuning pass 10
69.47
41.64
84.38
61.16
+ full finetuning pass @ 10
72.58
42.67
89.07
58.66
+ L2D pass @ 10
75.68
49.93
89.87
62.05
Table 14. Performance and aggregated statisties for the LoRA (Hu al., 2021) and full weight finetuning haselines both instruct and non-instruct versions of the Illama 3.2 IB L.M.
Mathematics
Coding
General knowledge
Overall
Method/Task
GSM8K
MATH
HumanEval
MBPP
MML.U
MMLU-Pro
Average Score
Parameters
Llama 1B Instruct
13.86
10.00
45.26
50.00
38.46
13.63
28.54
LoRA finetuning
26.29
11.06
42.45
47.20
38.24
14.56
29.97
10M
full finetuning
33.48
12.40
32.08
30.00
39.57
14.70
27.04
7615M
Llama 3.2 IB
2.05
2.10
16.98
11.60
26.51
11.20
11.74
LoRA finetuning
4.55
2.53
22.64
28.80
25.39
11.42
15.89
10M
full finetuning
17.42
5.68
23.75
12.80
28.62
11.74
16.67
7615M
21

---

Large Language Models to Diffusion Finetuning
Table 15. Performance of L.2D and our main baselines relaxed GSM8K (Cobbe et al., 2021) task implementation, with more permissive answer extraction, eight chain-of-thought prompls, and greedy sampling, matching the task implementation from Dubey (2024).
Mathematics
Method/Task
GSM8K
GSM8K (greedy/relaxed)
Parameters
Llama 3.2 1B Instruct
13.86
39.24
+ LoRA finetuning
26.29
41.24
3M
+ full finetuning
33.48
46.36
1235M
+ L.2
38.86
43.41
73M
+ L,2D (from full ft.)
46.89
48.11
1308M
Qwen 2.5 1.5B Instruct
13.56
31.06
+ LoRA finetuning
45.68
62.66
3M
+ full finetuning
50.45
63.18
1543M
+ L,2
53.03
59.85
103M
Llama 3.1 8B Instruct
51.97
81.21
+ LoRA finetuning
69.70
77.73
13M
+ full finetuning
65.53
75.53
8030M
+ L2D
75.61
80.98
281M
Qwen 2.5 7B Instruct
5.61
47.42
+ LoRA finetuning
70.08
80.53
10M
+ full finetuning
69.55
81.59
7615M
+ L2D
82.80
82.58
233M
Table 16. L2D comparison and integration with Rl-style RL. Summarized performance statistics. *Indicates modified task prompts. answer extraction, and evaluation compatible with <Chink>/<onswe>>>>>>>>>>>>>>>>>> style responses.
Method/Metric
Mathematics
Other tasks
All tasks
Parameters
Qwen 2.5 1.5B Instruct
14.80
$2.67
40.05
L2D
42.47
55.28
51.01
103M
66.33
23.66
37.88
1543M
L2D
69.35
28.73
42.27
1647M
Table 17. L2D comparison and integration with chain-of-thought scaling, Summarized performance statisties.
Method/Metric
Mathematics
Coding
tasks
Paramseters
Llama 3.2 IB Instruct
11.93
47.63
28.54
+ LoRA finetuning
18.68
44.82
29.97
3M
+ full finetuning
22.94
31.04
27.04
1235M
+ L2D
28.02
49.80
35.50
73M
+ CoT
13.97
48.81
29.64
+ LoRA finetuning and CoT
19.35
46.84
30.94
3M
+ full finetuning and CoT
23.97
34.27
28.48
1235M
+ L2D and CoT
29.04
50.29
36.00
73M
22

---

Large Language Models to Diffusion Finetuning
Table 18. Full per-task performance and aggregated statistics for the L,2D extensions from Section 4.
Mathematics
Coding
General knowledge
Overall
Method/Metric
GSM8K
MATH HumanEval
MBPP
MMLU
MMLU-Pro
Average Score Parameters
Llama 3.2 IB Instruct
13.86
10.00
45.26
50.00
38.46
13.63
28.54
+ L2D
38.86
17.18
47.80
51.80
41.99
15.35
35.50
73M
+ L.2D (127 steps)
38.86
17.92
52.20
51.60
41.87
14.96
36.24
73M
+ L2D (adaptive solver)
42.50
18.01
49.05
50.00
42.77
15.68
36.34
73M
L2D (full fo, ft.)
37.50
17.71
49.05
52.00
41.98
15.52
35.63
992M
+ LoRA finetuning
26.29
11.06
42.45
47.20
38.24
14.56
29.97
3M
+ L2D (from LoRA ft.)
40.15
18.24
45.91
51.00
42.79
14.98
35.51
76M
+
full finetuning
33.48
12.40
32.08
30.00
39.57
14.70
27.04
1235M
L.2D (from full ft.)
46.89
19.85
43.33
43.40
44.34
17.23
35.84
1309M
+ token search
36.44
19.07
48,91
49.80
35,33
13.44
33.83
+ L2D and token search
46.21
25.69
47.80
51.79
43.29
16.65
38.57
73M
L2D (guidance, wa
38.26
17.76
49.54
51.60
41.31
14.85
35.55
73M
+
L,2D (guidance, W, 1.5)
39.24
18.06
47.73
51.19
42.17
15.35
35.62
73M
L2D (guidance, tuned wy)
40.23
18.06
49.54
51.60
42.52
15.62
36.26
73M
Table Full per-task performance and aggregated statistics for L2D classifier- guidance extension from Section 4 evaluated with different classifier strengths Wi..
Mathematics
Coding
General knowledge
Overall
Method/Mctric
GSM8K
MATH
HumanEval
MBPP
MMLU
MMLU-Pro
Average Score
Parameters
Wg = 0
37.20
17.23
46.54
50.79
40.94
14.71
34.57
73M
Wi 0.5
36.89
17.62
46.54
51.19
41.06
14.70
34.67
73M
Wy =
38.26
17.76
49.54
51.60
41.31
14.85
35.55
73M
Wy 1.5
39.24
18.06
47.73
51.19
42.17
15.35
35.62
73M
W 2
38.86
18.04
47.73
50.20
42.32
15.32
35.41
73M
W = 3
40.23
17.71
46.54
49.60
42.52
15.62
35.37
73M
23


Reviews:
[{"review_id": "iKIKRGPDzY", "reviewer": "Reviewer_uZss", "summary": "The paper introduces L2D, a method that integrates the scaling properties of diffusion models into pre-trained language models (LMs) to enhance reasoning skills and computational scalability. L2D improves pre-trained LMs on math, coding, and various reasoning tasks, outperforming LoRA and full fine-tuning methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their feedback and the time they dedicated to our review.\n\n**Experiments**\n\nWe added clarifications and experiments to our revised work to address the reviewer’s concerns. We hope these will strengthen the argument that L2D is a novel orthogonal method, highly complementary with scaling approaches based on increasing generation length. \n\n**CoT scaling baselines**\n\nFollowing the reviewer’s feedback on CoT prompting, we made versions of our tasks with new CoT few-shot examples designed to elicit better and longer reasoning. In particular, these examples were obtained by prompting Claude Sonnet 3.7 to provide effective CoT based on the heuristics proposed in [1]. We note this change significantly increased inference time, especially for our multiple-choice tasks, going from the models generating a single letter answer directly to producing lengthy reasonings beforehand (averaging 84 new tokens). \n\n[Please find a summary Table of the added results at this link.](https://anonymous.4open.science/r/rebuttal_l2d-4B0B/cot_results_summary.png)\n\nAs shown, this tuned CoT prompting strategy indeed achieves improvements for both the base Llama model and our other finetuning baselines, albeit lower than our previous baseline results scaling test-time compute with token search (third section of [Table 2](https://anonymous.4open.science/r/rebuttal_l2d-4B0B/table2.png)) and L2D. Furthermore, in line with our other findings, using L2D models together with CoT prompting yields compounding test-time benefits, which we believe evidences the synergy between our method and this orthogonal approach.\n\n**R1-style RL for reasoning**\n\nWhile the R1 paper [2], spurring the recent focus on reasoning from RL, was uploaded on the web on January 23rd, the same date as the abstract submission deadline for ICML 2025 - we understand the importance of considering this relevant line of work.\n\nSince RL training requires expensive multi-node settings (far beyond L2D and the resources for the project) and appears mainly effective on very large LMs, we added results with the pre-trained DeepSeek R1 Distill Qwen 1.5B reasoning model. We not only used this model as an additional baseline but also as an extra base model from which to train L2D.\n\n[Please find a summary Table of the added results at this link.](https://anonymous.4open.science/r/rebuttal_l2d-4B0B/r1_results_summary.png)\n\nAs the DeepSeek R1 model is trained on a private dataset, heavily focused on Math, we find its performance exceeds the original Qwen 1.5B Instruct model on this task category. However, we find this comes at an expected actual loss in performance on coding and general knowledge, which our L2D approach avoids. Moreover, further fine-tuning this baseline with L2D achieves the highest results on Math, even surpassing the much larger 7B and 8B non-RL models, and recovers a large part of the performance loss on the other tasks. In line with the other results combining L2D with other traditional test-time scaling approaches, we believe these findings suggest that our new method should be viewed as complementary also to RL reasoning.\n\nHowever, we note that evaluating these reasoning models distilled from RL was over 10x more expensive than vanilla L2D and did not work out-of-the-box, requiring to modify the prompts and relax the answer extraction code for compatibility with `<think>/<answer>` style responses. \n\nFinally, we extended our conclusion to mention that training L2D itself with concurrent R1-style RL methods could also be another interesting future research direction, taking inspiration from recent work in RL finetuning of diffusion models in computer vision [4, 5].\n\n**Related work**\n\nWe would like to thank the reviewer for providing us with additional references to the related literature on test-time scaling of LMs and diffusion. We have included all their suggestions and also connections to the concurrent RL-based line of research (e.g., [2, 3]) in the new revision of our work.\n\n**Questions**\n\nAs described in Sections 3.1 and 4.1, all our main implementations used for the [Table 1 results](https://anonymous.4open.science/r/rebuttal_l2d-4B0B/table1.png) are already optimizing the diffusion path of L2D with LoRA, which is precisely what allows L2D to be over an order of magnitude more parameter-efficient than full weight finetuning. While optimizing all L2D parameters appears to further increase performance, especially on coding ([Table 2](https://anonymous.4open.science/r/rebuttal_l2d-4B0B/table2.png)), it comes with non-negligible additional costs comparable to the ones between traditional LoRA and full weight finetunings.\n\n[1] Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters, 2023.\n\n[2] DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via RL, 2025.\n\n[3] s1: Simple test-time scaling, 2025.\n\n[4] Training Diffusion Models with RL, 2024.\n\n[5] Diffusion Model Alignment Using DPO, 2024.", "claims_and_evidence": "The main claim of the paper is that the diffusion-based fine-tuning approach enables the pre-trained language model to scale at test time for reasoning.\n\nBy interpreting LMs trained with cross-entropy loss as single-step diffusion models, this framework is naturally facilitated, clearly supporting the convincing potential for test-time scaling through multi-step processes.", "methods_and_evaluation_criteria": "The method is technically sound.\n\nThe reviewer's main concerns lie in the evaluation.\n- Lack of CoT-like LM baselines. In experiments, there are no CoT-like test-time scaling baselines for LMs. L2D should be compared with these baselines to prove its genuine effectiveness on test-time scaling.\n- Lack of post-training baselines for reasoning. LoRA and full fine-tuning LMs on reasoning datasets seem not enough as fine-tuning baselines for reasoning tasks. More decent baselines using RL post-training for reasoning are essential to validate the effectiveness of L2D.", "theoretical_claims": "The proposed method is based on the diffusion formulation, but there are no theoretical claims in the paper that guarantee or analyze the efficacy or stability of the proposed approach.", "experimental_designs_or_analyses": "The experiments are generally well-designed, including tasks, data, and metrics, but the baselines sufficient for verifying the efficacy of the proposed method are not adequately compared.", "relation_to_broader_scientific_literature": "This work may have an impact by connecting the literature of LMs and diffusion through fine-tuning LMs with the diffusion framework. However, in my opinion, it has not been thoroughly examined how integrating diffusion into LM fine-tuning has an advantage over the existing works on test-time scaling in traditional LMs.", "essential_references_not_discussed": "Test-time scaling of LMs\n- Large language monkeys: Scaling inference compute with repeated sampling, ArXiv, 2024\n- Don't throw away your value model! Generating more preferable text with Value-Guided Monte-Carlo Tree Search decoding, COLM, 2024\n- Scaling LLM Test-Time Compute Optimally Can be More Effective than Scaling Parameters for Reasoning, ICLR, 2025\n- and much more on this topic\n\nTest-time scaling of Diffusions\n- Inference-Time Alignment of Diffusion Models with Direct Noise Optimization, ArXiv, 2024\n- Inference-Time Alignment in Diffusion Models with Reward-Guided Generation: Tutorial and Review, ArXiv, 2025\n- Test-time Alignment of Diffusion Models without Reward Over-optimization, ICLR, 2025", "other_strengths_and_weaknesses": "The reviewer's main concern is the lack of in-depth experimental and theoretical analysis regarding the advantages and disadvantages of the proposed method compared to existing LM reasoning frameworks.", "other_comments_or_suggestions": "Typo: Eq (2) might be L^{L2D} ~", "questions_for_authors": "Can diffusion fine-tuning be done using LoRA?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yBjSDGsvaR", "reviewer": "Reviewer_hDTm", "summary": "Authors provide a framework to combine the autoregressive LLM with diffusion models, to scale test-time compute in language reasoning tasks. \n\nDiffusion models are primary designed for continuous domains, with few exceptions of categorical diffusion models, but primarily designed for continuous domain. \n\nAuthors use some clever techniques to introduce the diffusion framework in an LLM. \n\nExperimental results on Maths and Coding reasoning tasks shows improved performance.\n\n\n## update after rebuttal\n\nMy original overall recommendation was 4: accept. My original minor concern was that the authors did not compare with best-of-N sampling in the experiments. The rebuttal addressed my concern. The original assessment has not changed, since I recommended accept.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their feedback and the time they dedicated to our review.\n\n**Experiments for test-time inference**\n\nSince L2D scales inference with a separate new \"diffusion path,\" we think it should be viewed as orthogonal to prior scaling approaches based on hand-designed heuristics and increasing generation length. To empirically confirm this, we would like to note that in the third section of [Table 2](https://anonymous.4open.science/r/rebuttal_l2d-4B0B/table2.png), we do compare with a tuned heuristic-guided token search strategy.  We believe these results not only show our method’s relative strengths but also validate how the two strategies do provide different and highly complementary benefits.\n\nNonetheless, following the reviewer’s feedback and questions, we collected even more experiments for our latest revision of this work, which we hope will strengthen the argument that diffusion is highly complementary with prior scaling approaches acting over the space of generated tokens: \n\n**R1-style RL for reasoning**\n\nWhile the R1 paper [1], spurring the recent focus on reasoning from RL, was uploaded on the web on January 23rd, the same date as the abstract submission deadline for ICML 2025 - we share the reviewer’s interest to analyze how L2D properties should be viewed in comparison.\n\nSince RL training requires expensive multi-node settings (beyond the resources for the project) and appears mainly effective on very large LMs, we added results with the pre-trained DeepSeek R1 Distill Qwen 1.5B reasoning model. We used this model both as an additional baseline and as an extra base model from which to train L2D.\n\n[Please find a summary Table of the added results at this link.](https://anonymous.4open.science/r/rebuttal_l2d-4B0B/r1_results_summary.png)\n\nAs the DeepSeek R1 model is trained on a recent private dataset, heavily focused on Math, we find its performance exceeds the original Qwen 1.5B Instruct model on this task category. However, we find this comes at an expected actual loss in performance on coding and general knowledge, which our L2D approach avoids. Moreover, further fine-tuning this baseline with L2D achieves the highest results on Math, even surpassing the much larger 7B and 8B non-RL models - as well as recovering a large part of the performance loss on the other tasks. In line with the other results, we believe these findings confirm that our new method should be viewed as complementary to RL reasoning.\n\nHowever, we note that evaluating these reasoning models distilled from RL was over 10x more expensive than vanilla L2D and did not work out-of-the-box, requiring to modify the prompts and relax the answer extraction code for compatibility with `<think>/<answer>` style responses. \n\nFinally, we extended our conclusion to mention that training L2D itself with concurrent R1-style RL methods could also be another interesting future research direction, taking inspiration from recent work in RL finetuning of diffusion models in computer vision [2, 3].\n\n**CoT scaling baselines**\n\nWhile some of our tasks already included CoT few-shot samples (e.g., GSM8K), following reviewer #1 (uZss) notable interest in this line of work, we made new CoT few-shot examples based on [4]. [Please find a summary of the results here](https://anonymous.4open.science/r/rebuttal_l2d-4B0B/cot_results_summary.png), and refer to reviewer #1 (uZss) response for further details.\n\n**Best of N**\n\nWe believe the token search baseline could be viewed as an advanced version of best-of-N scaling, where the tuned beam-search scores are used as the metric to assess which is the best response. Instead, best-of-N using ground-truth correctness assumes access to an oracle verifier and is typically only considered for coding, where the oracle could come in the form of a compiler and a set of test cases to solve. In fact, this is precisely what the \"pass@K\" metric used for Humaneval/MBPP considers, for which we provided further results in [Table 12](https://anonymous.4open.science/r/rebuttal_l2d-4B0B/table12.png).\n\nFollowing the reviewer’s interest and feedback, we extended these results by providing pass@K scores also for the other math and general knowledge tasks using the Llama 1B model, which could be viewed as an upper bound for any critic-based inference-scaling approaches:\n\n[Please find a summary Table of the added results at this link.](https://anonymous.4open.science/r/rebuttal_l2d-4B0B/pass@k_math_gqa.png)\n\nWe hope the reviewer will not hesitate to let us know if they believe it would be relevant to collect even more pass@K or other best-of-N analyses for our submission. \n\n[1] DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning, 2025.\n\n[2] Training Diffusion Models with Reinforcement Learning, 2024.\n\n[3] Diffusion Model Alignment Using Direct Preference Optimization, 2024.\n\n[4] Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters, 2023.", "claims_and_evidence": "Claims of improved reasoning abilities hold on experimental benchmark.", "methods_and_evaluation_criteria": "The method and the benchmark datasets make sense for the reasoning task.", "theoretical_claims": "The work is mostly empirical, and there are no theoretical claims to be verified.", "experimental_designs_or_analyses": "The experimental design is sound and makes sense, though I have a question (added below).", "relation_to_broader_scientific_literature": "The paper improves the \"reasoning\" ability of LLM using diffusion model, which are used in a lot of tools common people use everyday, so the work can have good impact.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "The paper is well-written and easy to understand. The problem is well motivated, and the experimental design is sound.", "other_comments_or_suggestions": "In the experimental section, I see authors have not compared with other test-time inference methods, such as best-of-N sampling (it's variants). Any reason for that? Or do you think that method is not a suitable baseline?", "questions_for_authors": "Authors have not compared their method with best-of-N sampling method, any reason for that? I am curious. \n\nAlso, there is no RL post-training baseline as well. \n\nAre these not relevant baselines? I might be wrong, so just wanted to pick authors brain on this.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XCPT2eYeY7", "reviewer": "Reviewer_4k8t", "summary": "This paper provided a novel perspective that treats language mode(LM) as a one-step diffusion model(DM). Thus, it proposes increasing the number of diffusion steps to boost the average score of the language model in test-time compute scaling. The methods show significant improvement in LMs in math, coding, and other reasoning.\n\n## update after rebuttal\nI have read through the author's rebuttal and confirmed that the methods are novel, However, I remain concerned that they have some performance issues that do not outperform the LoRA fine-tune baseline, indicating potential further improvement. Thus I keep my current score as Weak Accept.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their feedback and the time they dedicated to our review.\n\n**Questions**\n\n“*It is noticeable that in Coding-MBPP and GeneralKnowledge-MMLU tasks the performance is less than the LoRA fine-tuning or initial models. Is there any hypothesis for that?*”\n\nWhile L2D achieves the best performance on 22 out of the 24 task/model combinations across all our baselines analyzed in [Table 1](https://anonymous.4open.science/r/rebuttal_l2d-4B0B/table1.png), as the reviewer correctly points out, there are two exceptions for the Qwen 7B model where its performance is second-best. \n\nIn the MMLU task, the performance of L2D (71.11) fails to exceed the base Qwen 2.5 7B Instruct model (71.41). For this case, we would like to note that we designed our dataset to heavily focus on math and coding without emphasis on new real-world knowledge (preamble Section 4, Section 4.1, Appendix B), which we believe also explains why the other finetuning baselines get even lower results than the base instruct model (69.39 and 59.47). \n\nIn the MBPP task, the performance of L2D (76.79) is also slightly behind the LoRA baseline (79.60). While this is not the case for any other task, or even any other model other than Qwen 7B on the same task, we believe this could be an indication that L2D should be viewed not just to replace but also to be used in combination with traditional weight finetuning. The potential of combining these approaches is also supported by our extensions results in the second section of [Table 2](https://anonymous.4open.science/r/rebuttal_l2d-4B0B/table2.png), where we show that L2D can be effectively used after full and lora finetunings of the base model with compounding benefits.\n\nFollowing the reviewer’s interest, we added an extended discussion to our latest revision of this work to specifically address these result instances, as detailed above.\n\n**Other extensions**\n\nIn our latest revision, we also added new analysis and comparison of L2D with other concurrent work and traditional approaches to test-time scaling, which we hope will further strengthen our submission (please see the above response to reviewer #2 hDTm for details).", "claims_and_evidence": "The paper presents a well-articulated claim supported by strong evidence across various tasks, demonstrating clear reasoning. It highlights significant performance improvements in both tasks for recently open-sourced language models (LMs), spanning both small-scale models (1B, 1.5B) and medium-scale models (7B, 8B).", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are well-aligned with the problem and application. The methodology is clearly defined and builds upon well-established approaches, such as language models (LMs) and diffusion models (DMs). The key novelty lies in treating LMs as a one-step DM and integrating them within a fine-tuning framework. The benchmarks used are widely recognized, and the improvements are evident based on well-established evaluation metrics.", "theoretical_claims": "The proposed methods and evaluation criteria are well-aligned with the problem and application. The methodology is clearly defined and builds upon well-established approaches, such as language models (LMs) and diffusion models (DMs). The key novelty lies in treating LMs as a one-step DM and integrating them within a fine-tuning framework. The benchmarks used are widely recognized, and the improvements are evident based on well-established evaluation metrics.", "experimental_designs_or_analyses": "The experimental design and analysis are sound and valid, leveraging well-established benchmarks and solid base models. The experiments are well-structured, with clear improvements demonstrated over strong baselines. Additionally, the ablation studies are well-executed, providing insights into the contributions of different components. There are no major concerns regarding the validity of the experimental setup.", "relation_to_broader_scientific_literature": "The key contributions of the paper are well-situated within the broader scientific literature, building upon prior findings in both language models (LMs) and diffusion models (DMs). The work extends existing theoretical foundations by leveraging well-established theorems from previous studies but focuses on practical effectiveness rather than theoretical novelty. The novel approach of treating LMs as a one-step DM and integrating them in a fine-tuning framework aligns with recent trends in model unification and cross-paradigm learning. Additionally, the use of well-recognized benchmarks and clear improvements over prior baselines strengthen its connection to existing work, demonstrating meaningful progress in the field.", "essential_references_not_discussed": "There are no essential references missing.", "other_strengths_and_weaknesses": "see strength in other comments", "other_comments_or_suggestions": "## Strengths:\n- Practical Impact: The paper focuses on practical effectiveness rather than theoretical novelty, making it highly relevant for real-world applications.\n\n- Strong Empirical Results: The proposed approach shows clear and significant improvements over well-established baselines, demonstrating its effectiveness.\n\n- Well-Designed Experiments: The experiments are robust, leveraging widely recognized benchmarks and solid base models. The ablation studies provide useful insights into the contributions of different components.\n\n- Clear and Coherent Presentation: The methodology and findings are presented in a structured and logical manner, making it easy to follow.\n## Weaknesses:\n- Limited Theoretical Novelty: Since the work builds on existing theorems without introducing new theoretical advancements, its contribution is primarily practical.", "questions_for_authors": "It is noticeable that in Coding-MBPP and GeneralKnowledge-MMLU tasks the performance is less then the LoRA fine-tuning or initial models.  Is there any hypothesis for that?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XmEF6SnCJQ", "reviewer": "Reviewer_hLPT", "summary": "This paper proposes fine-tuning pretrained large language models (LLMs) using diffusion models to enable scalable test-time computation. By framing LLMs as single-step diffusions and introducing a small fraction of new parameters, the approach enhances multi-step reasoning, allows adaptive computational scaling, and incorporates guidance techniques while preserving the original model’s efficiency.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their feedback and the time they dedicated to our review.\n\n**Claims and Evidence**\n\nIn our [Table 1](https://anonymous.4open.science/r/rebuttal_l2d-4B0B/table1.png) results, across all 24 task/model combinations examined, our full weight finetuning and LoRA baselines improve performance by 1.7 and 6.23 on average over the base model. In contrast, L2D improves average performance by 11.52 (85% higher improvement than LoRA) and outperforms both finetuning baselines in 23/24 settings. \n\nAs the reviewer correctly points out, for one case, in the MBPP task with Qwen 7B, the performance of L2D (76.79) is only second best, slightly behind the LoRA baseline (79.60). While this is not the case for any other task, or even any other model other than Qwen 7B on the same task, we believe this could be an indication that L2D should be viewed not just to replace but also to be used in combination with traditional weight finetuning. The potential of combining these approaches is also supported by our extensions results in the second section of [Table 2](https://anonymous.4open.science/r/rebuttal_l2d-4B0B/table2.png), where we show that L2D can be effectively used after full and lora finetunings of the base model with compounding benefits.\n\nFollowing the reviewer’s feedback, in our latest revision of this work, we modified Section 1 to be more precise with our wordings and avoid general statements (such as Claim 1) and extended Section 4 to specifically address the MBPP results as detailed above.\n\nRegarding the scalability of L2D, as the reviewer also pointed out, the additional parameters of our method as compared to the LoRA baselines mainly come from storing the weights for the vocabulary of our new diffusion LM path (the module denoted \"MLP\" [at the bottom of our architecture diagram](https://anonymous.4open.science/r/rebuttal_l2d-4B0B/l2d_architecture_diagram.png)). We would like to note that the number of these parameters does not grow with model size, making our method increasingly more efficient in relative terms with larger LMs. Nonetheless, even in for the LMs considered in this work, the total number of optimized parameters is still only a small fraction of the original model’s weight (less than 6% for Llama 1B and 3.5% for Llama 8B), with L2D being over an order of magnitude more parameter efficient than full finetuning. \n\nFollowing the reviewer’s feedback, we added the above discussion about parameter efficiency to the latest revision of our work to provide a better context of our method’s parameter efficiency and scalability. Furthermore, we also mention that decreasing the diffusion space dimension (e.g., from 256 used in our work to 16) could be an option to explore in future work to make our method’s parameter count closer to LoRA, which has been shown viable by some of the diffusion language models learned from scratch such as [1].\n\n[1] Continuous diffusion for categorical data, 2022.\n\n**Clarity**\n\nFollowing the reviewer’s feedback, we tried to identify in the text multiple instances where we used “*multiple preposition phrases or clauses to describe one object*” and tried to improve clarity by breaking them up and referring back to the object explicitly. For instance, we rewrote the sentence at the start of Section 2.1 to:\n\n“Gaussian diffusion decomposes the problem of generating new samples from a target unknown distribution p* from a source distribution q = N(0, I) over multiple 'simpler' steps. The Gaussian diffusion decomposition effectively reuses the intermediate information computed in the model's attempts in each previous step.”\n\nIf the reviewer has any other specific example where text clarity could be improved, we hope they will not hesitate to point it out.\n\n**Questions**\n\nThe building blocks of the diffusion path are constructed from all layers in the MLP modules and only the layers of the pre-trained self-attention modules that are used to compute the queries (used in the diffusion path for cross-attention), as detailed in the second paragraph of Section 3.1.\n\nFollowing the reviewer’s question, we added this detail more explicitly a second time in the previous paragraph: “We implement the diffusion path with [...] the same number of blocks as the main path, each comprising a subset of its layers (from the MLP blocks and the query layers in self-attention).”\n\n**Quality and Other Extensions**\n\nWe hope our response in Claims and Evidence and the additional discussion provided some more interesting insights. Furthermore, in our latest revision, we also added new analysis and comparison of L2D with other concurrent work and traditional approaches to test-time scaling, which we hope will further strengthen our submission (please see the above response to reviewer #2 hDTm for details). Nonetheless, we hope the reviewer will let us know if there is any other specific part of our submission that they think we could further work on to improve its current quality.", "claims_and_evidence": "**1**. *Claim 1*:\\\n\"We show that L2D significantly improves four different LMs on math, coding, and a variety of reasoning tasks; and that its benefits can be both superior and complementary to traditional finetuning and search.\" \\\n*Evidence 1*:\\\nFrom the experimental results, L2D achieves approximately 5 more score on average than LoRA, but at the cost of more than 20 times number of parameters. For tasks such as coding on MBPP and general knowledge, the enhancements seem to be marginal. Finetuning on Qwen 2.5 7B Instruct for MBPP even demonstrates that LoRA outperforms L2D. It leaves the question on the effectiveness and scalability of L2D.\\", "methods_and_evaluation_criteria": "The proposed L2D is the compression of large language models achieved via fine-tuning for classifier-guided diffusion models. Because of the current lack of diffusion-based methods on language generation, it is natural to consider using diffusion models which can progressively generate contents, to predict next tokens. Since tokens are results generated from LLM, it is logically linear for me to fine-tune a diffusion path to compress LLM.\n\nThe evaluation criteria: scores, parameter sizes, strengths of guidance and the number of inference steps, are aligned with the evaluation of the compression performance mainly on efficiency and effectiveness. The benchmark datasets are the commonly used datasets to evaluate the performance of LLM and the compression for LLM. \n\nOverall, the proposed methods and evaluation criteria make sense for me.", "theoretical_claims": "There is no theoretical claim in this paper.", "experimental_designs_or_analyses": "Yes, the experimental design and analysis is sound for me.", "relation_to_broader_scientific_literature": "This work fills the research gap in diffusion models for language generation. It is novel to compress LLM via a diffusion path. It opens the possibility to bridge diffusion models which is ubiquitous for visual content generation, with LLM which is designed for language generation. The experimental results in Figure 3 clearly demonstrate the power of diffusion paths as compressions of LLMs can even achieve better performance than LLMs in language generation after a relatively small number of inference steps.", "essential_references_not_discussed": "No, I think all the key references are included and discussed.", "other_strengths_and_weaknesses": "**Weaknesses**:\\\n*a. Clarity*: This paper definitely requires to be polished on writing, especially the structure of the sentences and the phrasing. It is very confusing to have consecutive multiple preposition phrases or clauses to describe one object, as it is easy for the audience to be lost in these descriptions and forget the object.\\\n\n*b. Quality*: The insights from the experimental results should be clearly articulated and discussed, for example, the trade-off between scores and the number of parameters. Some preliminary theoretical insights or propositions will be beneficial.", "other_comments_or_suggestions": "NA", "questions_for_authors": "**1**. How are the subset of the layers of LM being selected to be the building blocks of diffusion path? Are they randomly selected or hand-picked? Is there any special requirements of the selection to guarantee the performance with intuitive or theoretical insights? Is it the main technique to achieve the scalability?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The reviews and discussions on the manuscript highlighted both strengths and weaknesses.

On the one hand, the manuscript clearly contributes to the study of large language models, from an approach via diffusions to a finetuning framework by integrating guidance techniques to achieve increased accuracy in answer questions on specific topics. The paper has been identified as mainly empirical, showing experiments with various models and benchmarks.

On the other hand, the paper required some revisions. As commented by reviewers, some comparisons were missing (e.g. with best-of-N sampling and comparison between the CoT reasoning baseline and the latest R1 reasoning model). There was a comment raised that it looks like in some task this work does not out-perform simple LoRA baseline. However, the rebuttals alleviated some concerns and the reviewers reiterated their mainly positive evaluation of this paper.
Decision: Accept (poster)