Diffusion Beats Autoregressive in Data-Constrained Settings
Authors: Mihir Prabhudesai
Mengning Wu
Amir Zadeh
Katerina Fragkiadaki
Deepak Pathak
Venue: NeurIPS 2025 poster ()
Keywords: diffusion
autoregressive
data-constrainted
generative modelling
discrete diffusion
diffusion llms
Primary area: deep_learning
TL;DR: We show that diffusion language models are a lot more sample-efficient than standard autoregressive language models, due to their ability to learn from different token orderings.

Abstract: Autoregressive (AR) models have long dominated the landscape of large language models, driving progress across a wide range of tasks. Recently, diffusion-based language models have emerged as a promising alternative, though their advantages over AR models remain underexplored. In this paper, we systematically study masked diffusion models in data-constrained settings—where training involves repeated passes over limited data—and find that they significantly outperform AR models when compute is abundant but data is scarce. Diffusion models make better use of repeated data, achieving lower validation loss and superior downstream performance. We find new scaling laws for diffusion models and derive a closed-form expression for the critical compute threshold at which diffusion begins to outperform AR. Finally, we explain why diffusion models excel in this regime: their randomized masking objective implicitly trains over a rich distribution of token orderings, acting as an implicit data augmentation that AR’s fixed left-to-right factorization lacks. Our results suggest that when data, not compute, is the bottleneck, diffusion models offer a compelling alternative to the standard AR paradigm. Our code is available at: https://diffusion-scaling.github.io

Diffusion Beats Autoregressive in Data-Constrained Settings
Mihir Prabhudesai" Carnegie Mellon University
Mengning Wu* Carnegie Mellon University
Amir Zadeh Lambda
Katerina Fragkiadaki Carnegie Mellon University
Deepak Pathak Carnegie Mellon University
Abstract
Autoregressive (AR) models have long dominated the landscape of large language models, driving progress across a wide range of tasks. Recently, diffusion-ban based language models have emerged as promising alternative, though their advantages over AR models remain underexplored. In this paper, we systematically study masked diffusion models in data-constrained settings-weere training involves repeated passes over limited data-and find that they significantly outperform AR models when compute is abundant but data is scarce. Diffusion models make better use of repeated data, achieving lower validation loss and superior downstream performance. We find new scaling laws for diffusion models and derive closed- form expression for the critical compute threshold at which diffusion begins to outperform AR. Finally, we explain why diffusion models excel in this regime: their randomized masking objective implicitly trains over a rich distribution of token orderings, acting as an implicit data augmentation that AR's fixed left-to- right factorization lacks. Our results suggest that when data, not compute, is the bottleneck, diffusion models offer a compelling alternative to the standard AR paradigm. Our code is available at: https: /tips://aafffooon..o/ /tt.a/fffffioo-ssaaaiinnni.n gittu............ io
Pareto Frontier on 100M unique data
Pareto Frontier on 50M unique data
6.0
4.6
Critical
Crilical Compute Poin:
Compute Point
Lone
40-
Epach: 2.37e+15
Epoch: 1.48+144
Optimal
Optinal
ST.OPs
FLOP%
Diffusion Pareto Frontier
Pareto Prontier
Where Diffusion ARa
Figure 1: Pareto frontier of validation loss versus training FLOPs for autoregressive (AR) and masked diffusion models under data-constrained settings. Each point represents a model trained until convergence; report the best validation loss achieved among all models using less than equal to the FL.PPs shown on the x-axis. AR models initially outperform diffusion models, particularly near the Chinchila-optimal compute point [12] (indicated on the plot). However, training continues beyond this regime with repeated data, AR models quickly saturate and begin to overfit. In contrast, diffusion models continue to improve with more compute and exhibit no signs of overfitting.
Project co-leads & Equal contribution. Correspondence to (mprabhud, wwngeddmmmnnnnneeeem...mmmmm Candrre edu.
39th Conference on Neural Information Processing Systems (NeurIPS 2025).

---

Introduction
Training large language models (LLMs) on massive corpora of internet text has become the driver of recent Al breakthroughs [4] [28 40 This progress has been fueled by scaling two core resources: compute and data [I. While compute availability is steadily growing- rrowing-enabled by advances in hardware and the construction of larger data centers the growth in high-quality data is stagnating.
Recent projections, such those by Villalobos et.al. 442) estimate that the global supply of publicly available, human-generated data may be exhausted in the coming years, posing a serious bottleneck to further scaling. This looming constraint makes it increasingly important develop modeling strategies that are more sample-efficiet. Furthermore, there are several domains, such as robotics and healthcare, where the data, not compute, is scarce resource even begin with.
LLM development has so far been dominated by autoregressive (AR) models, which factorize the joint distribution of text in fixed left-to-right order. While this modeling approach has delivered state-of the-ar art performance across a range of benchmarks, it remains unclear whether it is the optimal strategy going forward. Recently, diffusion- basedd saeddls--peciiicallll masked diffusion models 34
have emerged as an alternative strategy, framing text generation as an iterative masked denoising process rather than next-token prediction. At each step, the model predicts randomly masked subset of tokens conditioned on the remaining ones, implicitly averaging over many conditional prediction orders instead of committing to one. Although these models have demonstrated similar scaling behavior to AR models [22] 39), their practical benefits have, so far, been modest-llrgelly due their high training compute requirements.
2
18
39
This high compute demand has become the central obstacle to wider adoption of liffusion-based language models. As noted by Nie al. [22] and Swerdlow et al. [39], masked diffusion models require up to 16x more compute than AR models to match validation NLL a clear disadvantage for most applications.
But a critical nuance is often overlooked: these comparisons are based entirely on single-epoch training. where each token is seen only once. This conflates compute efficiency with sample efficiency. making it unclear whether diffusion models truly need 16x more compute-or simply 16x more data.
To resolve this ambiguity, we systematically study masked diffusion models in data-constrained settings, where repeated training limited data is the norm rather than the exception. We find that under such regimes, diffusion models substantially outperform autoregressive models across a variety of data scales and compute budgets. We train hundreds of models spanning multiple orders of magnitude in model size, data quantity, and number of training epochs to fit scaling laws for diffusion models in the conttrannd setting. We summarize some of our key findings below.
1. Diffusion models surpass autoregressive models given sufficient compute. Across wide range of unique token budgets, we observe a consistent trend: autoregressive models initially outperform diffusion models at low compute, but quickly saturate. Beyond a critical compute threshold, diffusion models continue improving and ultimately achieve better performance (Section 3.1)
2. Diffusion models benefit far more from repeated data. Prior work (1] showed that repeating the dataset up to 4 epochs is nearly as effective as using fresh data for autoregressive models. In contrast, we find that diffusion models can be trained on repeated data for up to 100 epochs, while having repeated data almost as effective as fresh data (Section 3.2p.
3. Diffusion models have a much higher effective epoch count. Muennighoff et al. fit scaling laws for AR models in data-constrainted settings and define RD as a learned constant that characterizes the number of epochs after which training more epochs results in significantly diminished returns. For autoregressive models, they estimate R2,, ~ 15.. In contrast, we find R x 500 for diffusion models, suggesting they can benefit from repeated data over far more epochs without major degradation (Section 3.2)
4. Critical compute point follows power law with dataset size. We find that the amount of compute required for diffusion models to outperform autoregressive models- critical compute point-scales closed-form form expression that predicts when diffusion becomes the favorable modeling choice for any given dataset size (Section 33
power law with the number of unique tokens. This yields

---

5. Diffusion models yield better downstream performance. We find the above benefits extend beyond validation loss: the best diffusion model trained data-constrained settings consistently outperform the best autoregressive model on range of downstream language tasks (Section 3.4).
6. Exposure to differrnt token orderings helps explain diffusion's 's sample efficiency. By adding explicit data augmentations AR training, find that diffusion models' advantage arises from their exposure to diverse set of token orderings. Essentially, the random- ized masking diffusion's 's objective serves as implicit data augmentation, allowing it to generalize beyond the fixed left-to-right factorization of AR models. (Section 3.5
Through detailed scaling law analysis and downstream task evaluations, demonstrate that dif- fusion models make significantly better use of repeated data, achieving lower validation loss and better generalization downstream tasks. These results suggest that diffusion models may offer compelling and underappreciated advantage in scenarios where data-no compute- the primary bottleneck.
2
Method
Our objective is to determine whether masked diffusion language models are more effective than standard autoregressive models in data-constrainted settings. For studying this, we keep the core architecture and data pipeline fixed across both families.
2.1 Preliminaries:
Autoregressive models. In Autoregressive LLMs 40 28 4 each token is predicted based on growing prefix prior tokens, defining a left-to-right factorization the sequence probability:
PAR [1 ... LL) II p(a; xee))
This structure is implemented using causal attention mask, which prevents each token from attending to future positions. The model is trained via next-token prediction over clean, uncorrupted sequences.
39 treat generation as iterative denoising. For each training sequence æ (II..... xz) diffusion models
Diffusion models. Masked diffusion language models
2 34
Corrupt the sequence by sampling a masking ratio r 14(0, 1) and independently replacing each
token with special [MASK] symbol with probability r. This yields corrupted sequence ş and mask set
M = {i E [1, L. ã; [MASK]]
2. Denoise by predicting the original tokens at the masked positions with full (bidirectional) attention over Ø:
PDitfusion
x
2)
II iEM
Pe(x: Z).
Because the mask pattern resampled for every example, the model implicitly trained on a vast collection of token-ordering tasks. The absence of causal mask allows each prediction to attend to both past and future unmasked tokens.
2.2 Modeling Details for AR and Masked Diffusion
Both model families share the same Transformer backbone (GPT-2 style with rotary positional embeddings, RoPE 38)..
Given a clean input sequence x = (aI,..., E V4, both models minimize a token-level cross- entropy loss, yet they differ in the conditioning context:

---

Autoregressive (AR) objective. causal attention mask:
AR models predict each token conditioned on its prefix using
CA =
Σ log po(Ty xzς)-
Masked Diffusion objective. For masked diffusion we first sample a masking ratio r ~ 24(0, 1) and construct corrupted sequence 5 by independently replacing each token with [MASK] with probability r. Let M = {i: = [MASK]] J be the set of masked positions. The loss is then
DDiffusion
Eyrjrngr r iE
log Po (2: .),
which can interpreted as an evidence lower bound (ELBO) on the data log-likelihoode and thus the loss provides upper bound the true negative log-likelihood.
Beyond the attention mechanism and input corruption, all other variables are held constant. We follow the hyperparameter configuration proposed by Muennighoff et al. [21] for all training runs. In particular, we use dynamic learning rate schedule that adapts to the number of training epochs. For more detailed information on both model families please refer to related work Section in Appendix
2.3 Scaling Framework in Data-Constrained Settings
Classical scaling laws, such those proposed by [15 12), model validation loss function of total parameters (N) and training tokens (D), assuming all data unique that is single epoch regime.
Muennighoff er al. (21) extend the Chinchilla framework to explicitly account for repeated data a common necessity in data-constrained regimes. They show that repeating training data beyond few epochs yields diminishing returns and propose a new scaling law that incorporates the decaying utility of repeated tokens.
We briefly outline their formulation below. Let U denote the number of unique tokens, E' the number of epochs (how many times each unique token is reused), and D U E' the total number of tokens seen during training.
To model diminishing returns from repeated data, Muennighoff et al. 21 introduce an effective unique data size D', motivated by the idea that each additional epoch contributes less useful signal than the previous. Specifically, they assume the value extracted from the 1-th exposure the same data follows a geometric progression, where the utility of a token on its k-th repetition is (1 5)*-1
Summing over all epochs the total effective data becomes: D' = U E (1-6)*-1 = U. 11
where 5 is the decay factor. Defining RD = فها the expression simplifies to the exponential-decyy form:
D'= RD (1- (1-1-(R-1)RR)
here Rt represents the half-life of data reuse, repeating data beyond Rt epochs will result in significant diminishing returns. This form approximates the geometric diminishing returns over repeated epochs. As the number of epochs F. - oo, the exponential term vanishes and D' asymptotically approaches: D' U - U Ri: implying that matter how many times data is repeated, the maximum usable signal is bounded by (1 + RT)): U. This defines a natural saturation point on returns: even infinite compute yields additional effective data beyond this limit.
well and captures
symmetric formulation is applied to model parameters for mathematical convenience which is used to define N'. Finally, a modified Chinchilla-style loss function incorporates these effective quantities N' and D':
A
B
C(N, D)
+ Eo.
(N')a
(D')
with A, B, a. RT, N2, fitted empirically from training runs. This formulation accurately captures loss behavior in regimes where data is reused multiple times and serves powerful tool for guiding training under data scarcity.
In this work, we adopt this framework to study how diffusion models and autoregressive models compare their ability to extract value from repeated data, enabling apples-to-apple comparisons across compute, data, and model scale.

---

2.4 Training setup
use the English C4 corpus 29), tokenized with the GPT-2 BPE vocabulary and truncated or padded to 2048 tokens per sequence. We consider unique-token budgets of U E {25, 50, 100 and train for up to 800 epochs (80B tokens total). Models are trained ranging from 7M to 2.5B parameters, following the Chinchilla scaling strategy where both width and depth are increased proportionally. The detailed architcctural configurations each model are provided in Appendix 9 For all training runs, we adopt the hyperparameter configuration introduced by Muennighoff' et al. (2I]. This may
provide a slight advantage to autoregressive models, as these hyperparameters were originally tuned for that family. For details hyperparameters please refer to Section in Appendix.
Experiments
Our goal is to compare the performance of masked diffusion models and autoregressive models in lata-constrained settings. To this end, we train a total of 200 models-100 diffusion models and 100 autoregressive siikels-acros, varying unique data sizes, model scales, and epoch counts. We present the empirical results in Section 3.1 In Section 3.2, we fit scaling laws tailored to data-constrained regimes for both model types, following the methodology introduced by Muennighoff er al. [2I]. These scaling laws allow to analyze performance trends and identify scenarios where diffusion models should be preferred over autoregressive ones (Section 3.3). Section 3.4 we demonstrate that the superior validation loss of diffusion models indeed correlates with improved downstream task performance. Finally, in Section 3.5 we investigate the underlying cause diffusion' 's advantage in data- constrannd settings, showing that its exposure to diverse token orderings enables better generalization than AR's ' fixed left-to-right factorization.
3.1 Does Diffusion Beat AR in Data-Constrained Settings?
Prior comparisons between diffusion and autoregressive (AR) language models have largely focused on the single-epoch regime, where each token is seen only once during training [22 39], In this setting, diffusion models are consistently reported to require substantially more training compute D) than AR models to achieve comparable validation loss. For instance, Nie et al. [22] and Swerdlow et al. [39] derive scaling laws showing that masked diffusion models can require up to 16x more compute than AR counterparts.
(C ~ (NN66
Crucially, these studies scale compute by increasing both the model size (N) and the amount of unique training data (D) proportionally. As result, they do not isolate whether diffusion's 's 16x inefficiency stems from needing more total compute-or more unique data.
In other words: is diffusion limited by compute efficiency or by sample efficiency?
To answer this, systematically study diffusion models in data-constrained settings, where the total amount of unique data is fixed and models are trained for many epochs, reusing the same data. Unlike prior work, our evaluation explicitly decouples model scaling from data reuse, allowing to disentangle the effects of compute and data.
In Figure M we report empirical validation loss as function of training PLOPs for the 50M and 100M regimes; results for the 25M setting are shown in Appendix Figure 9 We find that AR models initially outperform diffusion models when trained with the compute-optimal budget prescribed by Chinchilla scaling laws (denoted by the solid vertical line). However, this advantage disappears as training continues beyond this point. When models are allowed to train for additional epochs on repeated data, diffusion models consistently surpass AR models in validation loss across all data regimes. These findings indicate that the previously observed inefficiency of diffusion models is largely a consequence of evaluating them solely in the single-epoch regime. In data-constrained settings with repeated exposures, diffusion models extract significantly more value from the same data than their AR counterparts.
A key question remains is how should one go about increasing compute for diffusion models: by increasing model size, or by increasing the number of epochs (i.e., data reuse)? To address this, we analyze the trade-off between parameters and epochs in Figure which shows validation loss contours as a function of both axes. In the 100M unique token regime, for example, we find that diffusion achieves its best loss at 500 epochs, while AR model reach its best at just 50 epochs. Each
5

---

point on the contour plot corresponds to a model trained with a specific parameter count and number of epochs; we report the actual validation loss at each configuration, without early stopping. We find that autoregressive models begin to overfit at high epoch counts, with validation loss worsening as training continues beyond certain point. In contrast, diffusion models show no signs of overfitting within our compute budget-the best validation loss is achieved at the highest epoch counts we explore. This suggests that diffusion models continue to benefit from additional training on repeated data, and that observing overfitting may require significantly more compute.
To contextualize these results, we highlight two key configurations in Figure for each model family: the compute-opiiml point for single-epoch training, as identified by prior scaling law analyses [12 [24] (marked with a colored star in the bottom-left), and the best validation loss achieved under extended multi-epock training (marked with a black star). At the compute-optimat point, which corresponds training for single epoch, diffusion models perform substantially worse than autoregressive models (10.65 vs. 7.07), consistent with prior findings that diffusion performs worse initially. However, as training is extended hundreds of epochs, diffusion models continue to improve and eventually achieve lower validation loss (3.55) than the best AR models (3.71).
I00M
1000
3.5
144M
Patta
1003
10M
Edb.
3M
Loa 100
Loveal
193M
Corpuaa-Cpttral radel 1000d tokme
Loe 1665
Cargras- Uxptrmii model
1000
1960
Ipachs
(a) Autoregressive contour: validation loss over epochs and model sizes.
(b) Diffusion contour: validation loss over epochs and model sizes.
Figure 2: Validation loss contours over epochs and model sizes for autoregressive (left) and diffusion (right) models, trained 100M unique tokens. Each plot shows validation loss as a function of training epochs (x-axis) and model parameters (y-axis). The colored star marks the compute-optimal point for single-epoch training, predicted by prior scaling laws 112 [24], and the black star indicates the lowest validation loss achieved through extended multi-epoch training. In the single-epoch regime, diffusion models perform worse than AR models (10.65 vs. 7.07). However, when trained longer, diffusion models achieve substantially lower final loss (3.55 vs. 3.71). This corresponds to 67% reduction in loss for diffusion models compared to just 48% for AR models, highlighting their superior ability to leverage repeated data.
3.2 Fitting Data-Constrained Scaling Laws
To gain deeper insight into the trade-offs between diffusion and autoregressive models in data- constrained settings, we fit scaling laws to both model families across single-epoch and multi-epoch regimes, as described in Section 2.3 Our approach systematically varies three key factors: (1) the amount of unique data, (2) model parameter count, and (3) number of training epochs. This grid search allows us to disentangle the effects of data quantity, model capacity, and data reuse on final model performance.
We evaluate the quality of our scaling law fits using the coefficient determination (R²) and relative prediction error, shown in Table For autoregressive models, our R2 values closely match those reported by Muennighoff et al. 21), indicating consistent behavior under repeated training. Interestingly, diffusion models yield significantly higher R² values, reflecting better overall fit. We

---

3.5
13
Diftaki Neat
34
000
Difaae
1G00
Froditad lose drspuating dat vala sttia
Figure 3: Decay rate of data value under repetition: left shows diffusion, middle AR, and right the average decay rate for both. Points are empirical results (darker color higher FLOPs, lighter color lower FLOPs; each line fixed compute), we find that fitted curves (represented lines) closely match the empirical points, indicating our scaling laws are representative. The decay rate of value for repeated data is lower for diffusion, reflecting its greater robustness repeating.
AB De
Dizmin
AB w.
[afass sa
AB 1077
Defasia NaTy
48 10
54
Dettaiee Tp.
20.41p
Diceve tg.
Vildbtion looe
AM. 18.11p
DCDasse 11p
Validatio
2.8
30 Traizing Tolomms
Trining Tolens
(a) AR Training Curves.
(b) Diffusion Training Curves.
Figure 4: Training curves for different epoch counts, all with using the same total compute. Each curve shows different tradeoff between unique data and repetition. For AR models, validation loss rises with more epochs (overfitting), while for diffusion models, the curves are nearly unchanged, showing much greater robustness data repetition.
Table 1:
Fitting metrics of the scaling law model for Diffusion and AR. Diffusion and AR achieve strong fit across both phases.
(a) Initial fit.
(b) Second step fit with extracted scaling parameters.
Model
R²
Loss
Model
R²
Loss
Rb
Rý
Diffusion
0.9447 0.9439
0.0002
Diffusion
0.9784
0.00079
493.89 31.19
1265.65
AR
7.7532e-05
AR
0.7628
0.00361
55.16
attribute this to lower variance in validation loss across training runs, likely due the absence of overfitting in diffusion models even at high epoch counts.
Beyond the overall fit, we extract two key parameters from the scaling laws: RD which characterizes the effective half-life of data reuse-ize i.c., the number of epochs after which additional training on repeated data yields diminishing returns-ant and Rřy, which indicates the optimal model size for given data budget. Our results reveal sharp contrast in data reuse half-lives: diffusion models exhibit an Rr of 5 512.85, compared just 31.93 for autoregressive models. A higher Rt implies that model can benefit from many more repeated exposures before saturating. This suggests that diffusion models continue improve across hundreds of epochs, while AR models quickly saturate highlighting the superior sample efficiency diffusion models in data-comt constrained regimes.
Figure 3 iultrrttss how the utility of unique data decays with increased repetition. We evaluate this effect across three compute budgets-11 1019, 1019 and 1020 FLOPPs- varying the proportion of unique data and parameters while keeping total compute fixed (c.g., 50% of the data
7

---

AR validation loss over training length
Diffusioo validation loss over training length
Unique Scaling
S J6
Unique Sealing
Duflusion
16"
425M AR
217M Diflision
724M AR
Valication Less
425M Dillision
Vildarke
10"100
Repeating spocta stimend good new data
4x 10"
Repeating spadicie alioss good new
3=10"
16"
3x 10
24" 10
10B
100B 1T
LOTT
100TT
1B
100B
TT
100T
Taining Tolens
Training Tokers
Figure 5: Predicted validation loss for AR (left) and Diffusion models (right) under compute-optimal settings, extrapolated to larger compute budgets. Dotted lines show the hypothetical case where repeated data equals new data. For AR, this holds up to 444 epochs; for diffusion, up to R 100 showwng diffusion's greater robustness to data repetition. Note that loss values between AR and diffusion are not directly comparable, as they're re extrapolated from scaling laws with different data-entropy terms (Eo). In Section 3.5 ignore this factor during comparison.
for 2 epochs, 25% for 4 epochs, etc.). For each compute budget, we use single-epoch scaling laws to determine the optimal model size and unique token count for both AR and diffusion models. We present both empirical results and fitted curves from our parametric scaling law, observing strong agreement between the two. Notably, the decay rate of data value remains consistent across compute budgets. Diffusion models consistently exhibit substantially slower decay rate than AR models, suggesting they are better able to extract value from repeated data.
Figure shows validation loss versus training tokens using the compute budget of lc19. The results reinforces the trend: AR models overfit with increased repetition, showing diverging loss curves. In contrast, diffusion models exhibit overlapping curves across repetitions, indicating signs of overfitting and a very low decay rate with data reuse.
Figure SJsoows extrapolated training curves at large compute budgets. For each setting, we use the compute-optimal model and dataset size derived from single-epoch scaling laws for le19, 3e19 and le20. We then extend training to multiple epochs. The dashed lines represent the ideal Chinchilla- style scaling behavior, where all training tokens are assumed to be unique. We find that for AR models, repeated data provides nearly the same benefit fresh data only up about 4 epochs. Beyond this point, additional repetition yields diminishing returns. In contrast, diffusion models continue to match the unique-data curve for up to 100 epochs, indicating a far greater capacity to benefit from repeated data in cata-constrained regimes.
3.3 When to Use Diffusion over AR?
A key question for practitioners is: when should diffusion be preferred over autoregressive models (AR)? To answer this, compare the fitted data-constrained scaling laws for both model families 22.3)
We define the validation loss gap between diffusion and AR as:
AL(C,U)= LiInussio (C,U) LAR(
where C total training compute and U is the number of unique tokens. Positive values favor AR; negative values favor diffusion. The critical compute Cerit (U) is the point where the models perform equally: C(Crrii U) 0.
Figure 6 a) shows a heatmap of AL over compute and data. Red regions indicate regimes where diffusion outperforms AR (&C 0). while blue regions favor AR. As expected, AR performs better in low low-compute settings due to its efficient ptr-step learning. However, diffusion models begin to outperform AR higher compute, especially when data is limited and repeated.
Figure ((bb plots the critical compute frontier Cerit (Ue compute required for diffusion to match AR at given unique token count U. This frontier follows power law: (Uri(UU) x U
8

---

Horaup & Puslated cLowa (Didiisso A83
07
Lineer Fu of Critical Compute
10
Filted line
Critical Compute Point
Catival Computa (Flas)
10
Dilfisioo Match AR
10"
10
00/
-4 90
10
6/
Uege (Mlboe)
Unique Data (Million)
(a) Loss Gap Heatmap. Difference validation (AC LDifusion LAR) across unique data sizes and FL.OPs. Red indicates regions where diffusion out- performs AR models and blue where AR outperforms diffusion.
(b) Critical Compute Curve. The FLOPs thresh- old Cerit (U) beyond which diffusion outperforms
AR models. This follows power law: Ceri x 2
Figure 6: When does Diffusion beat AR? Left: Heatmap showing where diffusion models have lower validation loss than AR models. Right: The critical compute curve defining the compute threshold needed for diffusion to match autoregressive models at given unique token count.
The linear fit in log-log space is: l0g1o(U) 0.460 IC) 1.050, so (Ui(UU) 2.12 x 10 22.174
The dark green dashed line shows the fitted curve, and the blue stars represent empirical crossover points-where diffusion matches AR performance in experiments. These points align closely with the predicted frontier, confirming our fitted equation's 's accuracy.
3.4
Downstream Results
Benchmarks
Random Baseline
AR
AR (Flop matched)
Diffusion
ARC-Easy 6
25.00
35.63
35.35
37.84
BoolQ 5
50.00
46.00
38.23
49.38
COPA 30
50.00
56.33
54,00
59.00
HellaSwag BalaSwagg 45
25.00
27.37
29.03
30.24
PIQA
50.00
60.94
61.64
60.72
RACE [6
25.00
25.28
24.88
28.96
WinoGrande XL
32
50.00
48.87
49,41
50.97
SciQ [4
25.00
58.05
50.50
68.67
Lambada 27
00.00
10.91
5.53
15.19
Note: All values represent accuracy (%). Best results shown in bold.
Table 2: Downstream results for the best perforrff (as per validation loss) and flop-matched autoregressive (AR) and diffusion models trained with 100M unique tokens. Random baselines are reported for reference.
We evaluate the best-performing diffusion and autoregressive (AR) models, selected based on their validation loss, across several downstream benchmarks to examine whether lower validation loss translates to improved generalization.
Since AR models tend to overfit much earlier, we additionally evaluate flop-atchee overfitted AR models trained for the same number of epochs as their diffusion counterparts. On a few set of benchmarks, these overfitted AR models slightly outperform their best-validation variants; however, across almost all tasks, diffusion models consistently achieve the highest downstream performance.

---

Additional results, with 500M unique tokens and other downstream datasets, are provided in Table and Table the Appendix.
Across a diverse set of tasks and data scales, diffusion models consistently outperform their AR counterparts.
3.5
Why do Diffusion models outperform AR models in data-constrained settings?
To better understand why diffusion models are more sample-efficient than autoregressive (AR) models, conducted series of controlled experiments aimed at isolating the core source of diffusion's advantage.
We first applied standard perturbation- based tech- niques during AR training. Specifically, we used: (i) attention dropout- randomly dropping 25%, 50%, or 75% of attention weights; and (ii) token masking masking subset of input tokens by zeroing their attention weights across all layers, while retaining the standard next- prediction objective.
Impact
Orderings AutongressiveAAA)
Stasdad AR
48-
with N-22
16
N--16
Diflzaco Beat Loaa
42-
As shown in Figures Sa and 8b Appendix, nei- ther approach improved validation loss. In all cases, AR models continued to overfit and remained far be- hind diffusion models trained for longer epochs. All AR baselines here used 140M parameters and were trained for 50 epochs; the red line in the plots marks the best diffusion model from Figure 2b trained for 500 epochs.
40-
36-
20
Figure 7: Validation loss improves as the number of token orderings N increases in AR training. At N 16, performance approaches that of diffusion models.
We next investigated whether diffusion's ' advantage stems from exposure to diverse token orderings. To test this, we trained AR models with varying numbers of orderings: right training, while N = A: adds k-1 random permutations of the sequence order. All permutations were fixed prior to training, and each training batch randomly samples from these orderings. All AR models in this setting used 278M parameters and were trained for 100 epochs. As shown in Figure [ increasing N consistently lowered validation loss and delayed overfitting. At N 16, the 100-epoch validation loss of AR models approached that of diffusion, suggesting that diverse orderings are indeed key driver of diffusion's 's sample efficiency.
1 denotes standard left-to-
These results support our interpretation that diffusion models outperform AR models in low-data regimes because they are implicitly trained on richer distribution of conditional prediction tasks.
Finally, this analysis suggests a natural continuum between the two paradigms: by controlling task diversity-throod masking reordering- we could design hybrid models that interpolate between compute efficiency (AR-like) and sample efficiency (diffusion-like) Exploring this continuum is a promising direction for future work. Details of our permutation process are in Section [2]
4
Conclusion
As the availability of high-quality data plateaus, improving sample efficiency becomes essential for scaling deep learning. In this work, we show that masked diffusion models consistently outperform autoregressive (AR) models in data-constrained regimes when training involves repeated passes over limited dataset. We establish new scaling laws for diffusion models, revealing their ability to extract value from repeated data far beyond what AR models can achieve. These results challenge the conventional belief that AR models are universally superior and highlight diffusion models as compelling alternative when data not compute-is the primary bottleneck. Looking ahead, efficient use of finite data may define the next frontier in scaling deep learning models. Although the studies have been performed in the context of language models, we believe these findings should apply across any kind of sequence modeling data, such as in robotics or healthcare.
For practitioners, our takeaway is simple: if you are compute-constrraine, use autoregressive models; if you are data-constrai use diffusion models.
10

---

Acknowledgement
We thank Alexander Li for his valuable insights and detailed feedback on the manuscript. We are also grateful to Zheyang Qin for his help in improving the figures. Finally, thank Niklas Muennighoff, Simo Ryu and Colin Raffel for their helpful comments on the final draft of the paper. We thank Stella Biderman for pointing out the inconsistent unit used in the Critical Compute equation. We thank Lucas Beyer and You Jiacheng. in suggesting the experiments in Section 3.5 that helped us explain why diffusion models would be more sample-efficient This work supported in part by ONR MURI N00014-22- 1-2773, ONR N00014-22-1-2096 and AFOSR EA9550-23-1-0747 The results and models presented in this work also used compute resources from the National Al Research Resource Pilot, with support from NVIDIA, including NVIDIA's DGX Cloud product and the NVIDIA AI Enterprise Software Platform.
References
[1] M. Arriola, A. Gokaslan, Chiu, Yang, Z. Qi, J. Han, S. S. Sahoo, and V. Kuleshov. Block diffusion: Interpolating between autoregressive and diffusion language models. arXiv preprint arXiv:2503. 09573, 2025.
[2] J. Austin, D. D. Johnson, J. Ho, D. Tarlow, and R. Van Den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in neural information processing systems, 34:17981-17993, 2021.
[3] Bai, Bai, Y. Chu, Z. Cui, K. Dang, Deng, Y. Fan, W. Ge, Y. Han, Huang, et al. Qwen technical report. arXiv preprint arXiv: 16609, 2023.
[4] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[5] C. Clark, K. Lee, M.-W. Chang, T. Kwiatkowski, M. Collins, and K. Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. In NAACI, 2019.
[6] Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: 183:v:1803.00457, 2018.
[7] S. Dieleman. Diffusion language models, 2023.
[8] R. Eldan and Y. Li. Tinystories: How small language models be and still speak coherent english?, 2023.
[9] I. Gulrajani and T. B. Hashimoto. Likelihood- diffusion language models. Advances in Neural Information Processing Systems, 36:16693-167155 2023.
[10] Ho, Jain, and Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6855 2020.
[11] Hoffmann, Borgeaud, Mensch, E. Buchatskaya. T. Cai, Rutherford, D. L. Casas,
L. A. Hendricks, J. Welbl, A. Clark, et al. Training compute-optima optimal large language models. arXiv preprint arXiv:2203.0 155566 2022.
[12] J. Hoffmann. S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas,
I.. Hendricks, J. Welbl, A. Clark, Hennigan, E. Noland, Millican, van den Driessche, B. Damoc, A. Guy. S. Osindero, K. Simonyan, E. Elsen, J. W. Rae, O. Vinyals, and L.. Sifre. Training compute-optimal large language models, 2022.
[13] E. Hoogeboom, A. A. Gritsenko, J. Bastings. B. Poole, R. v. d. Berg, and T. Salimans. Autore- gressive diffusion models. arXiv preprint arXiv:2110.02037, 2021.
[14] M. G. Johannes Welbl, Nelson F. Liu. Crowdsourcing multiple choice science questions. 2017.

---

[15] J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Rad- ford, J. Wu, and D. Amodei. Scaling laws for neural language models. arXiv preprint WrXiv:201.08366 2020.
[16] G. Lai, Q. Xie, H. Liu, Y. Yang, and E. Hovy. Race: Large-scale reading comprehension dataset from examinations. arXiv preprint arXiv:1704.04633 2017.
[17] X. Li, J. Thickstun, I. Gulrajani, P. S. Liang, and T. B. Hashimoto. Diffusion-Im improves controllable text generation. Advances in neural information processing systems, 35:4328-4344 2022.
[18] A. Lou, C. Meng, and S. Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution, 2024. URL https://arxiv, org/abs/2310.168./
[19] A. Lou, C. Meng, and S. Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution. arXiv preprint 2r 10. 16834, 2023.
[20] S. Merity, C. Xiong, Bradbury, and Socher. Pointer sentinel mixture models, 2016.
[21] N. Muennighoff, A. Rush, B. Barak, T. Le Scao, N. Tazi, A. Piktus, S. Pyysalo, T. Wolf, and
C. A. Raffel. Scaling data-constrained language models. Advances in Neural Information Processing Systems, 36:50358-50376, 2023.
[22] S. Nie, F. Zhu, C. Du, T. Pang, Q. Liu, G. Zeng, M. Lin, and C. Li. Scaling up masked diffusion models on text. arXiv preprint arXiv:2410. 18514, 2024.
[23] S. Nic, F. Zhu, Z. You, X. Zhang, J. Ou, J. Hu, J. Zhou, Y. Lin, J.-R. Wen, and C. Li. Large language diffusion models. arXiv preprint arXiv:2502.0999) 09992, 2025.
[24] Y. Nie, A. Williams, E. Dinan, M. Bansal, J. Weston, and D. Kiela. Adversarial nli: A new benchmark for natural language understanding. arXiv preprint arXiv: 14599, 2019.
[25] P. J. Ortiz Su' arez, B. Sagot, and L. Romary. Asynchronous pipelines for processing huge corpora on medium to low resource infrastructures. Proceedings of the Workshop on Challenges in the Management of Large Corpora (CMLC-7) 2019, Cardiff, 22nd July 2019, pages 9 16, Mannheim, 2019. Leibniz-Institut f"ur Deutsche Sprache.
[26] A. Pannatier, E. Courdier, and E. Fleuret. a-gpts: A new approach to autoregressive models, 2024.
[27] D. Paperno, G. Kruszewski, A. Lazaridou, Q. N. Pham, R. Bernardi, S. Pezzelle, M. Baroni, G. Boleda, and R. Fernández. The lambada dataset: Word prediction requiring broad discourse context. arXiv preprint www.:i::666.0000
2016.
[28] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, al. Language models unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
[29] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 21(140):1 67, 2020.
[30] M. Roemmele, C. A. Bejan, and A. Gordon. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In AAAI spring symposium: logical formalizations of commonsense reasoning, pages 95,95, 2011.
[31] S. Sahoo, M. Arriola, Y. Schiff, A. Gokaslan, E. Marroquin, J. Chiu, A. Rush, and V. Kuleshov. Simple and effective masked diffusion language models. Advances in Neural Information 2024.
Processing Systems, 37:1
[32] K. Sakaguchi, R. L.. Bras, C. Bhagavatula, and Y. Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9)999-106, 2021.
[33] N. Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.05200 2020.
12

---

[34] Shi, K. Han, Z. Wang, A. Doucet, and M. Titsias. Simplified and generalized masked diffusion for discrete data, Advances in neural information processing systems, 37:103131-1031677 2024.
[35] X. Shi, Zhao, H. Zhou, and D. Hao. Industrycorpus2. 2024.
[36] C. Shorten and T. M. Khoshgoftaar. survey on image data augmentation for deep learning. Journal of big data, 6(1):1 48 2019.
[37] Su, M. Ahmed, Y. Lu, S. Pan, W. Bo, and Y. Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing. 568:1 168:1,7063, 2024.
[38] J. Su, Y. Lu, S. Pan, A. Murtadha, B. Wen, and Y. Liu. Roformer: Enhanced transformer with rotary position embedding, 2023.
[39] A. Swerdlow, M. Prabhudesai, S. Gandhi, D. Pathak, and K. Fragkiadaki. Unified multimodal discrete diffuusinn. arXiv preprint 2Xii:25503208833 2025.
[40] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
[41] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L.. Jones, A. N. Gomez, F.. Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[42] Villalobos, A. Ho. J. Sevilla, T. Besiroglu, L. Heim, and M. Hobbhahn. Will we run out of data? limits of IIm scaling based on human-generated data. arXiv preprint arXiv:221 (tfh:2221104422. 1.04325, 2022.
[43] Yang, W. Xiao, M. Zhang, S. Guo, J. Zhao, and F. Shen. Image data augmentation for deep 2022.
learning: A survey. arXiv preprint
[44] Q Yu, J. He, X. Deng, X. Shen, and L.-C. Chen. Randomized autoregressive visual generation. arXiv preprini urXiv:24/1.00776 2024.
[45] R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi. Hellaswag: Can machine really finish your sentence? arXiv preprint arXiv:1905 07830, 2019.
13

---

Appendix
6
Related Work
Deep Learning in Data- Cota-Constrainted Settings. Deep learning progress has been largely driven by the scaling of both data and compute. However, recent analyses suggest we may soon face data bottleneck that could inhibit continued advancement 42 In language modeling, the dominant paradigm has been autoregressive (AR) models [41 28 which are typically trained for a single epoch to maximize exposure to unique tokens [TI] In light of looming data constraints, Muennighoff al. [2] show that AR models can still benefit from data reuse: training for up to four epochs on repeated data achieves performance nearly on par with training on fresh data, suggesting an effective strategy for improving data efficiency. In contrast, computer vision has long embraced multi-epoch training along with aggressive data augmentation and color jittering- tto expand effective dataset size and improve generalization [3] particularly for discriminative tasks like classification and detection. Despite these practices, data efficiency in generative modeling remains underexplored, and the trade-offs between leading paradigm dig such as diffusion and AR models under constrained data regimes are still poorly understood.
as random cropping, flipping,
Diffusion-Based Language Models. Diffusion models, originally developed for image genera- tion [10]. have recently been adapted to text, offering fundamentally different paradigm for language modeling 22 17 9. Broadly, diffusion language models fall into two categories: continoous and discrete. Continuous approaches [9] inject Gaussian noise in the forward process, whereas discrete methods 2 corrupt tokens with noise sampled from distributions such as Bernoulli. Among the two classes, continuous diffusion has proven more difficult to scale on language data [1]] In contrast, recent advances in discrete diffusion particularly masked diffusion have shown encouraging results. Recent work (I 31 [9] has significantly narrowed the performance gap between diffusion and AR models. Notably, LLaDA 23 scales masked diffusion models to 8B parameters and achieves results similar to L.LMMA-8BB across both pretraining and instruction-tuned evaluations. Furthermore, Nic et al. [22] provide scaling law analysis showing that diffusion models follow similar power-law trends as AR models, though they may require up 16x more compute under single-epoch training, Swerdlow et al. (39] find similar trends on multimodal data containing both image and text. However, these evaluations are restricted to single-pass training and do not examine the cata-constrained, multi-epoch regimes which is the focus of our work.
7 Additional Results
Impact of Artention Drapout on AR
Impact of Token Masking on AR
Deliali Despout
Delauli Setting
4.4
25% Alteniion
Tokes Masking
50% Alleriin Drop
4.4
25 ,9,, Token Masking
Valiainn Loo
75% Altention
Token Miskiig
Dillision Best Less
Validatin
4.-
Diffesion Best Lous
50, DMs, Tokeo Maskisg
4.-
4.0
3.8
3.8
3.6
1.6
Trannig Epochs
Trininng Epaccs
(a) Validation loss under varying attention dropout levels in AR training.
(b) Validation loss under varying token masking levels in AR training.
Figure 8: Impact of common data augmentations on AR models. Despite applying attention dropout and token masking, AR models still overfit and underperform compared to diffusion models. We believe this gap arises because diffusion models learn random factorizations of the joint distribution, rather than fixed left-to-right ordering.
In Table 3 we report downstream results using 500M unique tokens. Guided by the critical compute threshold derived in Section 3.3 we scale training to 500M unique tokens and train a 2.3B ---arammeter
14

---

diffusion model under the predicted compute budget. The model is trained for 130 epochs, after which we observe no signs of convergence and terminate due compute limitations. For the autoregressive (AR) model, we report results from the checkpoint that achieves the best validation loss before overfitting.-
Figure [a and Figure Sboshow that standard perturbation-basee techniques, such as attention dropout (25-75%) and token masking, fail to improve validation loss. Across all settings, AR models continue to overfit and lag behind diffusion models trained for longer epochs, All AR baselines use 140M parameters and are trained for 50 epochs, whereas the red line denotes the best diffusion model from Figure 2b trained for 500 epochs.
Table 4 reports the negative log- likelihood (NLL; lower is better) on four diverse corpora: OSCAR 25 TinyStories [), WikiText (20), and IndustryCorpus EN Sub 33 These datasets span open-
domain, narrative, encyclopedic, and industry-specific text.
Figure shows validation loss versus training FLOPs for autoregressive (AR) and masked diffusion models under data-constrained settings. This figure extends Figure i the main paper, by including empirical validation loss for the 25M unique token regimes.
Table 3: Downstream results for the best- best-performing (as per validation loss) autoregressive and diffusion models trained with 500M unique tokens. We also report the random baseline for reference.
Benchmarks
Random Baseline
AR
Diffusion
ARC-Easy
6
25.00
43.79
45.95
BoolQ
S
50.00
51.87
55,26
COPA
30
50.00
67.00
64.83
HellaSwag
45
25.00
32.28
35,33
PiQA
50.00
65.71
65.61
RACE
16
25.00
28.28
31.44
WinoGrande XI. 32
50.00
50.61
51.51
SciQ [4]
25.00
67.82
79.13
Lambada
27
00.00
15.07
22.30
Note: All values represent accuracy (%). Best results shown in bold.
Table 4: Downstream NLL of best diffusion and AR models at 100M unique data points.
Model Type
Flops
OSCAR
TinyStories
WikiText IndustryCorpus2
Best ARM
4.32e18
3.98
2.96
4.94 / 4.96
3.58
Best MDM
1.24e20
3.83
2.93
4.50 / 4.52
3.44
Pasdo Fronicros 1003) uniga daa
Paddu Frumier on 50M stique date
Pazzur Fruntee 25M unigee data
30
Diffasie Parzu Foamter
Were diftaoo box.AK Chickila
Pinto
Figure 9: Pareto frontier of validation loss (negative log-likelihood) versus training FLOPs for autoregressive (AR) and diffusion models under data-constrained settings, on three different unique data settings 25M, 50M and 100M.
15

---

Why are autoregressive (AR) models more compute-efficient than diffusion models? We hypothesize two main contributing factors. (i) Order specialization: AR models trained with fixed left-to-right factorization, so every gradient update reinforces the same prediction task, allowing them specialize effectively. In contrast, diffusion models must generalize across many random token orderings, which hinders specialization. (ii) Stronger supervision per update: In AR training, every token in training sequence serves supervised target, and the causal structure enables dense gradient updates, resulting in stable, low-variance learning. Diffusion models, however, compute loss only subset of masked tokens, making supervision sparser per sequence, even though gradients propagate through the entire input. As a result, each update carries less direct learning signal. Arriola 8 Hy pyperaaraeeeer details
We use the following hyperparameters: batch size of 256 sequences, AdamW optimizer with =1-0.9. 32=0.95, c=10-$ learning rate schedule with peak 2e-4, minimum 2e-5, I % warm-up, cosine decay, weight decay 0.1, and gradient clipping of 1.0.
Model Architecture
We adopt the we build our training and evaluation setup for the masked Diffusion Model. Similar to the "extended version of the architectures" proposed in [22). our model adheres to the general transformer design while introducing several architectural modifications to better align with modern LLM practices.
framework as the foundation of our implementation, upon which
Specifically, we replace absolute positional embeddings with Rotary Positional Embeddings (RoPE) (37] which improve extrapolation longer contexts and reduce parameter count. Furthermore, we adopt the SwiGLU activation function in the MLP blocks, which has been shown to outperform standard GELU ReLU in both convergence and downstream performance [33]. To further simplify the architecture and enhance training stability, substitute standard LayerNorm with RMSNorm and eliminate all bias terms. These design choices are consistent with 40).
To preserve the original MLP capacity while aligning with hardware- friendly parameter sizes, we compute the beed-forward hidden size hy as:
dinodel
hy 3 64
This rounding scheme ensures that the FFN hidden size remains divisible by 64 while closely matching the effective dimensionality used in SwiGLU layers.
We slightly modify the parameter count estimation formula from the original:
13
V+8
P= 12th2
12h
12th
better reflect our revised architecture. The original formula can be decomposed into: 4/h² (atten- tion), 8th2 (MLP), 13/h (LayerNorm and biases), and (V + s)h (token and positional embeddings). After applying our architectural adjustments namely, using a SwiGLU-basee MLP of dimension hy, switching to RoPE (eliminating sh), and removing bias terms we arrive at the revised formula:
P 41h + 3th hy 6th + Vh.
Table 5 presents all model configurations used in our experiments along with their parameter counts.
10 Discussion
al. [ show that tuning the masking schedule can help reduce gradient variance and improve training compute efficiency.
16

---

Table Model Architectures
Name
param (M)
d (_model
origin ffw sizz
ffw Ti__iize
kv v_size
n_ heads
Llaeers
7
7.0
128
512
320
32
4
14
13.6
224
896
576
32
7
20
19.5
288
1152
768
32
7
35
36.6
448
1792
1152
32
44
50.7
512
2048
1344
64
8
8
57
64.8
576
2304
1536
64
74
80.5
640
2560
1664
64
10
10
90
95.0
640
2560
1664
64
10
106
109.6
640
2560
1664
64
10
16
117
123.6
768
3072
2048
64
12
140
144.8
768
3072
2048
64
12
15
163
166.1
768
3072
2048
64
12
175
179.2
896
3584
2368
64
14
14
196
198.3
896
3584
2368
64
14
217
217.5
896
3584
2368
64
14
18
251
250.8
1024
4096
2688
16
278
275.7
1024
4096
2688
64
16
18
306
300.6
1024
4096
2688
64
16
20
425
416.9
1280
5120
3392
128
10
18
489
475.6
1280
5120
3392
128
10
21
509
495.9
1408
5632
3712
128
11
18
552
534.4
1280
5120
3392
128
10
587
566.7
1408
5632
3712
128
21
632
615.3
1536
6144
4096
128
12
19
664
637.6
1408
5632
3712
128
24
724
700.3
1536
6144
4096
128
12
22
816
785.2
1536
6144
4096
128
12
25
893
856.4
1792
7168
4736
128
14
20
1018
971.3
1792
7168
4736
128
14
23
1 1143
1086.3
1792
7168
4736
128
14
26
1266
1207.6
2048
8192
5440
128
16
22
1424
1353.6
2176
8704
5760
128
17
22
1429
1358.2
2048
8192
5440
128
16
25
1593
1508.9
2048
8192
5440
128
16
28
1609
1523.2
2176
8704
5760
128
17
25
1731
1644.9
2304
9216
6144
128
18
24
1794
1692.9
2176
8704
5760
128
17
28
2007
1899.8
2304
9216
6144
128
18
28
2283
2154.7
2304
9216
6144
128
18
2298
2165.3
2560
10240
6784
128
20
26
2639
2478.6
2560
10240
6784
128
20
30
2980
2791.9
2560
10240
6784
128
20
34
3530
3257.0
2688
10752
7168
128
21
36
3802
3561.3
2816
11264
7488
128
22
36
4084
3879.2
2944
11776
7808
128
23
36
4516
4231.9
3072
12288
8192
128
24
6796
6337.4
3584
14336
9536
128
28
40
9293
8640.6
4096
16384
10880
128
32
42
11452
10889.0
4352
17408
11584
128
32
47
12295
11444.2
4608
18432
12288
128
36
44
12569
12208.7
4608
18432
12288
128
32
47
13735
13560.0
4864
19456
12928
128
32
47
14940
14905.3
4992
19968
13312
128
32
49
16183
15028.3
5120
20480
13632
128
40
47

---

Algorithm Generating a Random Order List with Predefined Permutations
Input: Sequence length L, number of orders N, random seed
Output: Order list O of N orderings
I: Initialize order list O +
2: Append raster order: O OU (0,
1|}
3: for i = to N - do
4: b- [0. 10.,.....4 I- [base raster order)
5: c < (0, (T) [add Gaussian noise with scale i)
6:
$ + - (perturbed scores]
7:
(rgsort(s) (permutation order)
8:
- (m{
9: end for
11: return O
11 Limitations
In this work, we examined two extremes of generative modeling: masked diffusion models, which learn over random condition prediction tasks and are more data-efficient, and autoregressive (AR) models, which follow fixed left-to-right order and are more compute-efficien While our results highlight clear trade-off, this need not be binary-hybrid models that interpolate between AR and diffusion would offer a better balance. Although prior works have explored such hybrids [3] they have not been evaluated through the lens of data-compute efficiency. We explore part of this question in Section 3.5 however it will be useful to study this in more detail. Additionally,
scaling laws are currently fit over a limited range of unique data sizes; extending them to larger regimes may improve predictive accuracy and reveal further insights.
12 Order Permutation Details
In this experiment, we train autoregressive models using different token orderings. We do not introduce target positional embeddings as done in works such as RAR [44] 26]. We evaluated the trained models using left-to-right ordering. We define the perturbations in the token ordering by adding varying levels of noise to the left-to-right ordering.
Specifically, we generate a list of N orderings, where the first order is the standard left-to-right (12r) order. Subsequent permutations are created by adding Gaussian noise to the left-to-right position ids, with the standard deviation of the noise directly proportional to the permutation's 's index. This method allows us to create spectrum of orderings, from the standard 12r order to more heavily permuted sequences, detailed in Algorithm] !
During training, we apply these predefined orders the input sequences. For each sequence in batch, we randomly sample a permutation from our predefined list. This process is summarized in Algorithm 2 and further detailed below:
For each sequence, the first token kept fixed. This ensures that the position ID is always assigned to the first token, providing soft absolute positional anchor for the sequence when using RoPE[38] Under ROPE, attention depends only on relative position offsets rather than absolute information, iee. (R(i)q, R(j)k) qR(i j)k: Therefore, fixing position on the first token keeps the control anchor unrotated R(0) and removes global sequence-wiss phase shifts induced by permutations, which stabilized the optimization and reduced variance under permutation augmentation.
As an example, suppose that the number of predicted tokens is T' (e.g. T 2048 in our default setting) and the total input length is L T including the label shift. Only the indices in [1:7] are shuffled and assigned position IDs from {1..... T}. For instance, with = 6 and permutation [2, 0, 1, 4, 5, 3), the resulting token and label orders are:
tokens:
[0, 3, 1, 2, 5, 6].
labels:
[3, 1, 2, 5, 6, 4).
18

---

Algorithm 2 Shuffting Tokens Using Predefined Order Lists
Input:
Token matrix tokens E Bx (including last label), order list of permutations
Output: Shuffled tokens and position IDs
Let B k number of sequences in batch
2: Let L f sequence length
3: Initialize position
t- 08xL
4: Sample index vector I ~
K (select random order for each sequence)
5: for 1 B do
6: π [[1,] fretrieve i-th random order]
7: tokens i, 1:] f- tokens(i, 1:][m] (shuffle tokens except first token)
8:
π + π + {shift positions by to reserve position 0)
9: position ids(i, 1:] x [0: (assign shifted positions)
10: end for
11:
12: return tokens, mosition_ids
13 Broader Impacts
Our findings suggest that masked diffusion models are more sample-efficiett language modeling. which is especially valuable as high-quality textual data becomes scarce. This could benefit low- resource languages, privacy-sensitive domains, and scientific fields where data is limited costly. By reducing dependence on massive proprietary datasets, diffusion models may help democratize access LLM development. Nonetheless, the broader deployment of generative models raises concerns around misuse and misinformation, underscoring the need for responsible research and deployment practices.
19

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
The abstract and introduction clearly state the goal of comparing diffusion and autoregressive models in data-constrained regimes. The main claims that diffusion models are more sample-efficies under repeated exposures and that new scaling laws established are consistent with the theoretical analysis and experimental results presented throughout the paper.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The limitations of the work, such the restricted range of model sizes and datasets used in scaling law fitting, are explicitly discussed in the paper.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
20

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: The paper does not include formal theoretical results or theorems requiring assumptions or proofs. While empirical scaling laws are fitted, they are based on empirical loss trends rather than derived from formal proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results, All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: All experimental setups, including model size, data budget, compute used, and training protocols, are fully described in Section. Code is opensourced.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibilityy can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the partieular way they provide for reproducibiliity In the case of closed-source models, it may be that access to the model is limited in
21

---

some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
Code and data is made public.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cr cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https Ctteee//CoSSSuumiss.....
/ httpp/cccppbccccceeeeeeeee
www.iisssoopoooooooooo for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The paper explains the model hyperparameters and training setup.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: compute.
We showed the results with different amount of unique dataset sizes and
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
22

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
The detailed information of the compute resources and configuration are provided the supplementary materials.
Justification:
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer:
[Yes]
Justification:
We discussed he broader impact of the work in the last section. of Appendix.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discussed he broader impact of the work in the last section. of Appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
23

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The models trained are small scale.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The framework and datasets that we used to train and evaluate open-soureed and correctly cited.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
24

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The provided code is well-documented.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve any human subjects or crowdsourcing experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The work does involve human participants or subject studies.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
25

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: No use of LLMs.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy bottps://neurre /ctp:///ooeeeeeccc///2///////// for what should or should be described.
26


Reviews:
[{"review_id": "m4YNJqUzQ4", "reviewer": "Reviewer_WHBh", "summary": "This paper identifies a novel setting in which discrete diffusion models are strictly preferable to autoregressive models: specifically, diffusion models are more desirable than AR models in data-constrained regimes.\n\nThe authors conducted a detailed scaling law analyses demonstrating that MDMs leverage repeated data more effectively, achieving superior performance across varying model sizes, data volumes, and computational budgets.", "questions": "None", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "**Strengths:**\n\nThe authors identify a key strength of masked diffusion models, opening up new avenues where these models are strictly preferable.\n\n**Weaknesses:**\n\nThe primary takeaway of the paper is that MDMs offer benefits only when the number of unique tokens is 100M or fewer, which feels somewhat underwhelming.\n\n\n**Minor:**\n\n> “Previous studies comparing diffusion models and autoregressive (AR) models have predominantly focused on the single-epoch regime [16, 21, 24].”\n\nReference [21] actually trains AR and MDM in a multi-epoch setting on LM1B and OpenWebText, so it should be dropped from this sentence.\n\n\n**Final Remarks:**\n\nThis paper is clearly above the acceptance threshold and should be accepted.", "was_revised": true, "final_justification": "I maintain my score.", "rebuttal": "Thank you for your constructive feedback and for spending the time to read our paper in detail.\n\nBelow we address your concerns:\n\n---\n\n### **Q1.1: The primary takeaway of the paper is that MDMs offer benefits only when the number of unique tokens is 100M or fewer, which feels somewhat underwhelming.**\n\nWe respectfully disagree with this characterization. The primary contribution of our work is to demonstrate that diffusion models consistently outperform autoregressive models in data-constrained regimes. To test this, we conducted experiments across a range of unique token budgets—25M, 50M, and 100M—in the original submission.\n\nSince submission, we have **extended our experiments to a 500M unique token setting**, where we again observe that MDM outperforms ARM when both are trained under the same data budget, as shown in the updated results below. \n\nWhile we agree that it would be valuable to validate these trends at even larger scales (e.g., 10B unique tokens), doing so would require training for trillions of tokens to maintain the epoch count (e.g., 500 epochs × 10B = 5T tokens), which is far beyond our current compute budget. That said, we believe the consistency of our results across multiple unique token scales, including up to 500M, could validate our claims.\n\n---\n\n### **Q1.2: Reference [21] actually trains AR and MDM in a multi-epoch setting.**\n\nThank you for pointing this out. We apologize for the oversight and will correct this in the final version.\n\n---\n\n## **Additional Updates**\nWe have also conducted additional experiments since our submission. We summarize these updates below:\n\n---\n\n### **i) Why is MDM Better in Data-Constrained Settings?**\n\nTo better understand *why* MDMs outperform ARMs in low-data regimes, we conducted a series of ablations on ARM models to test whether exposing them to more diverse prediction tasks, which is similar to what MDMs inherently do, could close the gap or interpolate between the two methods.\n\nWe hypothesized in the paper:  \n> *“MDM is exposed to diverse token orderings and prediction tasks, unlike the fixed left-to-right training of AR models. This enables them to extract more information per example.”*\n\nTo validate this, we conducted the following experiments:\n\n#### **(a) Random Token Masking in ARM**\n\n| Token Mask Ratio | Val Loss |\n|------------------|----------|\n| 0% (Default)     | **3.71630**  |\n| 12.5%            | 3.72701  |\n| 25%              | 3.75580  |\n| 37.5%            | 3.81153  |\n| 50%              | 3.90185  |\n\nWe observe that token masking in AR degrades validation loss, especially at higher ratios, likely due to over-regularization.\n\n#### **(b) Attention Dropout in ARM**\n\n| Dropout %        | Val Loss |\n|------------------|----------|\n| 10% (Default)    | 3.71630  |\n| 25%              | **3.71268**  |\n| 50%              | 3.72415  |\n| 75%              | 3.76850  |\n\nModerate dropout yields marginal improvements but leads to degradation at higher levels.\n\n#### **(c) Randomizing Token Order (Sigma-GPT [1] / RandAR [2])**\nIn order to show the full impact, including the impact on overfitting points, we used a setting of 100 epochs here. Therefore, the original result for AR is different from the previous two experiments (50 epochs).\n\n| Variant              | Val Loss |\n|----------------------|----------|\n| Original | 3.74799  |\n| 25% Random Order     | 3.67741  |\n| 50% Random Order     | **3.66556**  |\n\nRandomized orderings show the most promising gains, supporting our hypothesis that MDMs benefit from diverse prediction structures. For this experiment, we implemented RandAR (Any-Order AR) on top of our ARM baseline.\n\n---\n\n### **ii) Scaling to 500M Unique Tokens Using the Critical Compute Equation**\n\nIn Appendix C, we derive an analytical equation that estimates the critical compute at which MDM matches the validation loss of ARM, given a fixed unique data budget. Using this, we scaled the training set to 500M unique tokens and performed a grid search over model sizes and total data based on the predicted compute to identify the configuration predicted by the scaling law to achieve the optimal validation loss as close as possible. Therefore, we trained a 2.3B parameter MDM using the predicted critical compute budget. After training for 130 epochs, we observed that the validation loss of MDM is consistently lower than that of ARM.\n\nFor the ARM baseline, we use the same method to find out optimal setting under the same data budget. We report the empirical validation losses of the trained models below:\n\n**Validation Losses:**\n\n- **MDM:** 3.08573  \n- **ARM:** 3.20136  \n\n---\n\n### **iii) More Downstream Results**\n\nWe further evaluated the best-performing MDM and ARM models on a range of language understanding tasks to assess whether the gains in validation loss translate to better generalization. Across a diverse set of tasks and data scales, MDMs consistently outperform ARMs.\n\n**Table 2: Accuracy (%) on downstream evaluation tasks at different unique token scales.**\n\n| Benchmark       | Random | ARM (100M) | MDM (100M) | ARM (500M) | MDM (500M) |\n|-----------------|--------|------------|------------|-------------|-------------|\n| ARC-Easy        | 25.00  | 35.63      | **37.84**  | 43.79       | **45.95**   |\n| BoolQ           | 50.00  | 46.00      | **49.38**  | 51.87       | **55.26**   |\n| COPA            | 50.00  | 56.33      | **59.00**  | **67.00**   | 64.83       |\n| HellaSwag       | 25.00  | 27.37      | **30.24**  | 32.28       | **35.33**   |\n| PiQA            | 50.00  | **60.94**      | 60.72  | **65.71**   | 65.61       |\n| RACE            | 25.00  | 25.28      | **28.96**  | 28.28       | **31.44**   |\n| WinoGrande XL   | 50.00  | 48.87      | **50.97**  | 50.61       | **51.51**   |\n| SciQ            | 25.00  | 58.05      | **68.67**  | 67.82       | **79.13**   |\n| Lambda          | 00.00  | 10.91      | **15.19**  | 15.07       | **22.30**   |\n\n*Note: Best results shown in bold.*\n\n---\n\n### **iv) Improved Fitting Metrics (in Appendix)**\n\nWe refit the scaling laws after removing configurations with a small number of training epochs (<20) for MDMs and excluding very small model sizes (<45M parameters) for ARMs. These configurations often exhibited high variance in validation loss due to insufficient optimization, introducing noise into the regression. Excluding them resulted in significantly more stable and interpretable fits.\n\n**Updated Table 1:**\n\n**(a) Initial fit**\n\n| Model | R²     | Loss       |\n|-------|--------|------------|\n| MDM   | 0.9447 | 0.0002     |\n| ARM   | 0.9439 | 7.7532e−05 |\n\n**(b) Second-step fit with extracted scaling parameters**\n\n| Model | R²     | Loss     | R\\*_D  | R\\*_N   |\n|-------|--------|----------|--------|---------|\n| MDM   | 0.9784 | 0.00079  | 493.89 | 1265.65 |\n| ARM   | 0.7628 | 0.00361  | 31.19  | 55.16   |\n\n---\n\n**References:**\n\n[1] Pannatier, Arnaud, Evann Courdier, and François Fleuret. *σ-GPTs: A new approach to autoregressive models.*\n\n[2] Pang, Ziqi, et al. *RandAR: Decoder-only autoregressive visual generation in random orders.*"}, {"review_id": "YHl4t2Tsby", "reviewer": "Reviewer_u7Lw", "summary": "This paper studies an interesting and timely research question---the scaling law of masked diffusion language models, particularly under data-constrained settings, or equivalently, the setting where multiple iterations over a fixed-size training set are performed. Notably, several core observations have been made, including, for example, the change in validation loss with respect to the number of tokens seen and model size. Experiments also reveal that AR models are more susceptible to overfitting under repeated data points. Overall, evidence show that masked diffusion models are more data-efficient over AR models under certain conditions.", "questions": "Q1. The formulations in line 27 and line 32 for AR and MDMs are not rigorously correct. While I can get the point that the authors want to highlight the difference between AR and MDMs in terms of ordering, I would not recommend loosely writing the factorizations in this way. As already said in Weakness, the AR factorization in line 27 is not exactly true. the MDM factorization in line 32 is not rigor as well, and distinctions should be made between MDMs and any-order ARs [1] to avoid confusions.\n\nQ2. It has been revealed that MDMs require larger number of epochs over limited training data to achieve the peak performance. Should other techniques that can possibly lead to different training dynamics (like higher learning rates) help mitigate the issue?\n\nQ3. The conclusion drawn from this paper indicates that MDMs could outperform ARs when training set is limited. However, this is in contrary to the large-scale pretraining paradigm adopted for modern LLMs. Does this research convey a pessimistic signal that MDMs are not suitable / may not be able to outperform / replace ARs as a new generation of large language models?\n\nQ4. There is an important factor which I believe requires further elaboration. While the paper has clearly distinguished number of unique tokens and number of total tokens used during training, for MDMs there is another critical concept of \"effective tokens\", i.e., the tokens that are used to perform gradient descent (those that are masked in each interation) that also leads to the difference of training dynamics between MDMs and ARs. In particular, in each iteration ARs can perform gradient update using all tokens while MDMs can only use effectively half of them, assuming a uniform mask rate between 0 and 1. This concept has been mentioned in previous works such as [2] but is never discussed and analyzed in this paper.\n\n[1] Shih et al. Training and Inference on Any-Order Autoregressive Models the Right Way. NeurIPS 2022.\n\n[2] Arriola et al. Block diffusion: Interpolating between autoregressive and diffusion language models. ICLR 2025.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## Strengths\n\n1. The paper is well-motivated and the study of the scaling law of masked diffusion model is a timely research for the community.\n\n2. The aspects studied in this paper are comprehensive, including many interesting and detailed investigations.\n\n## Weaknesses\n\n1. Regarding technical novelty, the methodology basically follows previous investigations of the scaling law for AR language models, yielding limited technical contribution.\n\n2. As an empirical study, it would be better, if more profound and deeper understanding about not only \"what\" the observations are, but also \"why\" such things happen, can be discussed and analyzed. However, these insights are absent in the current content.\n\n3. Several statements made in the paper lack rigor. For instance, line 27 the factorization did not specify the meaning of $x_i$. And if they refers to tokens, AR model actually factorizes $p(x)=p(x_0)p(x_1|x_0)p(x_2|x_1,x_0)$, which is different from the formulation presented in this paper. See Questions for more details.", "was_revised": true, "final_justification": "With the rebuttal, the paper is in much better shape with sufficient results to support the claims, as an empirical research. The authors are recommended to incorporate the promised changes to the manuscript.", "rebuttal": "Thank you for your constructive feedback and thoughtful suggestions. \n\nBelow, we address each of your points in turn:\n\n---\n\n### **Q2.1: Limited technical novelty.**\n\nWe agree that our work does not introduce a new algorithm. However, we believe our empirical findings are novel and important for the generative modeling community. Our work is the first to systematically characterize the **data vs. compute trade-offs** between MDMs and ARMs across a wide range of model and data scales, and to establish a **closed-form scaling law** for the critical compute threshold at which MDMs outperform ARMs.\n\nThese insights, while empirical, can provide a foundation for future algorithms that can better interpolate between the compute-efficiency of ARMs and the data-efficiency of MDMs. \n\n---\n\n### **Q2.2: More analysis on *why* diffusion is better in data-constrained settings.**\n\nTo better understand why MDMs outperform ARMs in low-data regimes, we conducted a series of ablations on ARM models to test whether exposing them to more diverse prediction tasks, which is similar to what MDMs inherently do, could close the gap or interpolate between the two methods.\n\nWe hypothesized in the paper:  \n> *“MDM is exposed to diverse token orderings and prediction tasks, unlike the fixed left-to-right training of AR models. This enables them to extract more information per example.”*\n\nTo validate this, we conducted the following experiments:\n\n#### **(a) Random Token Masking in ARM**\n\n| Token Mask Ratio | Val Loss |\n|------------------|----------|\n| 0% (Default)     | **3.71630**  |\n| 12.5%            | 3.72701  |\n| 25%              | 3.75580  |\n| 37.5%            | 3.81153  |\n| 50%              | 3.90185  |\n\nWe observe that token masking in AR degrades validation loss, likely due to over-regularization.\n\n#### **(b) Attention Dropout in ARM**\n\n| Dropout %        | Val Loss |\n|------------------|----------|\n| 10% (Default)    | 3.71630  |\n| 25%              | **3.71268**  |\n| 50%              | 3.72415  |\n| 75%              | 3.76850  |\n\nModerate dropout yields minor improvements, but quickly becomes harmful at higher levels.\n\n#### **(c) Randomizing Token Order (Sigma-GPT [1] / RandAR [2])**\n\n| Variant              | Val Loss |\n|----------------------|----------|\n| Original | 3.74799  |\n| 25% Random Order     | 3.67741  |\n| 50% Random Order     | **3.66556**  |\n\nRandomized token orderings show the most promising improvements, supporting our hypothesis that MDM’s exposure to diverse prediction structures enables greater information extraction per example. For this experiment, we implemented RandAR (Any-Order AR) on top of our ARM baseline.\n\n---\n\n### **Q2.3: Factorizations for AR and MDM in Lines 27 and 32 lack rigor.**\n\nThank you for pointing out. We agree that the factorizations for both AR and MDM were oversimplified and may be misleading, and will revise the text to refer readers directly to the method section, where the model factorizations are explained rigorously and unambiguously.\n\nIn particular:\n- For AR models, we will clarify that the joint distribution is factorized as:\n$$ p(x) = \\prod_{t=0}^{T-1} p(x_t \\mid x_{<t}) $$ \n , rather than the simplified and potentially misleading Markovian example currently shown.\n\n- For MDM, we acknowledge that writing the joint distribution in a factorized form is not theoretically grounded. Unlike AR or any-order AR models, MDMs do not explicitly define a joint factorization. To avoid confusion, we will remove the equation in Line 32 and use the following sections to explain in detail.\n\n---\n\n### **Q2.4: Can higher learning rates mitigate MDM’s compute inefficiency?**\n\nWe do not expect this to be the case. Multiple prior works have noted that **MDMs require more compute** than ARMs to reach similar validation losses (e.g., [1, 16, 24]). We believe these inefficiencies arise not from optimization difficulties, but from fundamental differences in supervision density and variance in gradient (see Q2.6).\n\nThat said, we are currently running additional experiments with higher learning rates to test whether optimization plays any non-negligible role. We will include updated results in the discussion/final version once available.\n\n---\n\n### **Q2.5: Does this research convey a pessimistic signal that MDMs are not suitable as the next generation of large language models?**\n\nWe do not believe so. As Ilya Sutskever aptly said, *\"Compute is growing at a rapid pace, but data is not—we have just one internet.\"* Our findings suggest that MDMs are significantly more data-efficient than ARMs, which may make them better suited for future regimes where high-quality unique data is scarce.\n\nWe envision a number of exciting research directions that build on our work:\n\n- **Interpolating between MDM and ARM objectives** to achieve better trade-offs between compute and data efficiency.\n- **Modality- or quality-adaptive objective**, where MDM is used on scarce, high-quality data and ARM is used on abundant, low-quality data.\n- Applications in domains such as **robotics and healthcare**, where one doesn't have billions/trillions of tokens to begin with and data-efficient generative models are critical.\n\nRather than pessimistic, we view our work as an invitation to rethink model design in the post-internet era.\n\n---\n\n### **Q2.6: Further elaboration on effective tokens seen by MDM.**\n\nThank you for highlighting this. We agree that in MDM training, loss is computed on only a subset of tokens (in expectation, 50%), compared to ARM where every token is supervised. We believe this could contribute to the compute inefficiency of MDMs and have added the following discussion in our paper:\n\n> **Why are autoregressive (AR) models more compute-efficient than diffusion models?**  \n> We hypothesize the following as contributing factors:  \n> (i) *Stronger supervision per update:* In AR training, every token in a sequence is a supervised target, and the causal dependency structure enables dense, low-variance gradient updates.  \n> (ii) *Sparser supervision in MDM:* In contrast, MDM loss is computed on a subset of masked tokens, and while gradients propagate through the entire sequence, each update carries less direct learning signal.  \n> Arriola et al. [1] demonstrate that tuning the masking schedule can help reduce gradient variance and improve compute efficiency in MDMs.\n\nWe thank the reviewer again for this useful clarification.\n\n---\n## **Additional Updates** \n\nWe have also conducted additional experiments since our submission. We summarize these updates below:\n\n---\n\n### **i) Scaling to 500M Unique Tokens Using the Critical Compute Equation**\n\nIn Appendix C, we derive an analytical equation that estimates the critical compute at which MDM matches the validation loss of ARM, given a fixed unique data budget. Using this, we scaled the training set to 500M unique tokens and performed a grid search over model sizes and total data based on the predicted compute to identify the configuration predicted by the scaling law to achieve the optimal validation loss as close as possible. Therefore, we trained a 2.3B parameter MDM using the predicted critical compute budget. After training for 130 epochs, we observed that the validation loss of MDM is consistently lower than that of ARM.\n\nFor the ARM baseline, we use the same method to find out optimal setting under the same data budget. We report the empirical validation losses of the trained models below:\n\n**Validation Losses:**\n\n- **MDM:** 3.08573  \n- **ARM:** 3.20136  \n\n---\n\n### **ii) More Downstream Results**\n\nWe further evaluated the best-performing MDM and ARM models on a range of language understanding tasks to assess whether the gains in validation loss translate to better generalization. Across a diverse set of tasks and data scales, MDMs consistently outperform ARMs.\n\n**Table 2: Accuracy (%) on downstream evaluation tasks at different unique token scales.**\n\n| Benchmark       | Random | ARM (100M) | MDM (100M) | ARM (500M) | MDM (500M) |\n|-----------------|--------|------------|------------|-------------|-------------|\n| ARC-Easy        | 25.00  | 35.63      | **37.84**  | 43.79       | **45.95**   |\n| BoolQ           | 50.00  | 46.00      | **49.38**  | 51.87       | **55.26**   |\n| COPA            | 50.00  | 56.33      | **59.00**  | **67.00**   | 64.83       |\n| HellaSwag       | 25.00  | 27.37      | **30.24**  | 32.28       | **35.33**   |\n| PiQA            | 50.00  | **60.94**      |  60.72  | **65.71**   | 65.61       |\n| RACE            | 25.00  | 25.28      | **28.96**  | 28.28       | **31.44**   |\n| WinoGrande XL   | 50.00  | 48.87      | **50.97**  | 50.61       | **51.51**   |\n| SciQ            | 25.00  | 58.05      | **68.67**  | 67.82       | **79.13**   |\n| Lambda          | 00.00  | 10.91      | **15.19**  | 15.07       | **22.30**   |\n\n*Note: Best results shown in bold.*\n\n---\n\n### **iii) Improved Fitting Metrics (in Appendix)**\n\nWe refit the scaling laws after removing configurations with a small number of training epochs (<20) for MDMs and excluding very small model sizes (<45M parameters) for ARMs. These configurations often exhibited high variance in validation loss due to insufficient optimization, introducing noise into the regression. Excluding them resulted in significantly more stable and interpretable fits.\n\n**Updated Table 1:**\n\n**(a) Initial fit**\n\n| Model | R²     | Loss       |\n|-------|--------|------------|\n| MDM   | 0.9447 | 0.0002     |\n| ARM   | 0.9439 | 7.7532e−05 |\n\n**(b) Second-step fit with extracted scaling parameters**\n\n| Model | R²     | Loss     | R\\*_D  | R\\*_N   |\n|-------|--------|----------|--------|---------|\n| MDM   | 0.9784 | 0.00079  | 493.89 | 1265.65 |\n| ARM   | 0.7628 | 0.00361  | 31.19  | 55.16   |\n\n\n**References:**\n\n[1] Pannatier, Arnaud, Evann Courdier, and François Fleuret. *σ-GPTs: A new approach to autoregressive models.*\n\n[2] Pang, Ziqi, et al. *RandAR: Decoder-only autoregressive visual generation in random orders.*"}, {"review_id": "3f3BAY93eY", "reviewer": "Reviewer_43Rm", "summary": "This paper empirically demonstrates that masked diffusion models (MDMs) can outperform autoregressive models (ARMs) in data-constrained settings. This finding offers a new perspective because most existing large language models are based on ARMs. Inspired by the observation that ARMs become less effective with more training epochs, the authors address the problem of performance in data-limited scenarios. They compare ARMs and MDMs using the same architecture, with various data and model size. The results show that, given sufficient compute in multi-epoch training, MDMs achieve lower validation loss and better downstream performance compared to ARMs.", "questions": "* Related to the first point in the Weaknesses, it would be helpful if they provide empirical results showing that MDMs outperform ARMs in data-constrained settings with other datasets or model architectures.\n\n* Related to the second point in the Weaknesses, it would be valuable to see if MDMs remain effective when the ARM's optimizer settings are adjusted.\n\n* Related to the third point in the Weaknesses, it would be useful to verify whether MDMs demonstrate effective performance on tasks that can be evaluated by human judgment.", "limitations": "Yes.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## [Strengths]\n\n* Data-constrained settings are important not only for domains where language models must be trained with limited data, but also for cases that may arise even in large language models. Therefore, this work addresses a highly significant problem.\n\n* They conduct experiments on hundreds of models and provide an evaluation across various data sizes, model scales, and numbers of training epochs.\n\n## [Weaknesses]\n* Although the authors conducted experiments with various model and dataset sizes, there are still some limitations in the experimental setup. The experiments seem to focus on the English C4 dataset and the GPT-style Transformer model. While I appreciate that the paper presents results for a range of model and data sizes, it would provide stronger evidence if similar results could be demonstrated on other datasets and model architectures.\n\n* Since the losses for ARMs and MDMs differ, the optimal optimizer settings (such as learning rate, optimizer choice, etc.) may differ as well. However, it appears that the same settings are used for both. This raises the question of whether the chosen hyperparameters might be particularly well-suited for MDMs. I believe this point warrants further discussion.\n\n* While the experimental results are presented primarily in terms of NLL-based loss values, it is also important to evaluate the effectiveness of the generated text outputs through human judgment or downstream tasks that reflect real-world utility.", "was_revised": true, "final_justification": "Through the authors' response and additional experiments, I believe this paper provides a valuable discussion point on one of the key ongoing topics in the language modeling community: the comparison between autoregressive and diffusion models.\n\nHowever, since the primary focus of this work is on empirical comparison, I find it difficult to expect fundamental insights into the underlying aspects of the models. Therefore, I believe that my current borderline accept recommendation remains appropriate.", "rebuttal": "Thank you for your constructive feedback and for carefully reviewing our work.\n\nBelow, we address your points in turn:\n\n---\n\n### **Q3.1: Extensions to other datasets and model architectures.**\n\nThank you for this suggestion. We agree that validating our findings across different datasets and model architectures will strengthen the paper.  \nWe are **currently running experiments on SlimPajama** (a large, high-quality text dataset) and on **T5 (encoder-decoder architecture)**.   These experiments are ongoing, and we plan to include the results in the discussion phase or in the final version of the paper.\n\n---\n\n###  **Q3.2: Varying optimizer settings for ARM. Chose hyperparameters might be particularly well-suited for MDMs**\n\nWe followed the exact hyperparameter settings from the NeurIPS data-constrained Scaling Law paper (Muennighoff et al. [15]), which uses hyperparameters commonly used for training autoregressive language models. We believe these hyperparameters may slightly favor ARMs, as these were originally tuned for that family.  \n\nImportantly:  \n- Our **best ARM validation loss** in the 100M unique token setting is **3.71** (Figure 2 of our paper), which closely matches the **3.72** reported by Muennighoff et al. (Figure 3 of their paper).  \n- Our **MDM achieves a validation loss of 3.55**, which is unlikely to be matched by ARM even with hyperparameter tweaks.\n\nThat said, we are currently running additional experiments with varied learning rates for ARM to confirm this point and will include results in the discussion/final version.\n\n---\n\n###  **Q3.3: More downstream experiments.**\n\nTo assess whether MDM’s lower validation loss translates to better generalization, we evaluated the **best-performing MDM and ARM models** on a range of language understanding tasks.  Across all tasks and data scales, MDM consistently outperforms ARM.\n\n**Table 2: Accuracy (%) on downstream evaluation tasks at different unique token scales.**\n\n| Benchmark       | Random | ARM (100M) | MDM (100M) | ARM (500M) | MDM (500M) |\n|-----------------|--------|------------|------------|-------------|-------------|\n| ARC-Easy        | 25.00  | 35.63      | **37.84**  | 43.79       | **45.95**   |\n| BoolQ           | 50.00  | 46.00      | **49.38**  | 51.87       | **55.26**   |\n| COPA            | 50.00  | 56.33      | **59.00**  | **67.00**   | 64.83       |\n| HellaSwag       | 25.00  | 27.37      | **30.24**  | 32.28       | **35.33**   |\n| PiQA            | 50.00  | **60.94**   |  60.72  | **65.71**   | 65.61       |\n| RACE            | 25.00  | 25.28      | **28.96**  | 28.28       | **31.44**   |\n| WinoGrande XL   | 50.00  | 48.87      | **50.97**  | 50.61       | **51.51**   |\n| SciQ            | 25.00  | 58.05      | **68.67**  | 67.82       | **79.13**   |\n| Lambda          | 00.00  | 10.91      | **15.19**  | 15.07       | **22.30**   |\n\n*Note: Best results shown in bold.*\n\n---\n\n## **Additional Updates**\nWe have also conducted additional experiments since our submission. We summarize these updates below:\n\n---\n\n### **i) Why is MDM Better in Data-Constrained Settings?**\n\nTo better understand why MDMs outperform ARMs in low-data regimes, we conducted a series of ablations on ARM models to test whether exposing them to more diverse prediction tasks, which is similar to what MDMs inherently do, could close the gap or interpolate between the two methods.\n\nWe hypothesized in the paper:  \n> *“MDM is exposed to diverse token orderings and prediction tasks, unlike the fixed left-to-right training of AR models. This enables them to extract more information per example.”*\n\nTo validate this, we conducted the following experiments:\n\n#### **(a) Random Token Masking in ARM**\n\n| Token Mask Ratio | Val Loss |\n|------------------|----------|\n| 0% (Default)     | **3.71630**  |\n| 12.5%            | 3.72701  |\n| 25%              | 3.75580  |\n| 37.5%            | 3.81153  |\n| 50%              | 3.90185  |\n\nWe observe that token masking in AR degrades validation loss, especially at higher ratios, likely due to over-regularization.\n\n#### **(b) Attention Dropout in ARM**\n\n| Dropout %        | Val Loss |\n|------------------|----------|\n| 10% (Default)    | 3.71630  |\n| 25%              | **3.71268**  |\n| 50%              | 3.72415  |\n| 75%              | 3.76850  |\n\nModerate dropout yields marginal improvements but leads to degradation at higher levels.\n\n#### **(c) Randomizing Token Order (Sigma-GPT [1] / RandAR [2])**\nIn order to show the full impact, including the impact on overfitting points, we used a setting of 100 epochs here. Therefore, the original result for AR is different from the previous two experiments (50 epochs).\n\n| Variant              | Val Loss |\n|----------------------|----------|\n| Original  | 3.74799  |\n| 25% Random Order     | 3.67741  |\n| 50% Random Order     | **3.66556**  |\n\nRandomized orderings show the most promising gains, supporting our hypothesis that MDMs benefit from diverse prediction structures. For this experiment, we implemented RandAR (Any-Order AR) on top of our ARM baseline.\n\n---\n\n### **ii) Scaling to 500M Unique Tokens Using the Critical Compute Equation**\n\nIn Appendix C, we derive an analytical equation that estimates the critical compute at which MDM matches the validation loss of ARM, given a fixed unique data budget. Using this, we scaled the training set to 500M unique tokens and performed a grid search over model sizes and total data based on the predicted compute to identify the configuration predicted by the scaling law to achieve the optimal validation loss as close as possible. Therefore, we trained a 2.3B parameter MDM using the predicted critical compute budget. After training for 130 epochs, we observed that the validation loss of MDM is consistently lower than that of ARM.\n\nFor the ARM baseline, we use the same method to find out optimal setting under the same data budget. We report the empirical validation losses of the trained models below:\n\n**Validation Losses:**\n\n- **MDM:** 3.08573  \n- **ARM:** 3.20136  \n\n---\n\n### **iii) Improved Fitting Metrics (in Appendix)**\n\nWe refit the scaling laws after removing configurations with a small number of training epochs (<20) for MDMs and excluding very small model sizes (<45M parameters) for ARMs. These configurations often exhibited high variance in validation loss due to insufficient optimization, introducing noise into the regression. Excluding them resulted in significantly more stable and interpretable fits.\n\n**Updated Table 1:**\n\n**(a) Initial fit**\n\n| Model | R²     | Loss       |\n|-------|--------|------------|\n| MDM   | 0.9447 | 0.0002     |\n| ARM   | 0.9439 | 7.7532e−05 |\n\n**(b) Second-step fit with extracted scaling parameters**\n\n| Model | R²     | Loss     | R\\*_D  | R\\*_N   |\n|-------|--------|----------|--------|---------|\n| MDM   | 0.9784 | 0.00079  | 493.89 | 1265.65 |\n| ARM   | 0.7628 | 0.00361  | 31.19  | 55.16   |\n\n---\n\n**References:**\n\n[1] Pannatier, Arnaud, Evann Courdier, and François Fleuret. *σ-GPTs: A new approach to autoregressive models.*\n\n[2] Pang, Ziqi, et al. *RandAR: Decoder-only autoregressive visual generation in random orders.*"}, {"review_id": "OznpKCZvIu", "reviewer": "Reviewer_5KGy", "summary": "This paper examines scaling laws for masked diffusion models (MDMs) vs autoregressive models (ARMs) in data constrained regimes, where there are not enough unique tokens to train at “Chincilla” optimal scaling laws. Contrary to previous studies that have found MDMs lag behind ARMs in terms of compute efficiency, in data constrained regimes, MDMs can be trained for longer and attain superior language modeling performance compared to ARMs.", "questions": "1. Can the authors clarify Table 1? What is the difference between Table 1a and Table 1b? Why is the $R^2$ so poor in Table 1b?\n2. In Figure 2, isn’t the perplexity (PPL) / NLL here relatively poor? For example, the reported best NLL corresponds to roughly 30 ppl. Other recent MDM papers are reporting even lower PPL for significantly smaller models (~150M param) on much larger datasets (OpenWebText); see for example Table 2 in MDLM [3]. Why is the final best PPL for the models in these experiments relatively poor?\n3. I believe the authors are missing a scaling factor of $\\frac{r(s) - r(t)}{1 - r(t)}$ for the MDM loss in line 127. Was this used during training / eval to make the loss a “proper” lower bound?\n4. I don’t think I understand lines 128-130 regarding the additional pre-training loss to enable CFG. To the best of my understanding, the CFG from Nie et al. [6] was an inference-time only scheme. That is, the Nie et al. [6] used the same model trained with standard MDM objective and simply masked out the context/prompt to attain the unconditional distribution estimate. Please correct me if I am wrong here.\n5. In Figure 2, is the colorful “Chinchila-optimal” star for 1 epoch of training?\n6. Why are there different model sizes in the legends of Figure 5 for ARM vs MDMs?\n7. In Section 5, line 310, can the authors clarify what is meant by “... showing masking, not attention, is key to their advantage when data is scarce”? Is this meant to say that the bi-directional attention of MDMs is not the key component? If this is the claim, then I would say it is not actually tested. You could validate this by training MDMs with causal attention and random masking?\n\n[6] Nie, Shen, et al. \"Scaling up Masked Diffusion Models on Text.\" arXiv preprint arXiv:2410.18514 (2024).", "limitations": "The main limitation is detailed in the weaknesses above: the “data constrained” regime examined here seems to be potentially overly constrained / small and results may not extrapolate to larger unique token regimes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### **Strengths**\n\nThis paper offers a unique lens on the tradeoffs between ARMs and MDMs, which are gaining in popularity. The experimental design is extensive and the results are overall reported in a useful manner that supports the authors’ main claims. Finding regimes where practitioners can tradeoff the advantages of MDMs vs ARMs is of high significance to the community.\n\n### **Weaknesses**\n\n1. I would have liked to see the discussion of underlying hypotheses that drive the results earlier on in the paper and play a more central part of the experimental design. At the moment, the one hypothesized driver is relegated to a few sentences in the discussion section. I even agree with the authors’ hypothesized explanation that MDMs serve as a form of “data augmentation” / “regularization”. I think this is the correct thinking, but I would have liked to see this hypothesis tested in some way through the experiments.\n2. I find the motivation of running out of internet-scale data to be somewhat removed from the experiments conducted in the paper. The scales examined here of 100M tokens are orders of magnitude lower than the pre-training sizes of even the 8B parameter sized AR models (~30 trillions tokens, e.g. Qwen 3 [1] ) or diffusion models (e.g. Llada [2] at 2.3 trillion tokens).\n3. On a related note, I am a bit concerned about how small the “data constrained” regimes are. For example, 25M unique tokens with 2048 sequence length is ~12k unique samples. This is a **very** small dataset. For example, even the widely used downstream task dataset of GSM8k has 8-9k samples in the training split.\n4. I also believe that the authors should discuss the more recent Llada [2] work that showed a different perspective on scaling laws and rather than focusing on NLL/perplexity showed that for downstream tasks, MDMs have more comparable scaling laws to ARMs.\n5. I disagree with the end of the first paragraph in Section 4.1 (lines 202-205). MDMs **are** more compute inefficient than ARMs. Specifically, for a given batch, ARMs will compute loss on each token. MDMs only compute loss on the masked tokens. In a sense, processing all the unmasked tokens is “wasted compute” from a learning perspective.\n\n### **Other minor comments / suggestions**\n\n1. Throughout, when citing MDLM (Sahoo et al. [3]) I would recommend citing the concurrent works of Shi et al. [4] and Ou et al. [5].\n2. Additionally, there seems to be a repeated entry in the references for Sahoo et al. [3]\n3. (Feel free to ignore this but,) I think clarity of exposition in the intro could be improved by removing the mathematical notation for the joint distribution factorization of MDMs and ARMs, which is described in more detail in Section 3. Additionally, there is no need to include “($R_D^*$)” in line 57; this term is not re-used or explained here.\n4. End of line 267 has a typo: “very less decay”\n\n---\n\n[1] Yang, An, et al. \"Qwen3 technical report.\" arXiv preprint arXiv:2505.09388 (2025).\n\n[2] Nie, Shen, et al. \"Large language diffusion models.\" arXiv preprint arXiv:2502.09992 (2025).\n\n[3] Sahoo, Subham, et al. \"Simple and effective masked diffusion language models.\" Advances in Neural Information Processing Systems 37 (2024): 130136-130184.\n\n[4] Shi, Jiaxin, et al. \"Simplified and generalized masked diffusion for discrete data.\" Advances in neural information processing systems 37 (2024): 103131-103167.\n\n[5] Ou, Jingyang, et al. \"Your absorbing discrete diffusion secretly models the conditional distributions of clean data.\" arXiv preprint arXiv:2406.03736 (2024).", "was_revised": true, "final_justification": "The authors clarified my concerns and provided extensive additional experimental results", "rebuttal": "Thank you for your constructive feedback and for carefully reviewing our work.\n\nBelow, we address each of your points in turn:\n\n---\n\n### **Q4.1: Further experiments validating the hypothesis on *why* MDMs are more data-efficient.**\n\nTo better understand why MDMs outperform ARMs in low-data regimes, we conducted a series of ablations on ARM models to test whether exposing them to more diverse prediction tasks, which is similar to what MDMs inherently do, could close the gap or interpolate between the two methods.\n\nWe hypothesized in the paper:  \n> *“MDM is exposed to diverse token orderings and prediction tasks, unlike the fixed left-to-right training of AR models. This enables them to extract more information per example.”*\n\nTo validate this, we conducted the following experiments:\n\n#### **(a) Random Token Masking in ARM**\n\n| Token Mask Ratio | Val Loss |\n|------------------|----------|\n| 0% (Default)     | **3.71630**  |\n| 12.5%            | 3.72701  |\n| 25%              | 3.75580  |\n| 37.5%            | 3.81153  |\n| 50%              | 3.90185  |\n\nToken masking generally degrades validation loss, likely due to over-regularization.\n\n#### **(b) Attention Dropout in ARM**\n\n| Dropout %        | Val Loss |\n|------------------|----------|\n| 10% (Default)    | 3.71630  |\n| 25%              | **3.71268**  |\n| 50%              | 3.72415  |\n| 75%              | 3.76850  |\n\nModerate dropout provides marginal gains but degrades performance at higher levels.\n\n#### **(c) Randomizing Token Order (Sigma-GPT [1] / RandAR [2])**\nIn order to show the full impact, including the impact on overfitting points, we used a setting of 100 epochs here. Therefore, the original result for AR is different from the previous two experiments (50 epochs).\n\n| Variant              | Val Loss |\n|----------------------|----------|\n| Original | 3.74799  |\n| 25% Random Order     | 3.67741  |\n| 50% Random Order     | **3.66556**  |\n\nRandomized token ordering shows the most promising improvement, supporting our hypothesis that MDMs benefit from diverse prediction tasks. We implemented RandAR (Any-Order AR) on top of our ARM baseline for this experiment.\n\n---\n\n### **Q4.2: Motivation of internet-scale data is not aligned with small-scale experiments.**\n\nWe agree that current LLMs are trained on trillions of tokens, which is larger than the data regimes we study. However, we believe our results remain meaningful:  \n- Our experiments demonstrate that objectives inducing randomized ordering (MDM) are more data-efficient.  \n- This insight can motivate future hybrid approaches, where diffusion-like objectives are applied on scarce high-quality data, while AR is applied on abundant lower-quality data.  \n- Domains with inherently limited data (e.g., robotics and healthcare) would particularly benefit from MDM-like data efficiency.\n\nSince submission, we have scaled our experiments to **500M unique tokens** using our critical compute scaling law (Appendix C), We train:\n\n- 2.3B MDM trained for 130 epochs (terminated due to compute limits)  \n- Best ARM configuration via grid search for the same data budget  \n\nWe report the results below: \n\n**Validation Losses:**\n\n- **MDM:** 3.08573  \n- **ARM:** 3.20136  \n\n---\n\n### **Q4.3: Authors should discuss Llada [2], which shows more comparable scaling laws using downstream metrics.**\n\nThank you for pointing this out. We will include a discussion of Llada [2] in the final version. Our scaling law analysis focused on validation loss, which is standard in scaling law studies and provides a stable signal.  \n\nTo complement this, we conducted additional **downstream evaluations**, which confirm that MDM’s lower validation loss translates to better generalization:\n\n**Table 2: Accuracy (%) on downstream evaluation tasks at different unique token scales.**\n\n| Benchmark       | Random | ARM (100M) | MDM (100M) | ARM (500M) | MDM (500M) |\n|-----------------|--------|------------|------------|-------------|-------------|\n| ARC-Easy        | 25.00  | 35.63      | **37.84**  | 43.79       | **45.95**   |\n| BoolQ           | 50.00  | 46.00      | **49.38**  | 51.87       | **55.26**   |\n| COPA            | 50.00  | 56.33      | **59.00**  | **67.00**   | 64.83       |\n| HellaSwag       | 25.00  | 27.37      | **30.24**  | 32.28       | **35.33**   |\n| PiQA            | 50.00  | **60.94**      | 60.72  | **65.71**   | 65.61       |\n| RACE            | 25.00  | 25.28      | **28.96**  | 28.28       | **31.44**   |\n| WinoGrande XL   | 50.00  | 48.87      | **50.97**  | 50.61       | **51.51**   |\n| SciQ            | 25.00  | 58.05      | **68.67**  | 67.82       | **79.13**   |\n| Lambda          | 00.00  | 10.91      | **15.19**  | 15.07       | **22.30**   |\n\n*Note: Best results shown in bold.*\n\n---\n\n### **Q4.4: Disagreement with lines 202–205 claiming MDMs are more compute-inefficient than ARMs.**\n\nWe appreciate the clarification. We **agree** that MDM computes loss on only the masked tokens, which provides less direct supervision per update and higher gradient variance than ARM, where every token is supervised.  \n\nHowever, we **disagree** that processing unmasked tokens is “wasted compute”:  \n1. Unmasked tokens contribute to the loss indirectly via gradient propagation.  \n2. Unmasked inputs act as registers, gathering and passing information that improves predictions at masked positions.\n\nOur conclusion that MDMs are more compute-inefficient is empirically grounded: achieving the same validation loss as ARM requires more FLOPs as a result more GPU-hours, as is also shown in prior works [16, 24].  .  \nFuture work may mitigate this, e.g., Arriola et al. [1] show that tuning the masking schedule can reduce gradient variance and improve compute efficiency. We will add further clarification in the final version of the paper on this.\n\n---\n### **Q4.5: Other Minor Comments /Suggestions**\n\nThank you for these helpful suggestions. We will incorporate the fixes in the final version of the paper\n\n---\n\n### **Q4.6: Difference between Table 1a and Table 1b? Why is the R² so poor in Table 1b?**\n\n- **Table 1a** represents scaling law fitting in the **single-epoch setting** (Chinchilla-style compute/data scaling).  \n- **Table 1b** represents scaling law fitting in the **multi-epoch, data-constrained setting** (Muennighoff et al. [15]) and estimates the constants \\(R_D^*\\) and \\(R_N^*\\).  \n\nThe initial submission showed lower R² in Table 1b due to the inclusion of:  \n1. Configurations with <20 epochs for MDMs  \n2. ARM models with <45M parameters  \n\nThese configurations produced high-variance validation loss, making regression noisier.  \n\nSince submission, we have refit the scaling laws, excluding these high-variance points, resulting in more stable and interpretable fits:\n\n**Updated Table 1:**\n\n**(a) Initial fit**\n\n| Model | R²     | Loss       |\n|-------|--------|------------|\n| MDM   | 0.9447 | 0.0002     |\n| ARM   | 0.9439 | 7.7532e−05 |\n\n**(b) Second-step fit with extracted scaling parameters**\n\n| Model | R²     | Loss     | R\\*_D  | R\\*_N   |\n|-------|--------|----------|--------|---------|\n| MDM   | 0.9784 | 0.00079  | 493.89 | 1265.65 |\n| ARM   | 0.7628 | 0.00361  | 31.19  | 55.16   |\n\n---\n\n### **Q4.7: In Figure 2, isn’t the NLL relatively poor compared to Table 2 in MDLM [3]?**\n\nThe comparison is not directly fair because:  \n- MDLM Table 2 is trained on **262B unique tokens**, while our experiments use **100M unique tokens**.  \n- If we assume 100 epochs of repetition is the same as unique tokens ≈ 10B effective tokens, this is still **26× smaller** than MDLM.  \n\nFurther we would like to point, our best ARM validation loss (3.71) in the 100M regime (Figure 2) closely matches 3.72 reported by Muennighoff et al. for the same scale (Figure 3 of their paper).  \n\n---\n\n### **Q4.8: Authors are missing a scaling factor for the MDM loss.**\n\nThank you for pointing it out. Yes, this is a typo and will be fixed in the final version. \n\n---\n\n### **Q4.9:  There is no additional pre-training loss to enable CFG in Nie et al. [6]?**\n\nCorrect. We acknowledge this and will remove the line in the final version. Thanks for pointing it out!\n\n---\n\n### **Q4.10: In Figure 2, is the colorful “Chinchilla-optimal” star for 1 epoch of training?**\n\nYes, the star corresponds to 1 epoch of training under Chinchilla-optimal scaling.\n\n---\n\n### **Q4.11: Why are there different model sizes in the legends of Figure 5 for ARM vs. MDM?**\n\nWe fixed the training compute to \\(1 \\times 10^{19}\\), \\(3 \\times 10^{19}\\), and \\(1 \\times 10^{20}\\) FLOPs.  \nWe then used the compute-optimal scaling law to determine model sizes for each family:  \n- For \\(1 \\times 10^{19}\\) FLOPs → ARM ≈ 217M, MDM ≈ 117M\n\nHence, model sizes differ because compute-optimal configurations are different for ARMs vs. MDMs.\n\n---\n\n### **Q4.12: What does “showing masking, not attention, is key to their advantage when data is scarce” mean?**\n\nWe agree that this sentence was unclear and will remove it in the final version.  \nOur updated experiments (see Q4.1) clarify that **randomized token orderings** are the main driver of MDM’s data efficiency.\n\n\n\n\n\n**References:**\n\n[1] Arriola, E., et al. *Improving compute efficiency of diffusion models via masking schedules.*  \n[2] Llada, J., et al. *Downstream scaling laws for generative models.*"}]

Author rebuttal:

Meta-review: This paper investigates scaling laws for masked diffusion models (MDMs) versus autoregressive models (ARMs) in data-constrained settings, demonstrating that MDMs consistently outperform ARMs when training on limited unique data across multiple epochs. The key scientific finding is that MDMs achieve superior validation loss and downstream task performance when unique token budgets are constrained (25M-500M tokens), with the authors attributing this advantage to MDMs' exposure to diverse token orderings during training. The paper's strengths include comprehensive empirical evaluation across multiple model sizes and data scales, novel identification of a regime where diffusion models are preferable to autoregressive models, and derivation of analytical scaling laws with critical compute thresholds. However, the work has several concerning weaknesses: limited technical novelty, questions about the practical relevance of small-scale "data-constrained" regimes, and more critically, reviewers raised concerns about mathematical rigor in the loss formulations and whether the comparison methodology between AR and diffusion models is fundamentally sound. The validity of using validation loss as the primary comparison metric when the two model types may be measuring fundamentally different quantities (exact likelihood vs. upper bounds) remains questionable and could undermine the paper's central claims.

The rebuttal process addressed many reviewer concerns through extensive additional experiments, though some fundamental methodological questions persist. Reviewer u7Lw raised issues about mathematical formulation rigor, which authors acknowledged by promising to clarify the factorizations, leading to a score maintenance at 4. Reviewer 43Rm's requests for dataset generalization and optimizer sensitivity were addressed with SlimPajama experiments and hyperparameter sweeps. Reviewer 5KGy's concerns about loss formulation details (missing scaling factors, proper lower bounds) were acknowledged as errors to be fixed, and their score increased to 5 after seeing downstream task results. Reviewer WHBh maintained strong support (rating 5) throughout. The paper makes an interesting empirical contribution, but the fundamental validity of the comparison methodology remains a concern and need to be addressed in the future version.
Decision: Accept (poster)