Dual-objective Language Models: Training Efficiency Without Overfitting
Authors: David Samuel
Lucas Georges Gabriel Charpentier
Venue: ICLR 2026 Poster (2026)
Keywords: language model
pretraining
training objective
mixed training objective
masked diffusion
Primary area: foundation or frontier models, including LLMs
TL;DR: We simultaneously train language models on autoregressive and masked-diffusion objectives, resulting in flexible models that outperform the single-objective models in both settings.

Abstract: This paper combines autoregressive and masked-diffusion training objectives without any architectural modifications, resulting in flexible language models that outperform single-objective models. Autoregressive modeling has been a popular approach, partly because of its training efficiency; however, that comes at the cost of sensitivity to overfitting. On the other hand, masked-diffusion models are less efficient to train while being more resilient to overfitting. In this work, we demonstrate that dual-objective training achieves the best of both worlds. To derive the optimal balance between both objectives, we train and evaluate 50 language models under varying levels of data repetition. We show that it is optimal to combine both objectives under all evaluated settings and that the optimal balance is similar whether targeting autoregressive or masked-diffusion downstream performance.

Published as a conference paper at ICLR 2026
DUAL-OBIECTIVE LANGUAGE MODELS: TRAINING EFFICIENCY WITHOUT OVERFITTING
David Samuel' University of Oslo davisamu@uio.m no
Lucas Georges Gabriel Charpentier" National Library Norway lucas. chwrsectieepnnndee
ABSTRACT
This paper combines autoregressive and masked-diffusion training objectives with- any architectural modifications, resulting in flexible language models that outperform single-objective models. Autoregressive modeling has been popular approach, partly because of its training efficiency; however, that comes at the cost of sensitivity to overfitting, On the other hand, masked-diffusion models are less efficient train while being more resilient to overfitting. In this work, we demon- strate that -ual-objective training achieves the best of both worlds. To derive the optimal balance between both objectives, we train and evaluate 50 language models under varying levels of data repetition. We show that it is optimal to combine both objectives under all evaluated settings and that the optimal balance is similar whether targeting autoregressive or
downstream performance.
INTRODUCTION
The dominant paradigm for training recent language models has been autoregressive next-token prediction (Brown al., 2020). This approach is remarkably efficient in training, allowing models to quickly absorb vast amounts of text. However, this comes with significant drawback: tendency to overfit when training data is repeated (Muennighoff' et al., 2023). This issue is becoming increasingly critical as the community reaches the so-called data wall - the imminent exhaustion of available data required to train ever-larger models according to established scaling laws (Villalobos et al., 2024).
An alternative approach, muakeed-diffustio language modeling, offers compelling solution to the overfitting problem (Prabhudesai et al., 2025; Ni et al., 2025). Yet. this robustness comes at the cost these models are known to be much less sample-efficient than their autoregressive counterparts (Nie et al., 2025a). The complementary strengths of these two approaches suggest combining them as natural solution counteracting their failure modes.
DUAL OBJECTIVE
AUTOREGRESSIVE
MASKED DIFFUSION
Average poro-toot-eoe score
10
5
4B
8B
12B
16B
20B
24B
28B
32B
Training progress (number of tokens)
Figure 1: The dynamics of zero-shot performance. The three models are trained in a rather extreme setting 128 repetitions of the training corpus. The autoregressive objective (blue line) converges the fastest but also very quickly overfits; the maaked-diffusiie objective (red line) converges slowly but without being negatively affected by the high amount of repetitions. Combining both objectives together (purple line) results in fast convergence as well robustness to overfitting.
"Equal contribution. Work done at the Language Technology Group, University of Oslo.

---

Published as a conference paper at ICLR 2026
In this work, we show that it is possible achieve the best of both worlds by simultaneously training a single language model on both autoregressive and
objectives. The core idea is to use the training efficiency of the autoregressive objective for rapid initial learning while using the masked-diffusion objective to regularize the model and prevent from overfitting. The effectiveness of this dual-objective approach is illustrated in Figure In the extreme data-constrainee setting with 128 data repetitions, the purely autoregressive model learns quickly but then catastrophically overfits. The masked-diffusion model is immune to overfitting but converges very slowly. Our proposed dual-objective model combines the strengths of both and successfully leverages the given compute and data. The resulting models can be deployed as a standard autoregressive models with no inference overhead.
Building on this observation, we conduct a large-scale systematic study to find the optimal balance between these two objectives under varying degrees of data constraints. Our main contributions are:
We propose a dual-objective training method that combines autoregressive and losses, enabling single model to excel at both unidirectional and bidirectional tasks.
Through an extensive empirical study, we systematically map the relationship between data repetition, the ratio of training objectives, and final downstream performance. Demonstrating that our dual-objective approach is superior to single-objective training in all evaluated settings, for both autoregressive and masked-diffusion evaluation including the finding that dual-objective models outperform pure masked-diffusion models even in regular data settings.
We derive two practical recommendations for setting the optimal objective ratio when training in both regular and data-constrained regimes, providing concrete guidelines for future training of large language models.
2 BACKGROUND
Before diving into details of combining autoregressive and masked-diffusion models, we need to briefly describe those two modeling approaches and language modeling in general. As the name suggests, language models are statistical models (-(-) of the true language distribution of some training corpus D. The training corpus consists of sequences x (T1,42, ... ... D of subword tokens. The language models are trained by finding such parameters @ that maximize the likelihood estimation (MLE; Fisher, 1922; 1925):
argmax E
~
In this paper, we combine two popular approaches for computing po('), autoregressive language modeling and masked-diffusion language modeling.
2.1 AUTOREGRESSIVE LANGUAGE MODELING
Language models have long tradition and since their inception in the seminal paper by Shannon (1951), they have been factored into a chain of next-token prediction terms (ar; mm:e)::
logge pe (x)
log Po
CaR (x; 0).
1=1
Computation of the next-token likelihoods can be efficiently parallelized when modeled by trans- former networks (Vaswani et al., 2017), and thanks to their scalability, it has been the most popular paradigm behind the recent era of large language models (Brown et al., 2020).
2.2 MASKED-DIFFUSIIE LANGUAGE MODELING
Masked-diffusion language models have recently become popular alternative to autoregressive models (Austin et al., 2021; Lou et al., 2024; Sahoo et al., 2025; Ou et al., 2025; Nie et al., 2025b). Computing (-(-) with masked-diffusion is slightly more complicated than with autoregression. but the resulting language model learns to handle full bidirectional context, which can lead to increased performance on downstream tasks (Berglund et al., 2024; Samuel, 2025).
2

---

Published as a conference paper at ICLR 2026
First, following Austin al. (2021), we define the forward (and backward) diffusion process that gradually turns a sequence of tokens a into special mask tokens (and vice-versa). The diffusion process {æ'} depends on the time variable E [0. 1] so that xx and a< is fully masked sequence, The intermediate values are defined by the probability distribution q:
L
x Dis
Cuooaa az); where (u x.))
T mask.
into mask token with probability The forward process is fully reversible and we can accordingly define the backward process, which gradually unmasks sequence (Austin al., 2021). Using the results from Ou al. (2025), the probability distribution (0)x(T; æ²) governing the backward process can be modeled with iime-independent transformer language model with parameters O as po (m: æ²). This model can be fitted to the training data by minimizing the upper bound on the negative oog-likelhhodd estimate (Ou et al., 2025):
We see that each token can either remain unchanged
log po 1
E (ejo
Σ {ili] mask)
log (a
x²) dt LMD 0)
The integral can be equivalently written as the expectation over U(, 1), thus, it can be directly used training objective when estimated by Monte-Carte sampling (Metropolis & Ulam, 1949). Such a Monte-Carlo estimate can also be used inference-time for likelihood-baa based evaluation, similarly to Equation (2). Note that the resulting objective is very similar to the one used to train masked language models such as BERT (Devlin et al., 2019).
DUAL-OBJECTIVVE LANGUAGE MODELING
The method of combining autoregressive and masked (diffusion) objectives is mostly based on the
earlier GPT-BERT approach by Charpentier & Samuel (2024). They showed promising results for very small language models trained within the limitations of the BabyLM Challenge (Hu al., 2024). We extend their approach to masked-diffusee language models and to orders of magnitude larger computation scale.
Dujl-objective language models are trained by minimizing the following combined loss function, which is further explained below more detail:
JaLAR (2;; 0) + (1 LMD 0).
argmin
AUTOREGRESSIVE MODE
DIFFUSION MODE
next-token prediction with causal attention
next-token prediction with fill bidirectional attention
p(From xx.) pileft x.) p(to (x,) p(rightt (x.) pl. xxa)
pl Excmpple (x²)
p(masked (x²)
DUAL LANGUAGE MODEL
a single transformer-basee model for both parudigms
lefi
right
mask
mask
Figure Two modes of operation inside a single model. We use the same transformer architecture with the same parameters to do both diffusion and autoregression language modeling, the only difference between the two modes is the input sequence and the attention mask.
3

---

Published as a conference paper at ICLR 2026
Loss weighting The balance between the autoregressive objective CAR and objective LMD is controlled by the hyperparameter a. It is crucial for controlling the trade-off between training efficiency and overfitting robustness; its relation to the number of data repetitions is extensively tested by the following experiments.
In practice, naively mixing both objectives within single batch could result in reduced throughput. For this reason, we assign each GPU device to a single objective so that the computation graph remains simple and static, and can be efficiently compiled. To be specific, we distribute the training of each model across 256 devices, which allows for choosing between 256 + 1 values: a E {1/250 | 2566).
Diffusion as next-token prediction Our goal is to align LAR and LMD so that they can be parame- a slightly modified version of masked
terized by single transformer model. For this reason, language modeling called masked next-token prediction (MNTP: Lv et al., 2024). With this approach, the model always uses the hidden state at position to predict the next token position + 1 (we prove that this parameterization is as expressive as the standard approach in Appendix F). In this way, both modes of operation are unified they both perform next-token prediction, as illustrated in Figure 2. MNTP has also been used in recent work for adapting masked diffusion model from an autoregressive checkpoint (Gong et al., 2025; et al., 2025).
Standard transformer architecture The main benefits of using masked next-token prediction are that we can use exactly the same transformer architecture as standard autoregressive models, and we can optimize its parameters with both objectives at the same time. The only difference between the two modes of operation the inputs - they are either (partially) masked inputs with empty (fully bidirectional) attention masks, or full unchanged inputs with causal (unidirectional) attention masks.
4 EVALUATION
While it is common practice to only consider the value of loss on a held-out set when evaluating language models (Kaplan al., 2020: Hoffmann et al., 2022: Muennighoff et al., 2023), it important to measure the actual downstream performance to accurately assess the effect of different training configurations. This especially crucial when training with two incompatible training losses. That being said, we also report validation losses in Appendix G.
We evaluate our models on nine standard language modeling tasks in a zero-shot fashion. All tasks consist of context (which can be empty) and multiple different completions where one is correct and the others are incorrect. We evaluate the sum of the log- likelihood of each completion and assign the completion with the maximum sum as the prediction of the model. Table I lists the tasks:
Tasks
Table 1: The list of evaluation tasks. The ARCI datasets contain some examples with 3 completions rather than 4. All tasks are evaluated zero-shot.
Task
# Examples # Completions
Split
Reference
ARC-Easy (ARC-E)
2376
4
test
Clark et al. (2018)
ARC-Challenge (ARC-C)
1 172
1t
test
Clark et al. (2018)
BLiMP
67 000
Warstadt et al. (2020)
Commonsense QA (CSQA)
1221
val
Talmor et al. (2019)
HellaSwag (HSwag)
10042
val
Zellers et al. (2019)
MMLU
14042
test
Hendrycks et al. (2021)
OpenBook QA (OBQA)
500
4 test
Mihaylov et al. (2018)
Physical Interaction QA (PIQA)
1 1838
val 2
Bisk et al. (2020)
Social IQa (SIQA)
1954
val
Sap et al. (2019)
Evaluation setup We follow the guidelines of the OLMES paper (Gu et al., 2025) for the normal- ization of our log- likeliikoood estimations as well as the prompt format, with two changes: 1) we only evaluate in zero-shot fashion to simplify the setup. 2) only consider their "cloze" formulation of each task, which is more suitable for smaller models. For the BLiMP task, which is not considered in the OL OLMES evaluation suite, do not apply any length normalization and score each sample with
4

---

Published as a conference paper at ICLR 2026
the raw log-likelihood score. Since the BLiMP and MMLU tasks contain multiple sub-tasks (67 for BLiMP, and 57 for MMLU), we report their how each task is normalized can be found in Appendix C.
as the final score. More information on
Normalized score averaging To ensure fair aggregation of the different task scores, we first normalize the scores such that the random baseline of each task is and the maximum is at 1; similarly to the Open LLM Leaderboard (Fourrier et al., 2024). To achieve this we apply the following formula our scores: score(x,1 t) (* (mm where is the raw score, 7r is the random baseline and m, is the optimal score for task t. We then take the simple average of the normalized scores across all tasks as the final performance of our model.
4.1 AUTOREGRESSIVE (UNIDIRECTIONAL) EVALUATION
To evaluate the autoregressive capabilities of our models, we use Equation (2) to estimate the log- likelihood of each completion. Specifically, given a completion (w) and context (c), we calculate the conditional log-likelihood as log (w c) Σ: log po (w)) C, wex)
4.2 MASKED-DIFFUSION (BIDIRECTIONAL) EVALUATION
One possibility evaluate the maakedddiiffusttn capabilities of our models is to also leverage the training objective in Equation (4) and estimate the conditional log- of each completion by While provides
Monte-Carlo sampling. We describe this approach more detail Appendix accurate downstream scores, it is computationally expensive and less accurate than using simpler pseudo tog-likelihood (PLL; Wang & Cho, 2019; Salazar et al., 2020; Samuel, 2025) estimation.
PLL allows us do bidirectional evaluation more than ten times faster while being more accurate than Monte-Carlo sampling (Appendix J). Therefore, we use PLL for evaluating the bidirectional capability of our models. We fully describe this method in Appendix E. As visualized in Figure on the left, we specifically use the semi-autonn rumii-auoreeressivee variation of PLL proposed by Samuel (2025).
PSEUDO DIGLLIKEIHHODD (WITH MASKS)
MONTE-CARLO ESTIMATE (OG-LIKELHHOOD
log Pe
anc
<2>
Fask
mask
Sieer
fiee
xii
log P9
ent
Fask
Aec
Aéroe
fuur
log Pe
log Pe
meak
log.e
xir
<
car
Awo
Abbee
Jieer
jiev
mack
SW
log
meo
<E>
maxi
FAaK
mack
MARk
Figure 3: Visual representations of bidirectional evaluation methods. Pseudo log-likelihodd estimation (on the left) reaches accurate likelihood scores substantially faster than the (theoretically grounded) Monte-Carlo estimation (on the right).
5 EXPERIMENTS
5.1
PRETRAINING SETUP
We train each 170-million-parameter language model (with 360M non-embedding weights) on 32 billion tokens in total. A repetition factor R means we sample unique subset of size 328/R tokens and repeat it R times during training. This total token budget is more than 4x past the Chinchilla compute-optimat point (Hoffmann et al., 2022); we specifically decided to conduct the experiments in this regime as it reflects how modern language models are trained in practice. This compute budget also large enough induce non-trivial zero-shot downstream performance, enabling us to measure clear differences between different configurations.
Model architecture The language models have 24 layers with hidden size of 1 024, their self- attention operations are divided into 16 parallel heads, the feed-forward modules have intermediate size of 3 554, and the vocabulary is set to 51 200 tokens. As for the architecture itself, we follow the usual modifications of the original transformer recipe (Vaswani et al., 2017) pre-normalization
5

---

Published as a conference paper at ICLR 2026
(Nguyen & Salazar, 2019) with RMSNorm (Zhang & Senmrich, 2019), rotational positional embed- ding (Su et al., 2024) and Swish-gated linear units (Ramachandran et al., 2018; Shazeer, 2020).
Optimization The parameters are optimized by the Muon optimizer for faster convergence (Jordan et al., 2024), specifically its variation proposed by Liu et al. (2025). The learning rate is set to 0,007 and decayed according to the carmup-stable-decay (WSD; Hägele et al., 2024) schedule (without warmup steps and 2 048 steps of linear decay). In total, each model is trained for 8 192 steps with 4M tokens each global batch and with a sequence length of 048 tokens. The optimization is regularized by weight decay (with strength of 10 and by an auxiliary z-loss term (with strength of 10 10-+ Chowdhery et al., 2022).
Training corpus and tokenizer Even though we limit the training data to 32B tokens, we delib- erately choose a text corpus that is not excessively filtered and that is representative of large-scale web crawls used in practice. We randomly sample English documents with 32B tokens in total from the HPLT v2 corpus (Burchell et al., 2025), which combines extracted webpages from the Internet Archive and CommonCrawl. We also use a smaller disjoint subset to monitor the validation loss. To prevent potential bias from using an external tokenizer, we train a standard byte-level BPE tokenizer (Gage, 1994) with 51 200 subwords directly on the full training data.
5.2 SEARCHING FOR THE OPTIMAL
We trained and evaluated 50 language models in total, the results are plotted in Figure In order to deal with the noisy nature of this data and to better understand the relation between the amount of data repetitions and the optimal a, we use simple statistical models.
Interpolation with Gaussian process Specifically, we use Gaussian process regression (GPR; Williams & Rasmussen, 1995) with a composite kernel structure to model the relationship between data repetitions. a and downstream performance. The composite kernel consists of constant kernel multiplied by an anisotropic Matérn kernel (u = 1.5; Stein, 1999) combined additively with a white noise kernel to account for observation noise. The input features are standardized to zero mean and unit variance, and the output features are normalized. The kernel parameters are optimized by L-BFGS-B (Liu & Nocedal, 1989) using SciPy (Virtanen et al., 2020). The resulting interpolations in Figure 4 show regular structure while closely fitting the data with R over 0.99 in all cases.
The optimal autoregresive-ddiffsiion ratios The fitted Gaussian process is a probabilistic model of the downstream performance with regard to the amount of data repetition and a. Thus. we can transform this to the probability that a particular a is optimal for the given data repetition. More concretely, we can estimate the density of this distribution by sampling from the posterior of the GPR model. The result of this is visualized in the bottom part of Figure 4.
5.3 RESULTS AND DISCUSSION
The structure of Figure 4 becomes clearer once we identify which training settings result in overfitting during training.' The density of optimal a weights highlights that there are two regions to consider: 1) Regular-data region where a language model trained solely on the autoregressive objective does not overfit this roughly corresponds to 16 repetitions of training data and less, as also shown by Muennighoff et al. (2023). 2) conssrrontr region - roughly corresponding to 32 data repetitions and more- where overfitting is an important consideration.
In the first case, it is clearly beneficial to put more weight on the autoregressive training than on -iffee--iffusiin. Yet, training only autoregressively does not lead to any improvement in any experiments within the regular-data region. Even when evaluated purely autoregressively. the differences between a set to and 15/16 are negligible. Switching to bidirectional evaluation, the single-objective a 1 performs poorly while all models trained with a values between 255/256 and 15/16 perform similarly notably, they all substantially outperform models trained only with masked- diffusion. This is a key finding: even without any data constraint, the dual-objective models achieve stronger masked-ding masked-diffusion performance than pure masked-diffusion training. despite dedicating only a small fraction of training to the masked-diffusia objective. We hypothesize that the prevalence of
Here, overfitted training runs are those runs, in which the held-out loss starts diverging while the training loss keeps converging (Appendix G). Such runs are highlighted in Figure by x marks.

---

Published as a conference paper at ICLR 2026
(A) UNIDIRECTIONAL PERFORMANCE
(B) BIDIRECTIONAL PERFORMANCE
(AR)
83
X
83
255
23
X
22
1/16
3333
23
2/4
1/2
4/4
1/8
230
1/64
1/266
0
(MD)
128x
(C) DENSITY OF OPTIMAL (UNIDIRECTIONAL) (D) DENSITY OF OPTIMAL (BIDTRECTIONAL)
256
<3/6
#
15/18
3/4
1/2
1/4
1/16
1/64
1/2s
(MD)
1x
2x
4x
8x
16x
32x
64x
128x
256x
1x
2xc
4x
8x
16x
32x
64x
128x
256x
Number of repetitions
Numbber of repetitions
Figure 4: Interpolated unidirectional and bidirectional results. The (a) and (b) figures on top show the relation between repetitions (x-axis) and the nutoregressive-diffssi weight a (y-axis); the contours follow the Gaussian process model that interpolates the average performance of language models trained according the specified settings. The respective results are plotted either as crosses when the model overfitted during training, as circles. The (c) and (d) figures below visualize the estimated probability that particular a (y-axis) is optimal for a given number of repetitions (x-axis).
the autoregressive objective leads to fast convergence, and that the small amount of masked-diffusion balances its slower convergence by inducing useful modeling priors. This leads us to formulating the first practical recommendation:
Remark 1
Language modeling under regular data settings
When training language model in regular data setting (16 repetitions less), train with small amount of masked-diffuse d-iffusioon objective (a z 63/64) to achieve stronger bidirectional performance than pure masked-diffusee training without losing any autoregressive performance.
In the second data-constrained case, the relation between data repetition, and final performance seems more complicated. We risk overfitting by putting too much weight on autoregression and underfitting by focusing too much optimal a values is fairly narrow. On the other hand, the optimal values are surprisingly similar for the unidirectional and bidirectional performance. We can notice that the region of optimal values
masked-diffussion
evident from Figure the interval of
7

---

Published as a conference paper at ICLR 2026
is right beneath the region of values that lead overfitting, but the question is how to identify such an a. It is possible to have an alternative interpretation of the autoregressive--iffine and count the number of data repetitions that each objective is individually trained then can see that more than 32 autoregressive repetitions lead to overfitting while fewer than autoregressive repetitions lead to underfitting. Thus, based the empirical results, recommendation for this scenario is:
weights
Remark 2
Data-constrained language modeling
When training language model in a data-constrained setting (more than 32 repetitions), choose that exposes the autoregressive objective roughly 16 repetitions of the training data.
Table 2: The normalized autoregressive performance of selected models. We show the results all nine evaluated tasks for three repetition values; each repetition group contains the results of the best-perfomingg and of the autoregressive-onf| model. The scores for each task normalized so that 0% corresponds to random baseline and 100% is the perfect score. The best result for each dataset size is boldfaced.
ARC-C
ARC-E
BLIMP
CSQA
HSwag
MMLU
OBQA
PIQA
STON
Model configuration
Average
REPETITION
Dual (a 63/64)
5.7
28.6
63.7
35.1
31.1
17.6
40.9
14.3
Autoregressive (a = 1)
5.9
30.3
61.3
33.5
31.7
3.8
13.6
39.4
15.2
26.1
32 REPETITIONS
Dual (a 6/4)
3.3
28.0
57.9
31.1
26.4
3.6
14.4
36.1
14.6
23.9
Autoregressive (a 1)
5.0
24.9
53.3
28.5
25,4
3.8
9.9
33.3
14.2
22.0
128 REPETITIONS
Dual (a = 1/8)
1.7
23.6
56.1
24.8
14.2
1.6
8.5
28.1
13.3
19.1
Autoregressive (a
-1.0
12.3
33.2
6.8
8.1
1.1
-0.5
15.8
8.9
9.4
Generalization to larger language models An obvious question is whether the recommendations hold even at much bigger scale for larger language models. Reliably answering this question would require expensive experimentation, but we believe that the conclusions hold for two reasons. Firstly, according to our results, the optimal a values clearly correlated with overfitting of autoregressive language models. Since the overfitting behavior does not depend on model size according to previous work (Muennighoff et al., 2023; Prabhudesai al., 2025), we believe that the optimal values should also not change. Secondly, the relative burden of representing two modes of operation within the learned parameters decreases with model size, so we believe that the benefit of the dual training objective should actually increase with model size.
To put the abstract average scores into another perspective, we look at the individ- ual (normalized) scores per task in Table 2. The results show that the improvement in performance from using a dual objective is observed on a majority of tasks. This is especially true the more repetitions there are. The detailed scores also highlight how effectively the dual objective learns from limited data, reaching nontrivial performance even when exposed to just 256M tokens of training data (under 128 repetitions). We observe similar trends for masked-diffun as the number of repetitions decreases, the performance gap increases rather than decreases. Detailed performance the masked-diffusion evaluation can be found in Appendix
Detailed results
evaluation except that
5.4 GENERALIZATION TO PREFIX LANGUAGE MODELING
Prefix language modeling (Dong et al., 2019 Raffel et al., 2020; Wang et al., 2022) is promising alternative to the two training objectives investigated in this work. It processes the conditioning part (prefix, c in notation from Section 4. of a text fully bidirectionally while the completion part (w in Section 4.1) 1) is processed autoregressively. Given that models trained with both unidirectional
8

---

Published as a conference paper at ICLR 2026
and bidirectional attention, test whether the exposure both can induce generalization to prefix language modeling without any further training. We repeat the earlier autoregressive evaluation with prefix attention masks and plot the results in Figure 5.
AUTOREGRESSIVE
(AR)
8
X
33
40
43/c4
83
8
3/4
1/2
83
X
1/a
22
1/41
1/26
0
(MD)
2x
4x
16x
128x
256x
1x
2x
128x
256x
Number of reperitions
Number of reperitions
Figure 5: Interpolated prefix results. The figures show the relation between data repetitions (x-axis), (y-axis), and downstream performance (color-coded), The individual results are interpolated by GPR model. The right figure demonstrates the relative improvement of prefix-masked evaluation compared to fully unidirectional evaluation (blue color denotes decreased performance and red color denotes performance increase).
The right side of Figure shows the overall improvement of the prefix evaluation over the autore- gressive one. Notably, we can that it is reliably over percentage point better across most configurations that combine both training objectives. This finding leads to our third recommendation:
Remark 3
Induced prefix language modeling
The autoregressive performance of dujl-oojective language models can be reliably improved inference time by processing the conditional part of a prompt fully bidirectionally.
6 RELATED WORK
Combining autoregressive and masked (diffusion) language modeling This paper builds upon the GPT-BERT training objective by Charpentier & Samuel (2024), validating its effectiveness in more practical setting. However, there a long history of papers that tried to combine bidirectional masked language modeling with unidirectional autoregressive modeling: T5 (Raffel et al., 2020) and BART (Lewis et al., 2020) were the first to train with autoregressive fill-in-the-blant training encoder-decode transformer architectures. Later, Du al. (2022) proposed GLM, which uses the same objective as T5 while using a simpler decoder-only architecture with complicated scheme of positional encodings. CM3 by Aghajanyan et al. (2022) further simplifies training by not requiring any non-standard architectural modifications like the previous work. As objective, their work close our approach
objectives by relying
they also add autoregressive language-modee model trained with CMB can be used as any other autoregressive model at inference time, similarly to us. However, our objective also generalizes masked-di didd---fffoon language modeling and allows for fine-grained balance of the two objectives throughout training. More recently AntLM by Yu et al. (2024) proposed to switch from one objective to the other a curriculum fashion, starting with short autoregressive training, followed by long masked language training and finishing another short autoregressive training. While this does show promise, the transition from one objective to the other leads to forgetting of the previous objective whereas objective continuously learns both objectives. Other notable works include prefix language models (Dong et al., 2019; Raffel et al., 2020; Wang et al., 2022) and UI.2 (Tay et al., 2023).

---

at Published as a conference paper at ICLR 2026
Scaling of autoregressive and masked-diffusion models Concurrent works by Prabhudesai et al. (2025) and Ni et al. (2025) have demonstrated that maasked-diffusion models outperform autoregressive models in data-constrained training regimes. Our results confirm their findings but show that using either of these training objectives is never optimal - combining them together should always be better, not only in data-constrained settings.
Bidirectional masking of user and system prompts A recent paper by Katz et al. (2025) shows that using bidirectional mask on user and system prompts improves performance on a wide variety of tasks, in line with Section 5,4. However, for models to be able to use such masks, the authors first need train adapters. Our work shows that by training both autoregressive and the same time, we are able to induce the prefix mask without any additional training.
constrennd scaling laws Muennighoft et al. (2023) studies the scaling laws of autoregressive models in data-constrained settings with similar motivation to this paper. They show that autore- gressive models cannot meaningfully learn from more than 16 data repetitions we demonstrate that this value is at least an order of magnitude larger when training with the dual objective.
Autoregressive diffusion Our work shares motivation with the autoregressive-diffuiiin models proposed by Wu al. (2023). The diffusion process in that work is biased towards left-to-right denoising, which improved the decoding efficiency of the diffusion language models at that time. Similarly, Arriola et al. (2025) speeds up decoding of generating chunks of tokens where each chunk is decoded by a diffusion process. In both cases, the resulting models are still diffusion models- albeit faster; these approaches do not generalize over autoregressive and
models by autoregressively
language modeling as our method.
Fair MD-AR comparison The recent work by Xue et al. (2025) modifies masked-diffusion language models by parameterizing them with causally-masked transformers, which makes the diffusion models more comparable to standard autoregressive models - decoupling their architectural differences from differences in training objectives. Their conclusion is that masked diffusion alone is a suboptimal objective for language, which is also confirmed by our experiments (Figure 4). However, we found that by simply combining both objectives, can get the benefits of diffusion without losing any performance.
Approaching the data wall Large language models are known reliably follow the empirical scaling laws that describe how their performance should improve with increased compute, model size, and training data. Kaplan et al. (2020) first demonstrated these relationships, showing how the training loss decreases as a power law with respect to these three parameters. These laws were later refined by Hoffmann al. (2022), who showed that cottiaa--ptimal training requires scaling data and model size together. Related to our work, the scaling laws reveal fundamental problem: achieving each incremental gain in performance requires exponentially more training data. Thus, data-constrained language modeling is quickly becoming relevant field of study even for high high-resource languages such as English.
7 CONCLUSION
In this work, we addressed the fundamental trade-off between the training efficiency of autoregressive models and the overfitting resilience of masked-diffusion models. We have empirically demonstrated that a dual-objective training strategy successfully achieves the best of both worlds, resulting in models that converge rapidly without any performance degradation in data-constrained settings. Crucially, because this unification requires no architectural changes, the resulting models incur no inference overhead and can be deployed as standard autoregressive transformers.
We established that combining objectives is universally beneficial and derived practical guidelines for selecting the optimal a based on the degree of data repetition. Furthermore, we observed that the diffusion objective induces robust prefix language modeling capabilities. leading to superior performance on downstream tasks compared to standard autoregressive baselines. While training on hundreds of data repetitions may seem extreme today, the asymmetry between exponentially scaling compute budgets and the finite supply of high-quality text suggests that data constraints will become increasingly relevant for frontier model development. Our findings indicate that dual- objective training provides a robust and compute- efficient path forward that retains standard inference capabilities as the field approaches these fundamental limits.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
To ensure reproducibility of our work we provided the guidelines on how train language models on both objectives at the same time in Section 3. For our model parameters and hyperparameters we specified those in Section 5.1. We describe how we perform the evaluations, the number of mask tokens used for PLL, the prompt formats, and log-likelihood normalizations in Section 4, Appendix C, and Appendix We openly release our custom training and evaluation code at https: //github. com/ haalo//uualllaaaaageeeeeee models. The training code is based on the common and freely distributed deep-Icarning framework PyTorch (Paszke et al., 2019). The trained models are released openly under the Apache 2.0 license for further investigation at https: haapss:///gggnnaaaac ct/lto///lttcc -1m- 470m.
AUTHOR CONTRIBUTIONS
Both authors have contributed equally and should be considered shared first authors of this manuscript.
ACKNOWLEDGMENTS
The computations were performed resources provided through Sigma2 the national research infrastructure provider for perf-peeformance computing and large-scale data storage in Norway. We acknowledge Norway and Sigma2 for awarding this project access to the LUMI supercomputer, owned by the EuroHPC Joint Undertaking, hosted by CSC (Finland) and the LUMI consortium through project 465001890.
The efforts described in this paper were jointly funded by the University of Oslo and the HPLT project (High Performance Language Technologies; coordinated by Charles University).
REFERENCES
Armen Aghajanyan, Bernie Huang, Candace Ross, Vladimir Karpukhin, Hu Xu, Naman Goyal, Dmytro Okhonko, Mandar Joshi, Gargi Ghosh, Mike Lewis, and Luke Zettlemoyer. CM3: A causal masked multimodal model of the internet, 2022.
Marianne Arriola, Subham Sekhar Sahoo, Aaron Gokaslan, Zhihan Yang, Zhixuan Qi, Jiaqi Han, Justin T Chiu, and Volodymyr Kuleshov. Block diffusion: Interpolating between autoregres- sive and diffusion language models. In The Thirteenth International Conference on Learning Representations, 2025.
Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Structured denoising diffusion models in discrete state-spaces. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp.
Curran Associates, Inc., 2021.
Lukas Berglund, Meg Tong, Maximilian Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, and Owain Evans. The reversal eurse: LLMS trained on "a is b" fail to learn "b is a". In The Twelfth International Conference on Learning Representations, 2024.
Yonatan Bisk, Rowan Zellers, Ronan Le bras, Jianfeng Gao, and Yejin Choi. PIQA: Reasoning about physical commonsense in natural language. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):7432-7433 Apr. 2020. doi: 10. N0.1609/aaii.v4443 66239.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020.
Laurie Burchell, Ona de Gibert, Nikolay Arefyev, Mikko Aulamo, Marta Bañón, Pinzhen Chen, Mariia Fedorova, Liane Guillou, Barry Haddow, Jan Hajie, Jindfich Helel, Erik Henriksson,
11

---

Published as a conference paper at ICLR 2026
Mateusz Klimaszewski, Ville Komulainen, Andrey Kutuzov, Joona Kytöniemi, Veronika Laippala, Petter Mæhluum, Bhavitvya Malik, Farrokh Mehryary, Vladislav Mikhailov, Nikita Moghe, Amanda Myntti, Dayyán O' O'Brien, Stephan Oepen, Proyag Pal, Jousia Piha, Sampo Pyysalo, Gema Ramírez- Sánchez, David Samuel, Pavel Stepachev, Jörg Tiedemann, Dušan Variš, Tereza Vojtéchová, and Bernaheu.
Jaume Baragoza-Bernaheu. An expanded massive multilingual dataset for high-performance language technologies (HPLT). In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilchvar (eds.), Proceedings of the 63rd Anmual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Pupers), Pp. 17452-17485, Vienna, Austria, July 2025, Association for Computational Linguisties. ISBN 979-8-89176-25555 doi: 10.18653/v1/ 2025.acl-long 8854.
Lucas Georges Gabriel Charpentier and David Samuel. GPT or BERT:
Why not both? In Michael Y. Hu, Aaron Mueller, Candace Ross, Adina Williams, Tal Linzen, Chengxu Zhuang. Leshem Choshen, Ryan Cotterell, Alex Warstadt, and Ethan Gotlieb Wilcox (eds.). The 2nd BabyLM Challenge at the 28th Conference on Computational Natural Language Learning, pp. 262 283, Miami, FL, USA, November 2024. Association for Computational Linguistics.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Lev- skaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan, Hycontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan. Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowyez, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier- Hellsteel,,,,en, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. PaLM: Scaling language modeling with pathways, 2022.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? Try ARC, the A12 reasoning challenge, 2018.
Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers). pp. 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguisties. doi: 00.18653/v//N19-11
Li Dong, Nan Yang, Wenhui Wang. Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. Unified language model pre-training for natural language understanding and generation. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. GLM: General language model pretraining with autoregressive blank infilling. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers), pp. 320-335, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18633/1120222.cd
R. A. Fisher. On the mathematical foundations of theoretical statistics. Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of a Mathematical or Physical Character, 222:309-3666 1922. doi: 1922..00...
R. A. Fisher. Theory of statistical estimation. Mathematical Proceedings of the Cambridge Philosophical Sociery, 22(5):700-725, 1925. doi: 10.1011/50035504410000000
Clémentine Fourrier, Nathan Habib, Alina Lozovskaya, Konrad Szafer, and Thomas Wolf. Open LLM leaderboard v2, 2024.
12

---

Published as a conference paper at ICLR 2026
Philip Gage. A new algorithm for data compression. C Users J., 12(2):23-38 -3- February 1994. ISSN 0898-9788.
Shansan Gong, Shivam Agarwal, Yizhe Zhang, Jiacheng Ye, Lin Zheng, Mukai Li, Chenxin An, Peilin Zhao, Wei Bi, Jiawei Han, Hao Peng, and Lingpeng Kong. Scaling diffusion language models via adaptation from autoregressive models. In The Thirteenth International Conference on Learning Representations, 2025.
Yuling Gu, Oyvind Tafjord, Bailey Kuehl, Dany Haddad, Jesse Dodge, and Hannaneh Hajishirzi. OLMES: A standard for language model evaluations. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.). Findings of the Association for Computational Linguistics: NAACL 2025, pp. 5005-,033, Albuquerque, New Mexico, April 2025. Association for Computational Linguistics. ISBN 11-88-9119---- doi: 10. 10.18653/v1/2025 findings- naacl.282.
Alexander Hägele, Elie Bakouch, Atli Kosson, Loubna Ben allal, Leandro Von Werra, and Martin Jaggi. Scaling laws and compute-optimal training beyond fixed training durations. In Workshop Efficient Systems for Foundation Models II @ ICML2024, 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Mcasuring massive multitask language understanding. In International Conference on Learning Representations, 2021.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Oriol Vinyals, Jack W. Rae, and Laurent Sifre. Training compute-optimal large language models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 0781713871088.
Ari Holtzman, Peter West, Vered Shwartz. Yejin Choi, and Luke Zettlemoyer. Surface form Why the highest probability answer isn't always right. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 202 I Conference on Empirical Methods in Natural Language Processing, pp. 7038-7051, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v1/202 h0tp2/002..mmmmmmamm5.4....
competition:
Michael Y. Hu, Aaron Mueller, Candace Ross, Adina Williams, Tal Linzen, Chengxu Zhuang, Leshem Choshen, Ryan Cotterell, Alex Warstadt, and Ethan Gotlibb Wilcox (eds.). The 2nd BabyLM Challenge at the 28th Conference on Computational Natural Language Learning. Miami, FL, USA, November 2024. Association for Computational Linguistics.
Keller Jordan, Yuchen Jin, Vlado Boza, Jiacheng You, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks, 2024.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.
Shahar Katz, Liran Ringel, Yaniv Romano, and Lior Wolf. Segment-based attention masking for GPTs. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilchvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 19308-19322, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-251-1 doi: 10.18653/v 1/222/1/22222// 10055:/11//22.....oooog.g..o0 acl-long.9947
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-to-sequene pre-training for natural language generation, translation, and comprehension. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7871 187880, Online, July 2020. Association for Computational Linguistics. doi: 10. 10.18653/v1/2222.a main.703.
13

---

Published as a conference paper at ICLR 2026
David Lindner, János Kramár, Sebastian Farquhar, Matthew Rahtz, Thomas McGrath, and Vladimir Mikulik. Tracr: compiled transformers as a laboratory for interpretability. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA, 2023. Curran Associates Inc.
Dong C. Liu and Jorge Nocedal. On the limited memory BFGS method for large scale optimiza- tion. Math. Program., 45(1-3):503-528, August 1989. ISSN 0025-5610.
Jingyuan Liu, Jianlin Su, Xingeheng Yao, Zhejun Jiang. Guokun Lai, Yulun Du, Yidao Qin, Weixin Xu, Enzhe Lu, Junjie Yan, Yanru Chen, Huabin Zheng, Yibo Liu, Shaowei Liu, Bohong Yin, Weiran He, Han Zhu, Yuzhi Wang, Jianzhou Wang. Mengnan Dong, Zheng Zhang, Yongsheng Kang, Hao Zhang. Xinran Xu, Yutao Zhang, Yuxin Wu, Xinyu Zhou, and Zhilin Yang. Muon is scalable for LLM training, 2025.
Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution. In Proceedings of the 41 International Conference on Machine Learning, ICML 24. JMLR.org. 2024.
Ang Lv, Kaiyi Zhang, Shufang Xie, Quan Tu, Yuhan Chen, Ji-Rong Wen, and Rui Yan. An analysis and mitigation of the reversal curse. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.). Proceedings of the 2024 Conference on Empirical Methods in Natural Language Process- ing. pp. 13603-13615, Miami, Florida, USA, November 2024. Association for Computational Linguisties. doi: 10. 1/t65/5///22224.mmmmmmaaaa/
Nicholas Metropolis and S. Ulam. The Monte Carlo method. Journal of the American Statistical Association, 44(247):335 341, 1949. doi: 10. 10.1000/01621459, 191..1949999911.1
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? A new dataset for open book question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun iichichi Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2018. Association for Computational Linguistics. doi: 10. 10.18653/11/D18.. 12600
Brussels, Belgium, October-November
Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin A Raffel. Scaling data-constrained language models. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 50358-5033 2023.
Curran Associates, Inc.,
Toan Q. Nguyen and Julian Salazar. Transformers without fears: Improving the normalization of self-attention. In Jan Niehues, Rolando Cattoni, Sebastian Stüker, Matteo Negri, Marco Turchi, Thanh-Le Ha, Elizabeth Salesky, Ramon Sanabria, Loic Barrault, Lucia Specia, and Marcello Federico (eds.), Proceedings of the 16th International Conference on Spoken Language Translation, Hong Kong, November 2-3 2019. Association for Computational Linguisties.
Jinjie Ni, Qian Liu, Longxu Dou, Chao Du, Zili Wang, Hang Yan, Tianyu Pang, and Michael Qizhe Shieh. Diffusion language models are super data learners, 2025.
Shen Nie, Fengqi Zhu, Chao Du, Tianyu Pang, Qian Liu, Guangtao Zeng Min Lin, and Chongxuan Li. Scaling up masked diffusson models on text, In The Thirteenth International Conference on Learning Representations, 2025a.
Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou. Jun Hu, Jun Zhou, Yankai Lin, Ri-n Wen, and Chongxuan Li. Large language diffusion models. In ICLR 2025 Workshop on Deep Generative Model in Machine Learning: Theory, Principle and Efficacy, 2025b.
Jingyang Ou, Shen Nie, Kaiwen Xue, Fengqi Zhu, Jiacheng Sun, Zhenguo Li, and Chongxuan Li. Your absorbing discrete diffusion secretly models the conditional distributions of clean data. In The Thirteenth International Conference on Learning Representattinss 2025.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach De Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang,
14

---

Published as a conference paper at ICLR 2026
Junjie Bai, and Soumith Chintala. PyTorch: an imperative style, high-performance deep learning library. Curran Associates Inc., Red Hook, NY, USA, 2019.
Mihir Prabhudesai, Mengning Wu, Amir Zadeh, Katerina Fragkiadaki, and Deepak Pathak. Diffusion beats autoregressive in data-constrained settings, 2025.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21()), jan 2020. ISSN 1532-4435.
Prajit Ramachandran, Barret Zoph, and Quoc V. Le. Searching for activation functions, 2018.
Subham Sekhar Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin T Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and effective masked diffusion language models. In Proceedings of the 38th International Conference on Neural Informa- tion Processing Systems. NIPS '24, Red Hook, NY, USA, 2025. Curran Associates Inc. ISBN 9798331314385.
Julian Salazar, Davis Liang, Toan Q Nguyen, and Katrin Kirchhoff. Masked language model scoring. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), ), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 2699-2712, 12, Online, July 2020. Association for Computational Linguistics. doi: 10. 10.18655/1///222.oal aca.ma.nn....m.m..m.mmmm..aa.... main. 240.
David Samuel. BERTs are generative in-context learners. In Proceedings of the 38th International Conference on Neural Information Processing Systems, NIPS '24, Red Hook, NY, USA, 2025. Curran Associates Inc. ISBN 9798331314385.
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social IQa: Common- sense reasoning about social interactions. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP), pp. 4463 4473, Hong Kong, China, November 2019, Association for Computational Linguisties. doi: 10.18653/v1/D19.1 1454.
Claude E. Shannon. Prediction and entropy of printed English. Bell System Technical Journal, 30 (1):50-64, January 1951. doi: 10. 00002/..1538..1 1-----33---- 7305. 11111..b1166....
Noam Shazeer. GLU variants improve transformer, 2020.
Michael L. Stein. Interpolation of spatial data. Springer Series in Statistics. Springer- Springer-Verlag, New York, 1999. ISBN 0-387-98629-4, doi: 10.1007/978- 00.1007998---4412-1111 1494-6. Some theory for Kriging.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. RoFormer: Enhanced transformer with rotary position embedding. Neurocomput. 568(C), February 2024.
ISSN 0925-2312. doi: 10. 20016//.neucom.220 2023..220026
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsense@A::
question answering challenge targeting commonsense knowledge. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers). pp. 4158, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-14433
Yi Tay, Mostafa Dehghani, Vinh Q Tran, Xavier Garcia, Jason Wei, Xuezhi Wang, Hyung Won Chung. Dara Bahri, Tal Schuster, Steven Zheng, Denny Zhou, Neil Houlsby, and Donald Metzler. UL2: Unifying language learning paradigms. In The Eleventh International Conference on Learning Representations, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
15

---

Published as a conference paper at ICLR 2026
Pablo Villalobos, Anson Ho, Jaime Sevilla, Tamay Besiroglu, Lennart Heim, and Marius Hobbhahn. Position: Will we run out of data? Limits of llm scaling based on human-generater data. In Proceedings of the s1 International Conference on Machine Learning, (CML'24. JMLR.org,
2024.
Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland. Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, C Carey, ilhan Polat, Yu Feng. Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 11726112722 2020. doi:
Alex Wang and Kyunghyun Cho. BERT has mouth, and it must speak: BERT as Markoy random field language model. In Antoine Bosselut, Asli Celikyilmaz, Marjan Ghazvininejad, Srinivasan Iyer, Urvashi Khandelwal, Hannah Rashkin, and Thomas Wolf (eds.), Proceedings of the Workshop on Methods for Optimizing and Evaluating Neural Language Generation, pp. 30-36, Minneapolis, Minnesota, June 2019, Association for Computational Linguistics. doi: N0.18653/11/119-230
Thomas Wang, Adam Roberts, Daniel Hesslow, Teven Le Scao, Hyung Won Chung, Iz Beltagy, Julien Launay, and Colin Raffel. What language model architecture and pretraining objective works best for zero-shot generalization? In Kamalika Chaudhuri. Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.). Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 22964 22984. PMLR, 17-23 Jul 2022.
Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. BLiMP: The benchmark of linguistic minimal pairs for En- glish. Transactions of the Association for Computational Linguistics, 8:377-392, 2020. doi: 10.1 162/tacl_aa 000321.
Gail Weiss, Yoav Goldberg, and Eran Yahav. Thinking like transformers. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 11080-11090, PMLR, 18-24 Jul 2021.
Christopher Williams and Carl Rasmussen. Gaussian processes for regression. In D. Touretzky, M.C. Mozer, and M. Hasselmo (eds.), Advances in Neural Information Processing Systems, volume 8.
MIT Press, 1995.
Tong Wu, Zhihao Fan, Xiao Liu, Hai-Tao Zheng, Yeyun Gong, Yelong Shen, Jian Jiao, Juntao Li, Zhongyu Wei, Jian Guo, Nan Duan, and Weizhu Chen. AR-Diffusion: auto-regressive diffusion model for text generation. In Proceedings of the 37th International Conference Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA, 2023. Curran Associates Inc.
Shuchen Xue, Tianyu Xie, Tianyang Hu, Zijin Feng, Jiacheng Sun, Kenji Kawaguchi, Zhenguo Li, and Mhi-Ming Ma. Any-order GPT as masked diffuuson model: Decoupling formulation and architecture. In ES-FoMo III: 3rd Workshop on Efficient Systems for Foundation Models, 2025.
Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong. Dream 7B: Diffuusio large language models, 2025.
Xinru Yu, Bin Guo, Shiwei Luo, Jie Wang, Tao Ji, and Yuanbin Wu. AntLM: Bridging causal and masked language models. In Michael Y. Hu, Aaron Mueller, Candace Ross, Adina Williams, Tal Linzen, Chengxu Zhuang, Leshem Choshen, Ryan Cotterell, Alex Warstadt, and Ethan Gotlieb Wilcox (eds.), The 2nd BabyLM Challenge at the 28th Conference on Computational Natural Lan- guage Learning, pp. 331-331, Miami, FI., USA, November 2024. Association for Computational Linguisties.
Rowan Zellers, Ani Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a machine really finish your sentence? In Anna Korhonen, David Traum, and Lluís Mårquez
16

---

Published as a conference paper at ICLR 2026
(eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Pp. 4791-4800, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P193 1/P11-11119991911 1472.
Biao Zhang and Rico Sennrich. Root mean square layer normalization. Curran Associates Inc., Red Hook, NY, USA, 2019,
17

---

Published as a conference paper at ICLR 2026
A THE USE OF LARGE LANGUAGE MODELS
Large language models have been used to provide feedback, fix grammatical errors and improve the writing in this paper, in particular, we used the Claude family of language models from https: //claude, ai. addition, we used the autocompletion tool from GitHub Copilot when writing the code used in this work.
B ERRATUM: LOSS FORMULATION
The original version of this paper as well as the training code used the following formulation the loss function:
argmin [2 a 0) LMD ( 0)
(6)
The additional factor of two for CAR was supposed to balance the 1/4 term applied when computing LMD (recall Equation (4)). This is however not mathematically correct (Equation (4)) and unnecessary; the loss function is thus simplified this updated version.
C LOG-LIKELIHOOD NORMALIZATION
For the BLiMP task, which is not considered in the OLMES evaluation suite, we do not apply any normalization and take the raw log-likelihood. We also stick to the no-context form of this task, where the whole sentence considered the completion. We apply character length normalization to ARC-Easy. HellaSwag, MMLU, PIQA, and SIQA. Finally, we apply point-wise mutual information normalization (Holtzman et al., 2021), where the likeliiel of the context-nnformed completion is divided by the log-likelihood of the uncontrained context completion, this can be seen in Equation (7), to ARC-Challenge, Commonsense QA, and OpenBook QA.
[wl
Pg (w;
c
w wei)
PMI(w) = log
pe (w)
2m
where w is the completion, is the context, and u the unconstrained context (in our case, this would be "Answer:")
D MONTE CARLO ESTIMATION OF LOG-LIKELLHHON
To evaluate the -ifkfdd-iffusion capabilities of our models, we use Equation (4) with the same modification as for the autoregressive evaluation well as adaptation of Monte-Carlo sampling to estimate the log- ligllikel of each completion. Specifically, we estimate the following expected value:
1
1 T
E
Logpo(es Σ
x²)
dt
E
Loggpo (I; at)
(8)
(0 (1a)
{flo mask)
[ask
To reduce the variance of the estimation and get faster convergence, we take the expectation between N equally spaced points between 0 and 1. instead of taking the expectation over t ~ 4(, 1). Yet, accurate estimation still requires N 256, which unbearably slow especially when compared to simple autoregressive calculation of log-likelihood that requires only single forward pass.
E
PSEUDO LOG-LIKELIHOOD ESTIMATION
The base PLL equation can be described by a slight modification of Equation (2):
[w]]
log Po (w)
log (w:
c + wo + ... $ w;-1
(9)
+ mask
① w:+1 1 @ ... ① ww))
18

---

Published as a conference paper at ICLR 2026
This means that instead of doing single forward pass, need do |w| forward passes to estimate the PLL. However, using a single mask token could lead to underestimating the log- log-liielhoood of words split into multiple tokens. Therefore, we can further modify Equation (9) to have variable (but constant) number of mask tokens after the token we are trying to estimate:
log Pe
log Pa [w
Wo 1
(10)
+ mask + ... + mask
Witn A ... ① w mi))
where represents the number of [MASK] tokens. In our case, we take a combination of two different numbers of mask tokens (1 and 6), by taking the best score of the two for each task. The two values were chosen experimentally, more details on the results of each number of mask tokens can be found in Appendix H.
F PROOF OF LEFT-SHIFT CLOSURE
This section proves that when we parameterize masked-diffusion language models as bidirectional transformers with shifted output, we do not lose any expressivity compared to standard non-shifted bidirectional models. We prove constructively by defining a shift operation in the RASP language (which can then be compiled into equivalent transformer model).
Definition (RASP programs). The Restricted Access Sequence Processing language (RASP: Weiss et al., 2021) is sequence processing language that uses two types of variables: sequence operators and selectors; and two types of operators: element-wise and select-aggregate operators. Valid programs in RASP operations sequence operators formed by finite composition of element-wise and select-aggregate operators.
Sequence operators represent sequences of values (akin to hidden states in transformer models). tokens and indi ces two pre-defined sequence operators; the first directly returns sequence of the input tokens (tokens( Makkens("helloo ("hello") = [h e, 1, 1, o], and the second returns the positional [0, 2, 3, 4).
indices aandices""hellee
Selectors are binary matrices (akin to attention matrices transformers).
Element-wise operators are arbitrary clement-wise transformations on sequence operators (akin to feed-forward layers in transformers). For example (indices ( 2)("eeloo") [2, 4, 5, 6].
Select-uggregate operators consist of two sequentially applied operators select and aggregate (corresponding to the attention operation).
select(æ, y. p) is an operator defined on two sequence operators x and and an element- boolean operator p defined on two sequence operators; the result is selector matrix M, where Mij p(=;, Uz). For example, select([0, 1, [1, 3). <) results in a upper-triangular 3 x binary matrix (selector).
aggregate M, x:; c) an operator defined on a selector M, a sequence operator æ and a default value (usually set to and omitted for convenience). It produces sequence operator such that:
M::My=IJ| E Ej:,,
M, Tzz if | My 1}| 0,
otherwise.
Fact (RASP-transfomee reduction). For every valid program written in RASP, there exists an equivalent biddreiiditttionall transformer model that computes the same per-position operation; Weiss et al. (2021); Lindner et al. (2023).
Definition 2 (E-realizable functions). We consider programs defined on input alphabet Σ with special token 
 E E. A valid input sequence x (i), "2.,,,,..,,,, .r)) E X' every sequence where Tn 
 and all E. The output space y is made of sequences y (y1, (2...)n) E )
19

---

Published as a conference paper at ICLR 2026
where every element is probability distribution over the alphabet Σ: that is all y: [0, 1] and Σ, (9t), =.
function f X - V E-realizable if there exists transformer whose output on every input X equals f fxx position-wise. Let Ry be the class of all E-realizable functions.
Theorem (Left-shift closure). Ry is closed under unit left-shifts: for every f RR, there exists g E Ry such that for all X and i E [1,n 1]: g(x), f (note that f())2 and g(x) are not constrained).
Proof. The proof constructs suitable function 9 E Ry for any f Ry. The new function g will mirror function f and then shift its output so that g(x); = f())+11, the shift will be constructed in RASP so that g is E-realizable.
Let f E Ry; be any E-realizable function and set 7j as fully-bidirectional transformer that realizes f. so (2x);; f());; for all valid inputs E X and all positions E [1.
First, we define RASP selector S = select (indices + 1, indices, =), whose entries therefore satisfy Sūj = iff j = i (each row selects exactly the next position + 1, and the last row selects none).
Then, for any sequence operator = (possibly vector-valued), we define a RASP program shift(z) aggregate(S, 2; c), where is arbitrary and can be simply set to 2n- By construction of S and the definition of aggregate, we have shift(z)n = = 2n and for every
[L, 1]:
shift(z);; 10: Sij = 1}
Σ
2j 1i+1-
(11)
J: B:
Using Fact 1, there exists a transformer Tahift that computes the RASP program shift. Therefore, we can construct transformer T, as Tshift TT. This corresponds the function g we are looking for -or operates in the same input and output space as Ty, so E Ry; furthermore, this function satisfies for all æ E and [1.n n 1]: g(z), shift(f(z)) = f(x)
model with shifted transformer, it as expressive as the standard non-shifted parameterization. More specifically, masked diffusion is defined in Equation (4), and Po(z; x²) is typically implemented as a fully- bidirectional transformer model that outputs this probability at the ith position. When we set E as our subword vocabulary, we get that the space of all possible transformer realizations of (v; | x²) are the E-realizable functions Rg (Definition 2). Theorem shows that if we instcad expect the output at the (i 1)th position, do not lose any expressivity. Thus, transformer-basee dual-objective language models are generalization of standard masked-diffusion language models. Note that the left-shift closure in Theorem works up the first token which is guaranteed be the special 
 token in Definition as well as the actual implementation.
Corollary 1.1. Theorem implies that when we parameterize
20

---

Published as a conference paper at ICLR 2026
G VALIDATION LOSS CURVES
While we focused on actual downstream performance the main experiments, we also show the validation loss below demonstrate the training dynamics.
The validation curves in Figure focus on an extremely data-constrained scenario with 128 data repetitions. There, it is crucial to avoid overfitting, which can be achieved by increasing the proportion of masked diffusion during training. Note that the noise of some of the curves is only due to our implementation of measuring the validation loss the sample size be too small when the proportion of the respective training objective is low.
COMBINED VALIDATION LOSS
6.0
51/32
1/8
7/8
5.0
Nofidtttoo mifiem
4.5
4.0
3.5
3.0
2.5
AUTOREGRESSIVE VALIDATION LOSS
6.0
5.0
4.5
Volhiiiion md
3.5
2.5
VALIDATION LOSS
6.0
5.5
5.0
Valistston nor
4.5
3.5
2.5
2.0
Figure 6: Validation loss curves for 128 repetitions. These plots clearly demonstrate how training runs with high a (in red) overfit. Low a values are blue.
21

---

Published as a conference paper at ICLR 2026
Contrary the previous figure, Figure 7 shows validation curves for 4 data repetitions. Here, overfitting is not an issue; instead crucial to improve the learning speed by increasing the proportion of autoregressive language modeling.
COMBINED VALIDATION LOSS
6.0
256
1:32
1/2
5.5
1/1228
1/8
7/8
5.0
Validation ml loze
4.5
4.0
3.5
3.0
2.0
AUTOREGRESSIVE VALIDATION LOSS
6.0
5.5
5.0
Voriiizionn nd
4.5
4.0
3.5
2.5
MASKED DIFFUSDI VALIDATION LOSS
6.0
5.5
5.0
Paliiaaton Joxr
4.5
4.0
3.0
2.0
12024
2 048
3 072
096
120
144
16168
191
Stop
Figure 7: Validation loss curves for 4 repetitions. All losses monotonically decrease because overfitting is not concern in this setting. High values are plotted in red and low values shown in blue.
H EFFECTS OF NUMBER OF MASK TOKENS ON THE PLL
We first look at whether using single number of mask tokens can lead to good estimation of the PLL in general. For this, we evaluate five different models from to mask tokens and report the results in Tables 3 to 7.
22

---

Published as a conference paper at ICLR 2026
Table PLL performance depending on the number of mask tokens. We show the PLL performance on the 9 tasks of the model trained with an equal weight of masked-diffusion and AR and 32 repetitions with different number masks. Best results per task are boldfaced.
Table 4: PLL performance depending on the number of mask tokens. We show the PLL performance on the tasks of the model trained with masked-di maakkeddiffusion to autoregressive ratio (ar = 7/8) and 32 repetitions with different number of masks. Best results per task are bold- faced.
Task
musks
Number 4
ARC Easy
18.7
24.1
25.5
26.3
26.0
26.3
ARC Easy
18.2
25.6
27.1
28.2
26,9
27.5
ARC Challenge
4.7
3.3
3.8
2.7
1.9
2.6
ARC Challenge
1.9
3.2
2,4
2.6
3.6
4.7
BLIMP
63.9
62.5
BLiMP
56.9
Commonsense QA
29.4
32.8
33.9
34.1
34.1
34,1
Commonsense QA
24.2
29.1
29.0
29.4
29.4
29.4
HellaSwag
29.8
27.0
26.7
27.1
26.7
26.4
HellaSwag
25.2
25.7
26.6
27.0
26.8
26.8
MMLU
2.9
MMLU
1.9
4.2
OpenBook QA
9.1
7.7
8.5
9.3
7.2
6.9
OpenBook QA
9.9
10.1
12.3
10.9
10.1
9.6
PIQA
33.1
34.3
35,1
35.4
35.6
36.8
PIQA
31.0
34.7
36.1
36.0
35.0
35.9
SIQA
11.4
13.3
13.7
14.4
14.4
SIQA
14.2
13.7
14.1
Average
22.6
23.3
23.6
23.5
23.3
23.4
Average
20.6
22.6
23.3
23.2
23.0
23.3
Table PLL performance depending on the number of mask tokens. We show the PLL performance on 9 tasks of the model trained with 7 masked- -iiseeddifffssio to autoregressive ratio (a 1/s) and 32 repetitions with different number of masks. Best results per task are bold- faced.
Table 6: PLL performance depending on the number of mask tokens. We show the PLL performance on the tasks of the model trained with an equal ratio of masked-diffusee (a = 1/2) and 16 repetitions with different num- ber of masks. Best results per task are boldfaced.
and AR
Task
Number of masks
Task
Number masks
2
4
ARC Easy
16.3
20.8
23.9
24.0
24.9
24.9
ARC Fasy
16.8
23.7
25.8
25.8
26.1
26.1
ARC Challenge
5.7
3.9
3,5
1.8
3.3
2.2
ARC Challenge
7.2
4.4
4.4
4.8
3.2
4.5
BLiMP
69.5
67.6
60.7
BLiMP
60.7
60.6
Commonsense QA
25.4
29.7
30.6
31.1
31.1
31.2
Commonsense
29.7
33.8
35.1
35.1
35.2
35.2
HellaSwag
25.5
22.8
21.0
21.2
20.5
19.8
HellaSwag
30.5
27.9
27.8
27.9
27.2
26.8
MMLU
2.2
2.0
2.5
2.4
MMLU
1.3
2.4
2.5
OpenBook QA
13.1
12.0
15.2
14.4
13.1
13.9
OpenBook QA
12.3
12.0
13.1
11.2
11.7
11.7
PIQA
29.6
30.3
30.8
30.1
31.2
31.0
PIQA
33.8
34.6
36.0
34.7
36.3
37.0
SIQA
12.2
15.0
15.2
13.6
13.8
13.9
SIQA
13.9
15.9
15.3
15.9
16.1
Average
22.0
22.7
22.9
22.1
22.3
22.2
Average
23.5
24.2
24.9
24.2
24.3
24.5
Table 7: PLL performance depending on the number of mask tokens. We show the PLL perfor- mance on the 9 tasks the model trained with an equal ratio of masked-diffusiee and 64 repetitions with different number of masks. Best results per task are boldfaced.
and AR (a 1/2)
Task
Number of masks
ARC Easy
16.6
21.8
23.5
23.4
23.1
23.1
ARC Challenge
3.9
3,2
4,0
3.5
BLiMP
63.1
61.2
59.6
$7.5
56.9
56.9
Commonsense QA
24.6
27.6
28.5
28.7
28.7
28.7
HellaSwag
26.8
25.2
24.2
24.7
24.3
24.1
MMLU
1.2
3.1
3.0
3.2
3,4
3.2
OpenBook QA
8.5
11.7
10.1
PIQA
31.0
31.7
32.1
33.7
34.3
34.1
SIQA
14.3
12.3
14.3
13.1
13.3
13.5
Average
20.8
21.7
22.3
22.0
21.8
21.7
23

---

Published as a conference paper at ICLR 2026
We two clear trends from the results. The first is that the BLiMP and HellaSwag tasks better evaluated with a single mask token, rather than multiple. This could be due to the simpler language found in these datasets. The second trend is that ARC-Easy, Commonsense QA, PIQA. and SIQA tend to do better with multi-token masking, this could be due to the more complex answers using more infrequent words that have higher likelihood of being split into subwords. We therefore decide that using combination of a single token mask for some tasks and multiple tokens for others is the best solution. To find the optimal combination, results can be seen in Table 8.
all possible combinations. The
Table 8: PLL performance for combinations of one mask token and multi-mask token. Best results per model boldfaced.
Mask combination
Repetitions
1-2
1-3
1-4
4
1-5
1 6
32 1/2
24.1
24.5
24.6
24.7
24.8
32 7/8
22.8
23.6
23.7
23.5
23.8
32 1/s
23.5
24.3
24.0
24.1
24.2
16 1/2
24.9
25.7
25.4
25.7
25.8
64
22.3
23.0
22.9
22.9
22.8
Based on Table decide to evaluate PLL for all models with both single mask token and six mask tokens. Then we take the max performance between the two for each task.
I TOTAL COMPUTE RESOURCES USED FOR TRAINING
The training of 50 language models used in this paper was conducted on the LUMI supercomputer, each language model was trained 128 AMD MI250X GPUs (which is equivalent to 256 logical devices) using roughly 500 GPU hours. In total, the resources required for conducting all training runs equals approximately 75 000 GPU hours.
J PLL VERSUS MASKED DIFFUSION
Table 9 shows that the performance of the masked-diffusion model is in general lower than that of the combined (1 and mask) PLL. In addition, the two PL.. evaluations took about 2 hours to complete while the masked-diffusion evaluation takes 12 hours to complete MI250X AMD GPU.
Table 9: Normalized PLL versus Masked- -Diflusion evaluation. The scores for each task are normalized that 0% corresponds to the random baseline and 100% is the perfect score. The best result for each task is in boldfaced. We evaluate model trained with equal AR and masked- ratio (a 1/2) and 32 repetitions.
Task
PLL
Masked-Diffussin
ARC-Easy
26.3
27.1
BLiMP
65.2
56.5
Commonsense QA
34.1
32.7
HellaSwag
29.8
21.3
PIQA
36.8
32.0
K PREFIX-LM VERSUS AUTORREGRESSIVE-LLM ON OPTIMAL MODELS.
Table 10 shows that evaluating with the prefix mask almost always outperforms using the causal mask when the models are optimally trained. This is true in both the regular and constrained data settings.
24

---

Published as a conference paper at ICLR 2026
Table 10: Normalized autoregressive and prefix performance of selected models. The scores for each task are normalized so that 0% corresponds to the random baseline and 100% is the perfect score. The best result for each dataset size in boldfaced. The results for BLiMP are the same, since there is no context and the prefix evaluation defaults to the autoregressive one. The AR ratios for the models are 12.5% for the 128 repetitions, 75% for the 32 repetitions, and 98.4% for the single repetition.
ARC-C
ARC-E
BLiMP
CSQA
HSwaR
MMLU
OBQA
PIQA
SIQA
Model
Average
Autoregressive Prefix
28.6
63.7
35.1
31.1
4,9
40.9
14.3
26.9
6.5
31.0
63.7
40.0
31.2
4.5
16.5
42.1
15.2
27.9
Autoregressive
28.0
31.1
3.6
14.4
36.1
14.64
23.9
Prefix
6.3
28.9
57.9
33.1
27.1
4.3
15.2
36.7
15.4
25.0
128 REPETITIONS
Autoregressive
1.7
23.6
56.1
24.8
14.2
1.6
8.5
28.1
13.3
19.1
Prefix
1.3
24.1
56.1
28.5
12.4
2.3
10.9
30.9
15.2
20.5
L DETAILED RESULTS OF DIFFUSION-MASKED EVALUATION
Table 11: The normalized PLL performance of selected models. We show the results on all nine evaluated tasks for three repetition values; each repetition group contains the results of the best-performing and of the autoregressive-only model. The scores for each task are normalized so that 0% corresponds to random baseline and 100% the perfect score. The best result for each dataset size is boldfaced.
ARC-C
ARC-E
BLIMP
CSQA
HSwag
MMLU
OBQA
PI0A
SIQA
Model configuration
Average
REPETITIONS
Dual (a 3/4)
6.0
28.3
62.7
33.4
27.8
4.3
12.3
37.4
15.4
25.3
Masked-Diffusion (a-
-0.1
64.8
29.0
1.6
27.2
21.4
Dual (a 1/s)
2.8
23.3
63.5
30.5
25.0
2.1
12.8
31.8
15.2
23.0
Masked-Diffusion (a
3.3
19.2
63.3
29.2
22.1
2.6
9.3
28.3
12.0
21.0
Table 11 shows similar trends to those found in Table 2. The notable exception being for BLiMP where the performances are similar between both models. Unlike the autoregressive models, the performance of the purely masked-diffusion models is similar to each other. This is partially due to model not overfitting, but also to it not being sample efficient. the dual-objective models, the performance significantly increases increase the training data set size.
other hand we see for
25


Reviews:
[{"review_id": "0k94nPNopR", "reviewer": "Reviewer_cMFP", "summary": "This paper aims to introduce a unified training objective for language modeling that jointly optimizes autoregressive (AR) and masked-diffusion losses within a single transformer, while requiring no architectural modification. \nBy co-training a single transformer model on both losses, DLM leverages AR’s rapid convergence while regularizing with the diffusion objective to prevent overfitting. Extensive experiments across diverse benchmarks demonstrate that this hybrid objective consistently enhances performance over single-objective baselines.", "questions": "* Could you elaborate more on how closely does the proposed “diffusion mode” correspond to a standard masked-diffusion language model? For instance, how is the diffusion time variable integrated into the next-token prediction objective, is it sampled per token or globally for the sequence, and how does this choice affect model behavior? How does the new objective support parallel decoding or multi-token prediction? \n\n* How stable is the training process under the dual objective? I am wondering what would the loss curves look like and whether there are any significant tradeoffs between the two modes during training?\n\n* How does this work relate to a recent work [1] that proposed a modified decoder architecture to do any-order next-token prediction? \n\n[1] Any-Order GPT as Masked Diffusion Model: Decoupling Formulation and Architecture, arxiv preprint 2506.19935", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "This paper is well written and easy to follow. The narrative is coherent, and the topic is both timely and important for the current stage of large language model development.\n\nThe proposed DLM objective is conceptually simple and practically appealing. It requires no architectural changes and includes GPU-efficient adaptations, making the method easy to implement and broadly applicable. This design choice enhances the practical relevance of the work.\n\nThe empirical evaluation appears systematic and comprehensive. The experiments are carefully structured, covering a wide range of data regimes and objective ratios, which strengthens the credibility of the results and the generality of the conclusions.\n\nAlthough the work is primarily empirical, it offers new insights that meaningfully advance understanding in the community.", "weaknesses": "### The link to diffusion language model is a bit weak \n\nIt is unclear to me whether the “diffusion mode” in the proposed objective truly corresponds to a standard masked diffusion language model, as to my knowledge, no published work implements diffusion language model using a decoder. \nConceptually, the proposed objective appears closer to introducing stochastic regularization into standard autoregressive training—adding noise to the input sequence to improve generalization and reduce overfitting. \nMoreover, it remains ambiguous whether the resulting model supports parallel decoding, a defining property of diffusion-based LMs. \nWhile the paper claims that the model “can generalize to prefix LM,” it does not explicitly demonstrate diffusion-style generation. A clearer explanation of how the diffusion time-conditioning interacts with the next-token prediction objective would strengthen the technical grounding.\n\n\n### Lack of analytical justificaiton. \n\nThe paper’s conclusions are derived almost entirely from empirical evidence. While the experiments are thorough and convincing, additional theoretical or analytical insights would improve the work’s depth. For example, identifying sufficient conditions (some toy cases) under which the dual-objective training is guaranteed to outperform either individual objective would help explain why the method works beyond observation. \n\n\n### Incomplete discussion of training dynamics.\nThe paper focuses primarily on zero-shot, multiple-choice benchmarks but provides limited discussion of training behavior. Analyses of the loss curves (for both AR and diffusion modes), convergence patterns, and stability would give a more complete understanding of how the dual objective influences optimization and generative quality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "y9Al1dViO8", "reviewer": "Reviewer_59vc", "summary": "This work studies Dual LM, which combines the auto aggressive next token prediction and masked language modeling into a single model. To be specific, authors are investigating the potential to train one single LLM with two different training objectives. And the finding is we can achieve a better trade off between compute efficiency and data efficiency by adjusting the ratio of two training objectives.", "questions": "NA", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1) Very clear presentation. The lessons and insights are very presented so that readers can understand the problem well.\n2) Studying the data efficiency is a very important problem, for both large and small models, especially considering we are overtraining models for better quality and cheaper serving cost.\n3) The sweep on training objective ratio is informative. This can help to design better training schedule targeting on a specific data/compute efficiency.", "weaknesses": "1) The definition of sample-efficiency is confusing. I think it would be better to rephrase it as \"compute-efficiency\". The sample efficiency makes people confused whether the sample here means \"unique token efficiency\". \n2) Minor: I understand the research is usually for long term purpose, but to be very honest, the conclusion we achieved in this work is we only need dual LM for very aggressive repeats. This is anyway informative but not something we have to use now.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6OqkP7QexV", "reviewer": "Reviewer_apc1", "summary": "This paper studies the trade-off between sample efficiency and overfitting resilience in language modeling by combining autoregressive and masked-diffusion objectives in a single transformer architecture, without architectural modifications. The authors conduct an extensive empirical study, training 50 models across varying data repetition regimes, to uncover the optimal balance between the two objectives. They show that the dual-objective approach consistently outperforms single-objective baselines in both autoregressive and masked-diffusion downstream tasks, providing practical guidance on loss balancing and implications for prefix language modeling.", "questions": "- The paper builds on established ideas such as GPT-BERT and CM3, with the main contribution appearing to be the systematic analysis of data repetition effects. Could the authors clarify the specific technical or algorithmic innovations introduced by the proposed dual-objective framework beyond these prior methods?\n- Would the authors consider adding or discussing comparisons with related works (e.g. Block Diffusion and AR-Diffusion), which also address the AR–diffusion trade-off?\n- Can the authors provide evidence—empirical or analytical—that the optimal AR–MD ratios and the 16-repetition guideline (Remark 2) remain valid for larger models (≥ 1B parameters)?\n- Can the authors provide a more rigorous theoretical basis or empirical ablation for their objective weighting strategy?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The authors train and evaluate 50 language model configurations under a wide range of data repetition regimes, allowing for robust conclusions on objective balancing.\n- The dual-objective method is introduced without architectural changes, maximizing practical usability.\n- The paper distills its complex results into two practical, easy-to-understand guidelines for training models in regular and data-constrained regimes.", "weaknesses": "- The core idea of combining AR and masked objectives in a single decoder-only model is not entirely new and builds heavily on previous work, particularly GPT-BERT, which the authors acknowledge. The primary contribution is more of an extensive empirical investigation rather than a fundamentally new training paradigm.\n- While Section 6 discusses prior methods such as GPT-BERT and AntLM, the paper does not include empirical comparisons with these or other recent approaches (e.g. Block Diffusion [1], AR-Diffusion [2]) that address the same AR–diffusion trade-off. Including recent works as baselines under the same training setup would provide a clearer and fairer assessment of the proposed method’s advantages.\n- The paper assumes that the findings generalize to larger models, but this remains speculative, as all experiments are limited to the 470M scale. Given the unpredictability of scaling behavior, it is unclear whether the observed trends would hold for models beyond 1B parameters.\n- While Equations (2) and (4) define the individual objectives, the paper offers little theoretical explanation of how their joint optimization works. The rationale for specific weighting choices is empirical, and potential gradient interference or complementarity between the two losses is not analyzed.\n\n[1] Arriola, Marianne et al. “Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models.” 2025\n[2] Wu, Tong et al. “AR-Diffusion: Auto-Regressive Diffusion Model for Text Generation.” 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "x1yybzo43y", "reviewer": "Reviewer_Z29n", "summary": "This paper proposes Dual Language Models (Dual-LMs), which jointly train a single transformer with both autoregressive (AR) and masked-diffusion (MD) objectives. The key motivation is to balance sample efficiency (of AR training) and overfitting resilience (of MD training) without changing the model architecture. The authors conduct a large-scale empirical study on 50 language models to support the method's effectiveness.", "questions": "1. The paper’s results suggest that the dual-objective helps mainly under extreme data repetition. What prevents it from providing gains in standard (1–8×) training regimes? Is it fundamentally tied to overfitting mitigation rather than general sample efficiency improvement?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. Clear motivation and timely problem.\nThe work addresses a relevant issue as large-scale language model training increasingly encounters limited high-quality data and potential overfitting under repeated exposure.\n2. Practical recommendations\nThe derived empirical guidelines (e.g., small MD component under regular regimes, stronger MD ratio under high repetition) are actionable for practitioners designing large-scale LLM training recipes.", "weaknesses": "1. Limited contribution in standard settings.\nWhen trained with only 1× data repetition, Dual-LMs provide no significant improvement over pure autoregressive baselines. This weakens the overall claim of universal benefit; the method is primarily helpful under extreme data-reuse scenarios\n2. Practical rarity of extreme repetition.\nThe key advantage appears only under 128× repetition, which is rarely used in modern large-scale LLM training. In realistic compute-optimal regimes (1–8× repetition), the dual-objective gains are negligible.\n3. Learning rate confound.\nFor very high repetition, a smaller learning rate is typically required to maintain stability. Since the experiments maintain a fixed LR schedule, it is unclear whether the observed benefits of Dual-LMs come from the objective mixing itself or from suboptimal tuning in baselines.\n4. Method scope and positioning.\nOverall, the approach seems more suitable as a regularization mechanism for repeated training rather than a fundamentally new paradigm for LLM optimization. The contribution is empirical rather than conceptual, and its practical importance outside data-scarce settings is limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)