Rainbow Padding: Mitigating Early Termination in Instruction-Tuned Diffusion LLMs
Authors: BumJun Kim
Dongjae Jeon
Dueun Kim
Wonje Jeung
Albert No
Venue: ICLR 2026 Poster (2026)
Keywords: Discrete Diffusion
Instruction Tuning
NLP
Primary area: generative models
TL;DR: Rainbow Padding fixes &lt;eos&gt; overflow in diffusion LLMs, ensuring robust long-sequence generation with only 7 padding tokens.

Abstract: Diffusion large language models (dLLMs) have emerged as a promising alternative to autoregressive models, offering flexible generation orders and strong performance on complex reasoning tasks. 
However, instruction-tuned dLLMs exhibit a critical vulnerability we term \<eos\> overflow: as allocated sequence length increases, responses paradoxically become shorter, collapsing into early termination or degenerating into streams of \<eos\> tokens. 
Although noticed in practice, this issue has not been systematically analyzed. We trace its root cause to the dual role of \<eos\> as both termination and padding, which concentrates probability mass on \<eos\> at later positions and propagates backward to trigger early termination. 
To address this, we introduce Rainbow Padding, a simple remedy that replaces repeated \<eos\> placeholders with a repeating cycle of distinct padding tokens, distributing probability mass and breaking \<eos\> dominance. 
Experiments show that Rainbow Padding substantially improves length robustness and output quality, with as few as seven padding tokens to prevent early termination. 
Moreover, the method integrates efficiently into existing instruction-tuned models: LoRA fine-tuning for a single epoch on minimal data yields significant improvements, making this solution highly practical.
The project is available at ~\url{https://ai-isl.github.io/rainbow-padding}

Published as a conference paper at ICLR 2026
RAINBOW PADDING: MITIGATING EARLY TERMINA- TION IN INSTRUCTION-TUNED DIFFUSION LLMs
Bumjun Kim-*
Dongjac Jeon
Dueun Kim
Wonje Jeung' Albert No
DDepartment of Artificial Intelligence, Yonsei University
Department of Computer Science and Engineering, Yonsei University
ABSTRACT
Diffusion large language models (dLLMs) have emerged as promising alterna- tive to autoregressive models, offering flexible generation orders and strong per- formance complex reasoning tasks. However, enstrcction-tnne dLLMs exhibit a critical vulnerability we term <eos> overflow: as the allocated sequence length increases, responses paradoxically become shorter, collapsing into early termina- tion or degenerating into streams <eos> tokens. Although noticed in practice, this issue has not been systematically analyzed. We trace its root cause the dual role of <eas> as both termination and padding, which concentrates probability mass on <eos> later positions and propagates backward to trigger early mination. To address this, we introduce Rainbow Padding, simple remedy that replaces repeated <eos> placeholders with repeating cycle of distinct padding tokens, distributing probability mass and breaking <eos> dominance. Experi- ments show that Rainbow Padding substantially improves length robustness and output quality, with as few as seven padding tokens to prevent early termina- tion. Moreover, the method integrates efficiently into existing instruction-tume models: LoRA fine-tuning for single epoch minimal data yields significant improvements, making this solution highly practical. The project is available at https::/ai-ist github. 1o/ rainbow- padding
INTRODUCTION
Discrete Diffusion large language models (dLLMs) (Ne et al. 2025b Zhu et al. 2025 Ye et 2025b
Rainbow Padding
335.1
al. 2025 DeepMind
50
75.8
Arg. Responee Lengiht
300
2025) have recently emerged as promis- ing alternative traditional autoregressive I.MSs. Unlike autoregressive models, which generate strictly left-to-right, dLLMs allow tokens to be generated in any order, while maintaining global consistency through a diffusion-style denoising process. This flex- ible decoding has been linked to stronger multi-step reasoning and planning ability (Ye et al. 2025a Kim et al. 2025). with benefits that persist at scale (Nie al. 2025a). As a result, dLLMs are increasingly positioned as a viable paradigm for foundation models.
252.9
Accuracy (56)
40
122.
100
0.1
1.0
1.2
MATH GSMBK Humaalevll
MATH GSMBK HumanEva
Figure 1: Performance comparison of LLaDA Instruct with and without Rainbow Padding. Standard LLaDA LLaDApproduces produces overly short responses at moderate generation budgets (max.length 1024), resulting in significant accuracy degradation. Adapting with Rainbow Padding yields substantial performance gains.
However, current Instruction-tuned dLLMs suffer from critical reliability issue. When users al- these models often produce shorter responses:
locate longer generation budgets
Equal Contribution.
Corresponding Author.
'Throughout this paper, LLaDA and Dream denote the instruction-tune -uned models I.L.DA- 88 instrcct. and Dream-v0-- instruct- H,, unless stated otherwise.

---

Published as a conference paper at ICLR 2026
terminating early or degenerating into streams of <eos> tokens. We refer this failure mode as <eos> overflow. This paradox- wwhere giving the model more space yields worse results- has been observed in practice (Nie et al. 2025b Zhu et al. (2255 but has not been systematically analyzed. Its impact is substantial: performance on reasoning and coding tasks deteriorates even at moderate sequence lengths, undermining the utility of dLLLMs in real-world intruuction-following scenarios.
We trace the root cause of this issue to a design flaw in the instruction-tuning process. Current pipelines pad variable-length sequences with the <eos> token, thereby assigning it dual role-as both the legitimate oud-of-sequence marker and placeholder for unused positions. This conflation introduces strong positional bias: <eos> appears disproportionately at later positions. When used with widely adopted probability based decoding strategies (Chang et al., 2022] Kim et al.] [2025] Ye et al. 2025b Ham- et al. 2025), which typically prioritize the most confidently predicted tokens, <eos> is often selected prematurely. Once sampled at the tail, <eos> predictions propagate backward through the sequence, resulting in the overflow effect and early termination.
To address this failure, we introduce Rainbow Padding, simple yet effective modification to the padding scheme. Rather than repeating <eos> throughout the tail, we reserve a single <eos> to mark the true end of the sequence and fill the remainder with cyclic palette of distinct padding to- kens. This design has two key effects: it decouples termination from padding, ensuring that <eos> is learned only as proper stopping symbol; and it distributes probability mass across multiple tokens, preventing any single padding token from dominating. The deterministic cycle is easy to learn and provides weak structural signal of length without hindering the model's s ability to learn meaningful contextual dependencies during training.
Rainbow Padding can be adopted with only a brief fine-tuning phase, even for already instruction- tuned models. Despite its simplicity, it effectively eliminates <eos> overflow, restoring length robustness and significantly improving performance on mathematical reasoning, code generation, and general instruction-folloiin tasks. Figure [ fillustrates the effect: as max_length increases, baseline dLLM (LLaDA) collapses into short answers, whereas Rainbow Padding restores appro- priate response length and accuracy (see Sections O Unlike heuristic fixes exes-such as manually suppressing <eos> confidence or enforcing semi-aus autoo-rruoeeeeivveeeee block decoding with sensitive hy- perparameters-Rainbooo Padding resolves the issue as an inherent property of the model, achieved through a simple change in the padding scheme. is architecture-agnostic dataset-agnostic, ro- bust to decoding strategies, and lightweight to deploy, making it a practical standard for robust instruction-tuning of dLLMs.
Our contributions are summarized follows:
We define and measure <eos> overflow-a failure mode unique to dLLMs-at at both the task and token levels, demonstrating its severe impact on diverse benchmarks.
We analyze how confidence-bacte decoding amplities padding-induced bias and show how a structured, cyclic padding scheme breaks the overflow cascade.
We propose Rainbow Padding, a cyclic multi-pad scheme that restores stable length control with minimal training overhead. We validate its effectiveness through controlled ablations, showing that as few as seven distinct padding tokens are sufficient to resolve the issue, with robustness across a variety of decoding strategies.
PRELIMINARIES
Diffusion Language Modeling. Diffusion models approximate complex data distributions via a latent variable framework that consists of a forward noising process and reverse denoising pro- cess (Sohl-Dickstein et al.] 2015) [Song et al., 2021: Ho et al., 2020). While initially proposed for continuous domains such as images, recent work has extended diffusion to the discrete setting, showing strong promise for language modeling (Austin et al. 202 Nie et al. 2025a).
Among several formulations, the dominant approach for discrete text generation is masked diffusion. Let x = (x1, I22..... ) be a sequence of length L from vocabulary 1/ augmented with a special mask token [M]. The forward process is absorbing statee Markov chain where each token can only be corrupted into [M]. Formally, for each position i, let M, E {0, be a Bernoulli indicator
such that Mj 1 if 2; is replaced by [M]. Denote M {i M, 1} as the masked indices
2

---

Published as a conference paper at ICLR 2026
and M {i: M, 0} the unmasked indices. The learning task is then to model the clean conditional distribution
(z XM) e M,
that is, to correctly guess the masked tokens given the unmasked partial sequence.
Training proceeds by first sampling corruption rate À (O, 1) and then masking each position independently. M; ~ Bern(A). The model parameters are optimized by minimizing the cross- entropy objective
((0) x,, A,
Σ
M, log po x
i=1
where 1/A normalizes for the expected fraction of masked tokens.
This masked formulation is both simple and effective: it avoids the instability of more complex transition designs, supports continuous-tin time parameterizations (Lou et al., 2024| Sahoo et al.) 2024 [hhi al. 2024] Gong et al.] 2025), and allows weight sharing across timesteps, yielding time-independent estimator (Ou et al. (025). As result, state-of-the-art open-source diffusion large language models (dLLMs) such as LLaDA (Nie et al. (025b) and Dream (Ye et al. (2025b) adopt masked diffusion their framework, typically built on Transformer encoder architec- tures (Vaswani et al., 2017] Peebles & Xiel (023). Throughout this paper, we will use this masked diffusion formulation to analyze and improve instruction-tuned dLLMMs.
Decoding Strategies Diffusion Language Models. A central advantage of dLLMs over au- toregressive models is their flexible any-order decoding: rather than being locked into left-to-right order, the model adaptively chooses which masked positions to reveal first. This adaptive decoding what gives dLLMs their potential for tasks such as planning, constraint satisfaction, or coarse-to- fine generation, but it also makes them highly sensitive to the choice of decoding policy. Because training imperfect, different unmasking orders induce different distributions and can even change
failure modes (Chang et al., 2022 ZZheng et al. 2025 Kim et al. 2025 Ye et al. 2025b).
Several simple heuristics are widely used:
Confidence: select the position with the highest peak probability.
Margin: select the position with the largest gap between the top-1 and top-2 probabilities.
Entropy: select the position with the lowest predictive entropy.
These adaptive strategies all aim to reveal "easy" tokens first and use them as anchors for harder positions. In this work we adopt the confidence- based strategy, as is the most widely used and also the most sensitive to early termination.
Diffusion Language Models. Instruction-tuninn (IT) is essential to the success of large language models (LLMs). as it aligns pretrained models with user instructions and enables strong zero-shot gen-
Instruction-tuniigg
[M]
[M]
[M]
[M]
[M]
[M]
[M]
[M]
[M]
love
[M]
[M]
[M]
[M]
[M]
[M]
eralization across downstream tasks (Wei et af.] 2022 Zhang et al., 2023). During IT, the model is fine-tuned on batches of finstruction
response) pairs of varying lengths. To enable efficient batching, shorter re-
love
sky
[M]
sponses are padded to fixed length by appending padding tokens. For autoregressive (AR) LLMs, this convention harmless: the model learns to stop at the <eos> token and excludes padding tokens from the training objective.
love
<eos>
<eos>
<eca>>
<eoa>
Figure Inference in dLLMMs: to control response length, dLLMs must learn explicit padding tokens, unlike AR models. Current dLLMs <eos> as padding.
In contrast, diffusion LLMs (dLLMs) operate on the entire fixed-length sequence at every decoding step. Here, using padding tokens can problematic: the model repeatedly observes padding tokens in trailing positions and incorporates them into both at- tention and the loss function. Current dLLMss use <eos> as padding for convenience (see Figure 2. As result, the <eos> token is heavily overexposed, leading biased termination probabilities that interact strongly with adaptive decoding strategies. This conflation of padding and termination in- troduces critical vulnerability for dl dL.,Ms, examine in detail in the following section.
3

---

Published as a conference paper at ICLR 2026
3 EARLY TERMINATION IN INSTRUCTION-TUNED DLLMS
Fixed generation length. Unlike autoregressive (AR) models, diffusion large language models (dLLMs) require fixed generation length (max.length) that must be spec- ified advance. If the allocated length too short, re- sponses may be truncated; if is longer than needed, the remaining positions are filled with padding tokens. In prin-
17.1
17.7
Accuracy (%)
15
14.6
11.0
ciple, well-trained dLLMs should remain stable as long as exx_length exceeds the minimum required length.
0.9
128
256
512
1024
Paradoxical degradation. Surprisingly, we find that the performance of instruction-tuned dLLMs degrades sharply as max length increases. As shown in Figure B per- formance of LLaDA drops substantially at 512 and col- lapses further 1024 tokens. This behavior is counterin- tuitive: many benchmark tasks require long and detailed responses, so allocating more tokens should, anything, improve quality. Moreover, lengths of 512 1024 tokens are modest by modern LLM standards, making this degra- dation particularly striking.
Max Length
tasks for LLaDA under different max length settings. Performance degrades and can collapse max max.lengt. increases.
Figure Accuracy
Avg. Response Length
120
1199
MATH
HumanEval
60
Shorter responses with longer allocation. Closer in- spection reveals that longer max.length allocations ac- tually lead to shorter responses. Figure shows that aver- age response length, measured up to the first <eos> token, decreases max_length increases. In extreme cases, models produce almost no content, collapsing into degen- erate streams of <eos> tokens (Figure 5. We refer to this phenomenon as <eos> overflow: paradoxically, allocating more space makes the model terminate earlier.
29.7
9.3
1.4
1.4
128
Max Length
Figure 4: Average response length of LLaDA under different max_length settings. Despite the larger budget, the generated responses become shorter max.length increases.
[uuetiio]] Can you make a python function for factorial using recursion?
[Answer: max_length 128] Sure, def factorial(n):
if n==1:
return 1
return n ffactorial(n-1) <eos><eos>...
[Answer: max max.lenee length 1024] Sure, 4600x<002xx0022x0000000002222222
Figure 5: Illustrative example of <eos> overflow in LLaDA. When max.length is small, the model gen- erates a coherent answer and terminates normally. As max_lengt increases, the generation increasingly collapses into repetitive <eos> tokens that occupy most remaining positions, leading output dominated by premature end markers rather than meaningful content.
Root cause: dual of <eos>. This failure arises from instruction-tuning practices. Current dLLMs reuse the same <eos> token both to mark the natural end of a response and to fill unused positions padding. This dual use introduces two issues. First, the model cannot reliably distin- guish whether given <eos> indicates a true stopping point or padding, weakening its ability to learn correct termination. Second, because training batches contain responses of varying lengths, later positions are disproportionately filled with <eos>, Under masked cross-entropy training, the model's s predictions align with empirical token frequencies:
Ex [o <eos> xx))
Pr(z, <eos>],
and since Pr[m; <eos>| + as approaches the maximum length L, the model learns exces- sively high priors for <eos> at the tail.
4

---

Published as a conference paper at ICLR 2026
Ratio
0.8
0734
Po (eos))
Conkleus
0.6
Po(x: (t (eos> Xi+l (eos))
0.5-
Pe(x: ( (eos) Mii5 (eos)
0.4
po(n= (eos)
x++10 (eon)
0.4
0.2
0.0
800
256
512
1024
Token Position
Max Length
(a) <eos> confidence at each position with max_length 1024. <eos> later positions create cascading bias affecting earlier positions.
(b) Average <eos> token ratio among the first 50% unmasked tokens across different
max_iength using confidence- based decoding.
Figure 6: Excessive <ecs> generation in LLaDA MATH.
<eos> overflow. This bias is further amplified by adaptive decoding. Figure 6a shows that the probability of predicting <eos> rises steeply toward the sequence end, approaching 1.0 even before generation begins. Once tail position is sampled as <eos> due to this high probability, earlier positions are also biased toward the same outcome, as quantified by
po(:; <eos>
Titk <eos>),
which increases sharply even for positions 10 tokens earlier. This cascading effect, <eos> over- slow, causes termination probabilities propagate backward from the tail, ultimately collapsing the response. Figure [bb confirms this dynamic: longer max_length allocations lead to substantially higher fractions of <eas> tokens unmasked within the first 50% of decoding steps, preventing the generation of richer content.
Heuristic fixes. Although excessive <eos> generation at longer lengths has not been formally analyzed, several works have attempted to mitigate with hoc strategies. For example, Zhu et al. (2025) manually suppressed <eos> probabilities during decoding and reported modest gains. However, dampening confidence in this token often causes the model to overshoot the true response length, leading to failed termination and repetitive outputs (e.g., solving the same problem multiple times), which degrades quality (re- fer to Appendix D.2 for details).
MATH
HumanFyyal
40
Accuracy
21.9
20
11.0
1.9
Number of Blocks
Another approach, adopted by LLaDA, is block-wise de- coding in a semi-autoregreesivv manner. Here the sequence is partitioned into contiguous blocks, and later blocks re- main masked until earlier ones are fully generated. This prevents premature <eos> predictions at the tail but the cost of enforcing sequential sched- ule, introducing mismatch between training and inference. The restriction undermines a core strength of diffusion models-the ability to unmask tokens in arbitrary order with bidirectional con- teic-which is critical for tasks requiring multi-step reasoning or subgoal planning, where dLLMs often outperform AR models Ye et al., 2025a Kim et al. 2025 Ye et al. 2025b).
Figure Performance of LLaDA with different block numbers during semi-autoregressive block decoding at max length 1024. 1024.
Block-wiss decoding also introduces sensitive hyperparameter: block number. In practice differ- block numbers are chosen across benchmarks without principled rule, and performance varies substantially with this choice (Figure (7). This sensitivity highlights both the instability of heuristic fixes and the need for fundamental solution <eos> overflow.
RAINBOW PADDING: A SIMPLE REMEDY FOR EARLY TERMINATION
In the previous section, explain that excessive <eas> tokens sequence ends during instruction- tuning cause <eos> overflow. Replacing <eos> in padding regions with a single <pad> token is insufficient, as it reintroduces the same concentration of probability mass that causes overflow.

---

Published as a conference paper at ICLR 2026
<RAINIIGG PHASE>
MATH PROBLEM
Prompt Who said that the arc of history is long, but trends toward justice?
What the positive difference between
<EOS> PADDING
120% and $130%S 20?
Martin Luther King
<eog? <eos?
<eog>
<EOS> PADDING
PADDING (OURS)
RAINBOW PADDING
The <eos>
have that $2*8=256$,
Martin
Luther
King
<eos>
<pad,? <pad,
<e08><e087...
the remainder is \boxed(1).
<eos><pad,><pad
Cyclic-7 Padding
Figure 8: Overview of Rainbow Padding. (Left) During training, distinct <pad> tokens arranged in cyclic pattern, contrasting with current dLLMs that use <eos> padding. (Right) Models trained with <eos> padding suffer from <cos> overflow, which Rainbow Padding resolves.
Design. We introduce Rainbow Padding to resolve this issue. As shown in Figure the true end of response is marked with a single <eos> token, while all remaining padding positions are filled with cyclic sequence from a dedicated of K distinct padding tokens:
P {<pado>, <padi>..... <padk-1>).
Intuition. The intuition behind Rainbow Padding is simple. By reserving <eos> exclusively for genuine sequence termination, the model avoids learning inflated <eos> probabilities from padding usage. This corrects the biased prior so that <eos> probabilities reflect only authentic termination events. prevents probability mass from concentrating but sparsely, so the model learns them as low-probability placeholders rather than high-confidence guesses. Together, these effects suppress the dominance of <eos> while still providing a clear and predictable structure that signals sequence length.
the same time, distributing the padding region across distinct tokens a single symbol. Each <pad,,> appears regularly
Stabilized sampling dynamics. By reducing individual padding token confidence, Rainbow Padding reshapes the decoding process. Content tokens gain relatively higher probability and are revealed earlier under adaptive strategies such confidence-based decoding. This encourages the model to establish the meaningful content first, providing coherent context for subsequent reason- ing. Consequently, the <eos> token emerges semantically appropriate point-as the natural conclusion of content- than as an early, High-probbbbiliy guess.
Why cyclic, not random. One might consider randomly sampling each padding token from uniform distribution as an alternative way to distribute probability mass. However, this creates challenging stochastic prediction task that diverts model capacity from instruction following. We observe that models fail to learn appropriate padding placement under this random scheme.
In contrast, Rainbow Padding adopts simple deterministic cycle that easy to learn, allowing the model master the padding region with minimal effort. This preserves model capacity for learning instruction-respons pairs while eliminating the root cause of <eos> overflow. We discuss the expected properties of Rainbow Padding in detail in Section
5 EVALUATING RAINBOW PADDING
Experimental setup. We compare Rainbow Padding with standard <eos> padding under con- trolled conditions by performing supervised fine-tuning on pretrained LLaDA-Base and Dream- Base. Models trained with <eos> padding serve as baselines that replicate current instruction-tuned behavior. Following Dream's recipe, we combine Tulu3 (Lambert et al.) 2024) and SmoIL.M (AI- lal et al.) 2025), randomly sampling 0.5M examples. All models are fine-tuned with LoRA (Hu ct al., 2022) for three epochs under identical configurations, differing only in the padding strategy. Rainbow Padding uses seven distinct padding tokens in a deterministic cyclic pattern.
Evaluation spans two categories. First, for max_ length h robustness, we use length-sensitin rea- soning and coding tasks: MATH-500 (Lightman al. 2023), GSM8K (Cobbe et al. 2021). and HumanEval (Chen et al. (02T). For generalization, we use MCQ benchmarks (MMLU (Hendrycks et al. 2021), HellaSwag (Zellers et al. 2019)) with en = 3, following the LLLDD setting.

---

Published as a conference paper at ICLR 2026
MATH
GSM8K
HumanEval
MMLU
HelloSwag
Method
#Blocks
Acc.
len_length
Acc.
length
Acc.
res.length
Acc.
Acc.
32.6
282.1
75.5
125.2
129.3
LLaDA-Bas
Rainbow Padding 4
32.8
282.5
74.7
125.7
40.2
129.4
65.3
61.3
16
32.8
280.8
75.7
126.9
39.6
130.7
0.98
13.2
8.7
20.7
13.7
Padding
13.9
64.8
62.5
16
29.8
292.5
72.2
124.4
41.5
100.8
Table 1:
Performance of LLaDDA-Base after instruction-tuning with <eas> and Rainhow Padding max length 1024, except for MMLU, HellaSwag. '#Blocks' denotes the number of equal partitions used for autorrrsrreer block decoding; #Blocks corresponds to standard decoding without blocks.
MATH
GSM8K
HumanEval
MMLU
HellaSwag
Method
mux Jength
Acc.
res. res.length
Acc.
lesgthh
Acc.
res.length
Acc.
Acc.
Dream Dreaa-Basse Base
1024
34.3
942.0
77.3
142.8
48.8
130.2
Rainhow Padding
65.3
70.9
512
36.2
470.4
76.5
108.3
48.2
108.0
1024
0.0
0.1
9.1
3.1
22,6
10.6
<eos> Padding
512
2.9
0.4
9.1
2.5
24.4
10.9
Table 2: Performance of Dream-Base after iustruction-tuning with <eas? and Rainbow Padding under differ- ent max. max_length settings.
All experiments employ deterministic confidence-bere decoding [Chang et al.) 2022) without tem- perature. Block-wiss wise remi-autoregressiv decoding is included only for LLaDA (where it is natively
implemented) and disabled elsewhere. Further details are provided in Appendix C
In the results tables, res_length denotes the valid response length before the first <eos>, while max hength is the allocated sequence length.
Performance comparison. Table shows that Rainbow Padding consistently outperforms <eos> padding across all benchmarks when generating 1024-token sequences without block-wise decoding (#Blocks 1). The baseline exhibits early termination, producing significantly shorter responses, whereas Rainbow Padding restores length robustness and yields higher task accuracy - for example, 32.6% 0.6% MATH.
Under semi-autoregresssvve block decoding (#Blocks > 1), Rainbow Padding maintains stable ac- curacy across block numbers, while <eos> padding remains highly cnsifivee-pefformance drops sharply as the block number decreases. This highlights the brittleness of block-wise heuristics and shows that they become unnecessary once padding calibrated correctly.
Table 2 presents analogous results for Dream. Rainbow Padding consistently achieves robust per- formance across max_length settings, while <eos> padding underperforms. On general-purpose benchmarks (MMLU, HellaSwag), Rainbow Padding matches or slightly surpasses <eos> padding across both models, confirming that learning the cyclic pattern imposes minimal overhead while delivering strong gains on length-sensitive tasks.
6 ANALYSIS OF RAINBOW PADDING
In this section, we provide additional experiments to validate the key properties of Rainbow Padding discussed in Section
Decoding Behavior. Figure 9a shows that the average maximum confidence among padding to- kens decreases dramatically with Rainbow Padding compared <eos> padding. Across diverse examples, the maximum probability assigned to any padding token never exceeds 0.2 the ini- tial decoding step, confirming consistently now-confidence predictions that reduce the likelihood of premature padding selection.
As a result, decoding unfolds more naturally with Rainbow Padding: as shown in Figure 9b the model first generates meaningful content and only later fills the padded tail. In contrast, <eos> padding tends to produce <eos> tokens at later positions early the process, pushing the genera- tion of earlier content tokens the end and causing the cascade that leads to early termination.

---

Published as a conference paper at ICLR 2026
Eusiee Stop
Step 1101
Paukling
0.8
200
400
600
500
1000
Confidenes
0.6
<eoa> Padding (<e0s>)
Rainhow Padding
Rainbow Padding (Max <pad>)
0.4
200
600
800
1000
0.2
0.0
0
200
4003
600
800
1000
Token Position
Token Position
Farher Step
Later Step 150
(a) Average model confidence on padding to- kens at each sequence position during initial step. For Rainbow Padding, the red line shows the average maximum probability among <padg> through <pads> at each position.
(b) Average token decoding order under confidence- decoding. Darker positions indicate earlier decoding, showing content-first generation with Rainbow Padding. Bottom panel: detailed view (1-150), showing non-sequential decoding characteristic of diffusion models.
Figure Analysis of Rainbow Padding effects on model behavior using LLaDA-Base fine-tuned with <eos> or Rainbow Padding on GSM8K with max.length. 1024.
Universality across Decoding Strategies. Figure I0 shows
100
that Rainbow Padding yields stable performance across dif- ferent decoding strategies -marinn-bsed (Kim et al.) 2025)
margin
conf
entropy
90
Accurey (%)
and entropy-basee (Ye al., 2025b) performing similarly
60
to the confidence- based strategy (Chang et al. (2022) (our de-
fault setting), demonstrating that our method generalizes ro-
40
bustly across diverse unmasking strategies.
20
For confidence-based decoding, the benefit of Rainbow Padding is straightforward: directly lowering individual padding token confidence by distributing probability mass across tokens. This advantage extends to other decoding strategies (e.g.. margin-based or entropy-basedd as the cyclic pattern of Rainbow Padding prevents any single padding to- ken from dominating (Figure Da). These evenly distributed probabilities reduce probability gaps (creating low margins) and induce uncertainty (high entropy) among padding tokens, making padding positions less likely be unmasked early.
MATH
GSMSK
HumanEval
Figure 10: Performance of LLaDA- Base fine-tuned with Rainbow Padding under different decoding strategics at max max_ lnngt 1024.
Loss <pad>
15
Loss
Efficient Adaptation to Instruction-Tunee Models. Learning distinct padding tokens in Rainbow Padding may impose greater complexity than learning <eos> padding. We quantify this overhead by tracking training loss on padding regions during LLaDA-Base fine-tuning. While Rainbow Padding exhibits high padding loss initially, it converges to zero within 5% of an epoch (Figure []]. The model rapidly adapts the deterministic cyclic pattern.
10
0.00
0.05
0.10
0.15
0.20
Epoch
Figure 11: Training loss padding tokens quickly drops near zero.
This low learning complexity enables deployment of Rainbow Padding beyond instruction-tuning from scratch. The method can efficiently adapt existing instruction-tuna dLLMs trained with <eos> padding. To demonstrate this, we fine-tune LLaDA and Dream using LoRA for single epoch on the 0.5M dataset from Section 5) Our adaptation lightweight, requiring approximately six GPU-hours two H200 GPUs, compared to the original instruction-tuning which used 4.5M (LLaDA) and 1.8M (Dream) examples over 3 epochs. See Appendix [ experimental details.
that minimal adaptation effectively resolves early termination, consistently producing longer outputs with substantial performance gains. MATH accuracy improves dramatically from near 0% to 20% for both models, while L.LDDA achieves 73.8% accuracy GSM8K compared to 3% performance before adaptation. These results demonstrate that Rainbow Padding integrates into existing dLLMS with minimal burden while delivering significant performance improvements.
8

---

Published as a conference paper at ICLR 2026
MATH
GSMSK
HvmanEval
Method
Acc.
res_ength
Acc.
res_eength
Acc.
res_lentth
Vanilla
0.1
1.4
3.0
1.2
11.0
7.5
LLaDA
+Rainbow Padding
28.6
252.9
73.8
122.5
38.4
338.1
Vanilla
0.0
1.8
60.6
91.6
24.4
16.6
Dream
+Rainbow Padding
32.4
984.4
77.3
120.1
47.6
70.3
Table 3: Performance of LLaDA and Dream fine-tuned with Rainbow Padding. single-epoch LoRA adapta- tion effectively mitigates early lermination and yields signifieant accuracy gains.
Effect of the Number of Distinct Padding Tokens. While our main results use seven distinct padding tokens (<padh> through
#Pad
MATH
GSM8K
HumanEval
MMLU
1
21.9
15.9
39.0
64.4
3
33.3
58.3
37.8
62.8
<padg>), Rainbow Padding can be configured with varying numbers of token types in the cy- cle. Increasing the number of token types dis- tributes probability mass more evenly within the padding vocabulary, reducing the likelihood of early termination. However, this also in- creases the learning burden, as the model needs to distinguish among a larger set of tokens.
7
34.3
79.6
40.2
65.2
20
36.2
76.5
36.0
65.9
Table Accuracy of fine-tuned LLaDA -Base with Rainbow Padding using varying numbers distinet padding tokens. Fewer tokens (e.g.. 1-3) result de- graded performance, while using sufficient number yields comparable performance.
Table 4 shows that increasing the number of padding tokens improves performance across tasks. Using only three tokens proves insufficient fully mitigate early termination, resulting in degraded performance (e.g.. 58.3% on GSM8K). Beyond seven tokens, however, gains 20 tokens yields no further clear benefit. This suggests that using seven tokens strikes a good bal- ance between effectiveness and learning cost. Importantly, general- general-purpose purpose performance remains stable across all con- figurations, as evidenced by consistent MMLU scores.
K-7
0.5
0.3
0.2
0.1
GSMSK
Humambvaal
To understand why a modest padding-vocabulary size (K)
Lowest-confidence content- token distribution during the first 20 de- coding steps for
Figure 12:
sufficient, compare the expected padding-toker prob- ability (== 1/K) with the confidence of meaningful tokens during early decoding. Early padding is avoided as long 1/K remains below the confidence of content tokens. We analyze this by inspecting the lowest-confidence unmasked tokens over the first 20 decoding steps of I.L with <eos> masked restrict the analysis to content tokens.
As shown in Figure 12 when using seven or more padding tokens, the expected probability per padding token (1/K) falls well below content token confidences, preventing early selection. With only three tokens, however, this probability overlaps with content token confidences. This explains why very small padding vocabularies cannot fully avoid early termination (as in Table 4 whereas any modest vocabularies (seven or more) suffice, and larger values yield only diminishing returns.
7 CONCLUSION
We identified <eos> overflow critical failure mode in instruction-tunet diffusion LLMs: al- locating longer generation lengths paradoxically leads to early termination. This problem arises from the dual use of <eos> as both terminator and padding token, which inflates its probability and destabilizes decoding, To resolve this, we introduced Rainbow Padding, simple strategy that reserves a single <eos> for true termination and fills remaining positions with a cyclic sequence of distinct padding tokens. This design decouples termination from padding, spreads probability mass across multiple tokens, and prevents collapse into premature <eos> predictions. Experiments show that Rainbow Padding eliminates early termination, substantially improves reasoning and code generation performance, and integrates efficiently into existing models such as LLaDA and Dream with minimal fine-tuning. Overall, Rainbow Padding provides lightweight and practical fix to fundamental flaw, suggesting a new standard for instruction -onniiinionnnnnnggggg of dLLMs and reinforcing their potential robust alternative autoregressive models.

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
This work was supported in part by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. RS-2024-00457882, Al Research Hub Project), and the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. RS-2024-00410005, RS-2025-23525649,
REFERENCES
Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Martín Blázquez, Guilherme Penedo, Lewis Tunstall, Andrés Marafioti, Hynek Kydlíček, Agustín Piqueres Lajarin, Vaibhav Srivastav, et al. Smollm2: When smol goes Dig-data-centrie training of small language model. arXiv preprint 0727:7250..02737, 2025.
Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models discrete state-spaces. In NeurlPS, 2021.
Heli Ben-Hamu, Itai Gat, Daniel Severo, Niklas Nolte, and Brian Karrer. Accelerated sampling from masked diffusion models via entropy bounded unmasking. NeurIPS, 2025.
Andrew Campbell, Jason Yim, Regina Barzilay, Tom Rainforth, and Tommi Jaakkola. Generative flows on discrete state-spaces: Enabling multimodal flows with applications to protein co-design. In ICML, 2024.
Huiwen Chang, Han Zhang. Lu Jiang, Ce Liu, and William Freeman. Maskgit: Masked generative image transformer. In ICCV, 2022.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 2021.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Rejichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110. 14160...411.... 2021.
Google DeepMind. Gemini diffusioo,, 2025. URL
http:://blog.goggleeecchoo//
Itai Gat, Tal Remez, Neta Shaul, Felix Kreuk, Ricky TQ Chen, Gabriel Synnaeve, Yossi Adi, and Yaron Lipman. Discrete flow matching. NeurlPS, 2024.
Shansan Gong, Shivam Agarwal, Yizhe Zhang, Jiacheng Ye, Lin Zheng, Mukai Li, Chenxin An, Peilin Zhao, Wei Bi, Jiawei Han, Hao Peng, and Lingpeng Kong. Scaling diffusion language models via adaptation from autoregressive models. In ICLR, 2025.
Shansan Gong, Ruixiang Zhang, Huangjie Zheng, Jiatao Gu, Navdeep Jaitly, Lingpeng Kong, and Yizhe Zhang. Diffucoder: Understanding and improving masked diffusion models for code gen- eration. In ICLR, 2026.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In ICLR, 2021.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS, 2020.
Edward J Hu, Yelong Shen. Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In ICLR, 2022.
Xiangqi Jin, Yuxuan Wang, Yifeng Gao, Zichen Wen, Biqing Qi, Dongrui Liu, and Linfeng Zhang. Thinking inside the mask: In-place prompting in diffusion llms. arXiv preprint arXiv:2508, 10736, 2025.
10

---

Published as a conference paper at ICLR 2026
Jaeyeon Kim, Kulin Shah, Vasilis Kontonis, Sham M. Kakade, and Sitan Chen. Train for the worst, plan for the best: Understanding token ordering in masked diffusions. In JCML, 2025.
Jaeyeon Kim, Lee Cheuk-Ki Kit, Carles Domingo-Enrich, Yilun Du, Sham Kakade, Timothy Ngo- tiaoco, Sitan Chen, and Michael Albergo. Any-order flexible length masked diffusion. In ICLR, 2026.
Inception Labs, Samar Khanna, Siddhant Kharbanda, Shufan Li, Harshit Varma, Eric Wang. Sawyer Birnbaum, Ziyang Luo, Yanis Miraoui, Akash Palrecha, et al. Mereury: Ultra-fast language models based on diffusion. arXiv preprint arXiv:2506.17298 2025.
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brah- man, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, al. Tulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:24/ 1. 15144, 2024.
Jinsong Li, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Jiaqi Wang, and Dahua Lin. Beyond fixed: Variahle-length denoising for diffusion large language models. ICLR, 2026.
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In ICLR, 2023,
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution. In ICMI., 2024.
Long Ma, Fangwei Zhong, and Yizhou Wang. Reinforced context order recovery for adaptive rea- soning and planning. arXiv preprint arXiv:2508. 13070, 2025.
Shen Nie, Fengqi Zhu, Chao Du, Tianyu Pang. Qian Liu, Guangtao Zeng, Min Lin, and Chongxuan Li. Scaling up masked diffusion models on text. In ICLR, 2025a.
Shen Nic, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, JUN ZHOU, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large language diffusion models. In NeurlPS, 2025b.
Jingyang Ou, Shen Nie. Kaiwen Xue, Fengqi Zhu, Jiacheng Sun, Zhenguo Li, and Chongxuan Li. Your absorbing discrete diffusion secretly models the conditional distributions of clean data. In ICLR, 2025.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In ICCV, 2023.
Fred Zhangzhi Peng. Zachary Bezemek, Sawan Patel, Jarrid Rector-Brooks Sherwood Yao, Alexan- der Tong, and Pranam Chatterjee. Path planning for masked diffusion models with applications to biological sequence generation. In ICLR 2025 Workshop on Deep Generative Model in Machine Learning: Theory, Principle and Efficacy, 2025.
Subham Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and effective masked diffusion language models. In NeurIPS, 2024.
Jiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis Titsias. Simplified and general- ized masked diffusion for discrete data. In NeurIPS, 2024.
Jascha Sohi-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In ICML, 2015.
Yang Song. Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In ICLR, 2021.
Yuxuan Song, Zheng Zhang, Cheng Luo, Pengyang Gao, Fan Xia, Hao Luo, Zheng Li, Yuchang Yang, Hongli Yu, Xingwei Qu, et al. Seed diffuuion: A large-scale diffusion language model with high-speed inference. arXiv preprint arXiv:2508.00 02.8002933 2025.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, 2017.
11

---

Published as a conference paper at ICLR 2026
Zhe Wang, Jiaxin Shi, Nicolas Heess, Arthur Gretton, and Michalis Titsias. Learning gressive models with application to molecular graph generation. In ICML, 2025.
carnning-order order autore-
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In ICLR, 2022.
Zirui Wu, Lin Zheng, Zhihui Xie, Jiacheng Ye, Jiahui Gao, Yansong Feng, Zhenguo Li, Victoria W., Guorui Zhou, and Lingpeng Kong. Dreamon: Diffusion language models for code infill- ing beyond fixed-size canvas, 2025. URL htttps://hkunlp: http///kullp....ttubpgpp.eg wtt.g......boog...ooootoootttttt 1o/bblg//lll//5// dreamon
Zhihui Xie, Jiacheng Ye, Lin Zheng, Jiahui Gao, Jingwei Dong, Zirui Wu, Xueliang Zhao, Shansan Gong, Xin Jiang, Zhenguo Li, et al. Dream-coder 7b: An open diffusion language model for code. arXiv preprint arXiv:2509.01 142, 2025.
Jiacheng Ye, Jiahui Gao, Shansan Gong, Lin Zheng, Xin Jiang, Zhenguo Li, and Lingpeng Kong. Beyond autoregression: Discrete diffusion for complex reasoning and planning. ICLR, 2025a.
Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong. Dream 7b: Diffusion large language models. arXiv preprint arXiv:2508.15487 2025b.
Rumpeng Yu, Li, and Xinchao Wang. Discrete diffusion in large language and multimodal models: A survey. arXiv preprint arXiv:2506. 13759, 2025.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- chine really finish your sentence? In ACL, 2019.
Andrew Zhang, Anushka Sivakumar, Chiawei Tang, and Chris Thomas. Flexible-length text infilling for discrete diffusion models. In EMNLP, 2025.
Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, et al. Instruction tuning for large language models: A survey, arXiv preprint arxv::2308. 10792, 2023.
Kaiwen Zheng, Yongxin Chen, Hanzi Mao, Ming-Yt Yu Liu, Jun Zhu, and Qinsheng Zhang. Masked diffusion models are secretly time-agnostic masked models and exploit inaccurate categorical sampling. In ICLR, 2025.
Fengqi Zhu, Rongzhen Wang, Shen Nie, Xiaolu Zhang, Chunwei Wu, Jun Hu, Jun Zhou, Jianfei Chen, Yankai Lin, Ji-Rong Wen, et al. Llada 1.5: Variance-reduced preference optimization for large language diffusion models. arXiv preprint 1922, 2025.
12

---

Published as a conference paper at ICLR 2026
A RELATED WORKS
Masked Diffusion LLMs. Masked Diffusion Models (MDMs) have emerged as a prominent and models, using masking kernels. This approach provides a simple and principled training framework (Sahoo et al. 2024 Shi et al.] 2024). Also,
high-performing approach within
MDMs scale effectively across various tasks, with successful applications large-scale, such as
language (Nie et al.] 2025a b; Ye et al., 2025b Song et al. 2025 Labs et al. 2025 DeepMind 2025
and code (Wu al. 2025 Xie et al.
2025 Gong et 2026
A key strength of dLLMs is their capacity for any-order inference, where tokens can be unmasked in arbitrary orders rather than following a fixed schedule (Kim et al. 2025 Peng et al. (023). This flexibility is theoretically grounded in the underlying continuous-time Markov chain (CTMC) or flow-matching frameworks (Campbell et al.) 2024
Any-Order Inference in Difftusion Language Models.
Gat et al. 2024). In practice, a spectrum of probabilistic strategies that guide the decoding order based on the model confidence metrics have been introduced, such as maximum probability (Chang et al. 2022). probability margin (Kim al.) 2025]. and token entropy (Ye et al., (025bb) [enn- Hamu et al. 2025). Model confidence is used not only for token ordering but also for optimizing the entire inference process. For example, J al. (2025) use the model's confidence in the final answer to implement an early exit from the reasoning phase to speed up generation. Recent research
has also explored directly learning the generation order (Ma al. 2025 Wang et al. 2025).
Length Control. Adaptive control of response length is an important capability for practical use of dLLMs. and it can be achieved in various ways (Yu et al. (2025) Training-based approaches modify the model's architecture or objective. Zhang et al. (2025) enable variable-length generation by denoising continuous token positions alongside token values. (Wu et al. (2025) introduce special tokens like <| <xppand]]>> >> and < delete >> to dynamically adjust sequence length, while Kim
ct al. (2026) use an auxiliary network to predict the expected number of token insertions. In contrast, Li et al. (2266) employ training-free methods that adapt the length during inference by monitoring the model's confidence in the <eos> and expanding the canvas when confidence is low. However, these approaches do not specifically address crucial artifact that emerges during instruction-tuning of dL..Ms the artificial inflation of the <eas> probability. This issue results from the common practice of padding shorter sequences in instruction datasets with numerous <eos>. Our work is the first to isolate this instruction-tuning -uning-specific problem and propose a targeted solution to recalibrate the model's output distribution, thereby resolving the excessive <eos> generation its source.
B DETAILS FOR THE EXTRA ANALYSIS
B.I UNIVERSALITY ACROSS DECODING STRATEGIES
In Section 6 we showed that Rainbow Padding effectively reduces <eos> overflow regardless of the decoding strategy. These probabilistic heuristics have been proposed guide the decoding order. They can be formalized as follows. Given model-predicted probability po (I;; XM)), the position i' decode is determined as:
Confidence: i' = arg max max ma [ xxx]
Margin: arg max
[po = vi |xxa) - po(a:: = 02 x
Entropy: = arg min (po [xM)]],
where (v1, U2)) = arg Top: pa pa = v xM) and H(p) = - p(z) log)
The token for transfer is determined by selecting the token with the highest probability at that position. In existing dLLMs such as LLaDA and Dream, randomness during decoding can be in- troduced through distinct mechanisms. In LLaDA, we can perturb token logits with Gumbel noise as in (Zheng et al. 2025). In Dream, we can employ top-p. top-k, and temperature sampling: the token distribution first truncated by top-p top-k filtering, then temperature is used to control the sharpness. When temperature parameter is used, tokens are selected probabilistically by categorical sampling rather than max probability scheme. In both cases, stochasticity arises from
13

---

Published as a conference paper at ICLR 2026
either the injected noise probabilistic sampling. However, experiments do not rely these randomness inducing strategies order to isolate the effects our proposed method without the confounding influence of sampling heuristics.
C
EXPERIMENTAL DETAILS
C.1 SETUP FOR TRAINING
Rainbow Padding is designed for instruction-tuning pretrained models. In our main experiments (Table [2. we fine-tune pretrained models (LLaDA-Base, Dream-Base) directly. To demon- strate efficient adaptation to existing Instruction-tunee models trained with <eos padding, we also fine-tune instruct models (LLaDA, Dream) with Rainbow Padding as shown in Table 3
Both experiments use LoRA (Hu et al. 2022) with rank 32, applying LoRA to all linear layers without bias terms. Base model training requires 3 epochs. while instruction model adaptation requires only epoch. We use the Adam W optimizer (Loshchilov & Hutter] (219)) with learning- rate 5e-5 and batch size 48. All experiments use identical training data detailed below.
Dataset. We combine datasets from Tulu3 (Labbert et al.) 2024 and SmolLM2 (Allal al. following Dream's 's cnstruction-tuning recipe. We curate this dataset by filtering out extremely long sequences (> 4096 tokens) and multi-turn conversations, then randomly sample 0.5M examples.
2025
Pad token configurations. Since the <pad,> > tokens required for Rainbow Padding are not in- cluded in LLaDA and Dream vocabularies, we need to specify these tokens explicitly. Rather than expanding the vocabulary with new tokens, we select extremely rare existing tokens (e.g., sequences of signs like GG9G.. with > 60 repetitions) that rarely appear in conversations or tasks. We assign up to 20 distinct <pad> tokens for both models.
Hardware utilization. All the experiments utilize H200 GPUs. Training took approximately hours in total on two H200 GPUs.
C.2 SETUP FOR EVALUATION
For evaluation, we use the official Im-eval implementation with confidence-based decoding in de- terministic mode, setting all stochastic hyperparameters (e.g., temperature) zero. Unless otherwise specified, semi-autoregressive block decoding is disabled. For LLaDA, MATH-500 and GSM8K are evaluated on the full official test sets, whereas Dream is evaluated on subset splits due to time and computational budget constraints for long-sequence evaluation. The performance difference be- tween subset and full-set results for LLaDA is negligible. We therefore expect Dream to exhibit similar behavior under full-set evaluation.
Dataset.
We assess five tasks: MATH-500, GSM8K, HumanEval, MMLU, and HellaSwag.
Table 5: List of external models and datasets with corresponding sources, links, and licenses.
Asset
Source
Access
License
LLaDA
Nie et 2025h
Link
MIT License
Dream
et al. 2025b)
Link
Apache License 2.0
MMLU
Hendrycks ct al.
2021
Link
MIT License
GSM8K
Cobbe et al. 202
Link
MIT License
MATH-500
Lightman et al. 2023
Link
MIT License
HumanEval
Chen et al. 202
Link
MIT License
14

---

Published as a conference paper at ICLR 2026
D MORE ILLUSTRATIVE EXAMPLES
Here, we present some illustrative examples including actual prompt-response pairs.
D.1 FAILURE EXAMPLES
In Section we discovered that the max length significantly impacts the performance of instruction-tunee models. The examples are presented in Figure 13 16
D.2 HEURISTIC FIx: CONFIDENCE LOWERING
Simply lowering <eas> confidence is a symptomatic patch, not a fix. While this approach appears similar to Rainbow Padding in reducing premature terminations, naive logit penalties distort the modeled distribution and eliminate the learned length signal carried by <eos>, This often leads multiple gen model avoids termination when it should stop, producing second or third answer. Illustrative examples are shown in Figure 17
D.3 RAINBOW PADDING
By contrast, Rainbow Padding separates padding from termination and preserves the model's dis- tributional semantics after brief fine-tuning, preventing overflow without harming length control.
The examples are presented in Figures 18 21
D.4 RAINBOW PADDING TO INSTRUCTION-TUNED MODELS
We applied Rainbow Padding to already instruction-tuned models with single-epoch fine-tuning. yields significant improvement even with single epoch. The examples are in Figures 22 25
E
LLM USAGE
LLMs were used for editorial purposes in this manuscript, limited to rewriting and polishing human- written text for clarity, grammar, and flow. All content, ideas, analyses, and results are original and were developed entirely by the authors. The authors carefully reviewed all LLM outputs to ensure accuracy and originality.
15

---

Published as a conference paper at ICLR 2026
Question
Problem: What is the positive difference between 120% of 30 and 130% of 20? Solution:
Answer: Max length 128
First, we calculate 120% of 30:
120% x 30 36.
Next, we calculate 130% of 20:
130% 1.30 x 20 = 26.
Now, we find the positive difference between these two values:
36 26 10
Final Answer: The final answer is 10. hope it <orect.<eos><<es>.....
Answer: Max length 1024
Figure LLaDA-Instrn on MATH with 3-shot prompt. At max length 1024, the model produces an incorrect answer most remaining positions with <eas>.
Question
Problem: A eircle of radius 2 is inscribed in semicircle, shown. The area inside the semicircle but outside the circle is shaded. What fraction of the semicircle's 's area is shaded?
Answer: Max length 128
The area of the semicirele is (4)² = 8T. The area of the circle is T()) = 4T. The area inside the semicircle but outside the circle is 8 4. 4m. The fraction of the semicircle's
area that is shaded is 8m 2
The answer is: & <eos><ees>...
Answer: Max length 1024
The xxxexxxxxxxxxxxxxxxxxxxxxxxxxx20
Figure 14: Dream-Instruct on MATH with a 3-shot prompt. At max length 1024, the model fills most available positions with <eos>,
16

---

Published as a conference paper at ICLR 2026
Question
Write solution the following problem and make sure that passes the tests:
dof
Strongost_Extensioog (class_name cxtensions):
You will be given the name of class (a string) and list of extensions.
The extensions are to be used to load additional classes to the class. The strength of the extension is follows:
Example: for Erronnest__xtenssie ( (lmy_class', ('AA,, 'Be', ccc 1) my class.AAAA
Here the completed function:
Answer: Max length 128
def get www.ssttetgtheettt (ext):
cap sum (1 for in ext if isssuperr ()) em sum for ext if c.islover ())
return cap sm
max max_strength = -1 strongest_ext None
for i, ext in enumerate (extensicns):
10
strength = get get_strength (ext)
if strength > max_strength or strength max_strength: rax_strength strength
12
13
strongest_ext ext
14
return E' (class name [strongest_ext)
<ecs><ees>...
Answer: Max length 1024
on HumanFval (code generation). As max length increases, the model increasingly generates and fails produce code.
Figure 15:
17

---

Published as a conference paper at ICLR 2026
Question
Write solution the following problem and make sure that passes the tests:
def (
Implement the Function f that takes as parameter, and returns 1ist of size n, such that the value of the element index
the factorial of if even or the sum of numbers from to i otherwise. i starts from 1.
the factorial of i is the multiplication of the numbers from 1 to i 2 ... i). Example:
((5) = [1, 2, 6, 24, 15]
10
Here the completed function:
Answer: Max length 128
def factorial (n)
н** Calculate the factorial of "..
if 0:
return
else:
return factorial (n-1)
(n-1)
def sum(n) (n)
н** Calculate the sum of numbers from to n if -- 0:
10
return
else:
13
return sum (n-1) +
14
result []
for 1 in range (1, n+1);
17
if sesult.appent (factorial (1))
18
19
else:
result. append (sum )1))
return result
22
<eos><eos>...
Answer: Max length 1024
result []
for i in range (n): if i 8 2 == 0:
result append (factorial (1)
clse:
seuult.pppedd (sum (range (1, i+1)
return result
<eos><eos>
Figure 16: Dream-Instruct on HumanEval (code generation). At max length 1024, the model fails to generate complete solution.
18

---

Published as a conference paper at ICLR 2026
Question
Problem: A line has slope 3. Which of the following vectors possible direction vectors for the line?
pair x (3,0), y (0.2);
(A):
(B):
x x
(C):
2 +y
2
(D):
3 +y
5
(B):
(0,0)
-2
(F): 2/5
.x
(40)
(G):
,*xx
16)
4
(H):
33x
10
Answer:
Max length 1024 + EOS confidence lowering
5 2
The slope of the line is Z. so the direction vector of the line is
Therefore, the correct
options are B, and Final Answer: The final answer is B, G. H. hope correct.
Problem: Let f be function such that f(z) Find f(2). Solution: Since f(x) +
have f(2) = 1
1
1
Final Answer: The final answer is $. I hope it is correct. Problem:
2
Let f(x) = Find ʃ(2). Solution: Since f()) 1, we have f(2)
Final
Answer: The final answer is }. I hope it is correct.
Figure 17: LLaDA-Instrue on MATH with a 3-shot prompt. When applying the <eos> Ceos>-confidenee-loveri
heuristic (as in LLaDA 1.5), the model avoids termination and fills the remaining budget with unrelated content tokens.
19

---

Published as a conference paper at ICLR 2026
Question
Problem:
Find the positive base which the equation 5 25g 137, valid.
Solution:
Answer: Max length 1024 + Rainhow Padding
Write 25g in base as 26 + 5 and 137, in base as b² 3b 7. Then the equation becomes
5(26 + + b² 36 7.
Expanding both sides gives
10b + = = 6² 3 +
Rearranging gives
b 7h 18 = 0.
Factoring gives
(b 2) =
Thus, = 9 or b = 2. Since must be positive, the answer = 9 Final Answer: The final answer is 9. I hope it is correct. <eos><>pdo><>>>>> > <pada><ppdd>>>>>>
Answer: Max length 1024 EOS padding
<ec>><<eo>>ces>>>>e>>>ees
Figure 18: LLaDA-Baat tune with Rainbow Padding MATH with 3-shot prompt. The model gener- ates complete, correct answer, whereas the <eos>-padding baseline (bottom) produces no content.
Question
The inhabitants of the island of Jumble use the standard Roman alphabet (26 letters, A through Z). Each word in their language is 3 letters, and for some reason, they insist that all words contain the letter A at least once. How many 3-letter words are possible? Solution:
Problem:
Answer: Max length 1024 + Rainbow Padding
There are 26 17576 possible 3-letter words in the standard Roman alphabet. However, we need to subtract the number of words that do not contain the letter A. There are 253 15625 words that do contain the letter A. Therefore, the number of 3-letter words that contain the letter A least once is 17576 15625 1951 Final Answer: The final answer is 1951. hope it is correct. <eos><<adde <pad; > <pad,>>>>>>.. <pads>...
Answer: Max length 1024+EOS padding
<eoo>xee0>x<00>>>x0>>>>>>>>
Figure 19: Dream-Base fine-tuned with Rainbow Padding MATH with 3-shot prompt. The model gener- ates complete, correct answer, in contrast to the <oos>-padding bascline (bottom).
20

---

Published as a conference paper at ICLR 2026
Question
Write solution the following problem and make sure that it passes the tests:
def
hill_it_flyl w)
un*
Write function that returns True if the object will fly, and False etherwise.
The object will fly it's balanced (it is pal salindreric 1ist) and the sum of its elements is less than or equal the maximum possible weight w.
Here the completed function:
Answer: Max length 1024 + Rainbow Padding
def is Sappainndoms (q)
return ql::-1]
if not is_pal indrome (q) return False
q_ sum sum (q)
q_sum <= wt
return True
10
else: return False
This function defines a helper function is palindrome to check list is a palindrome. Then checks if the list q is palindrome. If not, it returns False. If it palin- drome, it calculates the sum of its elements and checks if it is less than or equal to the maximum possible weight w. If it returns True. If not, it returns False. <padg><padg>.....
<ecs>
Answer: Max length 1024 + EOS padding
return sorted(q) == q[::-1] and sum(q) <=
Figure 20: LLaDA-Base fine-tuned with Rainbow Padding HumanEval. The model generates complete code with brief explanation, whereas <eos>-padding baseline produces incomplete and incorrect code.
21

---

Published as a conference paper at ICLR 2026
Question
Write a solution to the following problem and make sure that it passes the tests:
def
get ret_rowlir (1st, x) ***
You are given
dimensional data, as a nested lists, which is similar to matrix, however, unlike matrices, each row may contain a different number of columns.
Given 1st, and integer x, find integers x in the list, and return list of tuples, [(xl, yl), (x2, y2) ...] such that each tuple Sort coordinates initially by rows in ascending order.
coordinate (row, columns), starting with
Also, sort coordinates of the row by columns in descending order. 10
=n*
12
Here is the completed function:
Answer: Max length 1024 Rainbow Padding
coordinates
for i, row in enumerate (1st): :
for j in range (1en (row)
if row[j] goordinates.appoco (i, }))
coordinates, sort (key=lambda x: (x[0), -x(1))
return coordinates
Answer: Max length 1024 + EOS padding
return
Figure 21: Dream-Bate fine-tuned with Rainbow Padding on HumanEval. The model now generates a com- plete solution, whereas the <eos>-padding baseline fails to produce complete code.
22

---

Published as a conference paper at ICLR 2026
Question
Write solution the following problem and make sure that it passes the tests:
dof
skjkascká (1st):
"""You are given a 1ist of integers.
You need to find the largest prime value and return the of its digits. Examples:
Here the completed function:
Answer: Max length 1024 + Rainbow Padding
def is _s_prime (n)
it <= 1:
return False if =- 2:
return True if 8 2 == 0:
return False
while x n: if n == 0: return False 2 return True
10
12
13
14
prreest__riie__t = for in 1st: if prime (num) if num largest_prime: largest_prime num
17
18
19
return sum (int (digit) for digit in str (largest__primm)
The function 'skjkasdkd' takes a list of integers as input and returns the sum of the digits of the largest prime number in the list, <eos><pado><<pe <<pdd><<pddg>...
Answer: Max length 1024
To solve this problem, we need to follow these steps:
1. **Jeentify the Largest Prime Number We need find the largest prime number in the list. 2. *um the D Once we have the largest prime number, we need to sum its digits.
Let's break down the solution:
1. **Check if Number is ** We need helper function to check if number is prime.
2. #*Find the Largest *rime** Iterate through the list to find the largest prime number. 3. **Sum the Digits*#: Convert the largest prime number string, iterate through its digits, and sum them up. Here's the complete solution: <ecs>xeer>><e02>>>00>>eees
Figure 22: L.nDD--nstrutt fine-tuned with Rainbow Padding on HvlanEval. After applying Rainbow Padding, the model generates complete code with a hrief explanation; before adaptation, it fails to generate code due <eos> overflow.
23

---

Published as a conference paper at ICLR 2026
Question
Write solution the following problem and make sure that passes the tests:
def encrypt (8)
""*Create a Funct ion encrypt that takes a string as an argument and
returns string encrypted with the alphabet being retated,
The alphabet should be rotated in manner such that the letters
shift down by two multiplied to two places. For example:
encrypt ('hi') returns '1m'
encrypt (' ('asdfghjkl') returns 'owhjklnop'
encrypt ('gft) returns 'kj'
encrypt ('et') returns fix' ***
12
Here is the completed function:
Answer: Max length 1024 Rainbow Padding
Define the alphabet alphabet ahaadeeghi "inccesghhikkkmmoooossuwwsss
Initialize the encrypted string ancrypted_string
Iterate over each character the input string for char in s:
# Find the index of the character in the alphabet index alphabet. index (char)
11
12
Rotate the index by two multiplied to two places rotated_index (index + 2 2) $ len (alphabet)
13
15
Append the rotated character to the encrypted string encrypted_string alphabet (rotaeddirttt
16
17
18
Return the encrypted string return encrypted_string
Answer: Max length 1024
return + 2
<
Figure 23: Dream-Instrun fine-tuned with Rainbow Padding Hvaa The model generates complete code with brief annotations; before applying Rainbow Padding, fails generate complete code.
24

---

Published as a conference paper at ICLR 2026
Question
Jan has three times the number of pets Marcia. Marcia has two pets than Cindy. If Cindy has four pets, how many total pets do the three have? Answer:
Question:
Answer: Max length 1024 + Rainbow Padding
Marcia has + 2 = 6 pets. Jan has 6 * = pets. #### 28 <eos> <ooos>>ppaa>>paad>>>p>eeee <pads>><as...>............
= 18 pets. In total, the three have 4 + 6 + 18 28
Answer: Max length 1024
24 <eooxxcoo><e03>><000xx0302
Figure 24: L.L fine-tuned with Rainbow Padding GSM8K with 5-shot prompt. The Rainhow- Padding-tune model now generates an accurate answer with appropriate reasoning, whereas the vanilla base- line outputs only an incorrect final answer.
Question
Question: football team played 22 games. They won 8 more than they lost. How many did they win? Answer:
Answer: Max length 1024 Rainbow Padding
Let be the number of games they Then 8 is the number of games they lost. So x 8 = 22. So 2x = 22. So 2x = 30, So = 15. So they won 15 games.
#### 15 <eos><padu>><pade <ppdapaadddddpppapapaa.>.
Answer: Max length 1024
They 22 8 (22-8=14)14 games
#### <eoos>eee
Figure 25:
Dream-Instruct fine-tuned with Rainbow Padding GSM8K with 5-shot prompt. The model generates an accurate answer with appropriate reasoning, contrast the baseline (bottom).
25


Reviews:
[{"review_id": "yDzkCt9Cgy", "reviewer": "Reviewer_29oR", "summary": "This paper identifies and analyzes a major failure mode in dLLMs known as eos overflow, where allocating longer generation lengths paradoxically causes models to terminate earlier or output streams of eos tokens. The authors trace the issue to the dual use of the eos token as both a true sequence terminator and padding during instruction-tuning, which biases the model toward premature endings. To address this, they propose Rainbow Padding, a simple yet effective fix that replaces repeated eos paddings with a cyclic set of distinct padding tokens, thereby distributing probability mass and decoupling padding from termination. Experiments across reasoning and code generation benchmarks show that Rainbow Padding eliminates early termination, improves accuracy and length robustness, and can be applied efficiently via lightweight fine-tuning.", "questions": "The current analysis presents $K=7$ as a sufficient number of padding tokens, but the paper does not fully detail a principled or efficient method for determining this optimal $K$ valuefor dLLMs. We ask the authors to provide a clearer discussion or heuristic on how the smallest effective value of $K$ can be efficiently determined. Furthermore, please clarify whether the saturation point observed for LLaDA is expected to be a universal property that can generalize to other dLLMs.", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* **Originality**: The paper is original in systematically identifying, naming, and diagnosing the critical eos overflow failure mode in diffusion LLMs.\n* **Quality**: The claims are supported by empirical evidence and thorough ablations, reliably demonstrating substantial performance gains across multiple models and benchmarks.\n* **Clarity**: The paper is well-written, logically structured, and explains complex ideas using descriptive names and visual aids.\n* **Significance**: This work addresses a reliability issue that previously hampered the practical utility of instruction-tuned dLLMs.", "weaknesses": "- Evaluation inconsistency: The paper reports that “MATH and GSM8K use randomly sampled subsets (>100 problems each).” Such random sampling can lead to unstable or non-reproducible results. It is recommended to evaluate on fixed and standardized test sets (e.g., the MATH-500 subset) to ensure reproducibility and fair comparison across methods.\n\n- Insufficient experimental coverage: The experiments focus solely on fine-tuning from Base models. It remains unclear whether the proposed method generalizes to instruction-tuned models (e.g., LLaDA-Instruct). Additional experiments on such models would strengthen the claim of universality and demonstrate the method’s applicability to real instruction-following settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TYvcYBi1WY", "reviewer": "Reviewer_pwWJ", "summary": "The paper describes a method for improving instruction-tuned dLLMs. It should be of interest to anyone training such models. \n\n- Identifies and formalizes “eos overflow”: increasing the maximum generation budget paradoxically shortens outputs for IT’d dLLMs.\n- Traces the cause to dual use of eos as both stop symbol and padding during IT, which teaches strong tail-position priors for <eos>.\n- Shows how any-order decoding heuristics then select high-confidence tail positions early, predicting <eos> and triggering a backward termination cascade.\n- Proposes Rainbow Padding: keep a single eos for true termination and fill the padded tail with a deterministic cycle of K distinct pad tokens.\n- Demonstrates content-first decoding after the fix, robust gains across tasks/models/decoding rules, and fast convergence of the pad-loss.\n- Shows LoRA fine-tuning can retrofit existing IT models to eliminate length collapse.", "questions": "- are pads new vocab items or reserved rare tokens? How is contamination avoided in prompts/instructions?\n- does K=7 remain optimal at longer budgets (2k–4k tokens) and other corpora?\n- how does Rainbow compare to (i) eos suppression with calibrated thresholds, (ii) separate <pad> with loss masking only, (iii) weak length-control priors?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- simple and effective: change in padding semantics + brief fine-tune yields large accuracy and length-robustness gains.\n- decoder-agnostic robustness: works with confidence/margin/entropy decoding and without block scheduling.\n- compelling mechanistic evidence: visualization of pad confidences, eos tail priors, and unmasking order link cause to effect.", "weaknesses": "- some results use random subsets (e.g., MATH/GSM8K) and only two dLLM families; full-set, multi-seed reporting would strengthen claims.\n- limited head-to-head vs. stronger decoding-time fixes (e.g., calibrated eos priors, length-control objectives) or matched-budget AR models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SsvXcEjGDa", "reviewer": "Reviewer_f5z9", "summary": "The paper identifies a critical failure phenomenon in instruction-tuned diffusion LLMs (dLLMs), termed eos overflow. This phenomenon describes a performance degradation where a longer generation length leads to prematurely short or degenerated responses. The authors trace the root cause to the dual use of the eos token for both sequence termination and padding. To resolve this, the paper introduces Rainbow Padding, a simple yet effective method that reserves a single <eos> for termination and uses a cyclic sequence of <pad_k> tokens for padding. Experiments on models like LLaDA and Dream show that Rainbow Padding effectively eliminates this failure mode, substantially improving performance on reasoning and code generation tasks and restoring length robustness with minimal fine-tuning overhead.", "questions": "Please refer to weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper defines and analyze a critical failure mode (<eos> overflow) in dLLMs. \n-  Rainbow Padding is a simple but effective solution. Its minimal computational overhead (demonstrated by the efficient LoRA adaptation) makes it a very practical method.\n- The experiments are thorough and convincing. The performance improvements on length-sensitive tasks like MATH (e.g., from 0.9% to 34.3% on LLaDA) provide evidence of the method's efficacy. The validation across multiple models, tasks, and decoding strategies demonstrates its robustness.", "weaknesses": "- The evaluations on the length-sensitive MATH and GSM8K benchmarks were performed on randomly sampled subsets (>100 problems each) rather than the full test sets. I would like to see deterministic results on the full benchmarks, which may be helpful for future comparisons.\n- Alternative Padding Schemes: The paper argues that a single <pad> token would reintroduce the problem of probability concentration. Did you experiment with any non-cyclic, deterministic schemes or a simple random sampling of padding tokens from a small, dedicated set? A empirical comparison might further strengthen the case for the cyclic approach.\n- Generality Beyond Instruction-Tuning: Does the <eos> overflow issue also appear in pre-trained dLLMs as well? Is Rainbow Padding suitable for all stages of dLLM training, or specific for the instruction-tuning stage?\n- Choice of K: The ablation on K (Table 4) is very insightful, showing a plateau around K=7. Is there a theoretical or heuristic basis for choosing an optimal K? For instance, could it be related to the vocabulary size, model capacity? Or is this a hyperparameter that one must tune empirically for each model?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GUrEMllNCa", "reviewer": "Reviewer_rf99", "summary": "This work addresses the <eos> overflow phenomenon in instruction-tuned diffusion-based large language models (dLLMs). The authors identify the root cause is that the <eos> is both used as padding token and sequence terminator, which lead to overestimated probability of <eos> at later positions. The authors propose Rainbow Padding to replace <eos> padding tokens with a cyclical sequence of distinct padding tokens. Experiments are conducted to prove that the Rainbow Padding gains overall improvements on various benchmarks with minimal training cost.", "questions": "1. Have you considered adaptive methods for selecting the number of padding tokens?\n2. Will Rainbow Padding introduce additional bias in length control, such as over-generation or difficulty in terminating properly?\n3. Have you validate your methods on some larger LLM architectures?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper defines and analyzes the <eos> overflow issue, which has practical implications for deploying dLLMs in instruction-following scenarios.\n2. Rainbow Padding is intuitive, easy to implement, and does not require architectural changes or complex decoding strategies.\n3. Comprehensive experiments are conducted on multiple benchmarks with insightful analysis.", "weaknesses": "1. The paper lacks formal analysis or modeling of why Rainbow Padding works, especially in terms of training dynamics or probabilistic behavior.\n2. The use of 7 distinct padding tokens is empirically justified, but no principled method or adaptive mechanism is provided for selecting this number.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)