UltraLLaDA: Scaling the Context Length to 128K for Diffusion Large Language Models
Authors: Guangxin He
Shen Nie
Fengqi Zhu
Yuankang Zhao
Tianyi Bai
Ran Yan
Jie Fu
Chongxuan Li
Binhang Yuan
Venue: ICLR 2026 Poster (2026)
Keywords: diffusion language model; long context LLM
Primary area: foundation or frontier models, including LLMs

Abstract: Diffusion LLMs have attracted growing interest, with plenty of recent work emphasizing their great potential in various downstream tasks; yet the long‑context behavior of diffusion LLMs remains largely uncharted. We present a case study of post‑training techniques for extending the context window of diffusion LLMs (i.e., LLaDA) without retraining from scratch. We show that a simple modification to the standard Rotary Positional Embeddings (RoPE) extension effectively accommodates the probabilistic modeling inherent in the diffusion process, enabling stable scaling to longer context ranges. We further compare masking strategies used during post‑training and analyze their impact on optimization stability and long‑range recall. Instantiating these insights, we introduce UltraLLaDA, a diffusion LLM with a 128K‑token context window that, in our empirical evaluation on long‑context tasks, significantly outperforms training‑free baselines. Our experimental results highlight the special positional extension as a key lever for scaling diffusion LLMs to extended contexts and offer practical guidance for practitioners seeking 128K‑scale context via efficient post‑training.

Published as a conference paper at ICLR 2026
ULTRALLADA: SCALING THE CONTEXT LENGTH TO 128K FOR DIFFUSION LARGE LANGUAGE MODELS
Guangxin He², Shen Nie², Fengqi Zhu², Yuankang Zhao?, Tianyi Bai", Ran Yan Jie Fu', Chongxwan Li", Binhang Yuan
HHKUST Renmin University of China University of Chinese Academy of Sciences +Shanghai AI Lab
ABSTRACT
Diffusion LLMs have attracted growing interest, with plenty of recent work em- phasizing their great potential in various downstream tasks; yet the long-context behavior of diffusion LLMs remains largely uncharted. We present case study of post-training techniques for extending the context window of diffusion LLMs (i.e., LLADA) without retraining from scratch. We show that simple modifica- tion to the standard Rotary Positional Embeddings (RoPE) extension effectively accommodates the probabilistic modeling inherent in the diffusion process, en- abling stable scaling to longer context ranges. We further compare masking strate- gies used during post-training and analyze their impact on optimization stability and long-range recall. Instantiating these insights, we introduce ULTRALLADA a diffusion LLM with a 128K-token context window that, in our empirical evalu- ation on long-context tasks, significantly outperforms training- freee baselines. Our experimental results highlight the special positional extension as a key lever for scaling diffusion LLMs to extended contexts and offer practical guidance for prac- titioners seeking 128K-scale context via efficient post-training.
INTRODUCTION
Diffusion-basee based large language models (LLMs) Gulraaani & Hashimoto 2023
Lou et al. 2024
Nie et al. 2025) have recently emerged as a promising new paradigm in natural language process- ing. Unlike popular auto-regressive LLMs (Liu et al. 2024 Dubey al., 2024 Yang et al. 2025a), which generate text token- token-by-tokenr by-token, diffusion LLMs employ an iterative denoising process over the entire sequence, offering potential advantages over rego-segressive language models (such as bidi- rectional global context awareness, flexibility in conditional control, and unified modeling across modalities (Li et al.] 2025).. Substantial research has already explored the scalability (Nie et al., 2025). multimodal extensions (Yang et al.) [2256] You et al. (2025). reasoning (Zhu et al. [025 Zhao et al. 2205),, and efficiency optimizations (Ma et al. [025]] Arriola et al., Wu et al. 2025) of diffusion LLMs. However, one critical aspect remains largely unexplored: how
effectively scale and extend the context window of diffusion LLMs beyond their original training limit? Con- cretely, the problem is about how to enable diffusion LLMs to handle vastly longer input sequences (e.g., up to 128K tokens) to achieve solid performance on long-context language modeling tasks.
Unlocking long-context capabilities would significantly broaden the applicability of diffusion LLMs. Many real-world language tasks (e.g. processing long documents, multi-turn dialogues, retrieval-based question answering (Liu et al., 2025a)) demand handling inputs far exceeding the few-thousand-tokee contexts used in standard training. If the context window of diffusion LLMs could be scaled to such lengths, we could explore if the advantage of diffusion LLM could be fur- ther amplified in maintaining coherence and handling complex dependencies across much larger texts than auto-regressive LLMs. Furthermore, preliminary observations suggest diffusion LLMs behave differently with long inputs, e.g., they exhibit stable perplexity under context length extrapo- lation, whereas rego-segre L.LM typically see perplexity blow up catastrophically once past their trained context limit (Liu et al. 2025b). To move forward, we believe it is crucial to fully realize the potential of diffusion LLMs in long context scaling to systematically compare their performances with auto-regressive LLMs in tasks requiring extensive long context.
Corresponding Binhang Yuan (bi doiyuan@ust.Nm .kk).

---

Published as a conference paper at ICLR 2026
Overall AVG Score: 1.0
Overall AVG Score: 0.2
1.0
0
10
10
20
0.8
20
0.8
30
30
40
SCOrD
Deptb
40
0.6
50
G0
60
0.4
60
0.4
70
80
90
Get
0.0
36
Ba YoOk
0.0
Context Langth
Context
(a) ULTRALLADA.
(b) LONGLLADA.
Figure 1: NIAH evaluation up to 128K context-length. ULTRALLADA can find all of the needles within the context window 8-32x longer than that LONGLLADA can handle.
Achieving this goal, however, is non-trivial. Prior research in scaling the context window of auto- regressive LLM indicates that naively increasing the position range of the model (for example, by extrapolating Rotary Positional Embeddings (Chen et al. 2023) beyond the length seen in training) can lead to mismatches with the model's 's learned positional dynamics and generation process. Such performance degradation could be inherited from the attention patterns learned on shorter contexts, which are not able to directly accommodate the extended positional embeddings. Recent attempts of scaling diffusion LLMs have further reveals some interesting observations, e.g., diffusion LLMs do not suffer total collapse in perplexity when input length grows (unlike auto-regressive LLMs), but they instead display "local perception" bias, which means that when diffusion LLM is given a context longer than was trained on, it tends to only utilize information from the most recent segment (e.g. the last 4K tokens) and ignore the content that lies far back in the prompt (Liu al., 20255). However, such observation was derived from a training-free method (L.u et al. (025h). Based on the prior research on auto-regressive LLMs, post-training based methods for extending the context length of LLMs are often superior to training-free free methods because they allow the model to fundamentally adapt its internal mechanisms to handle longer sequences (Liu et 2025a)
In this paper, we plan to explore how to effectively scale and extend the context window of diffusion LLMs by answering the following two concrete questions:
Q1. How can we effectively scale and extend the context window of diffusion LLMs derived from the existing methods originally designed for auto-regressive language models?
Q2. What is the corresponding potential performance boost when comparing the post-training based methods for diffusion LLMs with the training method (i.e., LongLLaDA)?
Contribution 1. To answer the first question, introduce diffusion-specific extension of Ro- tary Positional Embeddings that enables better long-contexs modeling. In particular, we develop a Diffusion-awan NTK method diffusion LLMs - that accommodates the iterative denoising process and permits RoPE extrapola- tion 128K tokens. We also investigate different masking strategies for light-weight post-training on extended contexts (including adaptive masking and end-of-document concatenation) and ana- lyze their impact optimization stability and long-range information recall. These methodological contributions provide principled foundation for scaling the context window of diffusion- based lan- guage models post-training setting, where we extend the context length of one sfuteoff-the-nt diffusion LIM, i.e., LLADA (Nie et al. 2025 to up to 128K tokens, namely ULTRRALLADA
adaptation of the Neural Tangent Kernel method tailored to
To answer the second question, we present comprehensive experiments validating the effectiveness of ULTRALLADA across multiple long-context benchmarks. We compare Ut.- TRALLADA against LONGLLADA and the original LLADA base model, observing that ULTRA- LLADA consistently outperforms these alternatives when handling extremely large context (e.g..
Contribution
Figure D On a diverse benchmark suite of long-context tasks, ULTRALLADA achieves superior performance, maintaining low perplexity and high task accuracy context length increases. This extensive empirical evaluation highlights ULTRALLADA 's state-of-tte of-ate---tth----ee and demonstrates the practical benefits of our lightweight post-training approach.
the-the-rt art long-context context capabilities
2

---

Published as a conference paper at ICLR 2026
2 PRELIMINARY AND RELATED WORK
Diffusion Language Model. Diffusion models have recently emerged as a promising alternative to the auto-regressive models for language generation (Li et al., 2025). Unlike AR LLMs that predict tokens sequentially, diffusion language models generate text through an iterative denoising process conditioned noised inputs. Among these methods, one popular paradigm known as masked
diffusion language model (Austin et al. 2021; Lou et al. 2023 Sahoo et al. 2024, Shi et al. 2024] Ou et al. 2024) utilizes forward process incrementally introduce noise into discrete data, and
learned reverse process to generate coherent sentences. Given a noise level t E [0, 1] and input data 2% E 1,, ,..;, K 1 where K is the vocabulary size and L is the sequence length, the forward process ( (zo)) independently replaces each token in 2o with a mask token with probability 4, and leaves it unchanged with probability 1-th t. Masked diffusion LLMs are usually trained to minimize the upper bound on the negative logliikli (Sahoo et al. 2024 Shi et al. 2024
Ou et al. 2024):
- -E--U[0011, [0.1,[0 [0.1), aurmpa 2e (20 (20)
log Po (a) (2x)
where denotes the mask token, and Pe is parameterized by a bidirectional Transformer [Vaswani ctt al. 2017). During inference, masked diffusion language models denoise starting from the con- catenation of prompt and a masked sequence. At each denoising step, pe predicts the original tokens for all masked positions. Afterward, a subset of the predicted tokens is re-masked, and the number of tokens to be re-masked is treated as hyperparameter that controls the trade-off between generation quality and decoding speed. LLaDA (Nie et al. 2025) and Dream (Ye et al.) 2025) are two of the representative open-source diffusion language models. LLaDA is trained from scratch, while Dream is obtained by post-training a pre-trained auto-regressive language model. Both mod- els achieve performance that competitive with strong LLMs such as LLaMA3-8B (Dubey et al., 2024) and Qwen2.5-7B (Qwen et al. (025).
Extending the Context Window of LLMs. Extending the context length of LLMs has become crucial for tasks such as long document processing and multi-turn conversations. Recent research has developed a range of techniques to extend context windows far beyond the few thousand to- kens during pre-training (Liu et al., 2025a). In state-of-the-art LLM transformer architectures (e.g.. LLaMA (Touvron et al., 2023)), rotary position embedding (RoPE) (Su et al., (024) is a popular position embedding method, which enhances traditional position embeddings by encoding posi- tions using rotating vectors in the complex plane and preserves relative positional relationships, enabling better generalization to extended contexts. Various methods have been proposed to extend the context window: Position interpolation (PI) (Chen et al. 2023) generates positional embed- dings for out-of www.o--raninngrrangg.. positions by interpolating between embeddings of nearby positions; ATK-Aware Scaling (Peng & Quesnelle| 2023) rescales the RoPE base to preserve the neural tan- gent kernel (NTK)-a linear approximation of the model's behavior-when extending the context; YaRN (Peng et al.) extends RoPE by dynamically scaling rotation frequencies based on layer depth and target context length to maintain rotational consistency across layers and contexts. Noted that most of the current context extending methods are based on auto-regressive LLMSs - only a very recent research, i.e., LongLLLaDA (Liu et al.] 20255), has initialized the research for diffusion LLMs, which integrates an NTK-based RoPE extrapolation into diffusion LLM based on wriniig-free paradigm. To make an apple-to-apple comparison, our training- based extension is also based on NTK, and we summarize its formulation below (Peng & Quesnelle 2023):
Turget
ANTK- -
Tiran
Here d denotes the (even) rotary dimension, Tuain and Turget represent the pretraining context length and desired extended context length; ANIK is the global scaling factor applied to the RoPE base. Note that NTK scaling multiplies the RoPE base by the same factor À across all rotary dimensions.
3 LONG-CONTEXT SCALING FOR DIFFUSION LLM
To answer Q1 (how to scale diffusion LLM context windows). we first revisit NTK-basee RoPE ex- trapolation from LONGLLADA (Liu et al. 2025b) in $3.1 propose new diffusion-aware NTK extension approach for ULTRALLADA in $3.2; and enumerate the details of our strategies for lightweight post-training on long-context data that mitigate cross-segment noise in $3.
3

---

Published as a conference paper at ICLR 2026
3.1 REVISIT NTK-BASED RoPE EXTRAPOLATION IN LONGLLADA
LONGLLADA (Liu et al. 2025b) applied -TK-aware RoPE scaling originally designed for auto- regressive LLMs to diffusion LLMs, demonstrating that simple RoPE base scaling can extend diffusion model's context length in a training-free manner. Concretely, given the RoPE base b, rotary dimension d. pretraining context length Train. and target context length I Target. LONGLLADA first computes critical dimension dent based on the LLM's pre-training context length Tuain then sets a scaling factor Abuseline via:
T Lurget بoي, dent = 2[ logs 2x 2x
d T Train dent=
Aasellnee = b-'
(3)
Key Observation and Discussion. Note that LONGLLADA utilizes the same Timin in context window extension as auto-regressive LLMs, although diffusion (with bidirectional attention) and auto-regressive language models perceive different effective relative span during pre-training. As result, directly applying uuto-regressive-orienne
assumptions to diffusion LLMs therefore, misesti- mates both the critical dimension and the scaling factor. In other words, such approach inherits extrapolation strategies from regrereegees LMs without accounting for diffusion-speciffe proper- ties (notably, bidirectional attention over the entire sequence). Consequently, long-contexi potential inherited from diffusion language modeling is not fully unlocked. This motivates us to propose a inffusion-specifee NTK extrapolation method (Diffusion- aware NTK) that explicitly incorporates diffusion LLM characteristics.
3.2 DIFFUSION-AWARE NTK IN ULTRALLADA
Motivated by the observed gap discussed in 3.1 we introduce difsusion-aware NTK, which refines the mapping from Tuain to critical dimension by accounting for diffusion-speciffe properties, yield- ing an adjusted scaling factor X" Comprehensively, our goal is to better exploit the properties of diffusion LLM's s properties to unlock their long-context capability. We define: Teup as the maximum relative span learned during pretraining on T mrger length data, and Feap as the required relative span after context-length extension to T Turget.
Diffusion-aware NTK. We first define a new scaling factor X' for RoPE based on an NTK-aware criterion tailored to diffusion L.Mss. Concretely, let d be the model's attention dimension and b the original RoPE base. We define the modified critical dimension d' as the largest even-indexed dimension whose RoPE sinusoidal period Ld,.. does not exceed Teap. Formally, we have:
= TEaa - ) d=== 21 2 logs 2
d Teap
(4)
2T
By reducing the angular frequency, this scaling mechanism increases the RoPE periods across all dimensions. In other words, we choose X' Equation 4 this way, so that we can effectively slow down the RoPE rotations, and lengthen the positional wavelengths along all attention dimensions. In particular, the formerly "well-trainedd" critical dimension d now has an expanded period covering Tecap: thereby unlocking the model's ability to attend over the extended context range.
It is crucial to note that diffusion LLMs have different effective Teap from auto-regressive LLMs. In diffusion model with bidirectional attention, each token attends to the entire sequence in both di- rections, during training it effectively learns relative positions in the range [ -Tirain - 1), T Tirain 1]. This symmetric coverage roughly doubles the span of learned positions (i.e., 2T (Taaa compared to an auto-segressive LLM, which only attends past tokens (relative positions [0, Timin 1). As result, a diffusion LLM can naturally handle a wider range of relative positions(see Appendix Cor the theoretical analysis). We account for this by setting Teap R 2Tiain and TEaap
iarget (ver- sus Teup z Train and Tecap 2 Tunget for outo-regressive models) when computing A. In practice, using this diffusion-awwe -warr Teap leads slightly larger critical dimension and more conservative (smaller) X' than the original NTK formula, i.c., Anwseline in Equation 3 For example, as illustrated in Figure 2a] for the 8B I.LADA model with T Tiain 4K, the baseline auto-regressive centric formula yields derit A 64 (period 44K tokens), whereas our diffusion-aware approach gives d(m & 70 (pe- riod ~8K tokens). For simplicity, we conduct training-fie free experiments to verify the effectiveness of this method. The preliminary results (Figure 2c 2b and this formulation suggests that incorporat- ing bidirectional coverage is essential to extend diffusion LLMs' context properly. A finer- grained sweep over nearby critical dimensions provided in Appendix DJ We adopt this Diffusion-awe aware NTK scaling for all extended-context experiments in the following sections.
4

---

Published as a conference paper at ICLR 2026
NTK
NIAH(32K)
I.((6KK)
Diffu- aware
79.36
35,38
Baseline
1..
Baseline 75.28
35,02
éritical dim (64)
pirru-aware
0.5
(b)
Training-free NIAH and LongBench scores.
eritical dim (70)
20
0.0
Diffu-aware NTK
18
Baseline NTK
Perplexey
16-
84 0
70 75 ao Context Length
35 40 RRo8 bimen en
0k
4k
8k
12k 16k 20k 24k 28k 32k
Context Length
(a) RoPE criticul dimension.
Training-free PPL on 32K context length.
Figure 2: RoPE critical dimension and training- free case study under different NTK scaling.
3.3 CASE STUDY OF MASKING FOR DIFFUSION LLM CONTEXT EXTENSION.
After re-scaling the positional embedding using the above method, further post-train the diffu- sion LLM on long-context data under the new encoding. We generate long input sequences from the PG19 corpus (Rae al., 2019) following the packing strategy of (Peng & Quesnelle (023): shorter documents are concatenated together to reach a target length (we use 64K tokens per packed sequence), while very long documents are split into consecutive 64K segments (carrying any remain- der to the next segment). This packing creates training sequences up to 64K tokens long without altering the overall token distribution.
A key challenge in long-context post-training is dealing with cross-document interference. When multiple unrelated texts are concatenated in a single sequence, a vanilla diffusion LLM (with global bidirectional attention) may attend across the document boundaries, causing tokens to mistakenly incorporate context from other documents. Auto-regressive LLMs also face interference when pack- ing data, but their strictly causal (unidirectional) attention (Figure 3a naturally limits some interac- tion between unrelated segments. In the diffusion paradigm, because any token can interact with any other token, we need explicit strategies to prevent spurious interactions between unrelated segments.
Masking in Diffusion LM Long Context Extension. Previous works on long-context training for auto-regressive LLMs have proposed several approaches: e.g., pack data certain length
regardless of the document boundary (Fu et al.) 2024 Touvron et al. 2023: Ratfel et al. 2020),
inserting special boundary tokens to mark segment ends (Peng al. 2023). using specialized attention masks to block inter-document attention (Gao et al. 2024; Shang et al.) (2025). However, the effectiveness of these strategies in diffusion LLMS (with fully bidirectional attention) remained under-explored, motivating us to compare them in our setting. We consider three post-training strategies of masking (illustrated Figure 3)
(i) Adaptive attention masking: We construct document-aware attention mask for each packed training sequence that allows full attention only within each original document. Any attention between tokens belonging to different source documents is masked out (set to 0). This effectively blocks cross-document influence while preserving bidirectional attention within each document (Figure schematically shows this masking for three segments). During post-training, the model learns under this segmented attention pattern, ensuring it does not rely on unrelated context.
(ii) End- find-of-done document (EOD) concatenation: We insert special und-of-documed token between documents when packing them. The model is still trained with standard full bidirectional attention over the entire sequence (no mask beyond the usual ones), but the EOD token provides learnable boundary indicator (Figure (3c).. The expectation is that the model will learn to treat EOD as separator and avoid blending information across it. Unlike Adaptive Masking, this method does not explicitly forbid cross segment attention, but rather gives the model a chance to infer document breaks from the EOD symbol.
(iii) Direct concatenation: As a baseline, we also include naively packed sequences with no special handling, i.e., documents are concatenated back-to-back and the model uses full bidirectional
5

---

Published as a conference paper at ICLR 2026
0
Allowed (1)
0-
Altn Allowed (1) FOD Position
0
Altn Allowed (1) FOD Position
EDD Position
3
3
3
Position ldx
Position idx
Position idx
5
5
5
6
6-
7
8-
8
8
7
4
Position
Position idx
Position idx
(a) Anto-regrescive Attention.
(b) Adaptive Attention (Diffuusion.
(c) Full Attention (Diffusion).
Figure 3: Different attention mechanism for Long-contes training.
Table 1: Perplexity on 128K long test data.
PPL
4K
8K
16K
32K
48K
64K
96K
128K
LLDDA-8B-Base
12.00
11.98
13.66
16.80
28.04
53.92
162.62
343.88
LongLLaDA
13.39
13.38
14.66
16.41
N/A
N/A
N/A
N/A
ULTRALLADA 11.27
11.15
11.66
11.32
11.96
11.62
11.09
10.45
attention the entire sequence with no boundary tokens. This is the simplest approach and may be prone to maximal cross-document interference.
We trained separate long-context models with each of the above strategies (all of them also using our Diffusion-awarr NTK position scaling). Empirically, both adaptive masking and EOD con- catenation dramatically reduce cross-document interference compared the direct concatenation baseline. The model post-trained with direct concatenation often produces incoherent results, pre- sumably due unrelated content bleeding together. In contrast, both alternative strategies yield much more coherent generations, with Adaptive Masking showing a slight advantage over EOD to- kens in our experiments. By either explicitly preventing cross-document attention or clearly marking the boundaries, these approaches allow the diffusion LLM to fully exploit its long-context capac- ity during post-training. Overall, our combined approach - applying Diffusion- awarre NTK RoPE scaling followed by long-context post-training with an appropriate long sequence data processing strategy (adaptive masking or EOD markers) substantially improves the model's 's ability to handle extended context windows. As we will show next, the resulting model (namely UtTRALLADA maintains coherent, low-perplexity performance even far beyond its original training context.
4 EXPERIMENTS
We thoroughly evaluate the long-context capabilities of ULTRALLADA and compare it against base- line models to answer Q2 (the performance gain of our post-training approach over training frree method), and against different post-training settings to further discuss Q1 (how effectively scale and extend the context window of diffusion LLMs). Experiments are conducted on four benchmarks that stress-test long-context capabilities of ULTRALLADA up 128K tokens:
PPL-128K: Language modeling perplexity evaluation on 128K-token test document from PG19. Lower perplexity better predictive modeling of long text. We estimate PPL via Monte- Carlo denoising likelihood on randomly masked tokens (cf. Eq. []) the same method used in LONGLLADA; while lower still indicates better performance, this metric is not strictly identical to auto-regressive language models' next-token PPI..
NIAH-128K: Needll-in-a-haasttcc long-context retrieval task (gkamradt 2023). where a single relevant sentence is embedded in long distractor context (up 128K tokens) and the model must retrieve it. We report retrieval accuracy using an inference configuration with output length set to 32, block size of 32, and 32 sampling steps.
LongBench-16K: A suite diverse long-con cong-context tasks (Bai et al. 2023) including question an- swering, summarization, code completion and other tasks, at context length of 16K. We report the aggregated score (weighted by question count) across all sub-tasks using an inference config- uration with output length set to 5 512. block size of 64, and 512 sampling steps.

---

Published as a conference paper at ICLR 2026
Table LongBench cut at 16K Evaluation. Sub tasks: single-document QA (SD), multi-document QA (MD), summarization (Sum), in-context learning (ICL), synthetic tasks (Syn), and code com- pletion (Code). AVG is an aggregated question- count- average score.
Model
AVG
SD
MD
Sum
ICL
Syn
Code
--D---B--Base
31,56
13.40
8.04
21.71
61.83
30.38
56.41
LongLLada
36.07
15.14
11.26
24.77
67.22
40.56
61.54
Ultaa (Ours)
39.98
18.84
14.99
27.58
70.59
50.44
63.57
Table 3: RULER with context lengths 4K to 32K. include Retrieval: Needle-in-ar -n-li--s--- (NIAH), Aggregation: frequent words extraction (AGG), question answering (QA), and Multi-hop Tracing: variable tracking (VT). AVG is question-count-weiggeee average score. " indicates failure.
4K
8K
Model
AVG
NIAH
AGG
QA
VT
AVG
NIAH
AGG
QA
VT
LLaDA-8B-Bas
86.17
99.19
45.84
67.50
100
41.69
46.66
42.84
23.50
36.00
LongLLaDA
87.41
97.84
73.82
53.00
100
65.20
76.28
41.20
49.50
56.00
LltA (Ours)
88.37
97.31
66.69
68.50
100
86.22
98.28
55.29
62.00
100
16K
32K
LLaDA-8B-Base
LongLLaDA
45.48
52.75
17.70
40.00
53.80
5.69
3.69
5.45
15.50
2.60
UltraLLaDA (Ours)
77.51
93.00
43.12
39.50
98.40
73.63
92.78
29.29
29.00
98.40
includes categories like retrieval, aggregation (e.g. common word extraction), question answering. and multi-hop tracing (variable tracking). We evaluate with context lengths up to 32K. Evaluations are conducted with context lengths up 32K, and we report both the question-coum--wiggtte average score and the breakdown by category. Results are obtained using an inference configuration with output length set to 64, block size of 64, and 64 sampling steps.
RULER-32K: A benchmark for long-context models (Hsieh et al. 2024)
For all evaluations, lower is better for perplexity, while higher is better for accuracy scores. We emphasize fairness in comparison: all models use the same base 8B LLADA initialization and were evaluated under identical conditions. ULTRALLADA was obtained via a lightweight post-training of 600 steps long data, whereas the baseline LONGLLADA model uses training-free RoPE extrapolation approach without additional training. In all cases we use deterministic decoding to climinate sampling variance, so do report confidence intervals or significance tests. For re- producibility, the training details and model hyperparameters are provided in Table 7 (Appendix B
4.1
MAIN RESULTS.
NIAH Retrieval. Figure compares our ULTRALLADA to the LONGLLADA baseline on the NIAH retrieval task for context lengths from 4K up 128K tokens. ULTRALLADA achieves 100% retrieval accuracy at every evaluated context length, all the way to 128K. In contrast, the base- line LONGLLADA (which extended 4K-trained model to 32K in a training-free manner) performs reasonably at shorter lengths (over 80% accuracy at 8K and 16K) but then drops sharply to around 20% 32K and fails entirely beyond 32K. In fact, LONGLLADA could not be evaluated past 32K due to its method's limitations. These results demonstrate that our post-training approach preserves excellent retrieval capability even with extremely long contexts (128K), whereas the training- baseline rapidly degrades context length increases. ULTRALLADA can successfully retrieve the "needle" in contexts that are 8-32x longer than those the bascline LONGLLADA can handle.
PPL. We next assess generative modeling quality text from PG19 and measured perplexity when the model is given increasing portions of context (4K up to 128K). Table reports the perplexity of the base model LLADA, the LONGLLADA baseline, and ULTRALLADA, The base LLADA model (trained on 4K context) experiences a perplexity in- crease from around 12 at 4K to around 344 at 128K, indicating it cannot maintain coherence beyond
long document. We took 128K-token test

---

Published as a conference paper at ICLR 2026
its training length. The LONGLLADA baseline performs even worse at shorter ranges notably, at 4K and 16K its perplexity is higher (worse) than the base model's - and achieves only a slight improvement at 32K (16.4 vs base 16.8). Furthermore, LONGLLADA cannot be extended beyond 32K (entries marked "N/A" in Table D. In contrast, ULTRALLAAA maintains a low and stable perplexity (11-12) across all lengths up 128K. This highlights the robustness of our approach in modeling extremely long sequences.
LongBench. We evaluate models on the LongBench benchmark truncated 16K context window (since many tasks in LongBench do not require more than 16K). Table 2 shows the aggregated scores (higher better) and breakdowns across six representative sub-tasks: single- and multi- document QA (SD, MD). summarization (Sum), in-context learning (ICL), synthetic reasoning tasks (Syn), and code completion (Code). ULTRRALLADA achieves the highest score on every sub-task, outperforming both the base model and the LONGLLADA baseline. Overall, ULTRALLADA'S 's average score is 39.98, a solid improvement over the baseline's 35.38 and the base model's 30.89. This demonstrates that our long-context training not only extends the context length but also yields quality gains on challenging tasks even at 16K (within the baseline's range). We attribute this to improved range coherence and understanding gained through our post-training procedures.
RULER. Table 3 reports results on the RULER benchmark with context lengths from 4K to 32K (covering retrieval, aggregation, QA, and multi-hop variable tracking (VT) tasks). ULTRALLADA again consistently outperforms both the base model and LONGLLADA at all lengths. The perfor- mance gap widens as the sequence length increases. At 8K, the baseline already lags behind (average score 65.20 vs our 86.22). By 16K the baseline's s average drops to 45.48 while ours remains 77.51. At the maximum 32K length, the baseline collapses to average of 5.69, essentially failing on most tasks, whereas
still achieves 73.63 average. Notably, prior work noted that diffu- sion LLMs struggled on the VT tasks long lengths (Liu et al. 2025b). We observe the same for maintains a near- pear-eerfect
the baseline: its VT score falls to 2.6 at 32K. In contrast, VT score of 98.4 at 32K, on par with its performance at 8K. fact, ULTRALLADA exhibits strong scaling on both the retrieval (NIAH) and tracing (VT) categories, where it achieves 90-100% ac- curacy across the board. The gains on aggregation (AGG) and some QA tasks are more modest, This suggests that diffusion LLMs, even when scaled to extreme context lengths, particularly excel at tasks requiring pinpoint retrieval of information and maintaining consistency (tracing variables) over long texts, while tasks that involve combining or reasoning over many pieces of information (aggregation, complex QA) remain challenging. Overall, the RULER results reinforce that our post- training method yields substantial performance boosts over the training-fres baseline, especially at the longer end of the context range, answering Q2
Beyond the main results reported here, we further provide additional long-context evaluations on more complex comprehension and reasoning benchmarks at longer context lengths in Appendix H At the same time, similar to prior observations in autoregressive LLMs, extending the context win- dow may introduce a trade-off between cong-context gains and short-context performance; we pro- vide additional analysis of this short-comtext regression in Appendix !|
4.2 ABLATION STUDY: DIFFUSION- VS. BASELINE NTK SCALING.
We perform an ablation study to isolate the impact of our Diffusion-awarr NTK scaling (Section 3.2 compared to Baseline NTK scaling used by LONGLLADA. In this experiment, we post-trained two variant models on long data: one using our Diffusion-aware X calculation and one using the Base- line NTK formula (all other training settings held equal, including the use of the EOD concatenation strategy for data packing). We then evaluated both models on LongBench and RULER tasks. The results are summarized in the first two rows of Table 4 and On the LongBench: Diffusion-awan NTK yields small but consistent gain in overall score: 39.80 vs. 39.44. On the RULER: At 4K, the baseline variant marginally higher (90.00 vs. 87.86). From 8K onward, diffusion- awarre scal- ing takes the lead and the margin widens with length: 86.30 vs. 85.30 at 8K, 82.99 vs. 79.54 at 16K, and 70.78 vs. 65.85 at 32K. This trend supports our premise: explicitly accounting for diffu- sion's bidirectional attention in RoPE scaling better preserves performance sequences grow. In sum, the gains observed with ULTRALLADA stem not only from additional training data but also from the positional encoding adaptation introduced by Diffusion-awand aware NTK. Appendix BeadiixxEproo provides additional long-context experiments comparing ceftusion-wwar NTK with the baseline NTK under identical post-training settings, showing the advantage of diffusion-awand NTK.
8

---

Published as a conference paper at ICLR 2026
Table 4: LongBench results: Diffusion-aware NTK and mitigating cross-document interference.
Model
AVG
SD
MD
Sum
ICL
Syn
Codc
Base-NTK + EOD-Cat
39.44
17.39
14.76
26.87
70.27
49.48
63.79
Diffu-NTK EOD-Cat
39.80
17.83
13.43
27.91
69.38
51.53
64.50
Diffu-NTK Adapt-Mask
39.98
18.84
14.99
27.58
70.59
50.44
63.57
Diffu-NTK Direct-Cat
38.77
17.29
13.22
27.54
70.22
50.07
60.92
Table 5: RULER results: Diffusion-aware NTK and mitigating cross-document interference.
4K
8K
Model
AVG
NIAH
AGG
QA
VT
AVG
NIAH
AGG
QA
VT
Base-NTK + BOD-Cat
90.00
99.41
65.40
72.00
100
85,30
97.63
52.70
62.00
99.2
Diffu-NTK EOD-Cat
87.86
96.56
63.22
72.00
99.20
86.30
99.22
51.70
63.60
97.60
Diffu-NTK- + Adapt-Mask
88.37
97.31
66.69
68.50
100
86.22
98.28
55.29
62.00
100
Diffu-NTK Direct-Cat
89.44
98.09
65.40
74.00
99.20
85.98
97.19
55.74
66.50
95.80
16K
32K
Basc-NTK + HOD-Cat
79.54
93.63
46.90
46.00
99.20
65.85
76.34
27.47
46.00
98.40
Diffu-TTK EOD-Cat
82.99
97.44
42.49
58.00
98.40
70.78
86.47
28.60
36.00
99.20
Diffu-NTK + Adapt-Mask
77.51
93.00
43.12
39.5
98.4
73.63
92.78
29.29
29.00
98.40
Diffu-NTK Direct-Cat
74.49
87.66
43.07
42.00
97.00
63.04
75.72
28.39
31.00
95.00
4.3
CASE STUDY: MITIGATING
INTERFERENCE.
Our second design study examines the importance of the long-context training strategy by com- paring the three long sequence data processing approaches from Section 3.3 Adaptive attention masking, EOD concatenation and Direct concatenation. We trained three diffusion LLMs on 64K- context data, all using the Diffusion-aware NTK scaling but each with of the different document processing strategies. We then evaluated each model on LongBench and RULER to see how the post-training method impacts performance. The results, shown the last three rows of Table 4 and 5] reveal that explicitly addressing cross coos-docummet interactions is crucial for long-context training. On LongBench, both boundary-aware strategies outperform naive packing: Adaptive Masking tains the best overall score, followed by EOD concatenation, while Direct concatenation lags. On Ruler, benefits of handling document boundaries grow with context length. At 8K, the EOD concatenation is best, narrowly ahead of Adaptive Masking and Direct concatenation. At 16K, the gap widens in favor of EOD concatenation over Adaptive Masking and Direct Concatenation. Cru- cially, at 32K, Adaptive Masking becomes clearly superior, surpassing EOD Concatenation and far outpacing Direct concatenation. Although Direct concatenation scores slightly higher at 4K than the boundary-aware variants, its advantage vanishes and then reverses length increases.
Takeaway. Between the two boundary-aware strategies, we observe some trade-offs. The EOD concatenation strategy tends perform better shorter moderate lengths. When the context is not extremely long, simply providing boundary tokens is sufficient and perhaps allows the model bit more flexibility (since it can still attend globally). However, longer length (e.g. 32K), the Adaptive Masking overtakes EOD concatenation. This indicates that for very long sequences with many concatenated documents, completely blocking cross-document attention yields the most robust results likely because it prevents any chance of confusion between unrelated content. Crucially, both strategies beat the Direct concatenation. This underscores the importance of explicitly handling document boundaries in long-context training for diffusion LLMs. In summary, our design studies confirm that both core components our approach the Diffusion-awarr NTK scaling and improved post-training data strategy necessary and effective for scaling diffusion LLMs to long context, answering both research questions Q1 and Q2 with a resounding positive result.
4.4 GENERALIZATION TO DREAM
To examine whether our training- based scaling strategy generalizes beyond LLaDA, we further ap- ply it to Dream, representative diffusion L.MM converted from autoregressive model. Dream na-

---

Published as a conference paper at ICLR 2026
tively supports context length of 2K and retains AR-style shifted-labe training objective, which structurally different from LLaDA's unshifted diffusion training objective. We extend Dream from 2K to 32K using our training-basse based pipeline and compare three variants: Dream, the original 2K model; Dream-NTK, the original Dream model with direct NTK scaling applied at inference time without post-training; and Dream-Finetune, the 32K Dream model obtained by post-training Dream with our long-context training pipeline. We evaluate these variants on NIAH-32K and LongBench- v2-Short (tasks with context lengths < 32K). The evaluation configuration and prompt template are the same as those used in Appendix E
Table 6: Generalization of our training- based scaling strategy to Dream.
Model
N-4K
N-8K
N-16K
N-24K
N-32K
LongBench-v2-Short
Dream
100.0
85.2
36.7
23.7
19.3
30.00
Dream-NTK
92.1
59.8
19.8
7.7
4.1
27.78
Dream-Finetune
Finetune
100.0
100.0
100.0
100.0
100.0
31.11
Key Findings. Directly applying NTK scaling to the original Dream model without post-training substantially degrades long-context performance, even falling below the original 2K Dream baseline. In contrast, our training-based approach yields the strongest overall performance: consistently outperforms both the original Dream model and the directly scaled Dream-NTK vari- ant on both NIAH-32K and LongBench-v2-Shoon long-context scaling strategy generalizes effectively to diffusion LLMs derived from autoregressive models and remains robust across different training objectives.
These results suggest that our training-based
5 CONCLUSION
In this paper, we discuss lightweight, post-training route to scale the context window of diffusion LLM to up to 128K tokens. Our approach introduces a Diffusion-aware NTK extrapolation that accounts for bidirectional attention, and complements it with post-training strategies with advanced long-context data masking. Together, these components yield a long-contex diffusion model, UL- TRALLADA, where the experimental results show that ULTRALLADA substantially outperforms training-fres scaling for diffusion LLMs and that both the diffusion-award positional treatment and boundary-aware packing necessary unlock long-context competence.
ACKNOWLEDGMENT
This work is supported by the HKUST startup grant R9895 from CSE; RGC-ECS project 26218024; RGC-NSFC project ter in Guangzhou Nansha Sub-Center.
We thank the support from National Supercomputer Cen-
REFERENCES
Marianne Arriola, Aaron Gokaslan, Justin T Chiu, Zhihan Yang, Zhixuan Qi, Jiaqi Han, Sub- ham Sekhar Sahoo, and Volodymyr Kuleshov. Block diffusion: Interpolating between autore- gressive and diffusion language models. The Thirteenth International Conference on Learning Representations.
Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in neural information processing systems,
2021.
Yushi Bai, Xin Lv, Jiajie Zhang, Hongehang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiy:2308. 14508, 2023.
Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306. 15595, 2023.
10

---

Published as a conference paper at ICLR 2026
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXhh:1803.05455 2018.
Tri Dao.
Flashattention-22: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.086611 2023.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama 3 herd of models. arXiv preprint arXiv:240..21783, 2024.
Yao Fu, Rameswar Panda, Xinyao Niu, Xiang Yue, Hannaneh Hajishirzi, Yoon Kim, and Hao Peng- Data engineering for scaling language models to 128k context. arXiv preprint arXiv:2402. 10171, 2024.
Tianyu Gao, Alexander Wettig. Howard Yen, and Danqi Chen. How to train long- Contecontext language models (effectively). arXiv preprint arXiv:2410.02660, 2024.
gkamradt. Limtest needle in haystack pressure testing 1lms. https://githh: com/ BtM.rradddLLLMettttttM InAHasstack 2023.
Ishaan Guirajani and Tatsunori B Hashimoto. Likelihood-based diffusion language models. Ad- vances Neural Information Processing Systems, 36:1 16693-1615,, 2023.
Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. Ruler: What's the real context size of your long-context language models? arXiv preprint arXiv:2404.06654 2024.
Tianyi Li, Mingda Chen, Bowei Guo, and Zhiqiang Shen. A survey on diffusion language models. arXiv preprint 1008555508. 2025.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint 1rX44:2412.19437, 2024.
Jiaheng Liu, Dawei Zhu, Zhiqi Bai, Yancheng He, Huanxuan Liao, Haoran Que, Zekun Wang, Chenchen Zhang, Ge Zhang, Jiebin Zhang, et al. A comprehensive survey on long context lan- guage modeling. arXiv preprint arXiv:2503.174000 17407, 2025a.
Xiaoran Liu, Zhigeng Liu, Zengfeng Huang, Qipeng Guo, Ziwei He, and Xipeng Qiu. Longllada: Unlocking long context capabilities diffusion 1lms. arXiv preprint 1XXi:25506.14229 2025b.
Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution. arXiv preprint arXiv:2310. 16834, 2023.
Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution. In International Conference on Machine Learning, pp. 32819-32848. PMLR, 2024.
Xinyin Ma, Runpeng Yu, Gongfan Fang, and Xinchao Wang. dkv-cache: The cache for diffusion language models. arXiv preprint urXiv:2505. 1578,,
Shen Nie, Fenggi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large language diffusion models. arXiv preprint arXiv:2502.09992 2025.
Jingyang Ou, Shen Nie. Kaiwen Xue, Fengqi Zhu, Jiacheng Sun, Zhenguo Li. and Chongxuan Li. Your absorbing discrete diffusion secretly models the conditional distributions of clean data. arXiv preprint 2460:24066033366 2024.
Bowen Peng and Jeffrey Quesnelle. Ntk-aware scaled rope allows laaaa models to have extended (8k+) context size without any fine-tuning and minimal perplexity degradation, 2023.
11

---

Published as a conference paper at ICLR 2026
Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarı: Efficient context win- dow extension of large language models. In The Twelfth International Conference on Learning Representations.
Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarm: Efficient context window extension of large language models. arXiv preprint arXiv:2309.0000 2023.
Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yugiong Liu,, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025.
URL https: http/axxvv..ooo org/ abs/ 2412 15115
Jack W Rae, Anna Potapenko, Siddhant M Jayakumar, and Timothy P Lillicrap. Compressive transformers for long-range sequence modelling. arXiv preprint arXiv: /911 1.0550700070.. 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1-67, 2020.
Subham Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and effective masked diffusion language models. Advances in Neural Information Processing Systems, 111333116-3300 2024.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021.
Ning Shang, Li Lyna Zhang, Siyuan Wang, Gaokai Zhang, Gilsinia Lopez, Fan Yang, Weizhu Chen, and Mao Yang. Longrope2: Near-lossless Ilm context window scaling. arXiv preprint arXiv:2502.20082, 2025.
Jiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis Titsias. Simplified and general- ized masked diffusion for discrete data. Advances in neural information processing systems, 37: 103131-103166 2024.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: En- hanced transformer with rotary position embedding. Neurocomputing, 568: 127063, 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiw:2302.13977 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Chengyue Wu, Hao Zhang, Shuchen Xue, Zhijian Liu, Shizhe Diao, Ligeng Zhu, Ping Luo, Song Han, and Enze Xie. Fast-dllm: Training-free acceleration of diffusion 11m by enabling kv cache and parallel decoding. arXiv preprint arXiv:2505.2266 226.12 2025.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388 2025a.
Ling Yang, Ye Tian, Bowen Li, Xinchen Zhang, Ke Shen, Yunhai Tong, and Mengdi Wang. Mmada: Multimodal large diffusion language models. arXiv preprint arXiv:2505. 15809, 2025b.
Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong. Dream 7b: Diffusion large language models. arXiv preprint a5Xiv:2508. /5487, 2025.
12

---

Published as a conference paper at ICLR 2026
Zebin You, Shen Nie, Xiaolu Zhang, Jun Hu, Jun Zhou, Zhiwu Lu, Ji-Rong Wen, and Chongxuan Li. Llada-v: Large language diffusion models with visual instruction tuning. arXiv preprint arXiv:2505.16933, 2025.
Siyan Zhao, Devaansh Gupta, Qinqing Zheng, and Aditya Grover. d1: Scaling reasoning in diffusion large language models via reinforcement learning. arXiv preprint 1rXiv:2504.122111 2025.
Fengqi Zhu, Rongzhen Wang, Shen Nie, Xiaolu Zhang, Chunwei Wu, Jun Hu, Jun Zhou, Jianfei Chen, Yankai Lin, Ji-Rong Wen, et al. Llada 1.5: Variance-reduced preference optimization for large language diffusion models. arXiv preprint arXiv:2505. 2025.
13

---

Published as a conference paper at ICLR 2026
A THE USE OF LLMS IN WRITING
We used LLM, namely OPENAI-GPTS, to polish the writing of this manuscript. No other generative AT functionality is used in the writing of this submission.
B TRAINING SETTING DETAILS.
For reproducibility, we symmarize the training details and model hyperparameters in Table 7
Table 7: Model training settings for all main results.
Model
Initialization:
LLaDA-8B SB-Base:
Tokenizer: the tokenizer used by LLaDA-8B MAAAA--B-BBsse Base;
RoPE/NTK:
DisSuson-awdn
NTK Baseline NIK (as specified per
experiment);
Attention: FlashAttention2
Dao 2023 ;
attention with Adaptive
Masking (bias-based) when spciched in experiment.
Optim.
AdamW (weight decay = 0.1, 81 = 0.9, 32 0.95); LR: cosine, decay iters = 400, warmup 3%, Grad clip = 1.0, peak LR 2e min LR 2e 2::
Batch size: 4M tokens per iteration with 600 iterations;
Dropout: attention 0.0, hidden 0.0.
Parallelism
Megatron with data parallelism (Zero-2) and context parallelism;
trained on a 128-GPU cluster.
Data
PG19 Rae et al. 2019) nulti-document concatenation (64K per chunk); Di rirect Concatenat on or ROD Saaoncatenaa on (as specified per ex-
periment).
C THEORETICAL ANALYSIS OF DIFFU-AWARE NTK.
Below, provide clearer theoretical justification, based on the structure of RoPE, for why diffusion model trained sequences of length Trrain naturally learns relative positions within the range [- 1), Tirain 1].
(1) RoPE makes attention scores depend relative offsets (j- Let the attention head dimension be We partition the query and key vectors into d/2 two-dimensional blocks:
q (go- 91, 92, ....... 4d-2, T k (ko, k1, 12 ky..... ... 2a- ka-1)
The m-th 2D block is associated with angular frequency 0m.
The RoPE transformation position defined as the block-diagonal matrix
R(i) = diag (o(())), R1 (i01), .i., Raf2-1
where each 2 2 rotation block is
c))) sin (i0m..
sin sin(i)m)) cos(if
Rm (i0m))
After applying positional encoding, the query at position and key at position j become
qi R(i)q.
k, = R(jjkk.
Their attention score is
Attn; = q. k, = à R(i) R()k. Using the group property of 2D rotations, for each block we have
Rm (i0m)) Pun (jom)) Rm ((j i)0m)-
14

---

Published as a conference paper at ICLR 2026
Therefore, the contribution of the m-th 2D block can be written
Attm (m) = (q (m)) Bm (Gi - =)0m ( (()) Am cos((j i)0m) + Bm sin((i - i)0m)
where the content-dependent coefficients are
Am 42m + 42m+1
Bn 92m K2m+1 92m++1 k2m-
Notably, An and Bm depend only the content vectors and are independent of positions i and j. Summing over all 2D blocks, the full attention score becomes
d/2-1
Attny Σ
[Am cos ((j ())mm) + Bm sin 1))mm]
This decomposition shows that all positional effects enter the attention score only through the rela- tive offset (j - -). In other words, RoPE explicitly parameterizes attention as a function of relative position.
(2) What range of relative positions does a diffusion model observe during training? During pre- training of a diffusion LLM, attention bidirectional, so both query and key positions can range over the full training sequence:
i E 10.1...., Tirain - 1}-
Therefore, the relative offset satisfies
j-i i 1), Tirain 1].
Since the RoPE attention score is explicitly function of - i), the diffusion model is directly exposed during training to relative positions over the entire range
[-TTrain 1), Train 1].
Equivalently, the model learns a RoPE-induced mapping from relative offsets to attention scores over this full bidirectional domain.
D EMPIRICAL VALIDATION OF THE CRITICAL- ESTIMATE.
To further validate the analytical estimate the diffusion-aware critical dimension, we conduct a finer-grained sweep over nearby settings and evaluate them on LongBench-v2-Shunn (tasks with context length < 32K) and NIAH (8K-33) (3K-2KK) The results are shown Table g We observe that d'mi 70, as predicted by our analysis, is already near-optimal, while d' 73 achieves the best overall performance across the two benchmarks. This suggests that our analytical estimate pro- vides strong initialization point, and that the remaining discrepancy likely arises because different relative offsets are observed with non -nifooofo frequency during training.
Table 8: Finer-grained sweep over nearby critical dimensions.
Critical Dimension LongBench-v2-Sheen
NIAH (8K-32K)
69
27.22
60.97
70
28.89
69.26
71
28,89
71.01
73
31.11
73.23
76
27.78
72.84
79
27.78
63.39
Evaluation Settings. For RULER, set the output length, block length, and number of denoising steps to 32, For LongBench-v2. we set the output length, block length, and number denoising steps to 4. The prompt template used for LongBench-v2 is:
15

---

Published as a conference paper at ICLR 2026
Please read the following text and answer the question below. <text>
(context)
</text>>
What is the correct answer this question: {question)
Choices:
(A) (choice_A)
(B) [choice_B]
(C) (choice_C)
(D) (choice_D)
Answer:
EFFECTIVENESS OF DIFFUSION-AWARE NTK
To further evaluate the effectiveness of diffusion-aware NTK under the same post-training setting, we extend the comparison between diffusion-aware NTK and baseline NTK to longer contexts be- yond those reported the main paper. Specifically, we PraluateRRLLES with a context length of 48K; LongBench-v2 with context length of 64K.
As shown in Table 9 diffusion-awart NTK consistently outperforms baseline NTK across all eval- uated tasks. Here, "LB-" denotes the corresponding LongBench-ve v2 task categories.
-ware Table 9: Comparison between diffusion-awee aware NTK and baseline NTK on long-context evaluation.
Method
LB-Easy
LB-Hard
LB-Short
LB-Medium
LB-Long
RULER
Diffusion-aware NTK
35.94
28.94
37.22
27.91
29.63
56.94 53.75
Baseline NTK
33.33
27.97
35.56
27.44
25.93
These results indicate that incorporating diffusion-specifie relative position spans yields more robust long-context behavior than directly applying AR-oriented NTK scaling.
Evaluation Settings. For RULER, we set the output length, block length, and number of denoising steps to 64. For LongBench-v2. we the output length, block length, and number of denoising steps to 4. The prompt template used for LongBench-v? is the same as that used in Appendix
F POST-TRAINING ON SHORT DATA.
We post-trained two variant models on short data (4K): one trained for 600 steps, aligned with long- context post-training settings, and another trained for 6000 steps to match the total number of tokens used in long-context training. Both models employed the Baseline NTK with the EOD concatenation strategy. We then evaluated them on RULER-4K, with results summarized in Table Interestingly, while long-context post-training leads to higher scores compared to the base model, short-context post-training shows no improvement on RULER-4K. This may indicate that the performance gains of the extended-context model do not stem from additional training data, but may from the context- length extension itself.
Table 10: Post-training using short data.
Model
AVG
NIAH
AGG
QA
ILaDA-8B-Base
86.17
99.19
45.84
67.5
100
UltraLLaDA
88.37
97.31
66.69
68.50
100
4K-60 1K-600steps
85.74
99.31
48.49
62.5
98.2
4K-6000steps
84.16
98.5
51.37
53.50
96.4
16

---

Published as a conference paper at ICLR 2026
G EXTRA EXPERIMENTS ON LONGBENCH-4K.
At the 4K context length, all settings deliver comparable performance. Notably, a similar pattern to that reported in Appendix where context-length scaling results in higher scores than base model.
Table 11: Extra LongBench-4K experiment results.
Model
AVG
SD
MD
Sum
ICL
Syn
Code
LLaDA-8B-Bal
33.45
14.35
12.58
25.62
65.08
28.32
59.71
Latr
35.55
16.67
12.73
25.78
65.38
29.9
67.25
Baseline-NTrE + EOD-cat
34.93
16.12
12.32
24.95
64.42
28.98
67.03
Diffu-NTK + EOD-cat
35.43
15.73
12.81
26.29
65.60
29.00
67.22
Diffu-NTK + Adapt-Mask
35.55
16.67
12.73
25.78
65.38
29.9
67.25
Diffu-NTK Direct-Cat
35.23
15.54
12.67
26.43
64.64
29.63
66.48
H ADDITIONAL LONG-CONTEXT EVALUATIONS BEYOND PPL AND RETRIEVAL
To further assess the robustness of ULTRALLADA beyond perplexity and simple retrieval-based evaluations, expand evaluation more complex long-context comprehension and reasoning tasks at longer context lengths. Specifically, we conduct additional experiments on RULER with a context length of 48K, and on LongBench-v2 with context lengths of 64K and 96K.
Table 12 summarizes the performance ULTRALLADA on RULER-48K, and Table 13 reports
results LongBench-v2 at 64K and 96K.
Table 12: Performance of ULTRALLADA on RULER with 48K context length.
UltraLLaDA
AGG
QA
VT
NIAH
RULER-48K
25.12
24.50
95.20
68.22
Table 13: Performance of ULTRRALLADA on LongBench-v2 at longer context lengths.
UlraLLaDA
Easy
Hard
Short
Medium
Long
LongBench-v2-64K
35.94
28.94
37.22
27.91
29.63
LongBencch-22966
38.54
27.97
37.22
27.77
27.78
These benchmarks cover broader range of long-context comprehension and reasoning tasks than perplexity and simple needle-in-a-haystack retrieval. The results show that UL.TRALLADA remains robust the context length increases, indicating that its gains extend beyond retrieval-heavy settings and generalize to more diverse long-context reasoning scenarios.
Evaluation Settings. in the Appendix D
The evaluation configuration and prompt template are the same as those used
I TRADITIONAL SHORT CONTEXT BENCHMARK CASE STUDY.
We evaluated both the base model and the context-extended model on Winogrande (Sakaguchi et al.) 2021) and ARC-e (Clark et al. 2018). For Winogrande, we used a 5-shot setting with two generated tokens and sampling step of 2. For ARC-c, adopted zero-shot setting with the same generation configuration. The results, summarized in Table [4, show that the context-extended model does not outperform the base model either benchmark. This may be because, similar to auto-regressive models, Diffusion LLMs also tend to experience performance degradation on tasks involving more complex reasoning after context-length extension.
17

---

Published as a conference paper at ICLR 2026
Table 14: Other Benchmarks.
Model
AVG
Winogrande
ARC-c
LLaDA-SB-Baws
79.95
78.53
81.36
Ultral- LaDA
77.42
73.48
81.36
Baseline-NTK EOD-cat
76.10
72.53
79.66
Diffu-NTK BOD-cat
76.35
72.69
80.00
J DETAILS ON TRAINING-FREB CASE STUDY.
We provide detailed experiment results on training-free case study in Section 3.2
NIAH. We compare Diffusion-aware NTK against the Baseline NTK the NIAH retrieval task across context lengths from 4K to 32K tokens. The Diffusion-aware NTK obtains retrieval accu- racy of 79.36%, outperforming the Baseline NTK (75.68%).
Overall AVG Score: 0.7936
Overall AVG Score: 0.7528
0
0-
10
20
1.0
20
1.0
0.8
30
0.8
Depth
40
0.6
Depth
40
0.6
Score
50
50
0.4
0.4
60
60
70
0.2
70
0.2
0.0
80
0.0
90
90
100
100
2K
4K
8K
16K
23K
31K
2K
8K
16K
23K
31K
Context Length
Context Length
(a) Diffuion-wwre NTK.
(h) Baseline NTK.
Figure 4: NIAH evaluation from 4K to 32K context-length.
LongBench. We compare Diffusion- NTK with Baseline NTK on the LongBench (cut at 16K)
benchmark. Diffusion-awaad NTK slightly outperfomrs the baseline.
Table 15: LongBench166 6K training-fres case study.
Model
AVG
SD
MD
Sum
ICL
Syn
Code
Baseline NTK
35.02
14.91
11.71
24.92
66.30
39.62
60.55
Diffu-aware NTK
35.38
15.14
11.26
24.79
67.22
40.56
61.53
18


Reviews:
[{"review_id": "ZxXuKRy8RY", "reviewer": "Reviewer_BN3Q", "summary": "This paper introduces UltraLLaDA, a diffusion LLM post-trained from LLaDA with a modified RoPE scaling method to extend the context length to 128K. UltraLLaDA proposes a diffusion-aware NTK/RoPE scaling method. The authors argue that standard NTK-aware RoPE scaling (as used in autoregressive long-context extension) is suboptimal for diffusion LLMs because diffusion models use bidirectional attention rather than causal attention. They propose a modified scaling that better reflects the positional distance statistics of bidirectional denoising. They then conduct long-context post-training on 64K length packed sequences, exploring several strategies for handling multiple concatenated documents: naive concatenation, EOD-token concatenation, and adaptive attention masking. The experimental results show that UltraLLaDA maintains near-perfect Needle-in-a-Haystack retrieval up to 128K and keeps perplexity stable out to 128K, whereas both the original LLaDA and the training-free method LongLLaDA collapse much earlier.", "questions": "1. Could you evaluate UltraLLaDA on more kinds of tasks beyond PPL and NIAH at longer context lengths? Are there any failure cases at 128K context length (because the current NIAH evaluation results seem perfect)?\n\n2. Could you apply the proposed method to models like Dream (which is trained from AR model) and SDAR-series models (which are also trained from AR models with block diffusion style)?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. **Good practical significance for diffusion LLMs.**  \n   Ultra-long context capability in diffusion LLMs has not been well explored. This work provides a realistic recipe to reach 128K with stable retrieval and usable perplexity, which is a meaningful capability milestone for dLLMs.\n\n2. **Comprehensive study of the proposed diffusion-aware NTK.**  \n   The paper presents a detailed analysis of the proposed diffusion-aware NTK, comparing it with LongLLaDA's baseline NTK, and shows the empirical improvement of diffusion-aware NTK in the training-free setting. This analysis is convincing.\n\n3. **Comprehensive experimental results and ablations.**  \n   This work conducts experiments on various long-context tasks, including Needle-in-a-Haystack, LongBench, and RULER, for all baselines (LLaDA, LongLLaDA) and UltraLLaDA trained with different sentence packing strategies. This thorough comparison supports the proposed method.\n\n4. **Clear writing.**  \n   The paper is clearly written and easy to follow.", "weaknesses": "1. **Core novelty feels incremental.**  \n   The main contribution of this work is diffusion-aware NTK. While the motivation (bidirectional vs. causal attention) is reasonable, the scaling rule itself shows only a moderate improvement over LongLLaDA’s baseline scaling when evaluated without post-training. From Table 4 and Table 5, even with post-training, the improvement for 4K–16K context lengths still seems small. The three long-context post-training sentence-packing strategies explored in this paper are also existing methods in autoregressive post-training. Applying and comparing them in the diffusion LLM setting is empirically valuable but not conceptually new.\n\n2. **Limited analysis of non-retrieval reasoning at longer context lengths.**  \n   Most ultra-long-context evaluations are on NIAH and perplexity stability. The paper does evaluate on LongBench with 16K context length and RULER at 32K, but there is less evidence for complex multi-document synthesis or instruction following at 64K–128K.\n\n3. **Lack of evaluation on more models.**  \n   The paper only conducts experiments on LLaDA model. There are multiple kinds of diffusion LLMs: models trained from scratch in diffusion style like LLaDA, models converted from AR like Dream, and block-diffusion-style models. This work should apply the proposed method to more diffusion LLMs to demonstrate generalization.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Up2B2e6pmO", "reviewer": "Reviewer_eBLu", "summary": "This paper introduces UltraLLaDA, a post-training approach designed to extend the context window of diffusion-based large language models (specifically LLaDA) from 4K to 128K tokens. The authors make two main contributions: first, they develop a diffusion-aware NTK scaling method that refines standard RoPE extrapolation to better suit the bidirectional attention mechanisms of diffusion models, using a context cap of approximately twice the training length rather than the single-length limit typical of autoregressive models. Second, they explore several masking strategies for managing multi-document concatenation during training, comparing adaptive masking, the use of explicit end-of-document (EOD) tokens, and direct concatenation. UltraLLaDAdemonstrates strong performance on long-context benchmarks such as NIAH, LongBench, and RULER, outperforming the training-free baseline, LongLLaDA.", "questions": "1. Can you provide theoretical or empirical analysis showing that diffusion models actually learn relative positions in the range [-2T_train, 2T_train] during pre-training?\n2. Why was NTK-aware/ABF RoPE scaling used instead YaRN? Is there a intrinsic problem with YaRN that prevents it from being adapted to diffusion models? SoTA models are mostly always using YaRN instead of NTK scaling.\n3. Have you tested this approach on other diffusion LLMs or other model sizes? Would it work without finetuning on larger model sizes? What about other architectures?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Systematic study of post-training methods for extending context in diffusion LLMs, addressing an underexplored area.\n2. Well-motivated technical approach. The diffusion-aware NTK modification is intuitive, accounting for bidirectional attention means the model sees roughly 2x the relative position range during training.\n3. Comprehensive evaluation. Multiple benchmarks (PPL, NIAH, LongBench, RULER) across context lengths up to 128k tokens\n4. Lightweight post-training (600 steps) makes the approach accessible.", "weaknesses": "1. Single base model. All experiments use only LLaDA-8b as the base. Generalization to other diffusion LLMs or different model sizes is unclear.\n2. LongLLaDA baseline cannot be evaluated beyond 32k, making comparisons incomplete at the longest contexts.\n3. Masking strategy conclusions unclear. Tables 4-5 show adaptive masking and EOD concatenation trading advantages at different lengths, but the paper doesn't provide clear guidance on which to use when. The difference between all methods are very small and could be attributed to noise.\n4. No comparison against YaRN, a more widely used RoPE interpolation method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9pJZjaygIn", "reviewer": "Reviewer_ev1L", "summary": "This paper presents UltraLLaDA, a method for extending the context length of diffusion-based large language models to 128K tokens. The approach introduces a diffusion-aware NTK scaling technique and explores various masking strategies for handling long-context documents. Experimental results demonstrate the effectiveness of the proposed method.", "questions": "See the weaknesses section", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written and easy to follow.\n\n2. The proposed diffusion-aware NTK scaling method is simple yet effective, showing a certain level of novelty.\n\n3. The paper conducts extensive experiments that clearly demonstrate the effectiveness of the proposed approach.", "weaknesses": "1. The proposed diffusion-aware NTK scaling is primarily based on an empirical assumption that a diffusion LLM can naturally handle a wider range of relative positions, with $T_{cap} \\sim 2T_{train}$ and $T_{Ecap} \\sim 2T_{target}$. It would be more convincing if the paper provided deeper theoretical justification or analysis for this assumption.\n\n2. The investigation of different masking techniques for long documents shows limited novelty, as similar findings have already been discussed in prior studies on large language models[1,2].\n\n[1] LongRoPE2: Near-Lossless LLM Context Window Scaling\n\n[2]  The Llama 3 Herd of Models, https://arxiv.org/abs/2407.21783", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kAv8K162O5", "reviewer": "Reviewer_S1Du", "summary": "This paper introduces UltraLLaDA, a method for extending the context window of diffusion-based Large Language Models (LLMs) to 128,000 tokens through an efficient post-training process. The research addresses a critical and largely unexplored area, as the long-context capabilities of diffusion LLMs have not been systematically studied.   \n\nThe authors propose two main contributions:\n\nDiffusion-aware NTK: A novel adaptation of the Neural Tangent Kernel (NTK) method for scaling Rotary Positional Embeddings (RoPE). The key insight is that diffusion models, with their bidirectional attention mechanism, learn a much wider range of relative positions during pre-training (approximately twice the context length) compared to auto-regressive models. By accounting for this property, the authors develop a more suitable RoPE scaling factor that enables stable extrapolation to very long contexts.   \n\nMasking Strategy Analysis: The paper systematically investigates data packing and attention masking strategies to mitigate \"cross-document interference\" during long-context fine-tuning—a significant challenge for models with global bidirectional attention. It compares adaptive attention masking (which blocks attention between concatenated documents) and end-of-document (EOD) token concatenation against a naive direct concatenation baseline.   \n\nEmpirically, UltraLLaDA demonstrates remarkable performance. It achieves 100% accuracy on the \"Needle-in-a-Haystack\" (NIAH) retrieval task at the full 128K context length. Across a suite of benchmarks, including Perplexity, LongBench, and RULER, UltraLLaDA consistently and significantly outperforms the base LLaDA model and a training-free extension baseline (LongLLaDA), with the performance gap widening as context length increases. Ablation studies confirm that both the Diffusion-aware NTK and the use of boundary-aware masking strategies are essential for achieving these results.", "questions": "Q1: In Section 3.2, the explanation provided for $T_{\\text{cap}}$ and $T_{\\text{Ecap}}$ being twice as large in diffusion LLMs compared to auto-regressive LLMs is intuitive. However, the argument would be significantly strengthened by a more detailed theoretical derivation or formal proof to rigorously support this conclusion.\n\nQ2: The paper compares the model's training-free performance at critical dim = 64 versus 70. It would be insightful to see an analysis of performance at other proximal values (e.g., 69 or 71). Investigating whether 70 represents an optimal or near-optimal setting would provide stronger validation for the optimization process of the Diffusion-aware NTK method.\n\nQ3: The evaluation of long-context capabilities currently concentrates primarily on retrieval tasks. To provide a more comprehensive assessment, please consider including benchmarks that evaluate long-context reasoning abilities, such as the relevant tasks in LongBench v2.\n\nQ4: The experiments cap the maximum sequence length extension at 128k. Could the authors clarify the rationale for this specific limit? It would be beneficial to understand the method's performance at even greater lengths or, alternatively, to discuss the method's ultimate scaling limit (e.g., what is the maximum extension factor this approach can effectively achieve?).", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "Addresses a novel and important problem: long-context extension for diffusion LLMs. As diffusion models gain traction, understanding how to scale their context window is crucial for their practical application and competitiveness.   \n\nThe model's performance is a standout feature. Achieving perfect 100% accuracy on the 128K NIAH task is a powerful demonstration of the method's effectiveness in long-range information retrieval. The consistent and significant improvements over baselines across multiple diverse benchmarks (PPL, LongBench, RULER) provide robust evidence supporting the authors' claims.   \n\nThe core technical contributions are simple, intuitive, and clearly justified. The adaptation of NTK scaling is based on a clear-eyed observation of the architectural differences between diffusion and auto-regressive models. Furthermore, the systematic study of masking strategies provides valuable, practical insights for training such models.\n\nThe paper includes thorough ablation studies that successfully isolate the impact of each key component (the NTK variant and the masking strategy). This experimental rigor greatly strengthens the validity of the conclusions and clearly demonstrates that both proposed techniques are necessary for the final performance.", "weaknesses": "The narrow set of baselines, all comparisons are internal to the diffusion model LLaDA and LongLLaDA. Considering that this is relatively new in diffusion LLM, this is understandable, but perhaps could consider migrating other methods commonly used in auto-regression models for comparison, such as PI and YARN, to provide more insights.\n\nThe appendix reveals that UltraLLaDA's performance on standard short-context benchmarks degrades after long-context fine-tuning. This is a critical trade-off common in context extension methods but is not addressed in the main body of the paper. Acknowledging and analyzing this limitation in the main text would provide a more balanced and complete picture of the method's characteristics.   \n\nThe evaluation could be strengthened by incorporating more challenging long-context reasoning benchmarks to provide a more comprehensive assessment of the model's capabilities beyond information retrieval.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)