Unveiling the Potential of Diffusion Large Language Model in Controllable Generation
Authors: Zhen Xiong
Yujun Cai
Zhecheng Li
Yiwei Wang
Venue: ICLR 2026 Poster (2026)
Keywords: diffusion large langauge models
controllable generation
structured output
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Controllable generation is a fundamental task in NLP with many applications, providing a basis for function calling to agentic communication. However, even state-of-the-art autoregressive Large Language Models (LLMs) today exhibit unreliability when required to generate structured output. Inspired by the current new diffusion-based large language models (dLLM), we realize that the architectural difference, especially the global information-sharing mechanism for language modeling, may be the key to unlock next-level controllable generation. To explore the possibility, we propose Self-adaptive Schema Scaffolding ($S^3$), a novel framework that enables dLLM to stably generate reliable structured outputs (e.g., JSON) by utilizing its innate reverse reasoning capability and global context awareness. $S^3$ initiates a schematic template directly in the output context as a starting state for dLLM, offering a more robust and general method than intricate prompt optimization. Experiments demonstrate that our method substantially unlocks the dLLM’s potential in controllable generation in terms of structure adherence, content fidelity, and faithfulness. These results establish new perspectives and practical pathways for deploying language models in controllable generation tasks.

Published as a conference paper at ICLR 2026
UNVEILING THE POTENTIAL OF DIFFUSION LARGE LANGUAGE MODEL IN CONTROLLABLE GENERATION
Zhen Xiong' Yujum 2,
Zhecheng Li
Yiwei Wang
University of Southern California Universiity of California, San Diego er eric2i github. 1o/dLIM-Ctr1Ge
University of Queensland 3 Ant Group
University of California, Merced
ABSTRACT
Controllable generation is a fundamental task in NLP with many applications, providing a basis for function calling agentic communication. However, even state-of-the art autoregressive Large Language Models (LLMs) today exhibit un- reliability when required to generate structured output. Inspired by the current new diffusion-based large language models (dI.MM, we realize that the archi- tectural difference, especially the global information-sharing mechanism for lan- guage modeling, may be the key to unlock next-level controllable generation. To explore the possibility, we propose Self-adaptive Schema Scaffolding (5"), a novel framework that enables dLLM to stably generate reliable structured outputs (e.g..JONN by utilizing its innate reverse reasoning capability and global context awareness. 5 initiates schematic template directly in the output context as starting state for dLLM,, offering a more robust and general method than intricate prompt optimization. Experiments demonstrate that our method substantially locks the dLLM's potential in controllable generation in terms of structure adher- ence, content fidelity, and faithfulness. These results establish new perspectives and practical pathways for deploying language models in controllable generation tasks.
INTRODUCTION
Controllable generation a fundamental task in the era LLMs. provides the foundation for stable tool use, agentic communication, and interaction with existing application programming in- terfaces (APIs). Existing works demonstrate that structured output still poses significant challenges even for state-of-the-art autoregressive LLMs. Many inspiring explorations have been conducted by previous researchers to address these challenges.
Prior autoregressive-baaed language model's methods pair grammar-driven finite-state automata (FSA) with constrained decoding to enforce structural constraints during generation (Koo et al. 2024 When token satisfies the grammar, all beams pruned and generation halts. More broadly, as instruction following has improved, practitioners have turned to prompt-engineering heuristics to elicit structurally compliant outputs. Yet hand-crafted prompts for diverse structural specifications are labor-intensive and yield inconsistent results across domains and complexity lev- els.
Existing approaches share fundamental limitation: they rely solely on language models' intrin- sic capabilities without additional mechanisms to guide generation trajectories. This stems from architectural constraints of autoregressive models: (1) left-to-right generation prevents global struc- tural coherence, as early tokens are generated without full sequence knowledge; (2) commitment to previously generated tokens limits backtracking when structural violations occur; (3) sequential dependencies inhibit simultaneous satisfaction of multiple constraints. Effective structured gener- ation thus requires global sequence planning, iterative refinement, and parallel constraint satisfac- tion- capabiiiiil that autoregressive architectures inherently lack.
"Corresponding author.

---

Published as a conference paper at ICLR 2026
Write short acmoetic using the word HOPE.
Diffusion- based Modeling Sure. Here is acrestic poem:
Here 1s
poos:
Sure. Here is the acrostic acrostic poen: "geoold on through darkest tires, "erreer
times,
ood darkest
Autoregnessive Modeling
times, *0 hills, the
Sure! "Held through the darkest kills, shine. The stars appean after store, Every storn pass with time." "
*0
the sepereace will core whispered riynes, *xerrevery store will pass, with time....
shine,
nice
reperush whispered rtynes,
**g* has
.gg* stom will
Global Aware
with
Iterative Refine
Global Aware
Iterative Refine
Illustrative comparison between autoregressive and diffusion-basse language modeling on tasks requires specific global structure control and token-space planning in advance.
Figure I:
Under this scenario, we realize that the diffusion-based large language model (dLLMM can serve as a natural alternative to traditional autoregressive (AR) generation by iteratively denoising corrupted inputs, enabling global context modeling potential and parallel token generation (Arriola et al. 2025
Nie et al., 2025; Yu et al., 2025). Recent studies show that dLLMs can match AR models in instruc-
tion following, in-context learning, and math reasoning tasks (Gong et al.] [222 You et al.
2025
while also offering enhanced controllability and faster inference (Huang & Tang 2025
Labs et al.
2025).
However, current open-sourced instruction-tuned dLLMMs (Nie et al. 2025 Zhu et al. 2025 Ye et al,
20256) fail produce well-structured outputs, often generating hallucinated content breaking structural constraints. Additionally, their inference speed remains limited. Analysis reveals that existing implementations (c.g., ceni-auttorgressive approaches Nie et al. (2025)) systematically undermine dLLMss advantages in global awareness and parallel generation.
To unveil its full potential, we introduce novel Self-adaptive Schema Scaffolding (S²) method to fully unlock dLLM's potential in controllable generation. Specifically, S inject a schematic tem- plate into the output context instead of instruction, providing a more compelling language prior for dLLM during generation. With our method, dLLMM can achieve significantly improved structured output quality. To quantify performance, we introduce a comprehensive framework that evaluates structured outputs along three key dimensions: structural adherence, content fidelity, and faithful- ness (with detailed definitions provided in Section 5). Experimental results show that our method marginally improves the performance structured output using dLLM compared with the commonly used prompting strategy.
Our main contributions be summarized as follows: 1) We analyze the architectural advantages of diffusion-based large language models (dI.LMs) compared with autoregressive models for con- trollable generation, focusing on prior's global attention mechanism and iterative refinement capa- bilities. 2) We propose Self-adaptive Schema Scaffolding (SS), a training-free method that enables dLLMs achieve higher structure output performance with fewer denoising steps. 3) We establish comprehensive structure output evaluation framework focusing on structural adherence, content fidelity, and faithfulness metrics. Extensive experiments show that our S achieves superior perfor- mance across all metrics with reduced computational complexity. Together, our work establishes new perspective and practical solutions for deploying dLLMSs for controllable generation tasks.
RELATED WORKS
Structured Output requires models generate in predefined formats (c.g., code, JSON, XML, or tables), supporting tasks such as entity extraction (Li et al.) 2024). classification Huang et al. (005)). , and correlation prediction Xiong et al. (205). Grramaa-coostrainee decoding enforees compliance with context-free grammars (Geng et al., (023) or type systems (Mündler et al. 2025) by adjusting next-token probabilities, without task-specific fine-tuning. Alternatively, planning- based or two-stage strategies first predict intermediate structures, such abstract syntax trees, and then realize the final output (Wang et al., 2025). Most prior work relies on autoregressive LLMs for their strong language modeling and instruction-followwe following ability (Wei et al. 2022). Diffusion-basse models remain underexplored, motivating us to analyze their potential for structured generation and propose an evaluation framework covering structure compliance, content fidelity, and hallucination.
2

---

Published as a conference paper at ICLR 2026
Autoregressive Large Language Models Autoregressive LLMs have become a general solution across NLP tasks, strengthened by advances such as longer context windows (Liu et al., 2025) multimodal integration (Han et al.
2025), and test-time scaling (Jachh et al. 2024 Guo et al.
2025 Gemini Team, Google 2025
They achieve state-of-the-art results on benchmarks includ-
ing MMLU (Hendrycks al. 2020), WehArena (Zhou et al., (2023), and AIME (Zhang 2025). but
still face persistent challenges in hallucination and controllability.
Diffusion-based Language Models Diffusion models have recently been extended to discrete text, offering an alternative to autoregressive generation (Li et al. 2023). Research has introduced discrete score-based processes, refined noise schedules, and faster sampling methods, all aimed improving efficiency and output quality. Building on these advances, large diffusion language mod- els such as LLaDA (Nie al., 2025 Zhu al.. 2025) and Dream (Ye al. (0256) demonstrate strong instructionfollowwiie tage of parallel generation and slows inference (Israel et al., (025). Motivated by these limitations, we focus structured output setting where diffusion models can better exploit their architec-
ability. Yet their reliance on multi-step denoising weakens the advan-
ture, and propose an inference pipeline that accelerates generation for practical applications.
PRELIMINARY AND NOTATIONS
3.1 AUTOREGRESSIVE LANGUAGE MODELING
Autoregressive large language models (LLMs) generate text by predicting the next token. During inference, instruction-tuned models (Ouyang et al. 2022 a query Q by sampling from the conditional distribution
Wei et al. 2022) produce a response A to
|A|
log Po(A)Q) = Σ log Polaclazis Q),
where A (a....., ) ) sequence length, and 0 are model parameters.
is the token at position t, act are the preceding tokens, |A| is the
Due to the sequential nature of autoregressive generation, there are two systematic limitations: mod- els cannot directly access future token information during generation, and previously generated tokens cannot be revised. These constraints may limit the model's 's performance on tasks that would benefit from lookahead planning or itcrative refinement.
3.2 DIFFUSION LANGUAGE MODELING
Diffusion models (Ho et al. 2020 Nichol & Dhariwal 2021), originally developed for continuous
image generation tasks, have been adapted to discrete language modeling (Austin et al. 2021a Nic
al. 2025 Ye et al. 2025b).
Diffusion- language models consist of two key processes: a forward noising process and a re- verse denoising process. Given tokenized sequence Xg, the forward process progressively corrupts the sequence by masking tokens, producing increasingly noisy sequences x, for E [o, 1]. As t ap- proaches 1, more tokens are masked until æı becomes fully masked sequence. The reverse process trains a neural network to predict the original tokens at masked position within x; for E (0, 1]. The pre-training objective can be formulated as:
min Er ti
Σ
lkl MJ log Po (ar
where M denotes the mask token, L. is the sequence length, and & represents the model parameters.
Recent research showed that diffusion large language model (dLLM) can be enstruttionfiiettunee by concatenating (+) user instruction Q as fixed prefix to the masked target sequence A, (t = 1, all masked initially), enabling conditioned generation. During inference, a single denoising step that the dLLMM take can be factorized as:
AA]
log (Q
(Q0A) A)= Σ
I[a = M log Ps Ps(a/ + A,)
(3)
1=1
3

---

Published as a conference paper at ICLR 2026
conted condition
Problem
Structured
Description
Compile
initialize
dennise
Output Task
Structure
Sekema
decouple
Construint
Noisy Scaffold
Structured Output
sclecuve ronas.
The overview of our method's pipeline. We begin by decomposing the original task struction into two components: problem description and a set of structural constraints. These constraints are compiled into a schema, which then used to initialize a noisy scaffold where mask tokens serve placcholders for missing content. The dLLM completes this scaffold by predicting the masked tokens, using the problem description as context to generate structured outputs. Ad- ditionally, apply selective remasking strategy that allows the model to iteratively refine its predictions and further improve generation quality.
Figure 2:
Compared with AR models, the multi-step generation process can potentially be extended for it- erative token edition and refinement (Havasi et al. 2025). More interestingly, the global attention mechanism of dLLM can largely improve its global context awareness and even including future planning capabilities (Ye et al. 2025b 2024 2025a) (See Appendix (G). Therefore, in this paper, exploit dLLM'ss superior global awareness and explore its potential for structured output.
METHODS
In this section, we first establish a rigorous theoretical framework for structured output generation (Section 4.1. Subsequently, introduce our training-free pipeline, Schema Scaffolding (S (Sec- tion 4.2 and its improved version Self-adaptive Schema Scuffolding (S (Section 4.3
4.1 TASK FORMALIZATION
Formally, we define structured output generation task as follows: given user query Q, and struc- tural specification S, the objective is to generate response A that satisfies both the semantic quirements of Q and the structural constraints defined by S. The structural specification S can take various forms, including but not limited to: cheeaa-based structures, format constraints, compo- sitional structures, and domain-specifie formats. The task can be mathematically formulated as a constrained optimization problem:
A' arg max PLM(A(Q, AEA(S)
where A(S) represents the space of all valid outputs conforming to structure S and PLM is condi- tional langugage model. In practice, searching entire A(S) is intractable due to the complexity of the token space.
4.2 SCHEMA SCAFFOLDING
we propose schemu scaffolding, training-fre free approach that explicitly incorporates structural constraints by pre-ooulaaiing the gen- eration context with structural templates.
Building on the theoretical insight we formulated in Section 3.2
The core idea of this approach to transform unconstrained generation into a structured fill-in- the-blank task: In detail, method operates by parsing the structural specification S to identify invariant structural elements (e.g.. brackets, delimiters, field names) and replacing variable content positions with mask tokens M. This creates structural scaffold A, that constrains the model's gen- eration space while preserving semantic flexibility (See Appendix [ for more details). To formalize why this approach is effective for diffusion models, establish the following result:
Theorem 4.1 (Scaffold- Guided Denoising Convergence). Let Xa be target structured sequence and x, be partially masked sequence at timestep t with scaffold defining fixed structural po- sitions. For diffusion language model trained with objective (Eq. 2 initializing the denoising
4

---

Published as a conference paper at ICLR 2026
process with structural scaffold S reduces the expected denoising enror by:
[ - xxllMM 2 Elllxo xollm] (1- [1]
(5)
where Ko is generated with scaffolding. Xo generated without scaffolding, IIl denotes error over masked positions only, and |8| represents the scaffold coverage ratio.
The proof deferred to Appendix F.1 way guide the denoising process, with error reduction proportional to the scaffold coverage. also confirms our later empirical findings where even minimal scaffolding achieves near-perfect structural adherence.
This result confirms that scaffolding provides a principled
Formally, the structured generation objective of our proposed method here is:
A = arg max PIM( PiM(A(Q.S) AEA(S)
R arg max (A)
A,ESC
(6)
arg max A,ESC
Σ 0,
1/a = M log Po(a;lQ,A.)
Where SC A(S) represents the constrained subspace of outputs sharing the structural template derived from S, and A, denotes the scaffold with all non-masked tokens fixed except position
4.3 SELF-ADAPTIVE SCHEMA SCAFFOLDING
While our previous strategy constrains dLLMM generation for improved structural compliance, it also introduces another challenge: how to determine the appropriate number of mask tokens for each variable content position? Although we can build structured fill-n-thheblank templates in advance, predicting the required length for each variable field remains problematic without prior knowledge of the target content.
A straightforward solution is to allocate ample mask tokens for each variable position, expecting the model to use only what is needed. Yet our analysis (Sec. 6.1 shows that dLLMs sensitive to sequence length: longer scaffolds often distort generation quality instead of enabling selective usage, leading under-utilization or hallucinated content. Another option introduce specialized padding tokens through fine-tuning. While appealing in principle, this violates our training -fee objective and risks embedding dataset-specific biases that harm generalization to unseen domains.
Motivated by these intuitions, we propose an improved method that leverages the semantic token null an placeholder. This approach preserves the training-fre ffre property while guiding dLLMs to naturally represent absent or variable-length content with null tokens.
Formally, extend our scaffolding framework incorporate adaptive length management:
A Rs arg max Σ 1(a 1 = M] log (asl As)
A,ESC 0, E,E
where Q+ represents our augmented prompt which guide the model to adopt null tokens to indicate absent values.
Our approach here empirically transforms the fixed-length scaffolding problem into adaptive generation task where dLLMSs can naturally handle variable-length fields and missing values. Fur- ther experimental results demonstrate that Self-Adapti Schema Scaffolding significantly improves overall structured output quality, particularly in scenarios involving optional fields or variable-length content.
5

---

Published as a conference paper at ICLR 2026
5 EXPERIMENTS
5.1
IMPLEMENTATION DETAILS
We use the LLaDA Nie al. 2025) model as our primary dLLM for experiments and the WikiBio dataset (Lebret et al. 2016) as our dataset. For reproducibility, we set the decoding temperature to zero during inference. See Appendix A for more details.
5.2 EVALUATION FRAMEWORK
Given the unique characteristics of structured output generation, argue that traditional accuracy- based metrics alone are inadequate for capturing the full spectrum model performance. Thus, we propose comprehensive evaluation framework that assesses outputs across three key dimensions: Structural Adherence, Content Fidelity, and Faithfulness. This multi-dimensional benchmark enables nuanced and reliable assessment of model's ability generate coherent, informa- tive, and trustworthy structured texts.
Structural Adherence measures how well-generated outputs conform to the target schema. Specif- ically, we define:
Structure Validity (SV): the proportion of outputs that are syntactically valid and parse without errors, capturing basic structural correctness.
Field Completeness (FC): the percentage of required fields that are correctly populated, indi- cating whether the model includes all mandatory schema components.
Schema Compliance (SC): the strictest structural metric, measuring the proportion of outputs that fully adhere the predefined schema, including correct data types, value constraints, and nested structures.
Content Fidelity evaluates the semantic accuracy of information within structurally valid outputs. We consider:
Precision/Recall (PR/RE): precision and recall scores computed over individual field types, providing detailed view of model behavior across content categories.
Fl Score (FI): the harmonic mean of field-level precision and recall, computed using both exact match and fuzzy match strategies accommodate minor textual variations that preserve semantic meaning.
Faithfulness assesses the degree to which generated content remains grounded in the source input, which is especially critical in extraction settings.
Hallucination Rate (IRR): the proportion of output fields that include information not present in or not reasonably inferable from the source text, directly reflecting the model's factual consistency.
Together, these metrics support systematic comparisons across models and highlight specific areas for targeted improvement in structured output generation.
5.3
MAIN RESULTS
This section presents a comprehensive evaluation our approaches against baseline methods across three critical dimensions: structural adherence, content fidelity, and faithfulness.
5.3. STRUCTURAL ADHERENCE
Direct prompting dLLM is inadequate for structural constraint. Our bascline evaluation reveals performance consistently below 65% across all structural metrics-Schema Validity, Field Com- pleteness, and Schema Compliance. Even with 32 denoising steps, the maximal score among these three critical metrics is above 87%, far lower than the expectation for realistic uFility. 3)
Our schema scaffolding methods marginally outperform the baseline method in terms of structural adherence. Both vamilla and self aelf-adaptive variants enable near- sear-perfect structural adherence with as few 8 denoising steps, achieving performance saturation 16 steps. This dramatic improvement

---

Published as a conference paper at ICLR 2026
Dascine
Schemn Seathuling
Self wlaptse Schera Soadokinng
Couppsscnss
Complrace
.003.99%
S4N W//
1.0
100519
1.0
.300.911
0.8
0.5
6.5
.712
50000
0.6
0100
664
0.04
.177
310
912
16 Stops
32
16
32
6
52
Steps
Figure 3: Structural adherence comparison across denoising steps and methods. Results show consistent improvements across all metrics using our schema scaffolding approaches, with near- perfect performance achieved in fewer steps.
carries practical significance beyond accuracy: since diffusion model inference scales linearly with denoising steps, our approach simultaneously reduces generation latency while enhancing struc- tural quality. These findings establish that schema scaffolding addresses the fundamental barrier to deploying diffusion- based language models for structured output generation, elevating them from impractical to highly effective for real-world applications.
5.3.2 CONTENT FIDELITY
For content fidelity, additional denoising steps counterintuitively do not guarantee improved content accuracy and we observe that performance sometimes degrades with extended iteration (Fig. 4 This pattern reflects the diffusion model may deviation from optimal solutions during extended iterative reverse process, what term the "overthinking" phenomenon.
Vanilla Schema Scaffolding demonstrates clear improvements in recall and Fl score relative to baseline, indicating enhanced coverage of rel- evant content. However, precision suffers no- tably across all denoising configurations. This trade-off emerges from the model's compen- satory behavior: when constrained by rigid schema requirements, it over-generates tokens to fill all reserved slots, particularly problem- atic when content length varies significantly across examples.
Computation Budget
Method
8 Steps
16 Steps
32 Steps
Baseline
0.404
0.403
0.409
Sa
0.465
0.463
0.463
S
0.340
0.331
0.331
Table 1: Hallucination rate (lower is better) comparison across various denoising steps. Our self-adaptive schema scaffolding (83) method consistently achieves the lowest hallucination rates, indicating superior faithfulness. Boldface indicates the best results.
Our Self-adaptive Schema Scaffolding (S3) re- solves this tricky situation. By incorporat- ing adaptive null token for surplus slots, the method prevents over-generation while main- taining comprehensive coverage. This simple yet effective recipe yields substantial improvements across all three metrics, establishing robust content fidelity without sacrificing structural compliance.
5.3.3 FAITHFULNESS
Our self-adaptive schema scaffolding (59) method demonstrates superior faithfulness performance, consistently achieving the lowest hallucination rates across all denoising steps (Tab. [ This clear advantage over baseline approaches establishes the superiority of our method in maintaining factual grounding while generating structured outputs.
Interestingly, vanilla schema scaffolding reveals elevated hallucination rates compared to baseline methods. The worse performance faithfulness represents a critical weakness that challenges our original design of the Schema Scaffolding approach. Our analysis reveals that this hallucination issue stems from the distributional shifts introduced by the imposed structural constraints, which create token distribution that diverges from the model's pre-training reverse process. Specifi-

---

Published as a conference paper at ICLR 2026
Dascine
Schemn Seathuling
Self wlaptre Schema Soadokinng
0.12
1::2
2.11
0.150
0:15
0.33
0.11
0.10
0.125
11120
0.310
001
50000
2.4m
5000
0.070
0.075
0.94
0.030
0.000
0.12
0.055
0.00
16
0.000
8
0.000
16
30
Stops
Stape
Figure Content fidelity comparison across denoising steps and methods. Our self-adaptive schema scaffolding consistently achieves the highest precision, recall, and Fl score across all set- tings.
cally, the rigid schema acts as a structural prior that misaligned with the diffusion language model's learned denoising process. This distributional mismatch forces the model into suboptimal denoising trajectories. When the structural prior demands content generation beyond what can be grounded in the source text, the model defaults fabricating plausible tokens satisfy schema requirements-a behavior that contradicts its training objective of faithful reconstruction.
Our self-adaptive approach mitigates this fundamental confiict by allowing the model acknowl- edge missing information through null tokens, thus maintaining alignment with its pre-trained denoising capabilities while respecting structural constraints.
5.3.4 ABLATION STUDY
To demonstrate the effectiveness of our approach, we conduct ablation studies comparing our method against baseline models that are incrementally enhanced with different techniques (Tab2). While few-shot learning and template-as-guidance approaches improve dLLM's structural adher- ence and faithfulness, their performance on fidelity remains inconsistent and shows limited im- provement. In contrast, our zero-shot method achieves superior data efficiency and computational efficiency. We also observe strong diminishing returns beyond 8 denoising steps, indicating that the structural constraints provided by the scaffold are realized rapidly during inference. Despite requir- ing fewer denoising steps, S delivers remarkably stable structural adherence, along with marginal improvements in both fidelity and faithfulness. We also tested additional datasets to validate the generalizability of our approach (See Appendix more results).
DISCUSSION
6.1 ROBUSTNESS AGAINST HALLUCINATION
Similar to autoregressive language models, diffusion-based language models (dLLMs) also suffer from hallucination-geee -etiiattionnggneattninn generating nonfactual content, faulty reasoning, or unsupported conclusions. Unlike open-ended dialogue or creative text generation, hallucination in structured output is particu- larly detrimental, that are central to these tasks.
directly undermines the reliability and trustworthiness of the output-qualitice qualities
Our vanilla schema scaffolding method enforces strong structural constraint, which inevitably in- terferes with the natural generation trajectory of dLLMSs. To mitigate the resulting hallucination, we observe that is possible to transform unfamiliar test-time scenarios into familiar training-ting cases by providing sufficient in-context guidance. Thus, we introduce the notion of special token, null, as a flexible placeholder. Once the model adapts this convention, can fill otherwise empty slots without resorting to fabricated content, despite not being explicitly trained to padding to- kens. This simple yet effective prior guidance enables dLLMs to substantially reduce hallucination, and it inspired the formulation of our improved $3 method.
8

---

Published as a conference paper at ICLR 2026
Structural Adherence
Content Fidelity
Faithfulness
Experiment
SVT
FCt
SCT
PR
RET
FI1
HR|
8 Steps
baseline
0.346
0.310
0.312
0.101
0.051
0.068
0.404
w/ few-shots
0,471
0.447
0.443
0.086
0.056
0.068
0.366
w/ template
0.475
0.432
0.431
0.088
0.081
0.084
0.388
5 (ours)
0.994
0.994
0.994
0.115
0.151
0.130
0.340
16 Steps
baseline
0.646
0.604
0.576
0.086
0.085
0.086
0.403
w/ few-shots
0.735
0.713
0.674
0.084
0.087
0.085
0.371
w/ template
0.794
0.734
0.738
0.091
0.139
0.110
0.390
S (ours)
0.997
0.997
0.997
0.113
0.147
0.128
0.331
32 Steps
baseline
0.869
0.839
0.792
0.070
0.092
0.080
0.409
w/ few-shots
0.890
0.870
0.824
0.092
0.114
0.101
0.358
w/ template
0.909
0.886
0.882
0.091
0.160
0.116
0.384
8 (ours)
0.997
0.997
0.997
0.110
0.144
0.125
0.331
Table 2: Ablation study results comparing different experimental configurations. Boldface indicates the best results. We use examples for few-shot learning (Wei al. (2022)) To follow previous practice, we also extend the instruction with complete schema (Wang et al. (2025)) as guidance. Our method consistently outperforms these alternative techniques, indicating a nontrivial and supe- rior improvement.
6.2 COMPLEXITY ANALYSIS
An existing bottleneck of dLLMs lies in their inference speed. Empirically, the multi-step denoising process introduces a latency that grows positively with the number of diffusion steps. Within each step, the global attention computation incurs a quadratic cost with respect to the total context length L. As result, the overall computational complexity of the reverse process scales as O(L²). To mitigate this cost, some implementations adopt semi-autoregressive decoding scheme with block- wise KV-caching, which partially reduces the computational burden. However, this design still compromises the core parallelism advantages of diffusion-baace based decoding.
For structured generation tasks, where the output structure is known can be approximated, our proposed method 5 introduces an alternative perspective and initialize the reverse process from partially denoised state rather than fully random one. This warm-start initialization serves structural prior, effectively providing a language scaffold that accelerates generation and enhances controllability. Asymptotically, 5 reduces the decoding complexity ((n.?)), where is tun- able hyperparameter that remains significantly smaller than L in practice.
CONCLUSION
In this paper, we explore the potential of diffusion large language models' global awareness for controllable generation structured output. We propose the novel Self-adaptive Schema Scaffold- ing method (S3) that guides dLLMs to adaptively generate fully controllable structured output by manipulating the reverse process and leveraging the innate global attention mechanism. Our compre- hensive evaluation framework demonstrates that S achieves superior structural adherence, content fidelity, and reduced hallucination rates. Through complexity analysis, we show that our approach maintains computational efficiency while enabling higher levels of controllability and hallucination control. We believe these findings establish dLLMs as a promising alternative for controllable gen- eration tasks.

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
The work is partially supported by the U.S. National Science Foundation (NSF) Grant CRII 2451683, an NVIDIA Academic Grants Program, a U.S. Bank Academic Research Award, the Uni- versity of California, Merced, and UC Merced Faculty Research Award. The views and conclu- sions are those the authors and not necessarily reflect the official policy or position of the U.S. Government.
REFERENCES
Marianne Arriola, Aaron Gokaslan, Justin T Chiu. Zhihan Yang, Zhixuan Qi, Jiaqi Han, Sub- ham Sekhar Sahoo, and Volodymyr Kuleshov. Block diffusion: Interpolating between autore- gressive and diffusion language models. arXiv preprint arXiv:2503.09) 095773 2025.
Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in neural information processing systems, 34:17981-111 1999-99993, 2021a.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang. Carrie Cai, Michael Terry. Quoe Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07753 2021b.
Pawet Budzianowski, "sung-Hsien Wen, Bo- Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašić. Multiwoz-a large-scale multi-domain wizard-of-oz dataset for task- 2018.
oriented dialogue modelling. arXiv preprint PrXii:1880.000
Gemini Team, Google. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodal- ity, long context, and next generation agentic capabilities. Technical report, Google DeepMind, June 2025. URL httssss:://storagg /stttaage..g..///. googleapis. [gemini v2_ report pdf Technical Report.
http:isccoooooooooommmmeeeeeeeee deepmind mell-media/gge /ttpm///dddiaaiimiii/// gemini/
Saibo Geng, Martin Josifoski, Maxime Peyrard, and Robert West. Grammar--onstrained decoding for structured nlp tasks without finetuning, arXiv preprint 2rXiv:2305..3371, 2023.
Shansan Gong, Shivam Agarwal, Yizhe Zhang, Jiacheng Ye, Lin Zheng, Mukai Li, Chenxin An, Peilin Zhao, Wei Bi, Jiawei Han, al. Scaling diffusion language models via adaptation from autoregressive models. arXiv preprint arXiv:2410.1788 17911 2024.
Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501.1229. 2025.
Longzhen Han, Awes Mubarak, Almas Baimagambetov, Nikolaos Polatidis, and Thar Baker. Mul- timodal large language models: A survey. arXiv preprint arXiv:2506.1000 2025.
Marton Havasi, Brian Karrer, Itai Gat, and Ricky TQ Chen. Edit flows: Flow matching with edit operations. arXiv preprint arXiv:2506.09018 2025.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300 2020.
Jonathan Ho. Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
Chihan Huang and Hao Tang. Ctrldiff: Boosting large diffusion language models with dynamic block prediction and controllable generation. arXiv preprint arXiv:2505. 00.225555.444... /4455, 2025.
Yuting Huang, Meitong Guo, Yiquan Wu, Ang Li, Xiaozhong Liu, Keting Yin, Changlong Sun, Fei Wu, and Kun Kuang. Appealcase: dataset and benchmark for civil case appeal scenarios. arXiv preprint arXiv:2505. 165 14, 2025.
10

---

Published as a conference paper at ICLR 2026
Daniel Israel, Guy Van den Broeck, and Aditya Grover. Accelerating diffusion Ilms via adaptive parallel decoding. arXiv preprint S.Kiv:250060041133 2025.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412.161111 2024.
Guillaume Jaume, Hazim Kemal Ekenel, and Sean-Philiipp Thiran. Funsd: dataset for form under- standing in noisy scanned documents. In 2019 International Conference on Document Analysis and Recognition Workshops (ICDARW), volume 2, pp. 1-6. IEEE, 2019.
Terry Koo, Frederick Liu, and Luheng He. Automata-based constraints for language model decod- ing. arXiv preprint urXiv:2407.08/00 2024.
Inception Labs, Samar Khanna, Siddhant Kharbanda, Shufan Li, Harshit Varma, Eric Wang, Sawyer Birnbaum, Ziyang Luo, Yanis Miraoui, Akash Palrecha, et al. Mereury: Ultra-fast language models based diffusion. arXiv preprint arXiv:2506. 17298, 2025.
Rémi Lebret, David Grangier, and Michael Auli. Generating text from structured data with appli- cation to the biography domain. CoRR, abs/1603.07771, 2016. URL http::/arxiv..o
00s//600.00777
Yifan Li, Kun Zhou, Wayne Xin Zhao, and Ji-Rong Wen. Diffusion models for non-autoregressive text generation: A survey, arXiv preprint arXiv:2303.06574, 2023.
Zhecheng Li, Yiwei Wang, Bryan Hooi, Yujun Cai, Nanyun Peng, and Kai-Wei Chang. Drs: Deep
question reformulation with structured output. arXiv preprint 24X1::2411.17993 2024.
Jiaheng Liu, Dawei Zhu, Zhiqi Bai, Yancheng He, Huanxuan Liao, Haoran Que, Zekun Wang, Chenchen Zhang, Ge Zhang. Jiebin Zhang, et al. A comprehensive survey on long context lan- guage modeling. arXiv preprint arXiv:2503. 17407, 2025.
Niels Mündler, Jingxuan He, Hao Wang, Koushik Sen, Dawn Song, and Martin Vechev. Type- constrained code generation with language models. Proceedings of the ACM on Programming Languages, 9(PLDI):011622) 2025.
Alex Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. Interna- tional Conference on Machine Learning, 139:8162-888 8171, 2021.
Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large language diffusion models. arXiv preprint XXXx:25022.0992, 2025.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27735-27756. 2022.
Darren Yow-Bang Wang. Zhengyuan Shen, Soumya Smruti Mishra, Zhichao Xu, Yifei Teng, and Haibo Ding. Slot: Structuring the output large language models. arXiv preprint arXiv:2505.0401 04005. 2025.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In Interna- tional Conference on Learning Representations (ICLR), 2022.
Zhen Xiong, Yujun Cai, Zhecheng Li, and Yiwei Wang. Mapping the minds of 11ms: A graph-based analysis of reasoning 1lm. arXiv preprint 2xXi:25005.13800 2025.
Jiacheng Ye, Jiahui Gao, Shansan Gong, Lin Zheng, Xin Jiang, Zhenguo Li, and Lingpeng Kong. Beyond autoregression: Discrete diffusion for complex reasoning and planning. arXiv preprint arXiv:2410. 14157, 2024.
11

---

Published as a conference paper at ICLR 2026
Jiacheng Ye, Zhenyu Wu, Jiahui Gao, Zhiyong Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong- Implicit search via discrete diffusion: A study on chess. arXiv preprint arXiv:2502. 250.::2502..9005,,, 2025a.
Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Lingpeng github.thh..... htwpll//bboooo/o htlpg///222//rrrreeeeeee
Kong. Dream 7b, 2025b. httthhtt///hkkkte
Zebin You, Shen Nie, Xiaolu Zhang, Jun Hu, Jun Zhou, Zhiwu Lu, Ji-Rong Wen, and Chongxuan Li. Llada-v: Large language diffusion models with visual instruction tuning. arXiv preprint arXiv:2505.16933, 2025.
Runpeng Yu, Li, and Xinchao Wang. Discrete diffusion in large language and multimodal models: A survey. arXiv preprint arXiv:2506. 13759, 2025.
Di Zhang. Aime_ (revision 6283828), 2025. URL hntgs://huugiiggaecee..
datasets/di--aa zhang fdu/AIME 1983_2020
Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et Webarena: A realistic web environment for build- ing autonomous agents. arXiv preprint arXiv:2307. 13854, 2023.
Fengqi Zhu, Rongzhen Wang, Shen Nie, Xiaolu Zhang, Chunwei Wu, Jun Hu, Jun Zhou, Jianfei Chen, Yankai Lin, Ji-Rong Wen, et al. Llada 1.5: Variance-reducee preference optimization for large language diffusion models. arXiv preprint arXhv:2505.192255 2025.
12

---

Published as a conference paper at ICLR 2026
A IMPLEMENTATION DETAILS
diffusion large language model, loaded from HuggingFace official with bfloat1e 6 for optimal memory efficiency and performance. All experiments were conducted on one single NVIDIA RTX 4090 GPU. We set max_nex_ea... www._/ewttooeeseeee..... 128, which suffices all structured output task, and temperatures for reproducibility. The WikiBio dataset serves as the primary benchmark dataset.
We use the GSAI GSAT-ML/LLLaD--1.
B SCHEMA COMPILING
The compilation step serves the bridge between the user's structural constraints and the diffusion model's initial state. Formally, given a schema S and a target format F (e.g.. ISON, XMI., YAMI), the compiler function C(S, F) generates scaffold sequence Xscaffold A key advantage of 8 is that the diffusion process agnostic to the underlying data format. As long as a standard parser exists for a given format, the schema can be compiled into scaffold. We demonstrate this universality below with JSON and XML examples.
Case 1: JSON Compilation For Biography generation task, the JSON schema defines fields for name and birth year.
Input Schema (JSON):
Compiled Scaffold (53 Initial State):
"type": "object", "properties"
"name": [MASK] [MASK]
"born": [MASK] [MASK]
"name": ("type": "string"), "born": ("type": "integer")
Case 2: XML Compilation Applying the same logic to a hierarchical format like XML requires no changes to the diffusion algorithm, only to the initial compiling parser.
Input Schema (XML XSD style):
Compiled Scaffold (83 Initial State);
<xs:element name "pmren">""so""
<person>
<xs:element name="name*
<name> [MASK] [MASK] < </name>
sype="xs:string"
<born> [MASK] [MASK] </born>
<xs:element name="born)
< person>
type="xs xope="xs:int//>>> int
</xs:element>
C REMASKING STRATEGY
By default, dLLMs can generate all tokens within the full context in parallel through single infer- ence step. However, for many general NLP tasks, this strategy is often suboptimal. For example, simultaneously generating all tokens that form a complete solution cantly harder than doing so incrementally via the autoregressive approach. To address this issue, prior works have explored various remasking strategies bridge the gap.
math problem is signifi-
One common strategy is the block-wise masking, which employs sliding- mechanism. The context divided into blocks, and each iteration, only the tokens within the current window taken into consideration, while those in following blocks are remasked for future regeneration. Thus, this strategy is also considered suni-autoregressivve. However, empirical results show that using block size of one, which essentially reverting to fully autoregressive process, often yields the best performance. We consider such a strategy largely eliminates the parallelism advantage of ceffusionbbased models.
Another type of works focus on low-confidence remasking, which selectively discards tokens with low confidence scores, such low log-probability or high entropy, relative to others in the currett
13

---

Published as a conference paper at ICLR 2026
iteration. This process is repeated until all tokens are finalized, allowing for more adaptive number of iterations. Some approaches further combine low-confidence filtering with block-wise remasking to support in-block autoregressive generation.
In contrast, our proposed S method adopts a simple yet effective top-K remask strategy, where K 0/nnn, O/n, with O denoting the total number of tokens to be generated and being tunable number of denoising steps. Compared to the block-wise remasking method, our approach is significantly more efficient and introduces an additional level of controllability by predefining the number of
generation iterations, offering a practical trade-off between speed and output quality.
Remasking Strategy
Steps
SVt
FCt
SCt
FIf
HR
Top-K (5a) -owfiofieeecee
16
0.997
0.997
0.997
0.128
0.331
16
0.994
0.993
0.992
0.121
0.348
Block-wisc
16
0.989
0.986
0.985
0.115
0.362
Table 3: Ablation of different remasking strategies on structured generation (WikiBio, 16 steps). Additionally, our additional experiment in Table 3 indicate that Top-K approach, adopted by our method, yields superior performance across all structural (SV, FC, SC) and content (F1, HR) met- rics. We attribute this to the deterministic nature of Top-K remasking, which provides precise iter- ation control (K₃- = O/n). Unlike Low- www-Confidence remasking, which may re-mask too few or too many tokens in early steps leading unstable convergence. Top-K ensures predictable computa- tional budget and steady refinement of the scaffolded structure. Block-wise remasking performed the worst, suggesting that structural constraints formats like JSON often require global, non- contiguous corrections rather than local block updates.
D ADDITIONAL GENERALIZABILITT EXPERIMENTS
To validate the effectiveness of approach beyond the primary WikiBio dataset, conducted extensive evaluations across diverse domains and model architectures. This section details the per- formance of 5 on code generation, dialogue state tracking, and form filling, well its transfer- ability to other diffusion language models.
D.1 DOMAIN EVALUATION
We extended our evaluation to three additional datasets that require distinct structural formats be- yond standard ISON:
Using the Mostly Basic Python Prob- lems dataset, requiring strict syntactic adherence to Python code structures.
Code Generation (MBPP) (Austin et al., 2021b):
Dialogue Structure (MultiWOZ) (Budzianowski et al. 2018): Evaluating dialogue state tracking, which requires structured output representing user intent and slot values.
Form Filling (FUNSD) (Jaume et al., 2019): A visual document understanding task re- quiring the extraction of key-value pairs structured formats.
As shown in Table 4 S demonstrates consistent improvements across all tasks. We observed absolute gains of 15 ~ 35% in structural adherence metrics (SV, FC, SC) compared to the baseline LLaDA model. This confirms that the scaffolding mechanism generalizes effectively to different structural constraints (e.g., Python indentation, dialogue states) without task-specific tuning.
D.2 CROSS-ARCHIFECTURE EVALUATION
To ensure our findings are not specific to a single architecture, we validated S3 on Dream-7B YYe al. 2025b), another frat-of-thee-al diffusion language model. Using the same hyperparameters described in Section 5, we compared the baseline generation against our scaffolding method.
The results in Table 5 show that 53 boosts the Structural Validity (SV) of Dream-7B from 0.723 to 0.991. This indicates that our method exploits the fundamental non-autoregressive nature of diffusion models rather than model-specific artifacts.
14

---

Published as a conference paper at ICLR 2026
Task
Dataset
Method
Structural Metries (SV / FC/ SCC
LLaDA Baseline LLaDA + S
0.645 / 0.618 0.592
Code Generation
MBPP
0.989 / 0.991 / 0.988
L.LDDA Baseline
0.712 / 0.684 / 0.651
Dialogue Structure
MultiWOZ.
LLaDA + +
0.982 / 0.985 0.981
LLaDA Baseline
0.701 / 0.673 0.649
Form Filling
FUNSD
LLaDA + S
0.994 / 0.993 0.992
Table 4: Performance comparison on diverse structured output tasks (Code, Dialogue, and Forms) using the LLaDA model. 8 significantly outperforms the baseline across Structural Validity (SV), Format Compliance (FC), and Schema Consistency (SC).
Method
SV
FC
SC
F1
Dream-7B (Baseline)
0.723
0.698
0.671
0.082
Dream-7B S
0.991
0.989
0.988
0.121
Table 5: Generalization of S to the Dream-7B diffusion model.
E PROMPTS
This section provides the complete prompts used in our experiments for both the baseline method and our proposed Self-adaptive Schema Scaffolding (Sa) approach.
The baseline method employs detailed prompt that explicitly specifies the desired output structure through a comprehensive JSON schema, as shown in Figure 5 This approach requires the model to understand and adhere to the predefined schema constraints based solely on the textual instructions.
In contrast, our Self-adaptive Schema Scaffolding (S²) method utilizes significantly simplified prompt (Figure (6) that omits explicit structural specifications. Instead, our method relies on the inherent structural constraints enforced by the scaffolding mechanism, demonstrating the token ficiency and clarity of 3
F
PROOF
F.1 PROOF OF THEOREM.
Proof. The diffusion model learns to predict p=(x//a;)) for each masked position where æ; M. With structural scaffolding, partition positions into: Fixed scaffold positions: S {i: x #
M, fixed by structure) Variable positions: V {i o = M}
The model's prediction at each masked position depends the context:
Po (x)) = Ps (
With scaffolding, the context includes correct structural tokens at positions S, providing stronger signal:
Po (agl with S) = (2 (=())e)(())
Since the scaffold tokens are correct by construction (they match the target structure), they reduce uncertainty in the conditional distribution. The error reduction is proportional the informativeness of the scaffold.
For each masked position, the expected error with scaffolding is:
E(e][S] < Blee] T(S)
15

---

Published as a conference paper at ICLR 2026
Baseline Prompt
Instruction:
Extract the following information from the provided document and return only JSON response with ad- ditional text or explanation: name, bate, birth-place datt, plate, nationality, occupation. The response must conform to this JSON schema:
"scheme";
*type": "object",
"properties":
"name":
"type"; (*etring", "noli"),
"description's *Pull name the persor"
"birthi_date":
"type"; (*string", "ngll"),
"description": "Birth cate in 130 cornat (YYY--00--D, WYY-- or YYr!""
"birthi_raace::
"type"; ("string", "noll"),
"description": *Place et birth reity, country tornst preterred)*
"deathi_date""
"type"; I*string", "noli"),
"description": "Desth date in 130 tornat (W1YY-0---,, WIY-H,, or YYT!!"
"deachi_piaco"
"type": (*string", "null"),
"description": *Flace et seath reity, country tornst preterred)*
"baticaristy":
"type"; (*string", "noll"),
"description": "Rationality or citizenshipp
"occupation":
"type": (*string", "sull"), "description': "Erimary occupstion or profassion"
"soquiraa": (*name", "bitch'Ldatee', "birthl_placs", "dostal_date", "deathl_place", "nationalisy", cocupation ion"!
If any information is not available in the document, use null for that field. [DOCUMENT TEXT]
Figure 5: The complete prompt we use for the baseline method.
Prompt
Instruction:
Extract information from the provided document and return only JSON response with no additional text explanation. If any information is not available in the document, use null for that field. [DOCUMENT TEXT]
Figure 6: The complete prompt for our Self-adaptive Schema Scaffolding (S method. Unlike the baseline approach, our method does require explicit structural information the prompt, as naturally enforces predefined structure constraints on the model's 's output.
where 1(5;25 is the mutual information between scaffold and target token.
Aggregating over masked positions and noting that structural tokens typically have high mutual information with content tokens (e.g., field names predict value types), we get:
E | xollm] S E[llxo xollu] (1- |S|
The equality holds when scaffold and content are independent, which rarely occurs in structured generation.
16

---

Published as a conference paper at ICLR 2026
G PLANNING ABILITY OF DLLMS
The future planning capability an architectural advantage of diffusion large language models due the global awareness
formalized in Section 3. Previous works (Ye et al. 2024 2025a) have shown that text diffusion models exhibit superior planning capabilities in small-scale, task task-specific contexts. Additionally, another work et 2025b) has also empirically proven that dLLM is sig- nificantly better than similar-scal AR models like (Llama3 and Qwen2.5) in benchmarks requiring planning ahead, such as trip planning, constrained arithmetic problems, and Sudoku.
H THE USE OF LARGE LANGUAGE MODELS (LLMS)
We used an LLM assist with the phrasing and grammar of the manuscript. The LLM was used strictly as a writing aid and did not contribute to the scientific ideation, methodology, results presented in this paper.
17


Reviews:
[{"review_id": "eX2n6zDmre", "reviewer": "Reviewer_3M4f", "summary": "The paper proposes an algorithm that utilizes schema scaffolding for controllable generation in diffusion models, theoretically demonstrates the feasibility of schema scaffolding, and conducts experiments across three key dimensions to validate the effectiveness of the proposed method.", "questions": "See the weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "To ensure global awareness, the paper employs a diffusion model for generation. Furthermore, it introduces a schema scaffolding mechanism to enable controllable generation and provides theoretical proof of its feasibility.", "weaknesses": "1. The paper (Figure 1) points out that autoregressive models lack global awareness, which is an advantage of diffusion models. To validate this perspective, the authors should provide experimental results from an autoregressive model baseline. \n2. The equations subsequent to Equation 3 are unnumbered, resulting in an inconsistent presentation. \n3. The experimental setup is relatively simplistic, employing a very limited number of baselines, which consequently lacks persuasiveness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "L7oG5CAMCw", "reviewer": "Reviewer_bXwN", "summary": "This work proposes a method that enables diffusion large language models (dLLMs) to provide contents with structured output (e.g., JSON). Empirical results show that the propose method ($S^3$) improve both structural adherence and content quality in the chosen downstream task.", "questions": "1. For the Wikibio dataset, do you use the biography text as the input to the prompt, and the contents of the infobox as the target output? It seems that these details are not explicitly mentioned in the paper.\n\n2. In autoregressive LLMs, it is previous shown by Tam et al [1] that enforcing LLMs to provide structured outputs degrade task performance (i.e., generated contents' quality) compared to when LLMs can \"speak freely\" without structured constraints. Therefore, a way to preserve both the structured adherence and content quality is to adopt a two-stage inference process: Let the model \"speak freely\" and focus on generating the contents first, and then convert the generated contents to structured outputs. Do diffusion LLMs (dLLMs) have similar properties? I think it would make the work more complete if this experiment is done, so that we know whether enforcing structured outputs degrade dLLMs' content generation ability.\n\n[1] Tam, Zhi Rui, et al. \"Let me speak freely? a study on the impact of format restrictions on performance of large language models.\" arXiv preprint arXiv:2408.02442 (2024).", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Generating reliable structured outputs is an important research direction and has many practical downstream applications, as evidenced by the fact that it is widely discussed and investigated in autoregressive LLMs literature. This work extends this research direction to a relatively less explored model of diffusion LLMs (dLLMs) and proposes a new method to improve generation of structured outputs.\n\n2. It is well-motivated to use dLLMs for structured outputs, as this task (which requires lookahead planning from known future tokens) aligns well with the properties of dLLM.\n\n3. Empirical results show that the proposed methods greatly improve structural adherence to the target JSON schema, while also preserve or even improve generated contents' quality (as shown in Figure 4 and Table 1).", "weaknesses": "1. Generalization across tasks: The experiments only use one dataset (Wikibio by Lebret et al., 2016), so it is unclear how well the proposed method generalize to other, especially more difficult, datasets.\n\n2. Generalization across types of structured outputs: Following the previous point, it would also be nice to include experiments on other types of structured outputs, such as XML or YAML.\n\n3. Although the authors state they use the Wikibio dataset, they didn't mention what's the input and what's the target output or include examples in the paper. I think more details need to be elaborated as how the authors use the dataset in their experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DEXxuppiGJ", "reviewer": "Reviewer_fQDV", "summary": "In this work, the authors propose a framework to enable diffusion-based LLMs for reliable structured outputs. A schematic template is initiated. Experimental results show that the proposed method marginally improves the structure outputs compared with the commonly used prompting strategy.", "questions": "1. Comparative perspective:\nHow does S3 perform against autoregressive structured decoding methods (e.g., JSON mode in GPT-4 or constrained decoding in PaLM/CodeLLaMA)?\n\n2. Diffusion step selection:\nHow is the optimal number of denoising steps (e.g., 8, 16, 32) determined? Could adaptive step scheduling yield further gains?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. Clear motivation:\nExploring diffusion LLMs for controllable text generation is a fresh and under explored research area.\n\n2. Conceptually good idea:\nThe use of schema scaffolding as a structural prior aligns well with diffusion’s iterative refinement mechanism. No additional fine-tuning or retraining is needed.\n\n3. Improved results:\nThe paper demonstrates substantial improvement over baseline diffusion models.", "weaknesses": "1. Limited experimental scope:\nOnly one dataset (WikiBio) and one diffusion model (LLaDA) are tested. Broader validation tasks (e.g., code generation, dialogue structure, form filling) would strengthen generality.\n\n2. Lack of comparison to AR-LM baselines:\nAlthough the paper motivates dLLMs as alternatives to AR models, it doesn’t include a direct comparison with strong AR methods like structured prompting or constrained decoding (e.g., CodeLLaMA, T5, or GPT-style JSON control).\n\n3. There is no enough details on how to compile constraints into a schema, which is then used to initialize a noisy scaffold where mask tokens serve as placeholders for missing content. Especially, how to use this compilation in other tasks?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SaGNJUk2jU", "reviewer": "Reviewer_CuYm", "summary": "This paper explores the potential of diffusion-based large language models (dLLMs) as an alternative to autoregressive (AR) models for controllable structured generation—a key capability for applications like JSON output, API integration, and agentic communication.\nThe authors identify key limitations of AR models (lack of global coherence, irreversible token commitments, and sequential dependency) and argue that dLLMs’ iterative denoising and global attention mechanisms offer inherent advantages for structured output.\nTo unlock this potential, they propose Self-adaptive Schema Scaffolding (S³), a training-free method that pre-initializes dLLM outputs with a structured schema scaffold (e.g., JSON skeleton) and uses adaptive null tokens to handle variable-length or missing fields.", "questions": "- How sensitive is S³’s performance to the number of denoising steps or the choice of remasking strategy (top-K vs. low-confidence)?\n\n- Can this framework be extended to multi-modal dLLMs (e.g., LLaDA-V) where structured outputs integrate vision and text?\n\n- How does S³ interact with instruction-tuned or RLHF-optimized dLLMs trained for task-following?\n\n- Would hybrid diffusion–autoregressive setups (e.g., block diffusion) inherit similar controllability advantages?\n\n- Have the authors evaluated the effect of schema complexity (nested vs. flat JSON) on scaffolding performance?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- First comprehensive analysis of diffusion LLMs in controllable structured generation.\n\n- S³ improves structure adherence and reduces hallucination without retraining.\n\n- Achieves near-perfect structure compliance (≈0.997) and lowest hallucination (0.33 vs. 0.40 baseline).\n\n- Provides clear theoretical rationale and visual intuition for why scaffolding aids denoising.\n\n- Training-free and computationally efficient; suitable for real-world schema-driven applications.\n\n- The new tri-metric framework (adherence, fidelity, faithfulness) sets a useful precedent for future structured LLM work.", "weaknesses": "- Experiments focus narrowly on WikiBio and JSON; applicability to more diverse formats (e.g., code, graphs) remains untested.\n\n- The study lacks comparison to strong AR baselines using grammar-constrained decoding or function-calling LLMs.\n\n- Only one dLLM (LLaDA) is evaluated; cross-model tests (e.g., Dream or Mercury) would strengthen claims.\n\n- While effective, it may not generalize to schemas without explicit “missing field” semantics.\n\n- Despite reduced denoising steps, dLLM inference (O(nL²)) may still lag AR models (O(L)), which could affect scalability in long-context tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)