Title: Next Semantic Scale Prediction via Hierarchical Diffusion Language Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Discrete Diffusion Model

Theory for Discrete Diffusion

Diffusion Language Modeling
Abstract: In this paper we introduce Hierarchical Diffusion Language Models (HDLM) -- a novel family of discrete diffusion models for language modeling. HDLM builds on a hierarchical vocabulary where low-level tokens with detailed semantics are surjectively mapped to high-level tokens with coarse-grained meanings. In the forward process, each token is independently perturbed to its higher-level ancestor with more abstract semantics according to the scheduler, while in the reverse process the model progressively predicts the next, more detailed semantics. Taken together, HDLM provides a general time-varying next semantic scale prediction process for language modeling. We derive closed-form expressions for the diffusion Evidence Lower Bound (ELBO), and show that HDLM can be implemented in a flexible manner while including the existing MDLM as a special case. We also propose practical training techniques based on the insights. Extensive text generation experiments validate the effectiveness of HDLM, which demonstrates consistently lower validation and generative perplexity than baselines.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Hierarchical Diffusion Language Models (HDLM), a discrete diffusion framework for language modeling.It structures tokens into semantic hierarchies where detailed tokens map to coarse-grained ancestors. HDLM's forward process independently transitions tokens to higher abstraction levels via scheduled diffusion. The reverse process progressively predicts finer semantics through time-varying next-scale prediction.This enable tokens to occupy different semantic levels at each diffusion step. Theoretically, HDLM derives a closed-form ELBO showing it generalizes prior discrete diffusion models, introduces stochastic perturbations for self-correction, and enables flexible training schemes. Experiments demonstrate HDLM outperforms existing diffusion models in language modeling perplexity and downstream language understanding tasks while requiring less training data, with larger semantic hierarchies yielding stronger self-correction capabilities.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths：\nProposes a hierarchical diffusion framework (HDLM) that introduces semantic hierarchies between word-level tokens and mask tokens. This replaces uniform/mask noise with coarse-to-fine semantic transitions, addressing key limitations in prior discrete diffusion models. Theoretical innovations include deriving closed-form ELBO expressions for hierarchical diffusion and proving HDLM generalizes existing models HDLM’s hierarchical design enables error mitigation during decoding, validated by significant gains in self-corrected perplexity. Techniques like stochastic perturbations and force transition improve robustness and training stability. HDLM achieves state-of-the-art self-corrected perplexity among diffusion models and consistent improvements over the baseline method GIDD across all datasets. The schematic diagram is clear and easy to understand.\n\nWeaknesses:\nAlthough Figure 1 is clear and easy to understand, the methodology section of the paper is rather obscure and suffers from poor readability. \nThe experimental section is insufficient, with only limited comparisons of perplexity and downstream task performance against a few methods. The experimental setup lacks clarity, particularly regarding implementation details such as the clustering process. Additionally, the authors did not investigate the model's performance across more hierarchical levels.\n\nFurthermore, while the paper proposes several tricks for HDLM, including Stochastic Perturbation \\& Self-Correction, Hard Training Mode, Auxiliary Loss, and Force Transition, the experimental section fails to provide ablation studies for these components. This omission makes it difficult to assess their necessity and individual contributions to the model's performance.\n\nQUESTIONS:\n- Can other models not compute the self-corrected perplexity for unified comparison?  \n- Did the authors conduct ablation studies on the tricks for HDLM mentioned in the aforementioned drawbacks? Presenting these would more convincingly validate the effectiveness of these designs.  \n- It is recommended to elaborate on the specific clustering algorithm and operations in the paper, as I believe this is a crucial step for HDLM, directly impacting the model's final performance.  \n- Adding noise to word vectors in the embedding space causes the perturbed vectors to follow a certain probability distribution around their original positions. Does this correspond to the transformation from precise semantics to coarse-grained semantics in the proposed method?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

comments: SUMMARY:
This paper introduces an innovative class of discrete diffusion models called Hierarchical Diffusion Language Models (HDLM). In essence, HDLM employs a hierarchical vocabulary structure where lower-level tokens convey more detailed semantics, while higher-level tokens offer a more coarse-grained semantic perspective. The authors present a robust theoretical framework alongside practical experiments to convincingly demonstrate the efficacy of HDLM.

STRENGTHS AND WEAKNESSES:
**Strengths**
- This paper is well-written, and the concept of HDLM is both novel and intriguing.
- The theoretical results and practical design principles are robustly established.
- With its self-correction step, HDLM outperforms all other discrete diffusion models.

**Weaknesses**
- The related work section lacks organization.
- In Table 1, the comparison of HDLM's sampling speed with different cluster numbers against other discrete diffusion and AR models is unclear. Additionally, does the self-correction step introduce extra computational costs?
- Why does the small HDLM outperform the base HDLM in Table 2?

QUESTIONS:
- When applying HDLM to image generation using discrete tokens generated by VQ-VAE, what modifications are necessary? In this context, would HDLM be a suitable choice?
- Why do you use different numbers of clusters for different model sizes in Table 2? Should the number of clusters be fixed before training, and how is this number determined?
- Why did you choose the DiT architecture over the more commonly used LLaMA or Qwen in LLMs? Could you compare HDLM's performance with the LLaMA architecture?

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Hierarchical Diffusion Language Models (HDLM), a novel class of discrete diffusion models for language generation. The core idea is to replace the standard denoising process, which typically transitions from a mask token directly to a data token, with a multi-stage, hierarchical process. In the proposed forward process, word tokens are first perturbed into \"coarse-grained\" semantic clusters before finally becoming a mask token. The reverse process learns to progressively refine these abstract cluster tokens back into specific, \"fine-grained\" word tokens, a scheme the authors term \"next semantic scale prediction\". The authors provide a theoretical foundation for this approach, derive an ELBO for training, and show that a previous model, MDLM, is a special case of their framework. They evaluate HDLM on language modeling and downstream tasks, proposing a new *self-corrected perplexity* metric to demonstrate the model's capabilities.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. The paper tackles the timely and important problem of improving discrete diffusion models for language. The core idea of introducing semantic hierarchies is novel, intuitive, and offers a principled way to improve the denoising process compared to standard techniques. I find the *next semantic scale prediction* framing a good conceptual contribution.\n\n2. The work is theoretically-grounded \n\n3. The paper demonstrates that its framework is a generalization of previous work, with MDLM being a special case where only one cluster token exists.\n\n**Weaknesses**\n\n4. The paper's primary claim of effectiveness rests on a new metric, i.e., *self-corrected perplexity*. While the motivation to capture the model's refinement ability is understandable, introducing a new metric and using it alone to show your own model's superiority is not fully convincing without external validation. This is especially problematic when the model performs worse on standard metrics and achieves only marginal gains on downstream tasks. Furthermore, perplexity values derived from the diffusion ELBO are an upper bound on the true perplexity, meaning they should technically be reported with a $\\leq$ sign. Reporting these as exact values in Table 1 is imprecise.\n\n5. The writing, particularly in the results section (5.2), is not always clear. The justification for the self-corrected perplexity metric is dense and its fairness is just asserted. Key details are also missing; for instance, the notation HDLM-X is used in tables, but 'X' (representing the number of clusters) is not explicitly defined in the captions. The abstract is also vague, using generic sentences like demonstrate the effectiveness without concrete, quantitative details.\n\n6. The paper fails to provide any justification for choosing the number of clusters for different experiments. This is a critical hyperparameter of the proposed method, and its impact on performance, complexity, and other trade-offs is not discussed.\n\n7. The paper frequently uses terms like \"detailed semantics\", \"coarse-grained meanings\", and \"richer semantics\". While the operational meaning (grouping tokens via clustering) is clear, the connection to linguistic semantics is loose and based on intuition rather than a more formal, well-defined concept.\n\nOverall, I think the idea is interesting, yet tested on a single setting and -- currently -- it brings only marginal improvements over simpler baselines.\n\n*Minor*\n\nThere are several typos (e.g., L192: \"as state\" -> \"as stated\", L262: “yet which does not” -> “yet this does not”?, Table 4: xi -> \\xi).\n\nQUESTIONS:\n8.  Could you add the self-corrected perplexity result for an HDLM with 1 cluster (= MDLM) to Table 1? \n\n9.  Given that the ELBO provides only a bound on the true likelihood -- whose tightness can vary between models -- have you considered evaluating the models using a more direct measure of generative quality, such as generative perplexity, which is a standard evaluation in the context of diffusion language models?\n\n10. The paper only considers one intermediate hierarchical level. You mention extending the theory to multiple hierarchies as a limitation. Could you briefly elaborate on the potential benefits and challenges of such an extension?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Hierarchical Diffusion Language Models (HDLM), a discrete diffusion model that operate on hierarchical semantic scales. Unlike standard discrete diffusion models that transition between clean tokens and masks, HDLM introduces intermediate semantic hierarchies where tokens progressively transition from fine-grained to coarse-grained representations. The authors derive closed-form ELBO expressions, demonstrate that existing models like MDLM are special cases of their framework, and show that HDLM achieves competitive performance on language modeling tasks while enabling self-correction through its hierarchical structure.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- Conceptually novel approach: The idea of hierarchical semantic diffusion with \"next semantic scale prediction\" is intuitive and well-motivated, addressing key limitations of existing discrete diffusion models (lack of semantic richness in masked tokens, inability to self-correct).\n- Strong theoretical foundation: The paper provides rigorous mathematical treatment with closed-form ELBO derivations and proves elegant properties like the invariance of token-level loss weights.\n- Unifying framework: Showing that MDLM is a special case of HDLM demonstrates the generality of the approach and provides theoretical validation.\n- Self-correction capability: The hierarchical structure naturally enables the model to refine predictions, as demonstrated by improved self-corrected perplexity.\n\nWeaknesses:\n\n- Insufficient insights into model behavior: The paper presents only quantitative metrics (perplexity, downstream task accuracy) without providing any understanding of how HDLM actually functions as a language model. Metrics alone are insufficient to understand a generative model's behavior.\n- No generation examples or analysis: The paper lacks any examples of generated text, making it impossible to assess generation quality, coherence, or how the hierarchical structure influences output. Readers cannot judge whether the model produces sensible text or how it differs qualitatively from baselines.\n- Missing generation process visualization: How does the hierarchical denoising process unfold? Examples showing the progression from mask→cluster→word tokens would illuminate the model's unique generation dynamics and help readers understand the practical implications of the hierarchical structure.\n- Lack of parameter sensitivity studies: How do key hyperparameters (number of clusters, clustering methods, noise schedules, force transition) affect generation behavior? Without these ablations, it's unclear which design choices matter and how to configure the model effectively.\n- No analysis of learned representations: What semantic patterns emerge in the hierarchical clusters? How does the model utilize the intermediate representations during generation? Such analysis would provide crucial insights into whether the theoretical benefits translate to meaningful semantic organization.\n\nQUESTIONS:\nNo\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-04"}