Title: Continuous Diffusion Model for Language Modeling
Venue: NeurIPS 2025
Accepted: yes
Keywords: Diffusion language model

Riemannian diffusion model

diffusion model for discrete data
Abstract: Diffusion models have emerged as a promising alternative to autoregressive models in modeling discrete categorical data. However, diffusion models that directly work on discrete data space fail to fully exploit the power of iterative refinement, as the signals are lost during transitions between discrete states. Existing continuous diffusion models for discrete data underperform compared to discrete methods, and the lack of a clear connection between the two approaches hinders the development of effective diffusion models for discrete data. In this work, we propose a continuous diffusion model for language modeling that incorporates the geometry of the underlying categorical distribution. We establish a connection between the discrete diffusion and continuous flow on the statistical manifold, and building on this analogy, introduce a simple diffusion process that generalizes existing discrete diffusion models. We further propose a simulation-free training framework based on radial symmetry, along with a simple technique to address the high dimensionality of the manifold. Comprehensive experiments on language modeling benchmarks and other modalities show that our method outperforms existing discrete diffusion models and approaches the performance of autoregressive models.


Peer reviews:
comments: SUMMARY:
This article proposes a continuous diffusion model for language modeling, which integrates the geometry of the underlying categorical distribution. It establishes a link between discrete diffusion and continuous flow on the statistical manifold, and based on this connection, introduces a simple diffusion process that generalizes existing discrete diffusion models. The article also puts forward a simulation-free training framework relying on radial symmetry, along with a straightforward technique to tackle the high dimensionality of the manifold. Experiments across language modeling benchmarks and DNA are conducted, showing that the proposed method surpasses existing discrete diffusion models.

STRENGTHS AND WEAKNESSES:
Strengths:
- The methodology of connecting discrete simplex modeling to the generative modeling on the sphere manifold is novel and reasonable, at least for me.
- The mathematical derivation is solid, though I didn't check the detailed proofs.

Weaknesses:
- I am not familiar with the language tasks, but I think I witnessed some more large-scale language generation benchmarks in the technical report of Qwen3 and Kimi-chat. I think the method in this paper can not scale to that level, and it is a little overclaim to say 'approaches the performance of autoregressive models.'.
- Detailed time-cost of RDLM compared to other discrete diffusion models and DDPM is needed.
- On CIFAR-10 experiments, the comparison to classical DDPM/EDM/rectified flow matching is beneficial. 
- Experiments on scalability should be given.

QUESTIONS:
see weaknesses

LIMITATIONS:
see weaknesses
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes Riemannian Diffusion Language Model (RDLM), a continuous diffusion model that treats each categorical token as a point on the positive orthant of a hypersphere. \n\nKey contributions of the work include:\n\n1. proving an explicit correspondence between discrete masked / uniform diffusion and geodesic flows on the statistical manifold\n2. designing a mixture path that smoothly blends the two processes to enable gradual refinement\n3. deriving a simulation‑free training objective that leverages radial symmetry to sample bridge states cheaply\n4. using dimension‑splitting that maps large vocabularies to products of low‑dimensional spheres for stable learning\n\nExperiments on Text8, LM1B, CIFAR‑10, and promoter‑DNA design show that RDLM is competitive with strong discrete diffusion baselines.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. By grounding diffusion in information geometry, the work unifies discrete and continuous views and offers a principled route to exploit iterative refinement without losing signal at state jumps.\n2. The paper contains detailed derivations of the flow equivalence, likelihood bound, and bridge approximation, with formal proofs moved to the appendix .\n3. Results on images and DNA sequences suggest the method works across different kinds of data.\n\n\n**Weaknesses**\n\n1. Important continuous text diffusion baselines such as [DiffuSeq](https://arxiv.org/abs/2210.08933), [TEncDM](https://arxiv.org/abs/2402.19097), and [LD4LG](https://arxiv.org/abs/2212.09462) are missing, making it hard to gauge relative progress.\n2. [GIDD](https://arxiv.org/abs/2503.04482), that also suggests combining masked and uniform discrete diffusion, is not discussed.\n3. The language results are limited to Text8 and LM1B, which are both relatively small and dated corpora. Other commonly used benchmarks for text diffusion models, such as RocStories, and OpenWebText, are not covered. This omission makes it difficult to compare the proposed method with other diffusion-based baselines. Furthermore, the field has advanced considerably, and evaluation on more challenging tasks, including conditional generation, would be necessary to demonstrate the method’s practical relevance.\n4. The work claims that the continuous approach offers advantages for controllable generation but provides no evidence on conditional text generation tasks, such as summarization.\n5. The bridge approximation is empirically validated with MMD, yet its impact on final likelihood in extreme dimensions is not thoroughly analysed.\n6. The paper also lacks a comparison with Gaussian diffusion on the simplex as an alternative approach to modeling discrete data, as explored in methods such as [SSD-LM](https://arxiv.org/abs/2210.17432) and [TESS](https://arxiv.org/abs/2305.08379).\n\nQUESTIONS:\n1. What happens if you use only the masked or only the uniform path? Include an ablation on LM1B; a clear gain would strengthen originality claims.\n2. Please clarify theoretical differences with GIDD and give a quantitative comparison.\n3. Did you run standard Gaussian diffusion on the simplex or the sphere before implementing the Riemannian diffusion approach?\n\nLIMITATIONS:\nThe dense theoretical exposition currently obscures the core ideas of the model. I recommend moving some of the derivations to the appendix and using the freed space to present more experimental analysis and comprehensive ablation studies in the main text.\n\nPlease refer to the questions and weaknesses outlined above for further details.", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

comments: SUMMARY:
This work studies the continuous diffusion modeling for the discrete language task and presents the Riemannian Diffusion Language Model (RDLM) based on solid theoretical guarantees. More specifically, they first show the relationship between continuous flow on the manifold. Based on this, this paper develops a complete training framework to train RDLM and achieve excellent results in text, image, and DNA sequence tasks.

STRENGTHS AND WEAKNESSES:
Strength:

1.	Each component of RDLM (including the theoretical foundation, mixture paths, and training framework) has been discussed and designed in detail.

I didn't find obvious weaknesses; I just have some questions (in the next section).

A minor Weakness:

1.	It would be better to add more visualizations for an easier understanding of the RDLM pipeline.

QUESTIONS:
1.	In line 726 of this paper, this work says that the mixture path process enhances the performance. Is this augmentation supported by experimental evidence (for example, RDLM without mixture path)?

2.	As this work shows the potential of RDLM, would it be possible to scale up RDLM to achieve great performance?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a continuous diffusion model for language modeling by projecting the categorical distribution to the positive orthant of a sphere, which is diffeomorphic to the manifold of distributions. This enables the benefit of continuous diffusion, such as efficient sampling. To make it practical for language modeling, several tricks are introduced, such as dimension splitting. Experiments on text and image demonstrate the potential of the proposed method.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper tackles continuous diffusion language models, which is an important topic in current AI research.\n\nWeaknesses:\n- While the proposed method is elegant and neat, it is unclear whether there is real benefit for large-scale language modeling. The experiments on language modeling remain relatively small-scale.\n- The concern about its effectiveness in large-scale settings is that, if we consider the ambient space $\\\\mathbb{R}\\^d$ in which the statistical manifold $\\\\mathcal{P}(\\\\mathcal{X})$ and the sphere $\\\\mathbb{S}\\^{d-1}$ are embedded, the gap between these two diffeomorphic manifolds becomes larger as $d$ increases. Intuitively, this makes transitions from some point to a token (i.e., $\\\\mathbf{e}\\_k$) much farther on the sphere than on the simplex. I am not sure if this is a critical reason why continuous diffusion models struggle to match the performance of discrete diffusion language models.\n\nQUESTIONS:\n- Line 135: It says \"providing numerous opportunities to correct wrong predictions during the process.\" I am unsure about this claim, because being continuous can also lead to error accumulation, while discrete transitions hold the possibility of entirely denoising the accumulated error.\n- Eq. (10): The left-hand side should be $p\\_\\\\theta(X\\_T)$? Since this is a distribution over $X\\_T$, it would be better to use a clearer notation for this.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}