Continuous Diffusion Model for Language Modeling
Authors: Jaehyeong Jo
Sung Ju Hwang
Venue: NeurIPS 2025 poster ()
Keywords: Diffusion language model
Riemannian diffusion model
diffusion model for discrete data
Primary area: deep_learning
TL;DR: We present a continuous diffusion model for language modeling that incorporates the geometry of the underlying categorical distribution.

Abstract: Diffusion models have emerged as a promising alternative to autoregressive models in modeling discrete categorical data. However, diffusion models that directly work on discrete data space fail to fully exploit the power of iterative refinement, as the signals are lost during transitions between discrete states. Existing continuous diffusion models for discrete data underperform compared to discrete methods, and the lack of a clear connection between the two approaches hinders the development of effective diffusion models for discrete data. In this work, we propose a continuous diffusion model for language modeling that incorporates the geometry of the underlying categorical distribution. We establish a connection between the discrete diffusion and continuous flow on the statistical manifold, and building on this analogy, introduce a simple diffusion process that generalizes existing discrete diffusion models. We further propose a simulation-free training framework based on radial symmetry, along with a simple technique to address the high dimensionality of the manifold. Comprehensive experiments on language modeling benchmarks and other modalities show that our method outperforms existing discrete diffusion models and approaches the performance of autoregressive models.

Continuous Diffusion Model for Language Modeling
Jaehyeong Jo', Sung Ju Hwang
KAIST DeepAuto.ai
{ harryjo97, sjhwang82 Hekaist.ac.o kr.
Abstract
Diffusion models have emerged as a promising alternative to autoregressive models in modeling discrete categorical data. However, diffusion models that directly work on discrete data space fail to fully exploit the power of iterative refinement, as the signals are lost during transitions between discrete states. Existing continuous diffusion models for discrete data underperform compared discrete methods, and the lack of a clear connection between the two approaches hinders the development of effective diffusion models for discrete data. In this work, we propose a continu- ous diffusion model for language modeling that incorporates the geometry of the underlying categorical distribution. We establish a connection between the discrete diffusion and continuous flow the statistical manifold, and building this analogy, introduce a simple diffusion process that generalizes existing discrete dif- fusion models. We further propose simulation-frs frroonn--eeeeee training framework based radial symmetry, along with a simple technique to address the high dimensionality of the manifold. Comprehensive experiments language modeling benchmarks and other modalities show that our method outperforms existing discrete diffusion models and approaches the performance of autoregressive models. The code is available at https: h //gi thub. cwm/com////a/ao http//arrrrjy/rbr///
Introduction
Discrete diffusion models [2] 39] emerged as promising competitor autoregressive models for the generative modeling of discrete data. These models have demonstrated competitive performance on tasks such as language modeling [49, 52] and code generation [20]. Unlike autoregressive models that generate data sequentially, diffusion models generate the sequence in parallel, allowing for bidirectional controllable generation and faster sampling.
However, discrete diffusion models do not fully harness the power of iterative refinement, which is the key generative modeling of continuous data such image synthesis [19, 48] and video generation [5, 46). In these models, the forward process progressively corrupts data through stochastic jumps between discrete states, modeled between these discrete states, which results in the loss of informative signals during refinement. Hence, discrete diffusion models often exhibit limited generative performance and reduced controllability.
Markov chain. Denoising is achieved through transitions
Several efforts have been made to adapt continuous diffusion models for discrete data, motivated by their advantages in controllability [26), efficient sampling [40, 41], optimized design choices [10, 34], and the potential to unify different modalities [35,56]. However, their performance often significantly lags behind that of discrete diffusion models. Early methods [24, 36] extended image diffusion models to discrete domains by applying unconstrained continuous relaxation. Other approaches [3, 54] project discrete data onto the probability simplex using the Dirichlet distribution as its prior over categorical distributions, but often fail to capture complex patterns. Recent works [12, 15] apply flow matching on the statistical manifold learn categorical distributions, but these methods are limited to short sequences and small vocabularies. particular, the connection between discrete and continuous diffusion remains poorly understood, hindering the development of unified diffusion framework.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

In this work, we present Riemannian Diffusion Language Model (RDLM), a continuous diffusion framework for language modeling that incorporates the geometry the statistical manifold into the diffusion processes. We establish connection between continuous flow on the statistical manifold and the discrete diffusion process, showing that the transition distribution can be modeled as a conditional flow on the manifold. Based on the analogy, we introduce a simple design of the diffusion processes on the manifold that generalizes previous discrete diffusion models. We further present a simulation- free training scheme that leverages radial symmetry, consisting of simple parameterization and maximum likelihood -based training objectives. Through experiments on language modeling, image modeling, and biological sequence design, we validate that framework outperforms existing discrete and continuous diffusion models.
2 Background
2.1 Discrete diffusion models
Discrete diffusion models [2, 39, 49, 52] define the diffusion process directly on discrete states using the Markov chains. The forward process describes the transition from the current state other states, which is formalized by multiplying the transition matrix Qr:
q( (x Cat(are; Q.Ft-1),
where 2, is the random variable for the discrete states and Cat(.) denotes the categorical distribution. The marginal distribution corresponds to repeatedly multiplying transition matrices over time steps:
q (z) Cat(r;; Qix) Cat( Qu...Quz)..
(2)
Austin et al. [2] introduced several designs of the transition matrices, including masked (absorbing state) and uniform diffusion, and has been extended to continuous time Markov chains (CTMC) [2. 6].
2.2 Statistical manifold of categorical distribution
Let X' = f1..... d} denote the discrete data space, and let A Ad-1 {p1) Pa) E TR. E, 1, 2 0} denote the (d -)-dieensiaa probability simplex. A categorical distribution over X can be parameterized by the parameters pl, Pa satisfying >, P; = I and p; > O. The statistical manifold P(X)) of the categorical distributions thus corresponds to the simplex A equipped with the Fisher-Rao metric [1, 47] (see Appendix A. There exists a diffeomorphism from the statistical manifold P()) to the positive orthant of the (d 1)-dimensional sphere $d-1;
T P(()) $ P. + u4 -P
(3)
which induces the geodesic distance dy (u, v) = cos" - (u. v) for u. v E 84 where (5.) denotes the Euclidean inner product. We provide more detailed explanation in Appendix A.I.
3
Riemannian Diffusion Language Model
We introduce a novel continuous diffusion model for language modeling. In this section, we present a single token generation framework, which generalize modeling sequences in Section 4.
3.1 Generalization of discrete diffusion
Continuous reparameterization of discrete data incorporate the geometry of the underlying categorical distribution, we leverage the statistical manifold to parameterize discrete data [12, 15]. Each point on the statistical manifold P(X) corresponds to the parameters a categorical distribution over the discrete sample space X' = {1. d). In this way, discrete data can be represented as continuous parameters of categorical distributions on the manifold.
Yet the Fisher-Rau Rach metric is ill-defined on the boundary of the manifold where the initial distribution of the parameterized data lies, leading to numerical instabilities near the boundary. To address this, we leverage the diffeomorphism (Eq, (3)) which maps P(X) to the positive orthant of a hypersphere sd-1 [12, 15]. where each point u sd. corresponds to Cat(:; (u)). This mapping enables

---

(a)
(b)
X x== Py
0.2
Z1
R
Y = Z.)
X
X;
Xo
e1 022
e1
((v/(10)
A²
g2
Masked Diffusion
Uniform Diffusion
X (1,2,m) m)
2,
Figure 1: Illustration of the continuous reparameterization of discrete data and two types of our generative process on hypersphere. (a) Example of a transition distribution of a discrete diffusion process modeled by continuous flow on d-dimensional sphere. (b) Illustration of the diffusion processes on 2 generalizing masked diffusion and uniform diffusion, respectively.
discrete data to be reparameterized as continuous states on gd-1 while preserving the geometry of the categorical distribution, which we illustrate in Figure I (a). The reparameterized data distribution
Pdota on the hypersphere can be written as Potaaa (æ) = Σd -m p&0(x ex) where ps denotes the probability of the k-th state, and eş are d-dimensional one-hot vectors. In the case of masked diffusion, the discrete sample space is augmented with an additional mask state m.
From discrete difflusion continuous flow Our key observation is that the transition distribution qe (xxzlzz) of a discrete diffusion process (Eq. (2)) is a categorical distribution on X. Therefore, modeling Ye equivalent modeling continuous flow on the statistical manifold P(X). We show in the following proposition that discrete diffusion models over X' can be modeled by continuous flow on ((X) and further on $ (we provide the full proof in Appendix A.2).
Proposition 3.1. The transition distribution of discrete diffusion processes be modeled by the continuous flow on the statistical manifold, and further on the hypersphere.
proof sketch. A flow S4-1 that interpolates yo and yı as geodesic is described by the ODE:
dY. dl log Ki dt
expy, (31), Yo yo,
dt
where exp"* denotes the logarithm map on the hypersphere. Then, for well-designed schedule and endpoint y/, the process Z, := क (Y,) on P(X) corresponds to the transition distribution discrete diffusion process. In particular, we obtain the masked diffusion process for y = i.e.,
the mask token, and the uniform diffusion process for y Σi e//vd.
Although discrete diffusion processes can be represented as a flow on the statistical manifold, this flow cannot be learned by a neural network. The network fails to generalize to points outside the geodesic that interpolates the prior and the data distribution, producing an incorrect vector field. Therefore, we present simple design for continuous diffusion model that generalizes existing discrete diffusion models.
3.2 Generative process on hypersphere
The task modeling the distribution of discrete data can be reformulated as modeling a distribution Pdota the hypersphere. Building upon the Riemannian diffusion mixture framework [33], con- struct a diffusion process on S such that its terminal distribution matches Pdata- The construction entails deriving diffusion mixture representation based on bridge processes defined on gd-1
We first derive a bridge process (X,){= on sd-1 from an arbitrary point ag E $d-1 e as follows (we provide detailed derivation in Appendix A.3):
cos"! ex) - (Xe,e) dt aB X æo,
dx, -tt
V1 V1-(X,,eay)
where 7t 02/ Je ddss and By denotes the Brownian motion defined on $d-1, Intuitively, the current state X, moves the direction that minimizes the geodesic distance to the endpoint, resulting in process that bridges the starting and end points. While different forms of the bridge process

---

exist, for example, scaling the drift or the diffusion coefficients, Eq. (5) yields specific transition distribution that enables simulation-free training, which we explain in Section 3.3.
From the bridge processes, we construct a generative process {x,] on S using the diffu- sion mixture representation (see Appendix A.4 for the formal definition of the diffusion mixture representation and the derivation of the generative process in Corollary A.8):
dx,- = Σ Prie(es)X.i) t) dt o. Xo L0.
(6)
where (,,t) denote the drift of the bridge process in Eq. (5). Here, Pric(es X,) represents the probability that ex will be the final outcome of the process at time T, given the current state X, at time t. Note that the construction guarantees the terminal distribution of the process to be Pdasta
An ideal generative process is that gradually refines the uninformative states to recover the original tokens. We analyze the convergence of the bridge process through its radial process r == d, (Xe, ex) described by the following SDE (see Appendix A.3 for the derivation using Itô's formula):
(d 1)o? cot dt + o,dW, r = cos" ex),
drf 7ork
(7)
where W. is I-dimensional Wiener process. For do > oT. the radial process converges rapidly in early time steps, making it difficult for a neural network to approximate accurately. We empirically find that the geometric schedule a, = 0 -T- af with op < or leads to gradual convergence.
Masked difflusonn Based Proposition 3.1, initializing the generative process in Eq. (6) with the mask token, i.e., = yields a mixture process that generalizes the discrete masked diffusion framework. The diffusion process starts at the mask token and progressively evolves toward one of the target tokens cx2 as visualized in Figure (b). From the perspective of the discrete diffusion model, mixture process smoothly interpolates the discrete jump from e. e, through intermediate continuous states X,, where the final token is determined by the probability PT (ex) (ex|X,) X,).
The fundamental difference is that discrete masked diffusion operates through direct jumps between token and the mask token, where any incorrect transition is irreversible. In contrast, our continuous ap- proach allows for gradual transitions, providing numerous opportunities to correct wrong predictions during the process. This leads to more accurate modeling of the underlying data distribution.
Uniform diffusion Based on Proposition 3.1. the generalization of uniform diffusion can be achieved by initializing the generative process of Eq. (6) with the barycenter of the simplex d projected onto $d-1, i.e. x e,/d) = Ei= e./vd. We visualize the diffusion process in Figure (b). Intuitively, the barycenter of d corresponds to the uniform categorical distribution over d categories, which serves as the stationary distribution of the discrete uniform diffusion process.
Mixture paths We derive new family of generative processes by constructing a mixture over the time marginals of generative processes {Q
S i n. (see Appendix A.5 for derivation):
X = 0sx
(8)
1=1
where X is the dimeddependent mixing schedule assigned to the i-the generative path. This construc- tion allows the resulting process to transition between different generative behaviors over time.
In partieular, propose simple yet effective mixture path built from mixing the time marginals of the masked diffusion and uniform diffusion, for a time-dependent schedule X as follows:
X maak + 2n)
with initial distribution Ago(c.m) ( (1-)))6 e,/ Va). This formulation generalizes the mixture paths used in discrete flow matching [51] and the state-dependent schedule [52].
Generalizing flow matching Notably, our framework generalizes previous flow matching methods
on the statistical manifold [12, 15). By designing the noise schedule in Eq. (5) to a, = oo -+
we obtain the conditional vector field of the flow matching models.

---

3.3 Simulation-Free Training with Radial Symmetry
Next, we introduce our training scheme. We present simple parameterization of our generative model and derive the likelihood bound and training objectives. Further, we present simulation-fre training method based on the radial symmetry of the hypersphere.
Model parameterization To use the diffusion process in Eq. (6) as generative model, its unknown drift should be learned through neural network, similarly to flow matching [8. 37] bridge matching [33]. Yet the drift of the mixture process diverges near the terminal time ',, which makes it
challenging learn. Therefore, instead of approximating the drift function directly, we propose to model the probability Pr Xx (x,) with a neural network 8g as follows:
po (X t) == softmax (se (x,,) t)) (rij(ei|Xx)). ..r. (ea
(10)
which converges to eş for some k as l + T. In the case of masked diffusion, set the probability Pri((e 6 X₂) to be zero for all I, indicating that the final state cannot be mask token. From Eq. (10), the drift of the mixture process in Eq. (6) is parameterized follows:
no (x,,))
Σ (po(X,,)) ex)1 k=1
(11)
Our parameterization shares similar properties with the discrete masked diffusion [49]: (1) Zero Mask Probabilities. The final state cannot be a mask token. (2) Carry-Over Over Unmasking. If X, converges to a token before the terminal time, converges to zero, and the state X, is carried over without changing to different token.
Likelihood bound We derive a tractable upper bound on the negative likelihood of our generative model by applying the Girsanov theorem on compact manifolds (De Bortoli al. [16], Corollary H.3). Specifically, we first establish a point-wise upper bound on the negative log-likelihood under the parameterized mixture process Q" using the KL divergence between Q and bridge process @ which is conditioned on endpoints vo and eμ. Applying the Girsanov theorem, we obtain the following variational upper bound (we provide a detailed derivation Appendix A.6):
- (log D
1()) S E X
7i (10 (m0 (x.))) di
(12)
J
where 1/t is the drift defined in Eq. (5). The point-wise likelihood bound yields upper bound on the negative log-likelihood of our generative model parameterized by ps:
Paata - log [=)]
S
Eeu-Paata X
x (X,,)) {
of
(13)
Jo
Objective Based on the likelihood bound Eq. (13), we introduce a maximum likelihood training objective for the model parameterization Pa in Eq. (10):
1 oF
[(()= Eey--eaaee X X~
Σ t).
-x,
t)
dt
(14)
2
J
This objective corresponds to minimizing the mean squared emror in approximating the drift term.
In particular, L(0) can be minimized by reducing the cross-entropy between the predicted probability pe (X,, t) and the target one-hot vector e₁. Therefore we present a cross-entropy-biit based training objective, analogous to those used discrete diffusion models [49, 52]:
- log (), ex)dd X-22
CE (0) = -eee--paee Pdnta
(15)
We show Appendix A.7 that minimizing the cross-entropy- objective Eq. (15) leads to minimizing (()), thereby ensuring maximum likelihood training. We experimentally find that the cross-entropy loss _CE (0) yields faster convergence training and leads to better performance than the mean squared error loss L()).
5

---

Importance sampling The difficulty of approximating the probability PT (rrr(rrX varies sig- nificantly across different time points While predicting X fairly easy in the later stage of the process, it is challenging to do so during the middle of the process. The training can be improved by training more on the challenging time points. We derive an equivalent objective by applying importance sampling over I, which reweights the time distribution to focus on specific interval:
4g (0) = Resepaata Beng ((1)) log (po( Xt, 1), ex)
(16)
X-Q"
where q is normalized proposal distribution over t. We find that a simple choice g(t) = + (1
2e) (a,6( with small e effectively concentrates sampling within the desired time interval.
Approximation of transition distribution Our training objective involves sampling X, from the bridge processes each iteration. Yet this introduces a significant bottleneck during training, as it requires simulating the process due to its intractable transition distribution on the d-dimensional sphere. Therefore, we present an approximate sampling method that bypasses the need for simulation, thereby enabling scalable training across large vocabularies.
We propose approximate the distribution p( Xo, as the push-forward of Gaussian distri- bution on the tangent space via the exponential map, i.e., the Riemannian normal. This approximation justified by the fact that Eq. (5) results from applying a time change [64] to simple bridge process (Eq. (59)). which yields transition distribution similar to Riemannian normal.
We parameterize the mean of the Riemannian normal distribution μ4 == EX,,/ | and its covariance Σ, == -ov [exp]], (X.)] using the parameters Or and p; follows:
(V1 1 Or cos do
(V1-0²)
XT +
Xo
Σ, = I
(17)
sin Øu
sin @u
where do == cos" (Xo, Xr). Intuitively, f4, represents the normalized centroid of the samples X and Σg captures to the covariance of the lifted samples in the tangent space Tw...
Parameters of Riemannian normal While the parameters at and Pt are generally intractable, we derive them from the I-dimensional projections of the mixture process. Our main idea is to express the parameters in terms of the projected processes zi == ( Xr) and = = (Xt o,T; Xo), where X denotes the diffusion process (X,) conditioned on fixed endpoints X, and Xy. These projected processes are modeled by the following 1 1-dimensional SDEs (see Appendix A.8 for the derivation using the Itô's formula and the radial symmetry of S
(d - 1)07
dz/-
Tt cos =! V1 (zT)²-
T
di Bov 1 1 (=7))
(18)
2
1/- cos
2T
(d - 1002
d2f
(T _0.T) 20
dt + or 1 (24)² dwo, 2t
(19)
(=.
with =/6 = (Xo, Xr) and = = 1. where WT" and WP denote 1-dimensional Wiener processes. In the case of masked and uniform diffusion, Xo is fixed a single point such that (Xo, ex) is identical for all non-mask tokens ex. As a result, the mean projections B=f" and Exς remain invariant with respect to the choice of Xr.
Based on the radial symmetry of gdd1, we derive the parameters a and pr from the mean projections Ezl and Ez!" as follows (we provide detailed derviation in Appendix A.9):
(E>f /E2 cos )0
0t
Pt P F2
BBE//
B2
o²
(20)
sin do + (F /Raa - cos dn)
where du == cos'! (Xo, X and FJ' denotes the inverse of a damped Kummer function (Eq. (115)).
For small values of d, we calibrate Pt by applying a constant scaling factor.
The mean projections EZY and Ezf' can be easily obtained by simulating the 1-dimensional processes Eq. (18) and Eq. (19). Therefore, prior to training our model po. we precompute the parameters (OL//K Pc/KK } once, using sufficiently large value of K. The procedure for this precomputation is outlined in Algorithm 3 in the Appendix.

---

Algorithm Training
Input: Initial point u, model Pa. vocabulary size d, token sequence length L, time distribution (().,
pre-computed (a./K=Ps/KKEE) For each epoch:
1: Sample token sequence & from the training set
2: Xo f- (2)² and X1 < (ONE-HOT( (s)
3: Φo - (cos") (Xi. x
4: and ac. Pt + INTERPOLATE (LaS/K: ; PE/K 11=0)
5: μ4 + o Xi + (V of Oe 00s 40
Xi)
< Eq. (17)
6: X, Nga-1 (m) . La)
x Ng (ut .
Sample from Riemannian normal Cross-entropy- loss in Eq. (16)
7: Lo + -Q(( log (po (x,;); X1)
8: Update 0 using Lo
Algorithm 2 Sampling
Input: Initial point u, trained model po. vocabulary size d, number of sampling steps M, token sequence length L, noise schedule dt
1: X (w) +- 0 and òt < 1/M
> Start from the initial point
2: for m = 1 to M do
3: w ~ (O Ia))
4: p (X))
5: T]e
Ek (p',ex))
pos (x' (X' -xx'
Parameterization in Eq. (11)
VI-(X',,e
eexxx: (1gōt + orvoto") )
6: X -
P> Geodesic random walk
7: t +
8: end for
9: 8 + (ARGMAX(X*))
Return: Token sequence $
During training, we can sample X, from the Riemannian normal distribution without expensive simulation of the bridge processes. Compared to the simulation-barl training, our approach yields x50 speedup. In Section 6.4, we experimentally demonstrate that the Riemannian normal provides an accurate approximation of the distribution of X,-
Generation of Token Sequences
We now extend the single-token modeling framework to the genera- tion of token sequences. Since each token the sequence reparameterized onto a d-dimensional sphere, a sequence of length is modeled on the product manifold (Sd-1)". This formulation allows the sequence-level diffusion to be treated as a joint process over the spherical components.
Modeling sequence of tokens
We model the generative process as system of SDEs {X. X where each X; evolves according to a diffusion process on gd-1 analogous to the single-token formulation in Eq. (6):
ax:
pl X = ex x (X), t) dt + ad 1 1 S n.
(21)
Here p( p(X} = eu x!!) denotes the probability that the i-th token corresponds to k-th state, conditioned on the current intermediate sequence X1n.. Using the parameterization defined in Eq. (10), we train neural network to predict y Xx X.)).. The training and sampling procedures
for modeling token sequences outlined in Algorithms and 2, respectively.
7

---

Dimension splitting of statistical manifold For a large vocabulary set, the corresponding statistical manifold becomes sigh-dimessional, which introduces two challenges: (1) Sharp transition. Bridge processes on high-dimensional spheres tend to exhibit sharp transitions near the terminal time. This high-dimensional convergence behavior makes the mixture process difficult for neural networks to learn. (2) High input dimensionality. The input to the network resides in high-dimensional space, requiring sufficiently large hidden dimensions to encode the data adequately. Models with limited capacity fail to learn the conditional probabilityy p(X}"" X!"").
To address these challenges, we introduce dimension splitting, simple technique to reduce the dimensionality of the parameterized manifold. Instead of mapping the k-th token directly to g we first represent the index It in base and then map the represntation to the product manifold ( for m:= [log, d]. Dimension splitting reparameterizes a sequence of length L to a product manifold (st)"L The resulting bridge processes on S with small b exhibit gradual convergence over time, making them significantly easier for neural networks learn. Dimension splitting significantly enhances the likelihood when used together with the mixture path (Eq. (9)).
Related Work
Discrete diffusion models Discrete diffusion directly models the Markov chain on the discrete data space. One-hot data distributions are gradually corrupted to a stationary distribution using specific transition matrices, and the noising process corresponds to the stochastic jumps between states in the Markov chain. D3PM [2] introduces discrete-time Markov forward processes with both uniform and absorbing state transition matrices, and has been generalized to the continuous-time Markov chain framework [6]. SEDD [39] proposes learning the score entropy of discrete states instead of predicting the mean. Recent works [49, 52] introduce continuous-ting masked diffusion models, which offer simpler likelihood bounds compared to previous works. We provide further discussions on comparison with discrete diffusion models in Appendix A.10.
Continuous diffusion models for discrete data Early approaches discrete data modeling either fully relaxed discrete data into continuous space [24] or embedded tokens into a latent space [18, 36], without imposing any constraint. However, continuous relaxation without constraint fails to capture the discreteness of the categorical distribution. Recent works operate directly in logit space [21, 29] or on the probability simplex [3, 54), but rely on imperfect assumptions that fail to accurately represent the underlying categorical distribution. Flow matching has been applied to the statistical manifold to model the categorical distribution [12, 15], but these methods are limited to short sequences and small vocabularies. We provide detailed comparison in Appendix A0
6 Experiments
6.1 Text generation
We evaluate our Riemannian Diffusion Language Model (RDLM) for text generation tasks on two language benchmarks: Text8 [42] and One Billion Words Dataset [7].
Baselines We compare against state-of -fhte-art diffusion and autoregressive models. Multinomial
Diffusion [29], D3PM [2], SEDD [39]. MDLM [49]. MD4 [52] are discrete diffusion models.
Plaid [23] and Bayesian Flow Network (BFN) [21] are continuous diffusion models. IAF/SCF [63], AR Argmax Flow [29], and Discrete Flow [58] are flow-based models, and ARDM [30] and MAC [53] are any-order autoregressive models. We also compare with the transformer AR model [61]. We provide further details on the baselines in Appendix B.2
Implementation details For all experiments, we use the same data split and context size following Lou et al. [39] and Sahoo et al. [49]. For Text8, we randomly sample contiguous chunks of length 256 as done in previous works [2, 39]. For One Billion Words, we use the same tokenizer in He et al. [25] with context size 128. We use diffusion transformer architecture [44] with rotary positional embeddings [55] for all the experiments and match the number of parameters as used in the previous works [39, 49]. For our model, we use the mixture path of masked and uniform diffusion (Eq. (9)) and apply dimension splitting for large vocabulary. We provide more details in Appendix B.2.
8

---

Table Test perplexity results on LMIB dataset. Baseline results taken from Sahoo et al. [49].
Table 3: BPD results on CIFAR-I0 test set. Base-
line results taken from Shi et al. [52].
Method
Param.
PPL (4)
Method
Param.
BPD (4)
Autoregressive
Autoregressive
Transformer-X Base [14]
0.46B
23.5
PixeIRNN [60]
3.00
Gated PixeICNN [59]
3.03
OmniNety [57]
100M
21.5
PixeICNN+ [50]
53M
2.92
Transformer [61]
110M
22.32
PixelSNAIL [11]
46M
2.85
Discrete Diffusion
Image Transformer [43]
2.90
BERT-Mouth [62]
110M
142.89
Sparse Transformer [13]
59M
D3PM Absorb [2]
70M
76.90
Discrete Diffusion
DiffusionBert [25]
110M
63.78
D3PM Absorb [2]
37M
S 4,40
SEDD [39]
110M
32.79
D3PM Gauss [2]
36M
3.44
rLDR [6]
36M
3.59
MDLM [49]
110M
27.04
rLDR Absorb [6]
36M
3.52
Continuous Diffission
MD4 [52]
28M
S 2.78
Diffusion-La LM [36]
80M
118.62
Continuous Diffusion
RDLM (Ours)
110M
28.44
RDLM (Ours)
28M
S 2.73
Text8 We first evaluate on a small-scale character-level language modeling task. The Text8 [42] dataset is character-level text modeling benchmark extracted from English Wikipedia. We train models short text chunks of length 256 and evaluate the performance using Bits Per Character (BPC). As shown in Table 1, framework outperforms all previous diffusion models, including both discrete and continuous methods. We also outperform any- order autoregressive models that generate texts flexible decoding order, similar to discrete diffusion models. We achieve similar generative perplexity and entropy com- pared to existing discrete diffusion models. We provide generated texts from RDLM in Appendix C.1.
Table I: Bits Per Character (BPC) re- sults on Text8 test set. Results are taken from the corresponding papers.
Method
BPC (1)
Autoregressive
Argmax Flow [29]
1.39
Transformer AR [61]
1.23
Discrete Flow [58]
1.23
Any-order Autoregressive
ARDM [30]
1.43
MAC [53]
1.40
Discrete Diffusion
Multinomial Diffusion [29]
1.72
D3PM Uniform [2]
1.61
D3PM Absorb [2]
1.45
One Billion Words We further evaluate RDLM on One Billion Words Dataset (LMIB) [7], a medium-scale real- world language benchmark with vocabulary size of 30522. We evaluate the performance using perplexity (PPI.), and the results are summarized Table RDLM outperforms most existing diffusion models and is compet- itive with the state-of-the-an discrete diffusion model [49]. Notably, ours significantly outperforms the prior contin- uous diffusion model [36], demonstrating the effectiveness of incorporating the geometry of the
SEDD Absorb [39]
1.39
MDLM [49]
1.40
MD4 [52]
1.37
Continuous Diffusion
Plaid [23]
1.48
BFN [21]
1.41
RDLM (Ours)
1.32
underlying categorical distribution. We provide a discussion of the results with MDLM [49] in Appendix B.2. The generated texts are presented Appendix C.2.
6.2 Pixel-level image modeling
We further explore applications of RDLM beyond the domain by applying order-agnostic image data. Each image is represented as set of discrete tokens with a vocabulary of size 256, removing information about pixel proximity. Note that this different from the experimental settings with image diffusion models [27, 34] that use spatial information. We compare RDLM against autoregressive models and discrete diffusion models that operate directly on raw pixel space, which we describe in Appendix B.3. As shown in Table 3, our method achieves the lowest Bits Per Dimension (BPD), outperforming the discrete diffusion models [2, 52] and autoregressive baselines [11, 13). We attribute this strong performance inherently continuous data to the continuous nature of our framework, which fully exploits iterative refinement, suggesting potential for unifying modeling across different modalities.

---

6.3 DNA sequence design
We demonstrate that our framework can be applied to bi- ological sequence generation. We evaluate our method on the promoter DNA sequence design task, which aims to generate valid promoter DNA sequences conditioned on transcription profiles. detailed description of the task is provided in Appendix B.4. Model performance measured by the mean squared error (MSE) between the predicted regulatory activity of the generated sequence and that of the original sequence corresponding to the transcription profile. Table 4 shows that our framework achieves the lowest MSE, outperforming the flow match- ing methods [15, 54] and the discrete diffusion model [2].
Table 4: MSE results on the generated promoter DNA sequences. Baseline re- sults are taken from Davis et al. [15].
Method
MSE (L)
Bit- Bit-Diffusion (bit) [10]
0.041
Rit-Diffusion (one-hot) [10]
0.040
D3PM Uniform [2]
0.038
DDSM [3]
0.033
Dirichlet-M [54]
0.034
Language Model
0.034
Fisher-Flow [15]
0.029
RDLM (Ours)
0.027
6.4 Analysis
Training objective We validate the effectiveness of our eross-entropy-based loss of Eq. (15) in Table 5. Compared to the mean squared error loss of Eq. (14), the cross-entropy loss provides faster convergence in training and better NLL. Furthermore, Table shows that applying importance sampling to the training objective as defined in Eq. (16) yields improved likelihood.
Approximation of transition distribution We validate that our approximate sampling method closely matches the true transition distribution of the mixture process. In Figure 2, we report the maximum mean discrepancy (MMD) [22] distance between the simulated transition distribution and the approximated distribution obtained using the Riemannian normal. The approximated distributions exhibit nearly identical MMD as the simulated distributions, indicating that he approximation is accurate and reliable. Notably, the discrepancy approaches zero in high-dimensional manifolds, where simulation becomes increasingly expensive, making simulation baseed training impractical.
Dimension splitting For datasets with a large vocabulary, such as the LMIB dataset, our dimension splitting technique (Section 4) results in a significant improvement. Table shows that directly training a model on discrete data with large vocabulary fails capture the underlying distribution, due to the high input dimensionality. In particular, the sharp transition near the terminal time for high-dimensional mixture process makes it challenging for neural networks to learn. In large vocabulary settings, we achieve the best result via dimension splitting, combined with modeling the generative process using mixture path of masked and uniform diffusion.
7 Conclusion
In this work, we introduced the Riemannian Diffusion Language Model (RDLM), a continuous diffu- sion model for language and discrete data. We present simple framework that generalizes discrete diffusion models, building on the connection between the transition distribution and continuous flow on the statistical manifold. We provide general designs for generative processes and introduce simulation-fii training scheme leveraging the radial symmetry. Through experiments on language modeling benchmarks. RDLM demonstrates strong performance over prior discrete and continuous diffusion models. We further extend our approach to other modalities, including image and biological sequence generation, where RLDM achieves consistently strong results.
8
Acknowledgements
This work was supported by National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. RS-2023-002562511 Institute for Information & communications Technol- ogy (Iomotion(IITP) grant funded by the Korea government (MSIT) (No.RS-2015 20S9-209-11199000 Artificial Intelligence Graduate School Program(KAISS) ning & Evaluation (ITTP) with grant funded by the Ministry of Science and ICT (MSIT) of the Republic of Korea in connection with the Global Al Frontier Lab International Collaborative Research. (No. RSS-022-0046944 & RS-2024-0050929 and artificial intelligence industrial convergence
Information Communications Technology Plan-
10

---

cluster development project funded by the Ministry of Science and ICT(MSIT, Korea)&Gwangju Metropolitan City.
References
[1] Shun-ichi Amari. Information geometry and its applications, volume 194. Springer, 2016.
[2] Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Struc- tured denoising diffusion models discrete state-spaces. In Advances in Neural Information Processing Systems, 2021.
[3] Pavel Avdeyev, Chenlai Shi, Yuhao Tan, Kseniia Dudnyk, and Jian Zhou. Dirichlet diffusion score model for biological sequence generation. In International Conference on Machine Learning. 2023.
[4] Nihat Ay, Jürgen Jost, Hông Vân Lê, and Lorenz Schwachhöfer. Information geomelry, vol- ume 64. Springer, 2017.
[5] Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurt, Joe Taylor, Troy Luhman, Eric Luhman, et al. Video generation models as world simulators, 2024.
[6] Andrew Campbell, Joe Benton, Valentin De Bortoli, Thomas Rainforth, George Deligiannidis, and Arnaud Doucet. continuous time framework for discrete denoising models. In Advances in Neural Information Processing Systems, 2022.
[7] Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. One billion word benchmark for measuring progress in statistical language modeling. arXiv preprint arXiv: 13 13,..000 2013.
[8] Ricky T. Q. Chen and Yaron Lipman. Flow matching on general geometries. In International Conference on Learning Representations, 2024.
[9] Ting Chen. On the importance of noise scheduling for diffusion models. arXiv:3301.10972 2023.
[10] Ting Chen, Ruixiang Zhang, and Geoffrey E. Hinton. Analog bits: using diffusion models with self-conditioning. In International Conference Learning Repre- sentation, 2023.
Generating discrete data
[11] Xi Chen, Nikhil Mishra, Mostafa Rohaninejad, and Pieter Abbeel. Pixelsnail: An improved autoregressive generative model. In International Conference on Machine Learning, 2018.
[12] Chaoran Cheng, Jiahan Li, Jian Peng, and Ge Liu. Categorical flow matching statistical manifolds. In Advances in Neural Information Processing Systems, 2024.
[13] Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv:1904.10509 2019.
[14] Zihang Dai, Zhilin Yang, Yiming Yang. Jaime G. Carbonell, Quoc Viet Le, and Ruslan Salakhut- dinov. Transformer-xl: Attentive language models beyond a fixed-length context. In Association for Computational Linguistics, 2019.
[15] Oscar Davis, Samuel Kessler, Mircea Petrache, ismail iIkan Ceylan, Michael M. Bronstein, and Avishek Joey Bose. Fisher flow matching for generative modeling over discrete data. In Advances in Neural Information Processing Systems, 2024.
[16] Valentin De Bortoli, Emile Mathieu, Michael Hutchinson, James Thornton, Yee Whye Teh, and Arnaud Doucet. Riemannian score-based generative modelling. In Advances in Neural Information Processing Systems, 2022.
[17] Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat gans on image synthesis.
In Advances in Neural Information Processing Systems, 2021.

---

[18] Sander Dieleman, Laurent Sartran, Arman Roshannai, Nikolay Savinov, Yaroslav Ganin, Pierre H Richemond, Amaud Doucet, Robin Strudel, Chris Dyer, Conor Durkan, al. Continu- ous diffusion for categorical data. arXiv:221 1..1v22211.1009,, 2022.
[19] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, and Robin Rombach. Scaling rectified flow transformers for high-resolution image synthesis. In International Conference on Machine Learning, 2024.
[20] Itai Gat, Tal Remez, Neta Shaul, Felix Kreuk, Ricky Q. Chen, Gabriel Synnaeve. Yossi Adi, and Yaron Lipman. Discrete flow matching, In Advances in Neural Information Processing Systems, 2024.
[21] Alex Graves, Rupesh Kumar Srivastava, Timothy Atkinson, and Faustino Gomez. Bayesian flow networks. arXiv:2308, 200i:2330,,7033,, 2023.
[22] Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. The Journal of Machine Learning Research, 13(1):723-773, 2012.
[23] Ishaan Gulrajani and Tatsunori B Hashimoto. Likelihood-basee diffusion language models. In Advances in Neural Information Processing Systems, 2024.
[24] Xiaochuang Han, Sachin Kumar, and Yulia Tsvetkov. Ssd-Im: Semi--utoregressivve simplex- based diffusion language model for text generation and modular control. arXiv:2210.17432. 2022.
[25] Zhengfu He, Tianxiang Sun, Qiong Tang, Kuanning Wang, Xuanjing Huang, and Xipeng Qiu. Diffusionbert: Improving generative masked language models with diffusion models. In Anmual Meeting of the Association for Computational Linguistics, 2023.
[26] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv:2207.12590 2022.
[27] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances Neural Information Processing Systems, 2020.
[28] Bhung-Chau Hon, Jordan Ramilowski, Jayson Harshbarger, Nicolas Bertin, Owen JL Rack- ham, Julian Gough, Elena Denisenko, Sebastian Schmeier, Thomas M Poulsen, Jessica Severin, al. An atlas of human long non-coding raa with accurate 5 ends. Nature, 543(7644):199-204, 2017.
[29] Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling, Argmax flows and multinomial diffusion: Learning categorical distributions. In Advances in Neural Information Processing Systems, 2021.
[30] Emiel Hoogeboom, Alexey A Gritsenko, Jasmijn Bastings, Ben Poole, Rianne van den Berg, and Tim Salimans. Autoregressive diffusion models. In International Conference on Learning Representation, 2022.
[31] Emiel Hoogehoom, Jonathan Heek, and Tim Salimans. simple diffusion: End-to-end diffusion for high resolution images. In International Conference on Machine Learning, 2023.
[32] Elton P Hsu. Stochastic analysis manifolds. Number 38 in Graduate studies in mathematics. American Mathematical Society, 2002.
[33] Jachyeong Jo and Sung Ju Hwang. Generative modeling on manifolds through mixture of riemannian diffusion processes. In International Conference on Machine Learning, 2024.
[34] Tero Karras, Miia Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. In Advances in Neural Information Processing Systems, 2022.
[35] Shufan Li, Konstantinos Kallidromitis, Akash Gokul, Zichun Liao, Yusuke Kato, Kazuki Kozuka, and Aditya Grover. Omniflow: Any-to-any generation with multi-modal rectified flows. arXiv:2412.01166 2024.
12

---

[36] Xiang Li, John Thickstun, Ishaan Guirajani, Percy S Liang. and Tatsumori B Hashimoto. Diffusion-Im improves controllable text generation. In Advances Neural Information Pro- cessing Systems, 2022.
[37] Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In International Conference on Learning Representations, 2023.
[38] Iyya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv:171 2017.
[39] Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion language modeling by estimating the ratios of the data distribution. In International Conference on Machine Learning, 2024.
[40] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ODE solver for diffusion probabilistic model sampling in around 10 steps. In Advances in Neural Information Processing Systems, 2022.
[41] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu, Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv:221 2022.
[42] Matt Mahoney, Large text compression benchmark. text. html, 2006.
net/d//
[43] Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In International Conference on Machine Learning, 2018.
[44] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023.
[45] Stefano Peluchetti. Non-denoising forward-time diffusions. Openreview, 2021.
[46] Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoory Vyas, Bowen Shi, Chih- Yao Ma, Ching- Yao Chuang, David Yan, Dhruv Choudhary, Dingkang Wang, Geet Sethi, Guan Pang, Haoyu Ma, Ishan Misra, Ji Hou, Jialiang Wang, Kiran Ja- gadeesh, Kunpeng Li, Luxin Zhang, Mannat Singh, Mary Williamson, Matt Le, Matthew Yu, Mitesh Kumar Singh, Peizhao Zhang, Peter Vajda, Quentin Duval, Rohit Girdhar, Roshan Sumbaly, Sai Saketh Rambhatla, Sam S. Tsai, Samaneh Azadi, Samyak Datta, Sanyuan Chen, Sean Bell, Sharadh Ramaswamy, Shelly Sheynin, Siddharth Bhattacharya, Simran Motwani, Tao Xu. Tianhe Li, Tingbo Hou, Wei-Ning Hsu, Xi Yin, Xiaoliang Dai, Yaniv Taigman, Yaqiao Luo, Yen-Cheng Liu, Yi-Chiao Wu, Yue Zhao, Yuval Kirstain, Zecheng He, Zijian He, Albert Pumarola, Ali K. Thabet, Artsiom Sanakoyeu, Arun Mallya, Baishan Guo. Boris Araya, Breena Kerr, Carleigh Wood, Ce Liu, Cen Peng, Dmitry Vengertsev, Edgar Schönfeld, Elliot Blanchard, Felix Juefei-Xu, Fraylie Nord, Jeff Liang, John Hoffman, Jonas Kohler, Kaolin Fire, Karthik Sivakumar, Lawrence Chen, Licheng Yu, Luya Gao, Markos Georgopoulos, Rashel Moritz, Sara K. Sampson, Shikai Li, Simone Parmeggiani, Steve Fine, Tara Fowler, Vladan Petrovic, and Yuming Du. Movie gen: A cast of media foundation models. arXiv:2410. 2024.
[47] C Radhakrishna Rao. Information and the accuracy attainable in the estimation of statistical parameters. In Breakthroughs in Statistics: Foundations and basic theory, pages 235-2477 Springer, 1992.
[48] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Seyed Kamyar Seyed Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. Photorealistic text-to-image imaie diffusion models with deep language understanding. In Advances in Neural Information Processing Systems, 2022.
[49] Subham Sekhar Sahoo, Marianne Arriola, Aaron Gokaslan, Edgar Mariano Marroquin, Alexan- der M Rush, Yair Schiff, Justin Chiu, and Volodymyr Kuleshov. Simple and effective masked diffusion language models. In Advances in Neural Information Processing Systems, 2024.
13

---

[50] Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P. Kingma. Pixelenn++: Improving the pixelenn with discretized logistic mixture likelihood and other modifications. In International Conference on Learning Representations, 2017.
[51] Neta Shaul, Itai Gat, Marton Havasi, Daniel Severo, Anuroop Sriram, Peter Holderrieth, Brian Karrer, Yaron Lipman, and Ricky TQ Chen. Flow matching with general discrete paths: A kinetic-optimal perspective. arXiv:2412.03487, 2024.
[52] Jiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis K Titsias. Simplified and generalized masked diffusion for discrete data. In Advances in Neural Information Processing Systems, 2024.
[53] Andy Shih, Dorsa Sadigh, and Stefano Ermon. Training and inference on any-order autoregres- sive models the right way. In Advances in Neural Information Processing Systems, 2022.
[54) Hannes Stärk, Bowen Jing. Chenyu Wang, Gabriele Corso, Bonnie Berger, Regina Barzilay, and Tommi S. Jaakkola. Dirichlet flow matching with applications to DNA sequence design. In International Conference on Machine Learning. 2024.
[55] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputingg 568:127063, 2024.
[56) Zineng Tang, Ziyi Yang, Chenguang Zhu, Michael Zeng, and Mohit Bansal. Any-to-any generation via composable diffusion. In Advances in Neural Information Processing Systems, 2023.
[57] Yi Tay, Mostafa Dehghani, Vamsi Aribandi, Jai Prakash Gupta, Philip Pham, Zhen Qin, Dara Bahri, Da-Cheng Juan, and Donald Metzler. Omninet: Omnidirectional representations from transformers. In International Conference on Machine Learning, 2021.
[58] Dustin Tran, Keyon Vafa, Kumar Krishna Agrawal, Laurent Dinh, and Ben Poole. Discrete flows: Invertible generative models of discrete data. In Advances in Neural Information Processing Systems, 2019.
[59] Aäron van den Oord, Nal Kalchbrenner, Lasse Espeholt, Koray Kavukcuoglu, Oriol Vinyals, and Alex Graves. Conditional image generation with pixelenn decoders. In Advances in Neural Information Processing Systems, 2016.
[60] Aäron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In International Conference on Machine Learning, 2016.
[61] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Informa- tion Processing Systems, pages
2017.
[62] Alex Wang and Kyunghyun Cho. BERT has mouth, and it must speak: BERT as a markov random field language model. arXivv1902.04094 2019.
[63] Zachary M. Ziegler and Alexander M. Rush. Latent normalizing flows for discrete sequences. In International Conference on Machine Learning, 2019.
[64] Bernt Øksendal. Stochastic Differential Equations. Universitext. Springer Berlin Heidelberg, 2003.
14

---

Appendix
A Derivations
A.1 Preliminaries
Statistical Manifold of Categorical Distributions For a discrete sample space X = {1, d].
d-class categorical distribution over X' is parameterized by number of parameters pl, Pd
such tat Ei=1; Ps 1. The parameter space corresponds the (d 1) --ddmennsinnal probability
simplex:
(PI.....Pd Pa) E Bdd
1, 1p
Ad-1
Σ Pi 1p=11pp 20. 1=
(22)
A natural choice of a Riemannian metric on the simplex is the Fisher-Rao metric [1, 47]. For an interior point p E Ad-1, the Fisher-Rao metric is defined follows:
gFR(P) y) (z. u)
y
E.y.
a.y E TAAT11
(23)
p
where the normalization by vp in the inner product is performed component-wise This induces geodesic distance on the simplex defined as follows:
d(p, ④) = cos
Vith P.QEA--1
(24)
where p and q corresponds to the parameters of categorical distributions. The probability simplex dd11 equipped with the Fisher-Re Rao metric is a Riemannian manifold called the statistical manifold of categorical distribution, denoted as ((t) throughout the paper. The tangent space at an interior point p identified as (p(((x)) = { x E TRd Ei= ii =0 =0}- For further details on the geometry of the statistical manifold, we refer the reader to Ay et al. [4].
Hypersphere g denotes the (d- (d-11)dimensional sphere (u=(u),.... un) E, ut = 1] and s = (u1, ud) Σ, n² = 1, u, 2 0} denotes positive orthant of $ The hyper- sphere gd- can be embedded into the ambient Euclidean space Rd which induces a canonical inner product (ax, y) Di æ,y; For discrete sample space X' {1,2, ... ,d], there exists diffcomorphism from (( to $ defined as follows:
P(X) - 54
Ps u, vp:.
(25)
-| - P(X) u, + P: = u.
The diffcomorphism induces the the geodesic distance on 1d-1.
dg(u, v) cos (u, v), u. u u.
cos-"((u,
(26)
for which the geodesic corresponds to the great circle connecting two points and v. The corre- sponding exponential and logarithm maps on gd-1 can computed as follows:
exp, x =cos([>>) u + sin([1=(|))
u E S x (Sa-1))
(27)
|
cos (u, v)
expit(e) V1
(u, v)))
u. g
(28)
(u, v)²
15

---

dt Additionally, define the radial distance r"(z) = dz(x, E I where dg denotes the geodesic
distance on Sd-1 Then we have the following identities:
(v. x)
Vr (x) =
(29)
v1 (v, x))²
Ar"(x) = (d 100o))rr((())))
(30)
v(1 (u, w) (v. x) (wn, x)
(v, w) cos r (xx) cos r (a:
(x)) (r
(31)
(n, 2) (1 (w, x))
sin r (at) sin r (x)
In particular, the logarithm map in Eq. (28) can be represented in radial distance:
exp (v) -r (<<(<)>rr"(
(32)
A.2 Connection Between Discrete Diffusion Models and Continuous Flow
In this section, we derive the connection between the discrete diffusion models and the continuous flow on hypersphere.
Continuous Flow on Hypersphere We first derive a useful lemma for continuous flows on hyper- spheres. The following lemma describes continuous flow on the hypersphere as a spherical linear interpolation.
Lemma A.I. Define a flow {Y:)==0 on Sd-1 from yo E Sdd-1 to yı Sd (300-9003
dY;
dlog Ne expy, (y1), Yo = y/0,
(33)
dt
dt
where [0, + [0, 1] is a scalar function satisfying Kg = and KT = 0. Then the flow Y, has closed form solution:
Y, sin(Oo - 0.). sin Or
sin 0g yo. 02 = Ne cos" (yo, yn),
(34)
sin P
which corresponds the spherical linear interpolation, i.e., slerp:
Y. = expy, (Ks exp (36))
(35)
Proof.
Let 0; == cos (Y.. Then Y, can be written as follows:
Y cos 0,y1 sin 0,w,,
(36)
where w; E Rdd an unit vector. From the definition of 0r, we have the following identity:
d0: 1 dY;
1
d Ki Or(y1 Yi cos 0c)
2 sin 0 dt
91
(37)
dt sin 0,
di.
sin O2
d log Kit 1 cos2 0:
dl dlog Kit Oz.
(38)
sin Og dt which yields representation of the flow Y, in Eq. (33) with respect to 0:
sin O2
dY, dd, y1 cos f.
(39)
dt dt sin A2
Using the result of Eq. (39), we can see that w; is a constant vector independent of t:
dw:
1
dY,
dcos 8, y/ sin O, (Yi cos0:211
d sin Og
(40)
dt
sin" 0g
1 d0:
14))
Y,s 0z) + sin² 0.y1 cos Y, + cos² 0:311 = 0.
(41)
sin 0,
7p
16

---

Therefore we get the closed form solution for Yg:
Y; = cos 9: + sin 0 Yo cos 00y1 sin(00 - 0t)
sin 0z
y/1
sin 00 yo.
(42)
sin 0o
sin 0o
where 0, ni.0 from Eq. (38). Note that the solution Eq. (34) is well-defined in the sense that sin 0o > 0 always holds. This is because ,,,Y,, villl S 1 as Y, and yı are S Finally, using the definition of 0c, we can show the following:
expy, (Y) 0 Y, YT cos
K2 ky0o Kg0oeo NG expy, (Yo).
(43)
sin 0g
which gives the spherical linear interpolation defined in Eq. (35).
Our key observation is that the transition distribution (e la)) of a discrete diffusion process (Eq. (2)) a categorical. Therefore, modeling qe equivalent modeling the continuous flow on the statistical manifold P(X). Here, we show that discrete diffusion models over X' can be modeled by continuous flow on 84-1. Specifically, we derive that the transition distribution of discrete diffusion processes can be modeled by the continuous flow on the hypersphere.
Masked Diffusion Model We first show that discrete masked diffusion models correspond to continuous flow on the statistical manifold starting from an absorbing state.
Proposition A.2. Define flow (Vi) on gá-1 from ek to em:
dY, d dlog expy (em): Yo - Ek Kig
2 sin (Voo)
(44)
de
dt
where em denotes the absorbing state (i.e., mask state) and or [0. 1] is some differentiable
noise schedule satisfying og R and 0n O. Then the random variable Z, = x (Y₁) R satisfies the following:
Z, = og + (1 or) em:
(45)
which is flow that interpolates ex and e. the probability simplex A
Proof. Using Lemma A.I with 0, cos (em ex) 1/22 have the representation of Y;:
Y, = sin - 0m + sin Ogex = v1 - agem + Varet
(46)
since 02 sin 4 = (a.)). Therefore, Z, has the following closed form:
Z. em)mm aykk.
(47)
which defines a flow that interpolates e₁ and em on the probability simplex Ad-1
Note that Z, is a random variable on d-1 representing the categorical distribution Ca (1-a, e.)) This corresponds to the transition distribution g( (ro) of a discrete masked diffusion model, where the transition matrix for the diffusion process is given as follows:
at 0
0
Qa ahsaab
(48)
Oie
1
1
O
Corollary A.3. The discrete masked diffusion process can be modeled by a continuous flow on Sd-1 that starts from the absorbing state C..-
17

---

Uniform Diffusion Model We also show that discrete uniform diffusion models correspond to continuous flow on the statistical manifold that starts from the barycenter of the simplex.
Proposition A.4. Define flow (Y)) g from ex to Σ e
dY;
dlog Ki expy,
d
di di
Vd
E;
Yo - =kk
(49)
sin" (vI- sin 0o) Kg 1
1
0o cos"l
(50)
80
vd
where a E [0. 1] a differentiable noise schedule satisfying ag and a1 Then the random variable Z, (Y,) E IRd satisfies the following:
+ 1 (d - 1)
Z
d e,
Ck:
(51)
d
which is a flow that interpolates e₁ and En= e Vd on the probability simplex A dd-1
ed/Vd
Proof. Using Lemma A.I with 0o cos" cos"2(1///0 (L/vd), have the following representation Yg:
Y, sin 0s)
sin Hu
Σ
Vd
1
C;
sin 0g ek
sin Og
(52)
1=1
sin(Ou 01)
Vddsin 02
sin 0z) vd -
+
ek-
(53)
Vd-1
Vd-1
Due to the definition of Kg, Zi has the following closed form:
11-0 Ot. e; + 1 (d - 1) ek:
Z, =
(54)
d
d
which defines flow that interpolates and E ei/Vd, i.e., the barycenter of the probability simplex A d-1
Note that Z, a random variable on A d-1 representing the categorical distribution:
at e, + 1 (d 1)a
Cat
Σ 1710
Ewo
(55)
d
which corresponds to the transition distribution ( (zo) of a discrete uniform diffusion model. The transition matrix for the uniform diffusion process is given as follows:
N
1
1
1-N
1
Qunis
(56)
1-N
Corollary A.5. The discrete uniform diffusion process can be modeled by a continuous flow on Sd-1 that starts from the barycenter of the probability simplex.
18

---

A.3
Generative Process on Hypersphere
On a general manifold M that is complete, orientable, connected, and boundaryless, the logarithm bridge process [33] from ano M æ1 M is defined follows:
of
dx, 71 exp )et + o,dB, Xo zo N
(57)
Se of
where exp2 (-) denotes the logarithm map on M point and B,M the Brownian motion defined on M. In the case of M gd-1, derive the logarithm bridge process from Zy ex:
cos" (K,, ex)(ex - (X,, dt + o, Xo 2o. V1 1-,((X,, ex)²
dX; Y
(58)
where we used the logarithm map of Eq. (28) and B is a Brownian motion defined on gd-1, is worth noting that Eq. (58) derived from applying the time change [64] simple bridge process:
1 cos't ex)lex - (x,, X,) dt aB X 20-
dx,
(59)
T-t
- (X,,
Note that the drift of the logarithm bridge process can be rewritten using the geodesic distance dy (::-) as follows:
dX, = cos" (X,, (X,, Vx) dX,,, ex) dt o Xo
(60)
The direction the drift corresponds to the direction that minimizes the distance between the current state X, and the endpoint eg. Since "t - as / - T, the bridge process converges the endpoint ex- The convergence behavior can be analyzed by examining the radial process r = (ex, Xt),
which we describe below.
Radial Process Let r == (w, X₂) for arbitrary point w E Sd-1 Then the bridge process of Eq. (58) can be rewritten follows:
(et - x
dx, %
dt dB Xo Zo.
(61)
sin
where r re. Then the SDE of rr can be derived using the Itô's formula follows:
= ( dr -
(f co X.)
a? Ar. dt + (vr ad +
(62)
sinr
2
Beltrami operator on Sd-1, re- spectively. From the identities in Appendix A.1 and the fact that (Vr." dB,)) is a I-dimensional Brownian motion ([32] Example 3.3.3), we get the following result:
where and A denote the Riemannian gradient and the Laplace-Beltrami
dr = [~x r (Ck. w) cos r cos r
(d (d-1)a} cot(r?") dt + a,WW,
+ 2
sin r sin r
(63)
0o = cos"t (xo. w)
where W, a 1-dimensional Brownian motion. For w et, we obtain simplified formulation:
dr! = - + (d (d-1)07
con(rr) dt + a.dWe r 2 5k
(64)
2
ifk=
r(rtirt) - cot(rf)cot(rr)
(65)
otherwise
19

---

A.4 Diffusion Mixture Representation
We provide the statement of the diffusion mixture representation from Jo and Hwang [33], which extends Peluchetti [45] Riemannian manifolds. We refer the readers Jo and Hwang [33] for detailed derivation of the diffusion mixture representation for general Riemannian manifolds. We consider Riemannian manifolds that are complete, orientable, connected, and boundaryless.
Proposition A.6. Consider collection of SDEs on a manifold . indexed by E A:
dx, = ( t)dt + o (x; 8) dB,,,
X ~ Po
(66)
with marginal distribution of X3 denoted by p². Let C be mixing distribution over A. Then a diffusion process on M described by the SDE:
dx, = (x,, ()dt o (X,,)) dBM, Xo ~ po
(67)
7(2 = (x, t) (x) ((dx), o(z,t) pz(x))
(foles pa(o) Po(x)
1/2
c(di)
(68)
where a = (o admits the marginal distribution pc:
pr(x) / ((()) po(x) = f (Dd())((a))
(69)
From the diffusion mixture representation, Jo and Hwang [33] construct the generative process mixture of the bridge processes on M as shown in the following proposition.
Proposition A.7. Let Po and be probability distributions Riemannian manifold M. Consider collection of SDEs that describes bridge processes on M from Po to y P1:
dx. F.V = (X, t)dt + o,dB.M Xo= =x,
(70)
with marginal distribution of x denoted by p. Then the following SDE defines a diffusion process that transports an initial distribution po target distribution pr:
dX; (X,,,, t)dt + o,B.M. Xo= ~ po.
(71)
1/ 1 (2, Pt (z) (z(2)
n(2,t)=
po po(dvol=)p (dvoly),
(72)
pr(z) /f (dv
(73)
In the case of M = gd-1 we derive the generative process for the reparameterized data distribution
Pdata (x) = - P15(x -x)x by mixing the logarithm bridge processes on @d-1 (Fq. (5)).
Corollary A.8. Let Pdota (x) = =ie
k= pto(x-es) be data distribution on Sd- Then the
following SDE defines diffusion process that transports the initial point Zo gd-1 to the distribution Pdata:
d Σ
dx, =
Priles x,) of (X,, 1) di + a.dBo Xo = Eo.
(74)
n (2, t) = cos"l (z, ex) - (2))))2)
(75)
V1 (2
where Pric(es X.) represents the conditional probability that the process will reach the endpoint 6₁ at time T. given the current state X, at time t.
20

---

A.5
Mixture Paths
We derive a new family of generative processes by constructing a mixture over the time marginals of generative processes. We first present proposition for mixing diffusion processes with general time-dependent mixing schedule.
Proposition A.9. Consider collection of n SDEs on closed Riemannian manifold M: dx; - 1/ (x; l)dt + (X 1) dB,,. X ~
(76)
with marginal distribution of X denoted by pí. Let X ((O, TT) satisfy X 2 0 and Ex=1 X3 = for I. Then there exists a diffusion process with the marginal distribution pt:
n
pr(x)=
Mpt(z)).
(77)
Proof. We show that there exists scalar potential 4 MM [0, 1j + R such that the following SDE defines a diffusion process that yields the desired marginal distribution:
dx, (x,,,, t)dt (Xx,t)dB;"
(78)
(1x,1) Σ (x, (x())
()(x1)
pf(x)
Σ Xaa (x,
(79)
Pe(x))
(((
(r((z)
1/2
txx Maa (x,, p((c))
(80)
(z())
where a² T Here, we assume that of and o² are bounded and at are uniformly elliptic. First, define a function f : + R that satisfies the zero-mean condition:
dyp [(x, = Σ dt phla): S.v f)dvol, Σ S pi(a))voll, 1=1 de
7
dxt
1
dx
Σ
(81)
dt
where we used the fact that Σ Mj for all As M closed, its Laplace-Beltrane operator is invertible on the subspace of zero-mean functions. Therefore, the Poisson equation ( = f(,t) admits weak solution 4.
From the definition of pt. we can derive the following equality:
@p((x)
((N))(())
Opf(x)
n
dx
Σ (x(
(82)
0t
Ot
Ot
dt
= Σ
N
div (zx()(()))(zz 1)) div (al(zz + (2)
(83)
1=1
div
(x,)
1 Σ +
Xddiv (a((2, + div(Ve(zz (84)
f=1
div
(gp(())) (x.1) (
(85)
Σ div (<(,)) (Vl(t) Nap((a) 1=1
(f((x)
+
pz(
(x(x))
where we used the product rule for divergence in (xp( (e()r) Napi(a
p.(
21

---

Reordering the terms in Eq. (85), we obtain the following result:
Öp((x)
()()))
V (
pi(() (r(
)
div P((x))
Σ
Xa (x,, 1/
Øt
(x)
px(),)
2
i=1
pi( (X,) Pr (x(
+ 2 div
xx, t)
Vp((x))
(86)
which corresponds to the Fokker-Plance equation for the SDE of Eq. (78). Therefore, the diffusion process described by the SDE in Eq. (78) has a marginal distribution in Eq. (77).
From Proposition A.9, derive new family of generative processes by constructing a mixture over the time marginals of generative processes {Q' : iisissn
Qamir
X
Σx Mx=1,
(87)
where X₁ is the time-dependent mixing schedule assigned the i-the generative path.
One example is creating a mixture path by mixing the masked diffusion and the uniform diffusion on gd-1, as defined in Section 3.2.
Corollary A.10. Let pl and Pt ! denote the marginal distributions of the masked diffusion and the uniform diffusion on Sd-1, defined in Section 3.2, respectively. Then there exists diffusion process on gd-1 whose marginal distribution at time satisfies:
(r) Appmask(aa (x) + (1 >e)p;
unif (æ),
(88)
where A E [0, 1] for all E [0.
A.6 Likelihood Bound
We derive the point-wise likelihood bound and the upper bound on the negative log-likelihood of our generative model, defined as the parameterized mixture process Qº with the drift 1Jo Eq. (11).
Let @ be a bridge process with starting point zo and endpoint eg- From the KL divergence between Q and @+ we derive point-wise upper bound on the negative log- likelihoood using the Girsanov theorem compact manifolds (De Bortoli et al. [16], Corollary H.3):
(o(e) = (6re)))) = D (Q) (0%)
(89)
2 . //-
s = Exxce
0r (To( n7* (X,, t))
dt (90)
where the inequality comes from the data- pioteerrooessngggg inequality. The woint-wise likelihood bound leads to the upper bound on the negative likelihood our model:
1 o, -1
2 (no(Xe, (X,,
Egepacta [- log Po (=)] <
Beropaara X-Q
dl
(91)
2
A.7 Training Objective
We show that minimizing the cross-entropy-bee based loss defined in Eq. (15) guarantees maximizing the likelihood of our generative model defined the parameterized mixture process in Eq. (11).
We start with deriving uniform bound for the drift of the bridge process defined in Eq. (5):
cos" (2, e:) - (z. e1)2)
= cos" (2, Th.
(92)
V1
(2, e2)
22

---

Then the triangle inequality gives the following:
2 ded 1| (a, t)|l2 - Σ (2, (p, t),
)
(po(2, 1), t), (x,
(93)
2
S n 72
Σ (po(zz,)) t), e1) 80n1)
1 22 log p)(
(94)
From Eq. (94), we derive the upper bound for the maximum likelihood training objective [()) in Eq. (14) as follows:
((0) -Be--Paas
of
())) (Xx,t) df
(95)
X X-Q
2
J
2 2r²
Bea-aaata X-4
log (po (Xe, t), ex
(96)
oi
2n²
T-c
Beaapaaaa Xx3
sup [E
- (po( 1), ex)dt
of
(97)
2
log (p:(Xx;))), ex ]
+ Rexeraata
of
X~2
$ BBC (0)+ F(c),
(98)
where F(c) denotes the last term of Eq. (97). Since X x~0 is the bridge process with endpoint ex- X, converges to e₁ t T and (po (XT <), ex. R 1 for sufficiently small >> 0. As result, F(c) z for sufficiently small c, which lead to the following result:
((0) S MC
(99)
for some constant M > O. Therefore, minimizing the cross-entropy-basse based loss CCE approxi- mately guarantees maximizing the likelihood.
A.8 Projected Processes
Let X denote the mixture process (X I7 on Sd-1 conditioned to the endpoints Xo = 2o and
XT 21. Then X,jo,r corresponds to bridge process described by the following SDE:
dX, = cos"! (1)(1))()) (X,, x1)x) dt o, Xo 2z.
(100)
V1 x1,
We can derive the projection 2T = (XioOTT 21) using the Itô's formula for fr(.) = (,,21):
( v | (X1), cos" (X,, xi (0) - x1) o dl
dz7 =
v1 x1,
(101)
+ to, V fT(i dB.)
( x - (81,01)) X,, 7t
cos" TT
(d - 1)o I
=
11 - (1,,,,)),)) (zf
di
2
(102)
+ +orv (=f))'dW,
cos' 2/" v/1- 1- -(=T)² (d (d- - 1)02
21
dt + o
1 1- l..
(103)
23

---

where we have used the identities that the Japlace-Beltraml operator defined on Sd-1 has a simple and tractable form due to the radial symmetry of the hypersphere.
[=r
(fr(z) - - 1) æ1). Note
Similarly, zf = (Xe, vo) can derived using Ito's formula for fo(z) = (z, xo):
cos 1 of
1)07 (zzg x1) - 2/ 2/ (d
de. = 7
dt + σiV (2)) 20
(104)
(2
2
Masked Diffusion Since the masked bridge process has 2t = em and æ1 ex with (em, ex)
for all non-mask token Cx, the projected processes are described as the follows:
dæf
cos" 2/
61, efelt
1)o2 24 (d 1d-11)m
dt + or ( (2))
(105)
V1
(2
2
with initial condition =f 0 for all l and w are I-dimensional standard Wiener processes.
Uniform Diffusion The uniform bridge process has 2o = Ei= ei/Vd and x = exs and the projected processes have simple form:
cos" 2 =t
(d - 1)
dzf-
7te
AI.x =(-tt
I #8
dl + 0z / - (=f)aw%
VI
(2t)²
(106)
1/vd ifl
At,
otherwise
with initial condition ofj 1/Vd all
A.9 Simulation-Free Training with Radial Symmetry
Here we derive the parameters of the Riemannian normal distribution from the projected processes:
dæp =
Y cos" 2 VI -
(d- (d-1)o?
21
T
dt + -tt
(107)
2
dzf =
cos'l 27
(T -2(z]]
(d- (d-1)o?
0 dt + (59)
(108)
(zt))
2
with initial conditions = (xn, X and = 1. From the definition =/- (X 1o, mn), establish the connection between the mean projection Ezπ and the parameters a₁ and Pt:
E2T =s Ez( cxpp, (p:z), 21), 2 Nr,, (0,I)
(109)
E. [cos(pe|==1)mm + sin( ||] x1
(110)
= =E, cos(pel|=]|) (1) + Ex sin(pt]1z1))
2 | 22
(111)
Eq (17)
Eg (os(p=))=)])
_t x1 (V1 (v1-af
as
Or on sin do
Fg. Φ1
(112)
sin Φu
Eg cos(pille ||=
sin duar + cos ou at
(113)
for := Xy)) where the last term in Eq. (111) is zero due to radial symmetry. Similarly,
Ezi (expx (piz), xo) Eg cos(pill=1)) ag
(114)
24

---

Notably, we have the following identity for 2 ~ NN (0, I):
Eg cos(pe||=1|) = e -
d
p
Falpi):
(115)
2'2"
2
where ifi denotes the Kummer function, also known as the confluent hypergeometric function. Therefore, the parameters 04 and Pr. can be derived from the mean projections Exy and lEz;
(Eef /Eaf cos Φ0)²
0t
sin do + (E2 /Eaf - cos on) Pt - P7 B2
/B///V aî
(116)
A.10 Comparison with Prior Work
Comparison with Discrete Diffusion Models Discrete diffusion models [2, 39, 49, 52] do not fully leverage the power of iterative refinement, which is the key to generative modeling of continuous data, for example, image synthesis [19, 48] and video generation [5, 46]. In discrete diffusion models, the progressive corruption during the forward process is modeled by stochastic jumps between states in Markov chains. Since denoising is achieved by jumping between states, discrete diffusion loses valuable signals during refinement, which limits the generative performance and controllability. In contrast, our RDLM takes a continuous approach using the geometry of the statistical manifold and the hypersphere, and therefore avoids the signal loss that occurs during state transitions in discrete diffusion models, fully leveraging iterative refinement.
Advantage of Continuous Approach Due to fully leveraging the iterative refinement, RDLM can generate higher-quality samples, outperforming discrete diffusion models across diverse domains. Furthermore, our continuous approach offers additional advantages: (1) Controllable generation: Using a continuous diffusion model enables direct application of guidance, e.g., classifier [17] and classifier-free guidance [26]. (2) Optimized design choices: Benefit from advancements in continuous diffusion, e.g., optimized noise schedule [9, 31, 34] and self-conditioning [10]. (3) Efficient sampling: Our framework supports efficient and scalable sampling strategies such DPM-Solver [40, 41J. In contrast, discrete diffusion models are restricted to using a simple ancestral sampling strategy.
Comparison with Flow Matching Our method outperforms previous works using flow match- ing [12, 15] due to three key contributions: (1) generalization of discrete diffusion, (2) parameteriza- tion and training objectives, and (3) scalability higher dimensions.
First, our method generalizes discrete diffusion models, the current state-of-the-art in language modeling, and introduces novel mixture path process that enhances performance. In contrast, prior works using flow matching [12, 15] lack a direct connection to discrete diffusion models, resulting in suboptimal design that leads to inferior performance. Notably, flow matching-bast approaches a special case of our method, as shown in Section 3.
Second, we introduce novel parameterization (Eq. (10)) and soss--etroop-basedd training loss (Eq. (15)), similar to the loss used in discrete diffusion models. This loss optimizes the likelihood during training, and when combined with our importance sampling loss (Eq. (16), achieves superior performance. In comparison, Cheng et al. [12] uses a simple flow matching loss that does not guarantee maximum likelihood optimization.
Lastly, prior works are restricted small vocabularies due the difficulty of learning generative process on high- ligh-dimensiona manifolds (i.e., large vocabulary). This issue arises from the rapid convergence problems and insufficient model capacity, as discussed in Section 4. We address these challenges with dimension splitting, which significantly improves performance and enables effective scaling large vocabularies.
B Experimental Details
B.1 Training and Sampling
provide the pseudocode our training and sampling schemes in Algorithm and Algorithm 2, respectively. We additionally provide pseudocode for pre-computing the parameters for the Rieman-
25

---

Algorithm 3 Pre-computing parameters of Riemannian normal before training Input: Initial point u, vocabulary size d, number of simulations N, number of discretization steps K, noise schedule ds, time change coefficient "
1: 0 and öt 1/K
2: 4/ f (u, e1}
Radial symmetry
3: ag + 0 and Po f- 0
4: a - (1b))* and b + (1)
>> Initialize N independent trajectories
5: for k 1 to K do
6: Wa, W, (1) N
7: a f ok/K and 7 c 7k/K
8: a - a + (70 a vI a² (d-1)o² 0 ot + OVI V8W
> Eq. (18)
9:
b- b+ (1 cox"
(y-er" vi-a² (wo - abb) (d-1)0² b) 81 + +oo 17 V5WW
> Eq. (19)
10: r MEAN(a) MEAN(b)
> Ratio mean projections
00)
11: OK/K + 1- 1-1g+(r vu)
> Eq. (20)
12: PK/K t Fd (b/ V k
> Eq. (20)
13:
end for
14: Return: {au/K: Pt)
nian normal a4 and in Algorithm 3. Note that pre-computing takes only once before training our model, and the computation time is negligible compared to the training time.
Likelihood Computation For computing the upper bound for NLL, we use the Monte Carlo estimation of the negative ELBO derived in Eq. (13). Note that we use simulated X,, instead of approximation from the Riemannian normal, for accurate computation.
Computing resources
For all experiments, we use NVIDIA RTX A5000 and H100.
B.2 Text Generation
Baselines We compare against state-of-the-art diffusion models. Multinomial Diffusion [29], D3PM [2], SEDD [39], MDLM [49], MD4 [52] are discrete diffusion models. Plaid [23] and Bayesian Flow Network (BFN) [21] are continuous diffusion models. We do not existing works for flow matching on the statistical manifold [12, 15] as they do not provide likelihood computation applicable for language modeling.
We also use the transformer AR model [61] and the following autoregressive models as baselines: IAF/SCF [63], AR Argmax Flow [29], and Discrete Flow [58] are flow-based models, and ARDM [30] and MAC [53] are any-order autoregressive models.
Text8 Text8 [42] is small character-level text modeling benchmark extracted from English Wikipedia. Following the previous works [2, 39, 49], we split the dataset into 90M/5M/SM with a fixed sequence length of 256. We use vocabulary size of 28, comprising 26 lowercase letters, white space token, and a mask token. We use a 12-layer diffusion transformer [44] following Lou al. [39] with 92.4N 4M trainable parameters. We train our model for 1M iterations with batch size 512 as done in previous works, using the same learning rate, optimizer AdamW [38], and exponential moving average (EMA) with decay rate 0.9999.
One Billion Words One Billion Word Benchmark is dataset extracted from the WMT 2011 News Crawl dataset comprised of single sentences from news articles. Following Sahoo et al. [49], we use the pert-baas-unccasee tokenizer and pad and truncate the sequences to length 128. We use a 12-layer diffusion transformer [44] with the hidden dimension of 768 and 12 attention heads, following Sahoo et al. [49] with 110M trainable parameters. We train our model for IM iterations
26

---

Table 6: Analysis of the dimension splitting (Sec-
Table 5: Comparison between the training objec-
tion 4). We compare NLL on LMIB test set. Top-
tives. We compare Bits Per Character (BPC) the Text8 test set.
K Feat. denotes adding additional features of
top-k indices of the input state.
Method
BPC ())
Method
NLL (1)
Drift MSE (Eq. (14))
1.36
Cross Entropy (Eq. (15)
1.34
wo dimension splitting
S 11996.9
Cross Entropy Importance Sampling
1.32
w/o dimension splitting Top-K Feat.
661.1
w/ dimension splitting.
S 428.5
with batch size 512 as done in previous works, using the same constant learning rate, optimizer AdamW [38], and exponential moving average (EMA) with decay rate 0.9999.
Comparison with MDLM Here we provide a detailed comparison with MDLM [49] on the language modeling task using the One Billion Words dataset.
First, we did not search for optimal training hyperparameters' (e.g., learning rate). Instead, we directly adopted the hyperparameters used by MDLM ensure fair comparison. However, because RDLM employs a continuous approach, it might benefit from different hyperparameter choices than discrete diffusion models. Due resource limitations, we could not explore these optimized settings.
Furthermore, MDLM was trained using the low-discrepancy sampler, which is crucial for reducing the variance of the ELBO during training, leading to better perplexity results. We did not use the low-discrepancy sampler during training, yet RDLM still achieved competitive results on the LMIB dataset.
Additionally, the reported RDLM and MDLM results are based on training up to million iterations, at which point RDLM had not yet fully converged. Extrapolating RDLM's validation perplexity through curve fitting shows that RDLM surpasses MDLLM after 10 million iterations. Due to resource limitations, we were unable to train beyond 1 million iterations.
B.3 Pixel-level Image Modeling
Baselines We compare against autoregressive models and diffusion models that directly model raw pixel space. PixeIRNN [60], Gated PixeICNN [59], PixeICNN+ [50], PixelSNAIL [11], Image Transformer [43], and Sparse Transformer [13] are autoregressive models. D3PM [2), rLDR [6], and MD4 [52] are discrete diffusion models.
Implementation Details We represent each image set of discrete tokens with vocabulary size of 256. We use the 10-layer diffusion transformer [44] for our model with 35M trainable parameters. We train 100k iterations with batch size 128 and AdamW [38] optimizer following Shi et al. [52].
B.4 DNA Sequence Design
The dataset contains 100k promoter DNA sequences, each paired with transcription signal pro- file. Each sequence consists of 1024 base pairs centered at the annotated transcription start site position [28], and the base pair has 4 categories (ATGC) conditioned the profile.
We compare our model against diffusion models and language models. Bit Diffusion [10] continuous diffusion model, D3PM [2] discrete diffusion model, DDSM [3] and Dirichlet Flow Matching [54] are diffusion model and flow matching model using the probability simplex, respectively, Fisher-Flow [15] is flow matching model using statistical manifold.
Baselines
Implementation Details Following the previous work [15, 54], we use the same data split of and identical model architecture consisting 20-layer 1-D CNN with 13.3M trainable parameters. We train our model for 100k iterations with batch size 256 and AdamW [38] optimizer. We evaluate the MSE on the generated samples conditioned on the prescription signals from the test set, using 300 generation steps following the previous work [15].
27

---

10:1
Approx:
Approx:
Approx.
cir Stmul 1000 stops
Simul 10.00 stops
Simul 1000 stops
Simul staps
Simul 502 steps
Simul. 500 steps
10
Simul
10
24-
10-4
Difluction Time
Diftusion
Time
Dimension
(o) Dimension
(4) Dimension 30522
Figure 2: Maximum mean discrepancy (MMD) distance between the simulated distribution Xo, Xr) and the approximated distribution. We report the results for dimensions 4, 256, and 30522.
C Generated Samples
C.1 Text8
We provide uncurated text samples generated by RDLM trained on the Text8 dataset.
o zero one british single payrock neurologically related condition is member of the original playboys oriental pbkr cat ii a boob one card featured the late one zero dippie dons as became pigus the monoseur engine shair which became th
h delivered from the new meeting the construction of modern shooting begins kinington resurrects the hark or corped a hopper nightlife subjecting to turn his attention at a joyable moment he able to explain that he is in recovery with a new orleans baby
wilder unrefreshed bup of lightmarks sinar joseph avaret in the cetleben key in one nine nine seven this report has been portrayed urban rumour as that he was ana eichy
pertified only at the head of
shrinking feathor of the civil directs against
s seven two chromosomes regainally regular and contain number of mignain gnaning pros zopods or cells whose podic configuration divided agong the faces of dna generally replaced by b as therus group are non mit and elanisten special cayits regularly are ca
nine four although portrayals of frel appearance the novel include leaked to bratally targeted audiences largely by steve roper dart mer upick and j pernan durk born one nine four zero s but stillly not they are created the western master and mag both
idment indicates two different types drop tales have different charges which train structures having
and light weight variations have lower weight impedients such as chawings starges and groove gloves shorter holes can be jumpliten don badld a horse
d deliberately rejected this a different post however saw al sh ibn misha rody revealed to be the lord curses of jesus one nine one nine he handled his journey to its historical map of the egyptians and was still nodged as he committed to reproete he
ovincial governors regelrant a cursami governor granted to a spanish cominic in one seven eight three mateo s teltacheutes lebmo alexius jeano and pan dosien dostre of ruguen cosst originating specifically the treaty of st louis the extinctions remain
28

---

C.2 One Billion Words
We provide uncurated text samples generated by our RDLM trained on the LMIB dataset.
[CLS] social recklessly the obvious support 2013. [CLS] they were elected off by the english authorities, whose party subsequently named as principal when lawrence tang had to hold the property until they were turned to down their heads in the back sky of which sank from matthews's a doorstep. [CLS] it has been pouring gladly with work and along the motorway, where certified sales will follow a new bone in the next several days to avoid commercial production problems, according to recommendations from both workplace and tropical mod. [CLS] he said he plans watchsty will greens the old draft plunging sara, but have medics announced she would make you the taxpayer? [CLS] duchess [CLS]
[CLS] of lieberman. [CLS] analysts say since 5, 000 people have held established council in 120 forums and levels, some have returned to the villages of the british capital, mideast and sprint. [CLS] his friends ring between ironing his body they forbid forrest. [CLS] seven babies missing and 27 french subcontinent and two development employees suffered injuries
securing of greece, a spokeswoman said immediately, while tneye wedang. [CLS] both questions has already been considered. [CLS] jackie has an hopeful major interest for dirty potter, pilots bullock's show, whether they have what hugh and mariusa other, [CLS] [CLS] is the problem that worth most of marriage to have single car he doesn't need. [CLS] mr obama will carry out more casualties however than president obama's followers, and it mild to form the first cumulative current division ofers holding the guantanamo men that arches to injustice.
shame roots
[CLS] phillips said : designer kaia kangaroo, 27, and herself rubbed jim reyes, the general patron of france light, have organized a building aimed at gunning film houses. [CLS] riding, london graduate college in edinburgh and a temporary exhibit mall in fasside, marked since the work are new sport, smaller schools racing has more [CLS]
[CLS]aceon)) that spain had submitted one time the main website on mass wireless, in carpcsllo. [CLS] not two of the beer bk known in the companies could have thousand stretch men ginger, and showed vulnerable cases, leaving you in the same f200m standard. [CLS] yet apius is accepted months since
quickly to associate they agreed that it getting waste into ulysses air before creation known as the bulletinsburg, which can be bowed with bracelet growth by speed. [CLS] rely will get another less energetic first turn victory. [CLS] more than 2, 000 people arrived, out [CLS]
bulletin energy americas
[CLS] more steadily increasing transit facilities with murray's tax breaks. [CLS] nonero moee enjoyed terrestrial wallino with the immoitunghrck most years. [CLS] those who run on hard sling are good with childhood often or later short - term temperatures. [CLS] top seeded henin is [CLS] downing richard finally happy huckabee, who didn't say in new hampshire and arkansas four years ago, vaclav with worldwide gains. [CLS] even if the huckabee god had the black annesies chosen to go on his way to combat [CLS]
shark seventh and isatin out in stanford.
[CLS] high school, was potya's poker high george she former congressional class flicked was a prosecutor. [CLS] coln has won the services of the sub area tustiw university, near fort dodge, pa. [CLS] one is the daughter of a metro with a problem but a tough neighborhood, retirement campus which, that day, was published by hyde for the little class united states attorney. [CLS] let's sell floral parachute civil court on lutheran case. [CLS] the virginia government says the ad, which
29

---

will add its new poll kind wednesday, had 10, drastically supervisors and 25 people. [CLS] [CLS]
[CLS] a memorandum posted to the university model google, which makes the copies to sell patients ep you programmes similar. [CLS] almost no day cbees will homemadei.
off sigmificant stake in every final
[CLS] many in the raf had sincerity at her twins guilty of battling apology from the bishops. " [CLS] the courts have replayled their option felomee when the fed tends its view of the aec investors chance. [CLS] that veteran, who claimed aredell mol for the milestone but on wednesday with their hay jade bridge, was doing the champagne board without everyone quarter a mips visit overnight. [CLS]
[CLS] the bbc's george washington is the first of 15, 000 people to put the calraircer range. [CLS] the uk's arp drilled a fence in the construction of eu hospitals on the trunk network one africa's most damaging places. [CLS] all looked after world over just um occasionallytau, which takes place victorious for schizophrenia consumed near the doc centre. [CLS] it is complicated by profits, not the greek pilot anchors, some of whom the very top cruise lay in the deep west of britain, which threatens developing dozens, and joined a conference in america to provide a full grand theft pad to [CLS]
D Limitations and Broader Impacts
Limitations While our approach has shown promising results language modeling tasks and other modalities, a performance gap remains in some tasks compared to autoregressive models. We hypothesize that this is because autoregressive models utilize model capacity more efficiently, as they learn from single, fixed ordering of tokens. One interesting direction for future work is to design position-dependen noise scheduler that converges sequentially from left to right, mimicking the autoregressive generation process. In addition, although the current framework can generate sequences up predefined maximum length, it is not capable of producing sequences beyond this limit. This limitation could potentially be addressed by incorporating aeni-autoregressive approach that generates text in block-wise fashion.
Broader Impacts Our work may provide future directions for multimodal generative models that capable of generating data from multiple domains, for example, text, images, and videos, simultaneously. Furthermore, our continuous approach may allow better controllability and improved quality with advanced sampling strategies. However, there is risk that someone could misuse our framework to produce harmful content.
30

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Abstract and introduction summarize the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations are discussed in Appendix D due to page limit.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
31

---

Justification:
We provide the derivations and proof for the theoretical results in Appendix A.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide pseudocode for training and sampling, and explain experimental details in the main paper and the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
32

---

Answer: [Yes]
Justification: We submitted codes for our work as supplemental material, and will open- source the code in the future.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. ec/
colloolioo olicy) for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //nips. cc/ mublic/guidess Caleee/cooSSubbieec bebbbmisssiins ssisnsoliccyyyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide the experimental details in Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[No]
Justification:
We follow the standard setting for the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
33

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide information the computer resources in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https:: ttttp:://neurip... wttpccccppppcceeeeeeeeeeeeececee httpps//eaarrrp..cc/pll////////
ctwiettthhhhhueeeeleeeeeeeaeeeee csGuidel ines?
Answer: [Yes]
Justification: Our work conforms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss societal impacts in the Appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
34

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work does not pose such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite the relevant works.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
35

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Released codes will be well-documented.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
36

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy (https /ttp:/::///e.p../eeeee /c/////oeeeeeeccc////////11 for what should or should not be described.
37


Reviews:
[{"review_id": "wZzxtq9h7b", "reviewer": "Reviewer_GUHT", "summary": "This paper proposes a continuous diffusion model for language modeling by projecting the categorical distribution to the positive orthant of a sphere, which is diffeomorphic to the manifold of distributions. This enables the benefit of continuous diffusion, such as efficient sampling. To make it practical for language modeling, several tricks are introduced, such as dimension splitting. Experiments on text and image demonstrate the potential of the proposed method.", "questions": "- Line 135: It says \"providing numerous opportunities to correct wrong predictions during the process.\" I am unsure about this claim, because being continuous can also lead to error accumulation, while discrete transitions hold the possibility of entirely denoising the accumulated error.\n- Eq. (10): The left-hand side should be $p\\_\\\\theta(X\\_T)$? Since this is a distribution over $X\\_T$, it would be better to use a clearer notation for this.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The paper tackles continuous diffusion language models, which is an important topic in current AI research.\n\nWeaknesses:\n- While the proposed method is elegant and neat, it is unclear whether there is real benefit for large-scale language modeling. The experiments on language modeling remain relatively small-scale.\n- The concern about its effectiveness in large-scale settings is that, if we consider the ambient space $\\\\mathbb{R}\\^d$ in which the statistical manifold $\\\\mathcal{P}(\\\\mathcal{X})$ and the sphere $\\\\mathbb{S}\\^{d-1}$ are embedded, the gap between these two diffeomorphic manifolds becomes larger as $d$ increases. Intuitively, this makes transitions from some point to a token (i.e., $\\\\mathbf{e}\\_k$) much farther on the sphere than on the simplex. I am not sure if this is a critical reason why continuous diffusion models struggle to match the performance of discrete diffusion language models.", "was_revised": true, "final_justification": "The authors addressed some of my concerns, therefore, I intend to maintain my original positive rating.", "rebuttal": "We sincerely thank you for your constructive comments. We appreciate your positive comments that our paper tackles an important topic in current AI research, and the proposed method is elegant and neat. We initially address your comments below:\n\n---\n>**C1:** While the proposed method is elegant and neat, it is unclear whether there is real benefit for large-scale language modeling. \n\n**R1:** We believe our method offers benefits for large-scale language modeling, both in terms of performance and the inherent advantages of a continuous diffusion framework. \n\nFirst, superior results on the LM1B dataset, which has a sufficiently large vocabulary set of size 30522, demonstrate strong potential for scaling to large-scale language tasks. Importantly, RDLM can scale to even higher dimensions due to its simulation-free training scheme (Section 4) and dimension splitting technique (Section 5). \n\nFurthermore, our continuous diffusion approach brings additional practical advantages that have been successfully leveraged in other continuous diffusion models. These include controllable generation and compatibility with advanced techniques such as self-conditioning. It also supports scalable and efficient sampling strategies like DPM-Solver, which can further improve inference speed and quality.\n\n---\n>**C2:** If we consider the ambient space $\\mathbf{R}^d$ in which the statistical manifold $P(\\mathcal{X})$ and the sphere $\\mathbb{S}^{d-1}$ are embedded, the gap between these two diffeomorphic manifolds becomes larger as $d$ increases. Intuitively, this makes transitions from some point to a token (i.e., $e_k$) much farther on the sphere than on the simplex. \n\n**R2:** We would like to clarify that the diffeomorphism between the statistical manifold and the sphere preserves their intrinsic geometric structure, and the notion of gap or distance between points on these manifolds is preserved under this mapping. In particular, the transitions from a point on the statistical manifold to a token $e_k$ on the simplex correspond directly to transitions on the sphere via the diffeomorphism, and the distances do not become inherently larger on the sphere compared to the simplex as $d$ increases.\n\n---\n>**C3:** Line 135: It says \"providing numerous opportunities to correct wrong predictions during the process.\", but being continuous can also lead to error accumulation, while discrete transitions hold the possibility of entirely denoising the accumulated error.\n\n**R3:** Our continuous diffusion process is an iterative refinement process, where every intermediate step progressively improves the prediction to the correct target. This iterative nature effectively mitigates error accumulation, which makes the diffusion model a powerful generative model across diverse domains.\n\nIn contrast, discrete transitions attempt to reach the correct target in a single jump, which is inherently more challenging and prone to irreversible errors. This is especially critical in masked diffusion models, where incorrect predictions cannot be easily recovered without additional correction mechanisms. Our experimental results consistently show that the continuous diffusion approach outperforms discrete diffusion models, highlighting the advantages of gradual and continuous refinement over discrete jumps.\n\n---\n>**C4:** Eq. (10): The left-hand side should be $p_{\\theta}(X_T)$? \n\n**R4:** In Eq. (10), we parameterize the model $p_{\\theta}$ to approximate the transition distribution from current state $X_t$ to final state $e_k$, i.e., $p_{T|t}(e_k|X_t)$. The model takes as input $X_t$ and time $t$, and outputs the the probabilities $p_{T|t}(e_k|X_t)$ for all tokens $e_k$. Therefore, the left-hand side should be $p_{\\theta}({X}_t, t)$ as in the paper. We will add a more detailed explanation to clarify this point."}, {"review_id": "mA8Jvcr0zk", "reviewer": "Reviewer_myHk", "summary": "This work studies the continuous diffusion modeling for the discrete language task and presents the Riemannian Diffusion Language Model (RDLM) based on solid theoretical guarantees. More specifically, they first show the relationship between continuous flow on the manifold. Based on this, this paper develops a complete training framework to train RDLM and achieve excellent results in text, image, and DNA sequence tasks.", "questions": "1.\tIn line 726 of this paper, this work says that the mixture path process enhances the performance. Is this augmentation supported by experimental evidence (for example, RDLM without mixture path)?\n\n2.\tAs this work shows the potential of RDLM, would it be possible to scale up RDLM to achieve great performance?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n\n1.\tEach component of RDLM (including the theoretical foundation, mixture paths, and training framework) has been discussed and designed in detail.\n\nI didn't find obvious weaknesses; I just have some questions (in the next section).\n\nA minor Weakness:\n\n1.\tIt would be better to add more visualizations for an easier understanding of the RDLM pipeline.", "was_revised": true, "final_justification": "As showing in the discussion phase, I still think the modeling for the continuous diffusion models for the text generation task. Hence, I vote to accpet this work. However, as discussed by Reviewer iVwZ, the extension to the latent space is also important and interesting in leading to a large scale continuous diffusion models for text tasks.", "rebuttal": "We sincerely thank you for your constructive and helpful comments. We appreciate your positive comments that our work is based on solid theoretical guarantees, achieve excellent results, and each component is discussed and designed in detail. We initially address your questions below.\n\n---\n>**C1:** It would be better to add more visualizations for an easier understanding of the RDLM pipeline.\n\n**R1:** Thank you for your valuable feedback. Due to the page limit, we were unable to include additional visualizations in the current version. We agree that visual aids would enhance understanding, and we will add concept figures illustrating the RDLM pipeline in the final revision.\n\n---\n>**C2:** Comparison of mixture path and masked or uniform path\n\n**R2:** When using only the masked or the uniform path, we achieved test perplexity of 1.40 and 1.41 on the Text8 test set, respectively, while using the mixture path achieved 1.32. This shows that using the mixture path results in significantly improved performance.\n\n---\n>**C3:** As this work shows the potential of RDLM, would it be possible to scale up RDLM to achieve great performance?\n\n**R3:** Indeed, RDLM has the potential to achieve better performance when scaled up with larger models. Larger models can more accurately learn the transition distribution $p_{T|t}(X_T|X_t)$, which is critical for the effectiveness of RDLM. Our empirical results on Text8 and CIFAR-10 datasets support this observation, where we observe improved performance with increased model size. We believe that further scaling of RDLM could lead to even greater performance gains."}, {"review_id": "OJAGH4eSW5", "reviewer": "Reviewer_iVwZ", "summary": "The paper proposes Riemannian Diffusion Language Model (RDLM), a continuous diffusion model that treats each categorical token as a point on the positive orthant of a hypersphere. \n\nKey contributions of the work include:\n\n1. proving an explicit correspondence between discrete masked / uniform diffusion and geodesic flows on the statistical manifold\n2. designing a mixture path that smoothly blends the two processes to enable gradual refinement\n3. deriving a simulation‑free training objective that leverages radial symmetry to sample bridge states cheaply\n4. using dimension‑splitting that maps large vocabularies to products of low‑dimensional spheres for stable learning\n\nExperiments on Text8, LM1B, CIFAR‑10, and promoter‑DNA design show that RDLM is competitive with strong discrete diffusion baselines.", "questions": "1. What happens if you use only the masked or only the uniform path? Include an ablation on LM1B; a clear gain would strengthen originality claims.\n2. Please clarify theoretical differences with GIDD and give a quantitative comparison.\n3. Did you run standard Gaussian diffusion on the simplex or the sphere before implementing the Riemannian diffusion approach?", "limitations": "The dense theoretical exposition currently obscures the core ideas of the model. I recommend moving some of the derivations to the appendix and using the freed space to present more experimental analysis and comprehensive ablation studies in the main text.\n\nPlease refer to the questions and weaknesses outlined above for further details.", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n1. By grounding diffusion in information geometry, the work unifies discrete and continuous views and offers a principled route to exploit iterative refinement without losing signal at state jumps.\n2. The paper contains detailed derivations of the flow equivalence, likelihood bound, and bridge approximation, with formal proofs moved to the appendix .\n3. Results on images and DNA sequences suggest the method works across different kinds of data.\n\n\n**Weaknesses**\n\n1. Important continuous text diffusion baselines such as [DiffuSeq](https://arxiv.org/abs/2210.08933), [TEncDM](https://arxiv.org/abs/2402.19097), and [LD4LG](https://arxiv.org/abs/2212.09462) are missing, making it hard to gauge relative progress.\n2. [GIDD](https://arxiv.org/abs/2503.04482), that also suggests combining masked and uniform discrete diffusion, is not discussed.\n3. The language results are limited to Text8 and LM1B, which are both relatively small and dated corpora. Other commonly used benchmarks for text diffusion models, such as RocStories, and OpenWebText, are not covered. This omission makes it difficult to compare the proposed method with other diffusion-based baselines. Furthermore, the field has advanced considerably, and evaluation on more challenging tasks, including conditional generation, would be necessary to demonstrate the method’s practical relevance.\n4. The work claims that the continuous approach offers advantages for controllable generation but provides no evidence on conditional text generation tasks, such as summarization.\n5. The bridge approximation is empirically validated with MMD, yet its impact on final likelihood in extreme dimensions is not thoroughly analysed.\n6. The paper also lacks a comparison with Gaussian diffusion on the simplex as an alternative approach to modeling discrete data, as explored in methods such as [SSD-LM](https://arxiv.org/abs/2210.17432) and [TESS](https://arxiv.org/abs/2305.08379).", "was_revised": true, "final_justification": "I appreciate the authors' response, but the core issue of baseline comparison remains unresolved. A key motivation for this work is the claim that \"Existing continuous diffusion models for discrete data underperform compared to discrete methods.\" However, this foundational assertion is not substantiated with references to the literature, nor is it validated empirically through direct comparisons with leading continuous methods like LD4LG and TEncDM.\nWithout such an evaluation, the paper fails to convincingly demonstrate superiority over other continuous diffusion methods.", "rebuttal": "We sincerely thank you for your constructive and helpful comments. We appreciate your acknowledgement of our key contributions, in particular, that our work offers a principled route to exploit iterative refinement without losing signal at state jumps. We initially address your comments below:\n\n---\n>**C1:** Missing related works \n\n**R1:**\nWe appreciate the reviewer for providing additional related works on text generation. We would like to clarify that DiffuSeq is a diffusion model designed for sequence-to-sequence tasks, whereas our work addresses general unconditional text generation. Moreover, TEncDM and LD4LG are continuous diffusion models that adapt image diffusion models to text, and they are conceptually similar to Diffusion-LM, Plaid, or Bayesian Flow Network, which we have included as baselines in our comparisons. We will add discussion with these works in the revision.\n\n---\n>**C2:** Difference with GIDD\n\n**R2:** We appreciate the reviewer for bringing to our attention the very recent work GIDD, which is a contemporaneous work (appeared on arXiv after March 1st). GIDD is a discrete diffusion model that interpolates masked and uniform diffusion by mixing masking and uniform noise. On the other hand, our RDLM is a continuous diffusion model using Riemannian diffusion process on hypersphere, which fully exploits the power of iterative refinement. \n\nIn particular, our mixture path formulation (Eq. (8)) can be seen as a continuous generalization of interpolating diffusion process in GIDD. Extending our Proposition 3.1., the mixture path built from mixing masked diffusion and uniform diffusion corresponds to the trajectory of transition distribution of GIDD process. Moreover, Eq. (8) allows interpolating more than two processes, while GIDD is limited to mixing masked and uniform noise.\n\nDuring the rebuttal, we could not make a quantitative comparison with GIDD as GIDD does not have checkpoints for the benchmarks used in our work, for example, Text8, LM1B, CIFAR-10, or DNA sequence design. We will include a detailed discussion with GIDD and add comparison in the final revision.\n\n---\n>**C3:** Language results are limited to Text8 and LM1B and other benchmarks such as RocStories and OpenWebText, are not covered\n\n**R3:** Text8 and LM1B are among the most widely used datasets for evaluating diffusion models in recent literature, especially for discrete diffusion models. Notable recent works such as SEDD (Lou et al., 2024), MDLM (Sahoo et al., 2024), and MD4 (Shi et al., 2024) also use these datasets for benchmarking. These works do not typically use RocStories as a benchmark. We believe that experiments on Text8 and LM1B, as well as image generation and DNA sequence design extensively validate the effectiveness of our framework. While we agree that results on OpenWebText would further strengthen our work, we could not run the experiments due to limited resources. We will make every effort to include OpenWebText results in the final revision.\n\n---\n>**C4:** Evaluation on more challenging tasks, including conditional generation, would be necessary\n\n**R4:** Unconditional generation of discrete data using diffusion models is itself a challenging task, with a notable performance gap compared to autoregressive models. Recent works such as MDLM, MD4, and Discrete Flow Matching, have also focused primarily on unconditional generation tasks without extending to conditional generation. In our paper, we focus on validating the effectiveness of our new continuous diffusion framework for modeling language and discrete data in an unconditional setting. While conditional generation tasks are important, addressing them is beyond the scope of the current work.\n\n---\n>**C5:** The work claims that the continuous approach offers advantages for controllable generation but provides no evidence on conditional text generation tasks\n\n**R5:** We would like to clarify that we did not claim our method to have advantage for controllable generation. In introduction lines 29–30, we explained that previous works have adapted continuous diffusion models for discrete data by their advantage on controllability. For example, Diffusion-LM uses classifier-guidance to diffusion model for controllable text generation. While our continuous diffusion process can similarly incorporate classifier-guidance, we have not explored this direction in the current work.\n\n---\n>**C6:** The bridge approximation is empirically validated with MMD, yet its impact on final likelihood in extreme dimensions is not thoroughly analysed.\n\n**R6:** MMD results show that our approximation on the transition distribution is accurate. This leads to superior results on test perplexity in the LM1B dataset, which has a high dimension of 30522.\n\n---\n>**C7:** Comparison with Gaussian diffusion on simplex or sphere\n\n**R7:** \nFirst, we would like to clarify that Gaussian diffusion are ill-defined on non-Euclidean space like simplex or sphere. This is because the Gaussian distribution itself is not well-defined in these spaces, and diffusion process must be redefined to account for the underlying geometry and metric of the manifold. \n\nWhile Gaussian diffusion models can be applied in the ambient Euclidean space, recent literature on generative modeling over manifolds, for instance, RDM (Huang et al., 2022) or RSGM (De Bortoli et al., 2022), demonstrate that Riemannian diffusion outperforms Gaussian diffusion when modeling data that reside on non-Euclidean manifolds. Our experimental results further support this, showing that our method significantly outperforms Diffusion-LM, a Gaussian diffusion model designed for text data.\n\nFurthermore, prior works used as baselines in our paper, for example, DirichletFM (Stark et al., 2024) and Fisher-Flow (Davis et al., 2024), show that naively adapting flow matching to simplex, i.e.., LinearFM, underperformed Riemannian approaches. We will include this discussion in our revision and add LinearFM as our baseline.\n\n---\n>**C8:** What happens if you use only the masked or only the uniform path?\n\n**R8:** When using only the masked or the uniform path, we achieved test perplexity of 1.40 and 1.41 on the Text8 test set, respectively, while using the mixture path achieved 1.32."}, {"review_id": "H8m0PdBaMI", "reviewer": "Reviewer_FzDR", "summary": "This article proposes a continuous diffusion model for language modeling, which integrates the geometry of the underlying categorical distribution. It establishes a link between discrete diffusion and continuous flow on the statistical manifold, and based on this connection, introduces a simple diffusion process that generalizes existing discrete diffusion models. The article also puts forward a simulation-free training framework relying on radial symmetry, along with a straightforward technique to tackle the high dimensionality of the manifold. Experiments across language modeling benchmarks and DNA are conducted, showing that the proposed method surpasses existing discrete diffusion models.", "questions": "see weaknesses", "limitations": "see weaknesses", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The methodology of connecting discrete simplex modeling to the generative modeling on the sphere manifold is novel and reasonable, at least for me.\n- The mathematical derivation is solid, though I didn't check the detailed proofs.\n\nWeaknesses:\n- I am not familiar with the language tasks, but I think I witnessed some more large-scale language generation benchmarks in the technical report of Qwen3 and Kimi-chat. I think the method in this paper can not scale to that level, and it is a little overclaim to say 'approaches the performance of autoregressive models.'.\n- Detailed time-cost of RDLM compared to other discrete diffusion models and DDPM is needed.\n- On CIFAR-10 experiments, the comparison to classical DDPM/EDM/rectified flow matching is beneficial. \n- Experiments on scalability should be given.", "was_revised": true, "final_justification": "My point of view has not changed; I still value the novelty of this paper and question its practical usage. My score is pretty marginal, and I keep my score between 3 and 4.", "rebuttal": "We sincerely thank you for your constructive and helpful comments. We appreciate your positive feedback that our work is novel and reasonable, and the mathematical derivation is solid. We initially address your comments below:\n\n---\n>**C1:** I think the method in this paper can not scale to Qwen3 and Kimi-chat, and it is a little overclaim to say 'approaches the performance of autoregressive models.'\n\n**R1:** Thank you for your valuable feedback. We would like to clarify that our work proposes a novel language diffusion framework aimed at improving previous diffusion models for language modeling. Our primary focus is on advancing the modeling approach rather than scaling to very large modern LLMs such as Qwen3 or Kimi-chat.\n\nRegarding the comparison to autoregressive models, this comparison is made within the context of models of similar scale. We do not claim that our method matches the performance of state-of-the-art large-scale autoregressive models, but rather that it approaches the performance of autoregressive models at comparable model sizes. We will clarify this point in the revised manuscript to avoid any overclaim.\n\n---\n>**C2:** Detailed time-cost of RDLM compared to other discrete diffusion models and DDPM is needed.\n\n**R2:** RDLM does not require additional computational cost during sampling compared to the discrete diffusion models in terms of the number of model evaluations. Specifically, to generate samples, RDLM simulates the generative process modeled by the SDE in Eq. (6) (please see Algorithm 3 in the Appendix), and simulating Eq. (6) with a fixed step solver (e.g., Euler-Maruyama) requires the same number of model evaluations as simulating the discrete diffusion process. Therefore, in terms of sampling time-cost, RDLM is comparable to discrete diffusion models, as both approaches rely on an equivalent number of model evaluations per sample generation.\n\n---\n>**C3:** In CIFAR-10 experiments, the comparison to classical DDPM/EDM/rectified flow matching is beneficial.\n\n**R3:** CIFAR-10 experiment in our work addresses a fundamentally different task compared to the image generation tasks tackled by DDPM or EDM models. Our CIFAR-10 experiment focuses on pixel-level image modeling framed as an order-agnostic set modeling problem of discrete tokens, each drawn from a vocabulary of size 256, following the approach in MD4 (Shi et al., 2024). This formulation intentionally removes spatial information about pixel proximity, which is a key inductive bias exploited by classical image diffusion models through architectures like U-Net. Given this distinct setting, we excluded DDPM and EDM models from our baselines, as their assumptions and modeling strategies do not directly apply.\n\n---\n>**C4:** Experiments on scalability should be given.\n\n**R4:** Our experiment on the LM1B dataset, which features a sufficiently large vocabulary of size 30,522, demonstrates strong potential for scaling to even larger vocabularies. Notably, RDLM is designed to scale to higher dimensions efficiently due to its simulation-free training scheme (as detailed in Section 4) and the dimension splitting technique (described in Section 5)."}]

Author rebuttal:

Meta-review: This paper proposes RDLM, a continuous diffusion model for language. The idea is to project the categorical distribution of tokens onto the positive orthant of a sphere, allowing for the use of continuous diffusion processes. The reviews are mixed - the reviewers generally appreciate the novelty of the work but raise concerns about lack of baselines and potential scalability concerns. 

Reviewer iVwZ voted strongly for rejection and based that decision on the lack of comparison to related work including LD4LG and TEncDM. After checking the literature, AC stands with the authors that these are not standard baselines in continuous diffusion LMs - as the authors pointed out, these methods rely on pretrained models, which is a different setup from this paper. The authors chose to compare with diffusion LM and Plaid - which is more relevant baselines that do not rely on pretrained models. Therefore, I did not factor this point into my decision.

Given that another key reviewer was on the margin (FzDR pointed out their decision stands in between 3 and 4), I recommend acceptance - the paper's methodological novelty represents a valuable contribution to the field.
Decision: Accept (poster)