Discrete Modeling via Boundary Conditional Diffusion Processes
Authors: Yuxuan Gu
Xiaocheng Feng
Lei Huang
Yingsheng Wu
Zekun Zhou
Weihong Zhong
kun Zhu
Bing Qin
Venue: NeurIPS 2024 poster (2024)
Keywords: Diffusion Models
Discrete Modeling
Language Generation
Discrete Image Generation
Primary area: diffusion_based_models

Abstract: We present an novel framework for efficiently and effectively extending the powerful continuous diffusion processes to discrete modeling.
Previous approaches have suffered from the discrepancy between discrete data and continuous modeling.
Our study reveals that the absence of guidance from discrete boundaries in learning probability contours is one of the main reasons.
To address this issue, we propose a two-step forward process that first estimates the boundary as a prior distribution and then rescales the forward trajectory to construct a boundary conditional diffusion model.
The reverse process is proportionally adjusted to guarantee that the learned contours yield more precise discrete data.
Experimental results indicate that our approach achieves strong performance in both language modeling and discrete image generation tasks.
In language modeling, our approach surpasses previous state-of-the-art continuous diffusion language models in three translation tasks and a summarization task, while also demonstrating competitive performance compared to auto-regressive transformers. Moreover, our method achieves comparable results to continuous diffusion models when using discrete ordinal pixels and establishes a new state-of-the-art for categorical image generation on the Cifar-10 dataset.

Discrete Modeling via Boundary Conditional Diffusion Processes
Yuxuan Gul
Xiaocheng Fengit
Lei Huang Yingsheng Wut
Zekun Zhou
Weihong Zhong'
Kun Zhu
Bing Qin it
Harbin Institute of Technology Peng Cheng Laboratory
(yxgu,xcfeng lhuang, sonhangg.www,, zkzhou, whzhong, kzhu, qinb),r .it.hht.... eut.eddu.cmm en
Abstract
We present an novel framework for efficiently and effectively extending the power- ful continuous diffusion processes discrete modeling. Previous approaches have suffered from the discrepancy between discrete data and continuous modeling. Our study reveals that the absence of guidance from discrete boundaries in learning probability contours is one of the main reasons. To address this issue, we propose a two-step forward process that first estimates the boundary as a prior distribution and then rescales the forward trajectory to construct a boundary conditional diffusion model. The reverse process proportionally adjusted to guarantee that the learned contours yield more precise discrete data. Experimental results indicate that our approach achieves strong performance in both language modeling and discrete image generation tasks. In language modeling, our approach surpasses previous state-of-the-ar continuous diffusion language models in three translation tasks and summarization task, while also demonstrating competitive performance compared to auto-regressive transformers. Moreover, our method achieves comparable results to continuous diffusion models when using discrete ordinal pixels and establishes a new state-of-the- for categorical image generation on the CIFAR-10 10 dataset.
1 Introduction
Discrete modeling is essential due to the natural prevalence of discreteness in numerous domains, including proteins [Madani et al. [220 2023), images (Parmar
2018 Dosovitskiy et [021]
and natural language [Sutskever et al. 2014 Brown et al. 2020 Recent dominant framework
for discrete modeling the Transtormer Vaswani et al. 2017 with an autoregressive manner.
While achieving impressive performance, it does suffer from slow step-by-step generation process, especially for long sequences. Continuous Diffusion models (Sohl-Dickstein et al. 2015 Ho et al.) 2020], on the contrary, exhibit the ability to recover high- nigh-dimensional data from noise in parallel with limited iteration steps. Although proved be effective in continuous data generation (Rombach Kong et al.
et al.
2022
2021 they continue to encounter challenges in discrete modeling [Austin
al.
2021 Chen et al.
2023b Li et al. 2022 Gong et al.) 2023b).
In this paper, we reveal significant discrepancy pertaining to the modeling of discrete data using continuous diffusion models. Current approaches represent discrete sample with a vector point in the continuous space. The diffusion process learns a neural network to model the probability distribu- tions that recovers this continuous point from Gaussian noise. However, the discrete data actually corresponds to an area in the continuous space rather than single point, where the oversimplified assumption leads to mismatch between learned probability contours and the boundary of the discrete area. Take language generation as example, word is represented with embedding vector in the embedding space. To generate this word, it impractical strictly enforce the predicted vector be an exact match to the embedding. On the contrary, vectors around this embedding can also generate
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

the same word, thereby defining the collective area they encompass as discrete area of this word. As illustrated in Figure [g. suppose the learned probability density function po (x) and two points x and x" are sampled in the same density contour where (x') = po (x"). It is obvious that x² lies in the discrete area and is able to recover the discrete data while x" can not. This means that the diffusion model only learns a simplified scenario that does not match the real probability distribution.
To address the issues above, we pro- posed to take the boundaries of dis- crete areas as priors, as shown in Fig- ure IBB where boundary curves are re- garded as oracle contours. As grad- ually approaches the discrete bound- ary, the learned density contours of diffusion models are expected to trans- form from Gaussian distributions to the boundary distribution. Therefore, we propose to divide the forward pro- cess into two steps. First is the bound- ary estimation where we precisely cal- culate the stopping time to and po- sition Xr, at which the forward tra- jectory cross the boundary. Then rescale the trajectory for both train- ing and inference stages make the sampling probability of noisy point X, conditioned on the boundary. To make the boundary estimation tractable (ap- pendix A) and eliminate randomness in conditional state transitions Xt. - Xt, we utilize the Ordinary Differential Equations (ODEs) to describe the forward trajectory.
(4)
Episiaal
Vrstsi
Diservte Area
No
Figure I: (A) Blue and green curves are the learned prob- ability density contours of the diffusion model for two data points. The red area is the discrete area of the blue data Xo and the boundary of this area naturally density contour. The discrete boundary is complex hypersurface in the high- dimensional continuous space and we simplify into a red line for convenience of description. As observed in the mag- nified part, the learned contours deviate from the boundary contour, resulting in inconsistent probability densities and gradient directions. (B) We consider the discrete boundary as priors for the diffusion process to estimate more appro- priate probability distribution, where the learned contours expected to follow the shape of the discrete boundary.
Our approach is experimented in both language modeling and discrete image generation. On three machine translation datasets (IWSLT14 4 DE-EN (Cettolo et al.] 2012). WMT14 EN-DE, WMT16 6 EN-RO) and a text summarization dataset (GIGAWORD (Rush et al. 2015)) for language modeling, our proposed approach not only significantly improves existing diffusion models to most 7.8% but also achieves competitive performance to autoregressive transformers. For image generation on CIFAR- 10 Krizhevsky al. 2009), our model realizes comparable result to continuous diffusion models with discrete ordinal pixels and establishes a new state-of-the-art for categorical pixels.
2 Preliminaries
Diffusion Models
To model real distribution (xx)), diffusion models utilize a forward process
(x(o)) with 7 steps to gradually add Gaussian noise T(x) = NN I) into the data distribution,
where pr [x xn) = (x). There are different architectures for the forward process. A common
approach Ho et al. 2020 considers the forward process as the Markovian process, where Pe Pe(xXx)) =
1 (x, combines a series of Gaussian distributions. Thus the forward process follows a Gaussian distribution that (x(x)) = _ /Llá,, Xa. (1- I)) (Variance Preserving) Pa (x(x)) N(xx, I (Variance Exploding) [Song et al., 2021b), where noise scheduler Oe monotonically decreases from 1 to O and o, increases from su sufficiently small to the maximum pairwise distance
between all training data points. To recover data from noise, diffusion processes train neural networks x*(X,,) t) to predict xo (other equivalent targets include e and V log p(x:)) from X₁ Pe(x[xo):
Lo = Ee-ila,, No xx [[|x] Xy (x,, 1]]
Samples are generated with series of reverse state transition p( p(x,-1/ X,, ()),
Flow Matching Another architecture (Lipman et al. 2023 utilizes the ODEs and defines
time-dependent flow function (x( = (Xo)xx + (4((x)) (Xn) that maps (x = [x)) =
(05 det døy (x) = ((((((x)), (x)) where be and or can be the same as in diffusion
dx

---

Goaaka Canrectio
1= K79
Probalality Contours
Resealed Forwed
Figure 2: (A) Rescaled Probability Contours. The bold curve 1o is the density contour of one standard deviation. As the time t decreases from T to O, the rescaled contours will gradually fit the discrete boundary and probability densities will also concentrate this boundary. (B) Rescaled Forward Trajectory. Original forward trajectory X + Xt, > X, is rescaled to be a boundary conditional trajectory X, X, that starts from X; Xto The rescaled forward distribution p. (R, Xx) is transformed from the discrete boundary to Gaussian distributions.
4. Recovering data from noises relies on the vector field (x(xxx) that generates the probability path with the ODE dx)-<(x) = ( (x))d, & 0 - T. Neural networks up (x, are trained to estimate the vector field u, (x)) via the following objective:
models more straightforward form that /4 (1 & )xx and
uo(oi(xx): dxT()
Lo x00 (x0) xr (x))
dt
Besides, the vector field is proved to have the form:
(x(x) (xp)
(x(x)x))
(x (x()))) (x(x), where apostrophe indicates derivative to tt.
3 Methodology
As illustrated in Figure our objective is to refine the probability density contours of Pt (x(xx)) so that they better fit the boundaries of discrete samples while still allowing for the ease of sampling. Let x denote the samples from real distribution (xo). Obtaining boundary-aware corresponding noisy pe (x, Xto to X )xx,, dfg, where to random variable distributed according to when the diffusion trajectory and the discrete boundary intersect, and Xt, is the corresponding sample point at ty. Then the forward process is rescaled in two steps:
data x at time E [1, T] Pe (x
(x = f (x( x to, Xo
p(Xto= to xxo) dx,, dtg-
Trajectory Rescaling
Boundary
where the latter term is calculate the discrete boundaries and the former term is to rescale the forward trajectory. In order to make the equation tractable and ensure that x and X4, are on the same trajectory, model the forward process with flow functions di (x) and extend the notation
(x) u(xo, t) x v(xo, t) x, (x(x [+))
(5)
where ut-) and v(-) are coefficient functions and sampling x, from p. (x(xo) equals
x; = (E(E)), E e~ xx) = NO,
(6)
3.1 Estimate Discrete Boundaries
Before figuring out the joint distribution p(Xt, (o)x)), let's start by discussing how to verify whether an arbitrary point x in the continuous space belongs the discrete area of xg- Suppose Xg. which exists in the continuous space S, is the representation vector of discrete random variable in discrete space with K states. Besides, J is another discrete random variable i..dd with I. We define the discrete area of Xg in the continuous space S as:
C (Vx E SIf(x, I) > f(x, #I),
3

---

where fxx, I) is a function assessing the likelihood of an arbitrary continuous point inside the discrete area of Xg. For instance, language modeling, K is the vocabulary size. J.J E K" are two different sequences of n tokens and X IR (g( is a sequence of mn-dimensional vector embeddings for Z. f(x. I) is the dot similarity function. Oz collects all vectors in the embedding space that will be decoded to generate I and excludes vectors associated with any other token sequences J.
Given a noisy point Xt, locating at the boundary between Cr and Cy. we can get |f (x I) fx J]| based on previous definition. Replacing Xt, with eqs. (5) and (6), there is:
f Xo + Vta C, I) f(u,, + C,)
(8)
In language modeling and categorical images, f(+) a linear projection function that: []) = v -((,
Uso -
(9)
Further simplification of this equation can not be universally applied to all arbitrary forms of us, and Vtg. Therefore, we calculate separately for several commonly occurring special cases.
Diffusion Process
For variance preserving, there is uf + v? and we have:
U1, = -1/VL+ (
(1(x. (1ite.3)) [X0, [(x0: -
2
and Vea =1 -\/V 1+ ( (
/++
[( [(E,I)
f(x0,I)-
- J))
(10)
For variance exploding, there are u, = 1 and or. We can obtain:
Ut and Vi = (f(E,T) -(e,, (f(x: f(xo, J]).
(11)
Flow Matching
For optimal transport, there u, + V = and similarly we get:
use -1/(1- f(xo.I) (ffo)))f(xxx-- f(x.. -(,))) ((E
and Vto =1/ =1/(1+ f(e.J) (1++)))))) f(xo. f(xo
(12)
As result, to can be directly derived by inverting the coefficient function or vt. which depends on the choice of noise scheduling strategies. Since their differences do not affect our results, we omit the detailed calculation (appendix D and denote this process with function G(-):
to G(xo E), where u(xo, G(xo, E)) Uto and v(xo. G(xo, E)) Vto
(13)
It's worth noting that to is not a scalar but a vector, where the dimension the number of elements in Xg. If xo is sequence of tokens, to [1, If is a RGB image with 3-channel x h-height x w-width of pixels, to E [3x7/xx Furthermore, the corresponding noisy sample Xt, is derived as:
Xto u(Xo, G(xo; e))xx v(xo, G(xo, E (E),
(14)
which is a time-independent function of the Gaussian noise e. It's worth mentioning that both p(to x)) and p(x,, X1) are intractable, since G(xØ, E) and (G((...)) (E) are not invertible to e. Different es can be mapped to a same to or Xt., Fortunately, there is an one-to-one mapping between and the [xe, : to] pair. We denote the boundary flow function and the corresponding inversion as
(E) (# (E); G(xo. })],
v ([xt, tol) (Xt. u(xo, (o)xo)/v(xxx to), (15)
and the joint boundary distribution is calculated as
p(x to (x)) = []] tul).
(16)
The support set of Xt, is restricted to the boundary contour, while other regions in the space are assigned a probability of 0. To obtain the complete boundary, it necessary to iterate over all possible choices of I and perform pairwise comparisons with Z. The complexity is O(n K), where elements Xo is independently iterated. In practical implementation, obtaining the tightest boundary only requires step of parallel calculation and extra min(-) function over all to candidates.

---

Confidence Factor The discrete area defined by eq. 7 represents an ideal scenario in which the confidence of the boundary is insufficiently reliable for practical application. Due the intractability of obtaining the probability density function across the entire discrete area and calculating its confidence interval, we employ an empirical strategy. This approach involves utilizing a confidence factor, denoted as ranging from 0 to 1, which is multiplied by to to strike balance between confidence and discreteness. Therefore, r = implies the exclusion of discrete priors, causing the discrete area to collapse into a single point, which is the original diffusion process. As the value of r increases, the modeling of discrete boundaries improves at the expense of reliability. Empirically, when the model is conditioned with good guidance, setting larger value for allows us to obtain better discrete priors. However, in the case of unconditional modeling, maintaining reliability becomes more crucial to prevent oscillations and even collapses during training.
3.2 Rescale the Forward Trajectory
In this section, we introduce how to formulate the forward trajectory conditioned on discrete bound- aries and derive the rescaled noisy sampling distribution. We start with the boundary-independeer forward process Pr (x) xo). Let Xr denote noisy point at time sampled from pe (x(x)), there is
6, (x: u (xxo: ()xa) (xo f) given eq. (5). Equations (13) and (14) provide the corresponding
[xg,) fol pair on the same trajectory, which is deterministically calculated with no randomness:
X tol = (E:), where 6, = (x: u(xo. t)xo) /v(xo, t).
(17)
To model the transition probability Px(xt to Xt, xo), we utilize the Dirac delta function 8(x) = li -Voo (00, ((0,ooI), I²)), which can be loosely thought of as aggregating all probability densities toward the origin, assigning
infinite density the origin and zero densities elsewhere. Therefore, we have px(x,, to Xx, xo) = 8 ([xto tol (e:)) Then the forward process, conditioned on the discrete boundary, is simply derived via Bayes' rule:
0,
[Xto tol * ((G)
Pe (x(x)(x))
pe(xx|xxto lo, xn) = pe fo|x,, xn)
t00 x Pr (x,x(x))
p(xto: to to)xo)
otherwise
p( (1 4u (xo)
(18)
Since (x) > 0 and p(xi; tolxo) > 0. Pe Pe (2oo to. xo) is also delta function that Pe (x(xx Xto lo, xo) = (x: u(xo: ()xo v(xo. 1) ([xto: (o])).
(19)
Based on the translation property of the Dirac delta function, i.e. f f a)dr = f(a), the original forward process (x [x,) |th, v (xx) = (x,) naturally ignores the influence of discrete boundaries, even if the boundary information is explicitly added as a condition.
To enable the discrete priors, we propose simple and intuitive approach: rescale the forward trajectory. As shown in Figure the original forward process flows from Xo to random noise e, and we reset the starting point to Xt. Accordingly, the intermediate noisy points x,, t E [1, T] will be proportionally mapped on this new path, which is
x X,, T= T(t, to) x to + (T' - - to)/T
(20)
u(xo, Tt,, (o)) + v(xo. T(t, to) ([x10 to]). Similar to eq. 19). the rescaled conditional forward process a Dirac delta function:
Pe (&e Xto: : to. Xo) 5 (x, u(xo. Ttt, fo))xo - vxo, T(t, to)) T- (Ixa; fol))
(21)
However, Pe (&c (xo) faces the same problem of irreversibility as in eq. 14 and we derive it as:
/ (i T|x0)dT = / pe pe 7 7/x : to, xo) (0) (xx, : to (xxx) tojdr
Pr (Xe
(22)
= -Lw,, v o W)] r])dd= = / [V r])dr.
Obtaining the probability density function requires gathering together the probability densities of the same location &, with different r, which is intractable. Fortunately, we only need to sample noty points from this probability distribution & ~ px (x)), which is easy to implement:
* (Xa, T(t, G(X,, e))) X + v vxx,, T(t, G(Xo, e)))e, e~ (xx).
(23)
3.3 Recover Data from Noise
5

---

Training Objective Theoretically, the diffu-
Algorithm I Training
sion neural networks can be trained as eq. (
1: repeat
where the rescaled vector field is derived as dr di' However, since low
2:
Xo g(xo), π T()) N
dh d&, di
dr
3:
Uniform( (1...., ,
error estimation on Xo is of significant impor-
4:
T = T (1, G(xo; 6)) // eqs.
13 and
20
tance to our trajectory rescaling method, accord-
5:
X = u(xo, x)x, + v T)
// eq.
23
ing to eqs. (10) to (13) we convert the objec- tive to an upper bound of the eq. (2) (See ap- pendix F for more details) and train a neural network X (R,, t) to predict Xy directly:
6:
Take gradient descent step
Vellxo - X XX (x,, t)l
// eq.
24
7: until converged
Lo (x0r~((00),,-1 (. (x(x)) ||xx
xe (Se, t)]1
(24)
The training procedure is demonstrated in algorithm and key steps are summarized in the line 1
Reverse Process A direct approach that fol- lows the flow matching is to solve the ODE of dx) - Xa df, v/y (x) ~ ineffi-
Algorithm 2 Sampling
1: t = T, T
2: ê = &, ~ -N(,,I)
I/ Initialing
(x). This form of transformation cient with Xo- to-prediction during inference be- cause we have to solve the equation of r
3: for A/ = Aty..., At, do
St E>'
T
4:
8a = Xo (x(,
I1 Pseudo Target
5:
I Al
I/ Updating I/ eq. 25
T (t, Xg, >--u(xe,7>x, )) to get the with
v(xo.T)
6:
T = T (t, G(Xo. E))
respect to the change of x; and X, in real time.
7:
X u( T T)X0 + (x T)
Therefore, we provide a deterministic reverse
8:
? = y ([xTT]
// Trajectory Alteration
process as an alternative, which is a special case of DDIM [Song al. [021a] or the ODE
9: end for
10: == X X (x,,,)
|/ x1 - X
with discrete timesteps. Given the time intervals
11: return Xg
At [At].... At,l, Σ>> At T we general-
ize the boundary conditions [Xto (o) in px, [[X,|xt,. lo. xo) of eq.
21 and Vp ((xto(io)))) of eq.
15 to
any arbitrary condition pairs [xx; r] and obtain the reverse process:
de: TA [x: T] Xo)
xi A+
X
&n. TA 800 v(xo TA T(t At, G(&g, E))
(25)
AA -
where &o (x, t) and TA is the previous timestep of r on the same rescaled trajectory.
Sampling from the reverse process is illustrated in algorithm 2 Similar to the sampling process of DDIM [Song et al., [2021a], it starts from the Gaussian noise, iteratively predicts the pseudo target &a. have
and updates the reverse trajectory. However, since the and è are mutually conditioned, keep track of the I, r, &, and é during each iteration and split the update of into an asynchronous step (line 8). Because reverse trajectory keeps changing due different pseudo targets Xo predicted by learned neural networks, which brings severe instability, sometimes simply fixing the initial path (removing the line g exhibits better performance in experiments.
4 Language Modeling
Recent diffusion language models [Li et al. 2022 Gong et al. 2023b inherit the embedding-rounding
framework that sentence with discrete tokens W [ -....... w., is embedded continuous space via a trainable embedding layer EMB(W) = [EMB ( EMB (w,)). The vocabulary set
K that Yw, K. Besides, the token embeddings are used the target points xo [x] x(j].
xi
(, for continuous diffusion trajectories. Hence, generating tokens from embeddings is:
exp(f(x) w:))
p(W xx) = =-" p( P( Σ
(26)
EjEK exp(f(x).
where f(x,j) EMB(j) x is the dot production distance. It's also the function assessing the likelihood of point x inside the discrete area of j. The coefficient functions follow the DDPM HHo et al. 2020), which are u(xn; 1) Va, and v(xo, 1) vT 04. Besides, the objectives are
Lg = EW Σ
[[EMB(..) - xo (x 0112/11
(27)

---

Table 1: Result of BLEU scores on machine translation and ROUGE scores on text summarization.
IWSLT14 DE-EN
WMT14 EN-DE
WMT16 EN-RO
GIGAWORD ROCGE-1/2/L1
Models
BLEU (BLRU-1/2/3/40)
BLEU (BLEU- 1/2/3/4/fh
BLEUU (BLBAC112/3/41
Auto-Regressive Modeling
Transformers
34.31 (67.3/4..627:919.1
28.01 (58.2/33.5/21.7/4444
34.05 (63.1/39.//776/11))
37.57/18.90/34466
Ours+Rerank 35.02 (68.7/43.22.222011
27.67 (57.9/33.2/21 3/14.3)
34,33
37.49/18.68//4.822
Diffusion Process
D3PM
27.61 (65.4/37.7222.111.2))
22.94 (54.9/28.8/16.911000)
27.84 (59.8/34.9/22.//1)
33.92/14.96/31.72
DiffuSeq
28.78 (-
15.37
25.45
31.17/12.23/29222
SeqDiffuSeq
30.03
17.14
26.17
31.90//2.33/29.22
Difformer
31.58
24.80 (58.7/32.0/19.7/12.5))
30.08 (64.4999.5/6655188..))
35.47/15.
SEDD
31.87 (68.7/41.827.21188..
24.98 (59.2/3../20.1/122.99
29.38 (62.2/38.0/24.//16..
34.33/15.22/3.00
Dinoiser
31.91 (67.14409/26..711.7))
24.77 (57.2811001900012200)
31.49
35.17/11..3/32.53
Ours
33.42 (68.0442.027..7188.))
26.69 (57.1/32.320.441334)
33.15 (63.4/99.//2.4419.222)
36.44/16.09/33356
and additional rounding objective, which is commonly used in language modeling,
C. log po (W
log po ( ( (x),
(28)
The final training target is given by L = Lo + Ly, where the Xg of the same token sequence W keeps changing because the embedding layer EMB is trainable, which makes the model hard to be trained. Since previous work does not model discrete areas, large number of noisy samples inside this area will make L, too small to guide the training of the embedding layer, leading to mode collapse.
Experimental Setup Datasets used for experiments include three translation tasks (IWSLTIC 14 DE-EN (Cettolo et al. 2012). WMT14 EN-DE, and WMT16 16 EN-RC and one text summarization task (GIGAWORD [Rush et al. 2015). We mainly follow the setting of Gao et al. [2022] which is inherited from previous non non-auto-regressive text generation works [Gu et al. 2018 2019 Ghazvininejad et al.,
2019]. where translation datasets are distilled [Kim and Rush 2016 Base line are mainly continuous diffusion language models. DiffuSeq Gong et al. 2023b] and Dequusee (Yuan et al. 2022] are derived from Diffusion-LM [Li et al. 2022]. Difformer (Gao et al., [022] and Dinoiser Ye et al.) 2023) are recent empirical studies highlighting that scaling up the noise is beneficial for language modeling. We also compare with discrete diffusion language models, including D3PM [Austin et al., 2021] and SEDD [ou et al. 2023]. Since SEDD is a pre-trained language model, we configure its framework and train it from scratch specifically for our tasks. In addition, auto-regressive transformer Vaswani et al. 2017] is still one of the most powerful architectures for language generation.
Our boundary conditional diffusion language model is constructed from Difformer [Gao et al. 2022 where the model configuration is transformer-iwss -ransformer-ivst--deeen in FAIRSEQ framework [Ott et al. 2019
for IWSLT14 DE-EN and transformer-buse for other datasets. Sentences are tokenized with Byte-Pair Pair Encoding (Sennrich et al. 2016) and evaluated by detokenized BLEU (Papineni et al. 2002 for machine translation and ROUGE [Lin] 2004) for summarization. During training. the diffusion step is T = 2000 and the confidence factor = for translation tasks since they have strong conditions, while r = 0.5 for summarization. Sentences are generated deterministically with 20 steps.
Our approach achieves the state-of-the-art compared with continuous diffusion language models and outperforms the two discrete baselines on three machine translation and one text summarization tasks. Our method shows advantages, with 73.6% significant improvement at most on WMr14 EN-DE, over DiffuSeq [Gong et al. 2023b) and SeqDiffuSeq [Yuan et al.) 2022), which are two basic methods directly applying diffusion process to language modeling. Compared with recent strong diffusion language models like Difformer (Gao) et al. 2022) and Dinoiser (Ye et al. 2023], which have deployed various effective noise scheduling strategies on diffusion processes from the empirical perspective, our model is still superior with at most 3.07 advancement of BLEU score WMT 16 EN-RO. This implies the effectiveness of modeling discrete priors. In addition, we illustrate the performance of ruto-reeeessive modeling. where we use the transformer (aaswani al. 2017 to rerank the generated sentence candidates (7
Results Performances are demonstrated in Table 1
7

---

Table 3: Analysis on the training objectives.
Objectives
Ex, XX - Xo 1|
Ex, [x( xo)
ūt (xx
Ex, [p(5o E Cx,)]
BLEU
Exp (eq. 24p
8.44
1.56
51.81%
33.42
Lig
8.41
1.55
52.34%
33.49
length beam 3 sentence beams) of our model. The reranked performance can even outperform transformers on IWSLT14 DE-EN and WMT16 EN-RO.
Ablation Our approach is a general frame- work applicable to almost all continuous dif- fusion models, providing them with discrete
Table 2: Ablation studies.
Models
IWSLT14
WMT
Base (Difformer)
31.58
30.08
boundaries as priors. We choose Difformer [Gao
+ forward only
33.02
32.86
et al. 2022] as the base model and follow the configurations. As proved in eq. (19), the orig-
+ forward & reverse
33.42
33.15
Optimal Transport
32.77
33.65
inal forward process will ignore the discrete pri- although explicitly demonstrated. We conduct ablation experiments on the rescaling module. As illustrated in Table 2 our approach rescales the trajectory of both forward and reverse processes on Difformer. Only rescaling the forward trajectory is also effective but sub-optimal due to the inconsis- tent distribution during inference. Due to computational cost and fair comparison, our method leaves room for improvement. For example, replacing the forward trajectory with optimal transport in Flow Matching, u(xo, () = - 1/TT and vxxo, l) = L/T, achieves better performance on WMT16.
Analysis Our training objective, eq. (4 2 an upper bound of the eq. (2). We demonstrate the influence of this approximation in Table on IWSLT14 DE-EN to reveal the thought of our formula. On the one hand, Lx. brings theoretical errors at constant scale. On the other hand, Lx, mitigates some experimental errors from the neural networks. The first row Lx,, is the objective we used in eq. (24) and the second row La. [x,, } [| (
t)) (xx, d& [| is directly derived from the eq. ( The first two columns represent the error expectations of Xo and ū, on the test (((,,,,)
set. It is easy to observe that, with the dynamic coefficient dr T-xxG(xo.e) (appendix F). the di value of xo 's error (8.44) is much larger than the u, 's error (1.56). Therefore. Lx, is beneficral for reducing the impact of the prediction eπтоr from the neural network. The third column in Table B illustrates the one-step accuracy of predicting X₂ and the fourth column is the BL.FU score on test set. Experimental results show that optimizing the upper bound has negligible impact on the final performance (only a 0.2% drop of the BLEU score), while can improve the efficiency of the loss calculation during the training phase.
T
5 Discrete Image Generation
Image pixels are usually treated as real numbers in continuous space since adjacent pixel values exhibit linear continuity. They are essentially discrete and quantized data with finite state space, such as 256 states in RGB format. We utilize two discrete image representations. One is binary coding provided by Bit Diffusion [Chen et al.) 2023b] that converts a sub-pixel with 256 integers to a 8-bit binary code. It is more efficient it stores ordinal relationships, but the representation space it constructs will be sparse. Another is pixel embedding, which is a more discrete form of representation because the relationships between pixels are thoroughly broken down and reconstructed by learning the embedding representation. Each pixel is regarded as a one-hot vector and transformed with an embedding layer EMB as used language. Furthermore, we design an intermediate state to demonstrate the correlation between discreteness and modeling difficulty, which is initializing fixed embedding with binary coding. The optimization target for binary coding is the MSE loss, and pixel embeddings take the same objective as in language.
Experimental Setup
We use CIFAR- 10 Krizhevsky et al., 2009 for discrete image generation.
The evaluation metric is FID [Heusel et al. 2017), which compares 50K generated samples with the training set. Our image generation model is constructed on Bit Diffusion [Chen et al. 2023b), where
the architecture is U-Net [Ronneberger al. 2015 with 3 stages, 256 channels and 3 residual blocks
8

---

(A) Bit Diftusion repro (FID 10.37)
(B) DDIM (FID 4.04)
(C) Ours (FID 3.86)
Figure 3: Generated images Bit Diffusion repro, DDIM, and Ours on CIFAR- 10.
per stage. Diffusion steps are 1000 for both the train-
Table 4: FID scores CIFAR-10 10.
CIFAR-10 (FID 4)
ing and inference stages. The model is trained for 1.5M
Models
200K 500K Final
steps with the learning rate of 1 and batch size of 128.
Since the training script and detailed hyperparameters of
Continuous Pixels
Bit Diffusion are not available, we have to reproduce it by
DDPM
3.17
ourselves and our boundary conditional diffuuson model shares exactly the same configuration. Our confidence
DDIM
4.04
Discrete Ordinal Pixels
factors are = 0.5 for all three settings. Other baselines include D3PM [Asstin et al. 2021] and rLDR [Campbell
D3PM GAUSS
7.34
LLDR-0
8.10
et al. 2022 which are discrete diffusion models. SDDM
LLDR-10
3.74
[Sun et al. 2023 utilizes vector quantization from VQ-
BINARY CODING (UINT8):
GAN[ et al.
2021] continuous space for discrete
data. We also compare with DDPM (Ho et al.] 2020 and
Bit Diffusion
3.48
DDIM [Song al.
2021a continuous pixels.
Bit Diffusion repro 22.12
13.23
10.37
Ours
8.17 5.03
3,86
Results For binary coding, as shown in Table our approach outperforms the reproduced Bit Diffusion and attains competitive results to state-of-the-as models. For pixel embedding where ordinal information is decon- structed and reconstituted, our method exhibits a notable improvement of 3.81 FID score fusion. Moreover, in the case of categorical pixels, this advantage increases 8.25, positioning our approach with trainable embedding as a new state-of-the-art solution. Ad- ditionally, as deterministic diffusion processes, our model with binary coding can slightly exceed the performance of DDIM, where the generated samples are Figure
FIXED EMBEDDING:
Bit Diffusion repro
19.69 16.61
12.96
Ours
12.32 10.09
9.15
Categorical Pixels
D3PM UNIFORM
51.27
replicated Bit Dif-
D3PM ABSORBING
30.97
VECTOR QUANTIZATION:
D3PM-VQ
16.47
LLDR- VQ
40.06
SDDM-VQ
12.23
TRAINABLE EMBEDDING:
Bit Diffusion repm
33.09 27.21
19.26
Analysis We analyze the influence of the confidence factor in Table 5 The factor r is selected from [0, 0.2. 0.3, 0.5), where = 0 is the reproduced Bit Diffusion that discards the discrete priors. As the confidence factor increases, the impact of discreteness gradually improves, simultaneously enhancing the model's performance across all three settings. Since there is no guidance for unconditional image generation, do not use a larger factor to prevent mode collapses.
Ours
21.17 15.32 10.99
Related Work
Table 5: Confidence factors.
Discrete Modeling Auto-regressive models have demonstrated a domination over dis- crete modeling, especially for text generation
Models
= 0 0.2 0.3
0.5
BINARY CODING
10.37 7.39 5.33
3.86
FIXED EMBEDDING
12.96
11.35
10.80
9.15
Vassani al.] [2017] [Boonn al.] [2200]
TRAINABLE EMBEDDING
19.26
15.32
11.56
10.99
Achiam et al. 2023]. However, the computation

---

cost increases drastically as the size of sentence length or the image resolution increases. Diffusion
models DScks-Dickstein et al.) 2015 Ho et al. 2020, Dhhrrwal and Nichol 2021 Saharia al.
2022] can genérate data in parallel, but are tailored for continuous problems. To generalize diffusion models for discrete data, the most straightforward methods define discrete processes in discrete spaces [Sohl-Dickstein et al. [015 Hoogeboom et al. [2021b] Austin et al.) 2021 Camphell et al.)
2022, Zhang et al., 2023 Sun et al. 2023 Lou et al., 2023]. which will be bothered by large number of discrete status. Besides, simplified version of discrete diffusion processes is recently used in
language modeling [He et al. 2023] Chen et al. 2023a Approaches in another line argue to located
discrete data in continuous spaces, which more flexible and efficient, with the mapping functions including binary bits [Chen et al. 2023b] and embeddings
et al. 2022 Gong et al. 2023b a Yuan
et al. 2022 Gulrajani and Hashimoto 2023 Han et al.
2023]. Other generative models adapted
for discrete modeling includes Variational Autoencoders
KKnggma and Welling 2014), Generative
Adversarial Networks (Hjelm al., [2018] Fedus al.
2018), and Normalizing Flows [Lindt and
Hoogeboom
2021
Hoogeboom et al. 2021a a
lan et al.
2022
Diffusion Models with Deterministic Trajectory Deterministic diffusion process is usually used in the inference stage to speed up sampling, where DDIM [Song et al. 2021a] derives a serial of non-Markovian diffusion processes and the deterministic
is special case from this implicit
perspective. Additionally, deterministic diffusion processes can be converted to ordinary differential equations (Song et al.] 2021b), , which is utilized by recent sampling acceleration approaches such as DEIS Zhang and Chen 2023 and DPM-Solvers [Lu et al.) [2022b a Zheng et 2023). Our approach requires deterministic forward trajectory to eliminate the randomness between the boundary point and sampled point. Flow matching Liu 2022 Lipman et al. 2023 Albergo and Vanden-
2023 Liu al. 2023] is a collection of generative models that employ ordinary differential equations to facilitate both forward and reverse processes. They can be regarded as generally equivalent to Diffusion models. Therefore, we extend the framework of flow matching for our method.
7 Conclusion
We studied the gap between discrete modeling and continuous spaces, focusing on the inconsistency between probability density contours learned by continuous diffusion models and discrete boundaries. We have proposed novel and general approach to address this issue by enabling continuous diffusion models to be conditioned on discrete priors, which is achieved via discrete boundary estimation and trajectory rescaling. An important limitation is that our method is designed for continuous diffusion models, where discrete diffusion models constructed specially on the discrete state space would not encounter the problem. However, discrete diffusion models also possess their own shortcomings, and the practical applications of continuous diffusion models are more extensive. We believe that our method has the potential to advance the development of unified and general diffusion models. By bridging the gap between discrete and continuous modeling, we hope to inspire new possibilities for modeling complex systems and phenomena.
Acknowledgements
Bing Qin is the corresponding author of this work, We thank the anonymous reviewers for their insightful comments. This work was supported by the National Natural Science Foundation of China (NSFC) (U22B2059, grant 62276078), the Key R&D Program of Heilongjiang via grant 20227X01A32, the International Cooperation Project of PCL., PC1.2022001 and the Fundamental Research Funds for the Central Universities (Grant No.HIT..CCFF30230188,
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:23033.08774 2023.
Michael Samuel Albergo and Eric Vanden-Eijnden. Building normalizing flows with stochastic interpolants. In The Fleventh International Conference on Learning Representations, 2023. URI.
htttss://ppereevicee
10

---

Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Structured denoising diffusion models in discrete state-spaces. In A. Beygelzimer. Y. Dauphin, Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc., 2020.
Andrew Campbell, Joe Benton, Valentin De Bortoli, Tom Rainforth, George Deligiannidis, and Arnaud Doucet. A continuous time framework for discrete denoising models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances Neural Information Processing Systems, 2022.
Mauro Cettolo, Christian Girardi, and Marcello Federico. WIT3: Web inventory of transcribed and translated talks. In Mauro Cettolo, Marcello Federico, Lucia Specia, and Andy Way, editors, Proceedings of the 16th Annual Conference of the European Association for Machine Translation, pages 261-268, Trento, Italy, May 28-30 2012. European Association for Machine Translation.
Jiaao Chen, Aston Zhang, Mu Li, Alex Smola, and Diyi Yang. A cheaper and better diffusion language model with soft-masked noise. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4765-4775, Singapore, December 2023a. Association for Computational Linguistics.
Ting Chen, Ruixiang Zhang, and Geoffrey Hinton. Analog bits: Generating discrete data using difffusion models with self-conditioning. In The Eleventh International Conference on Learning Representations, 2023b.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. In M. Ranzato, A. Beygelzimer. Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 8780-8794. Curran Associates, Inc., 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. URL httttt::///p:eee/////// nett/foruum.d--iiccFaaayy
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873-12883, 2021.
William Fedus, Ian Goodfellow, and Andrew M. Dai. Maskgan: Better text generation via filling in the In International Conference on Learning Representations, 2018
Zhujin Gao, Junliang Guo, Xu Tan, Yongxin Zhu, Fang Zhang, Jiang Bian, and Linli Xu. Dif- former: Empowering diffusion model on embedding space for text generation. arXiv preprint arXiv:2212.09442, 2022.
Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. Mask-predict: Parallel decoding of conditional masked language models. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP--ICNILP), pages 6112-6121, Hong Kong, China, November 2019, Association for Computational Linguistics.
11

---

Shansan Gong, Mukai Li, Jiangtao Feng, Zhiyong Wu, and Lingpeng Kong. DiffuSeq-v2: Bridging discrete and continuous text spaces for accelerated Seq2Seq diffusion models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 9868-9875, 9875, Singapore, December 2023a. Association for Computational Linguistics.
Shansan Gong, Mukai Li, Jiangtao Feng, Zhiyong Wu, and Lingpeng Kong. Diffuseq: Sequence to sequence text generation with diffusion models. The Eleventh International Conference on Learning Representations, 2023b.
Jiatao Gu, James Bradbury, Caiming Xiong, Victor O.K. Li, and Richard Socher. Non-autoregressive neural machine translation. In International Conference on Learning Representations, 2018
Jiatao Gu, Changhan Wang. and Junbo Zhao. Levenshtein transformer. In H. Wallach, H. Larochelle,
A. Beygelzimer, F. d' d'Alché-Bue, Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Ishaan Gulrajani and Tatsunori Hashimoto. Likelihood-based diffusion language models. In Thirty- seventh Conference on Neural Information Processing Systems, 2023.
Xiaochuang Han, Sachin Kumar, and Yulia Tsvetkov. SSD-LM: cami-eetoregressivve simplex-base -base diffusion language model for text generation and modular control. In Anna Rogers, Jordan Boyd- Graber, and Naoaki Okazaki, editors, Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 11575-11596, Toronto, Canada, July 2023, Association for Computational Linguistics.
Zhengfu He, Tianxiang Sun, Qiong Tang, Kuanning Wang, Xuanjing Huang, and Xipeng Qiu. DiffusionBERT: Improving generative masked language models with diffusson models. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), pages 4521-4534, Toronto, Canada, July 2023. Association for Computational Linguistics.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge a local nash equílibrium. Advances in neural information processing systems, 30, 2017.
Devon Hjelm, Athul Paul Jacob, Adam Trischler, Gerry Che, Kyunghyun Cho, and Yoshua Bengio. Boundary seeking gans. In International Conference on Learning Representations, 2018.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusson probabilistic models. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1840-6851, Curran Associates, Inc., 2020.
Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling, Argmax flows and multinomial diffusion: Learning categorical distributions. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 12465. Curran Associates, Inc., 2021a.
Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling, Argmax flows and multinomial diffusion: Learning categorical distributions. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021b.
Yoon Kim and Alexander M. Rush. Sequence-level knowledge distillation. In Jian Su, Kevin Duh, and Xavier Carreras, editors, Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1317-1327, Austin, Texas, November 2016. Association for Computational Linguistics.
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In International Conference on Learning Representations, 2014.
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. In International Conference on Learning Representations, 2021.
12

---

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori Hashimoto. Diffusion- LM improves controllable text generation. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries, In Text Summariza- tion Branches Out, pages 74-81, Barcelona, Spain, July 2004, Association for Computational Linguistics.
Alexandra Lindt and Emiel Hoogeboom. Discrete denoising flows. In ICML Workshop Invertible Neural Networks, Normalizing Flows, and Explicit Likelihood Models, 2021.
Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In The Eleventh International Conference on Learning Repre- sentations, 2023.
Qiang Liu. Rectified flow: A marginal preserving approach to optimal transport. arXiv preprint arXiv:2209.14577 2022.
Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning generate and transfer data with rectified flow. In International conference on learning representations (ICLR), 2023.
Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion language modeling by estimating the ratios of the data distribution. arXiv preprint arXiv:2310. 16834, 2023.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint 1.00022 2022a.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. DPM-solver: A fast ODE solver for diffusion probabilistic model sampling around 10 steps. In Alice Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022b.
Ali Madani, Bryan McCann, Nikhil Naik, Nitish Shirish Keskar, Namrata Anand, Raphael R Eguchi, Po-Ssu Huang, and Richard Socher. Progen: Language modeling for protein generation. arXiv preprint arXiv:2004.0344 2020.
Ali Madani, Ben Krause, Eric R Greene, Subu Subramanian, Benjamin P Mohr, James M Holton, Jose Luis Olmos, Caiming Xiong, Zachary Z Sun, Richard Socher, et Large language models generate functional protein sequences across diverse families. Nature Biotechnology, 41(8): 1099-1106, 2023.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. Proceedings of NAACL-HLT 2019: Demonstrations, 2019.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Pierre Isabelle, Eugene Charniak, and Dekang Lin, editors, Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguistics.
Niki J. Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In International Conference on Machine Learning (ICML), 2018. URL http://procceddings mlr.
frees/v80/ae arress/v80/parma// parmar 1irrmar18a.. html
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. Proceedings of the IEFF/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR), pages 10684,16695, June 2022.
13

---

Curran Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. Medical Image Computing and Computer- Assisted InterentionnMICCA. 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234-241. Springer, 2015.
Alexander M. Rush, Sumit Chopra, and Jason Weston. neural attention model for abstractive sentence summarization. In Lluís Mårquez, Chris Callison-Burch, and Jian Su, editors, Proceedings the 2015 Conference on Empirical Methods in Natural Language Processing, pages 379-389, Lisbon, Portugal, September 2015. Association for Computational Linguistics.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, Jonathan Ho, David Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. In Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages Associates, Inc., 2022.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Katrin Erk and Noah A. Smith, editors, Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 1715-1725, Berlin, Germany, August 2016. Association for Computational Linguistics.
Jascha
Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Francis Bach and David Blei, editors, Proceed- ings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 2256-2265, Lille, France, 07-09 Jul 2015. PMLR.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Interna- tional Conference on Learning Representations, 2021a.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. bosse- generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021b.
Haoran Sun, Lijun Yu, Bo Dai, Dale Schuurmans. and Hanjun Dai. Score-based continuous-time discrete diffusion models. In The Eleventh International Conference on Learning Representations, 2023.
Ilya Sutskever, Oriol Vinyals, and Quoc Le. Sequence to sequence learning with neural networks. Advances in neural information processing systems, 27, 2014.
Shawn Tan, Chin- Wei Huang, Alessandro Sordoni, and Aaron Courville. Learning to dequantise with truncated flows. In International Conference on Learning Representations, 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Jiasheng Ye, Zaixiang Zheng, Yu Bao, Lihua Qian, and Mingxuan Wang. Dinoiser: Diffused conditional sequence learning by manipulating noises. arXiv preprint MXXi:2332210002 2023.
Hongyi Yuan, Zheng Yuan, Chuanqi Tan, Fei Huang, and Songfang Huang. Seqdiffuseq: Text diffusion with encoder-decoder transformers. ArXiv, abs/2212. 10325, 2022.
Pengze Zhang, Hubery Yin, Chen Li, and Xiaohua Xie. Formulating discrete probability flow through optimal transport. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Qinsheng Zhang and Yongxin Chen. Fast sampling of diffusion models with exponential integrator. In The Eleventh International Conference on Learning Representations, 2023.
Kaiwen Zheng, Cheng Lu, Jianfei Chen, and Jun Zhu. Dpm-solver-v3: Improved diffusion ode solver with empirical model statistics. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
14

---

A Stopping Time for Forward Process
The forward diffusion process X (x,,, 2 0} is markovian stochastic process with transition probability p(x; (xx])x=-)>== /V (x1; -1, (1 0;) I). And stopping time to with respect to X is random time such that for each 2 0, the event {to n is completely determined by the total information known up to time n. (xo....,X.) X.
Suppose the random variables {xn] are in a one-dimensional space and the forward process starts with Xo O. Besides, let A, Xo E A be the discrete area belonging to Xy that for each points in area will be regarded as Xg during data generation. Our expected stopping time is defined as:
to min(n 2 O, X $ A},
which represents the first time x, leaves area A. We can write the probability of stopping time as:
P(to = 0) P(xo & A) 0
P(oo = 1) = P(xu E A, & A)
= N N(xi va1xu, (1 - a1) dxi
XI#A
Ptoo = 2) = P(xo E A,x1 E A, $ A)
P(xo E A,x1 EA P(x2 $ A|x1 A)
- -/ N N(xi (1 (1-a1) I)
N VX22: Va2xi, (1 02) 1) dxi dx2
P(to = =n) P((xo E A...., xn-1 E A, $ A)
II N (xx (1 - 0;) 1) dx
x XSSEE
Since the diffusion process established in continuous space, calculating the probability of the stopping time requires integrating over each intermediate state X1m-1, rather than a simple state transfer as the discrete space. Hence, directly obtain the stopping time is intractable. Additionally, even if we are able to get probability of the stopping time, we can only get distribution over the time dimension, without knowing the exact time of x, leaving area A. Therefore, we need to climinate randomness from the state transition x;-1 x, and find deterministic forward trajectory to estimate the stopping time.
B Properties of Dirac Delta Function
There several useful properties of Dirac delta function:
Symmetry Property: 5(-x) 8(x)
Scaling Property: 5(arr) 5(x)
al
Translation Property: f f(x)(( a)dx f(a)
C Bridging Flow Matching and DDPM
In this work, we utilizes the framework of Flow Matching to model the diffusion processes, where the forward process is defined by flow functions in eq. ( Although having different mathematical forms, essentially equivalent to traditional diffusion processes. Here, we provide an alternative form from the perspective of state transfer Pe (x, x,-)).
15

---

C.1 Deterministic Forward Process
Equation (5) gives the definition pr (x)(xx)) [x))) where d (x) X, Vix. Here we provide the equivalent derivation of Pt (xc (Xo) from the perspective of diffusion processes:
/ pe pe (x)x
Px(x, (X0)
(29)
= -
ff p(x, (x)) II
where p(x)|x0) (N Xo. viI) is the first step of the forward process at which the global noise is introduced into the forward trajectory. The state transfer probability of forward process Po (x,xx/x--111 (x-1,xx)) 8(x, u, 1=45-1(Xx-1)) is Dirac delta function. Therefore,
/ II Po (xa x-- xo dxx2111
px(x)1x0)=
(30)
/ p2(x21>1 xo)p(x (xo)dx1,
Q:
where we denote the integral of x as Q1. Based on
Qo x(1 xn) .V Vuixo, viI) Q2 X2 X1, Xo) = (x2 - 12
V21/1 (x1))
V2
V2u1
X2
X1
112
X0
V1
V1
(31)
V1
V1
X1
X2
111
Xg
V2
V2
(Symmetry Property of Dirac Delta Function)
and the Translation Property of the Dirac delta function, we can calculate Q1 as:
Q
p2(X2 X1, xo) p(x) (x)) dx
8(w-a)
f(z)
x x::X]
(32)
where
V1
Viu2
a X2
+
u1
X0
V2
V2
01 Nugxo I.I.)
Then we can continue the deviation of pr (x,, (xn) as:
/ Qu II P (x, /x, Xo)dxx-t-
(x(x((x0)=
/ Q p. (x, X, xo)
(33)
- pe(xx((x Xx- Qu
Q.- -N vII
Therefore, the probability distribution of x, conditioned on Xo follows a Gaussian distribution (u, v2),, which is the same as in original DDPMs when the coefficient functions are defined as u, Vou and v, /i - ōz. This provides an important benefit that the Flow Matching and diffusion models share the same training procedure.
16

---

C.2
Deterministic Reverse Process
The reverse tranfer probability follows Bayes' rule:
Pr- (x1-1(x0)
p(x, X,, xo) = pe (x1//x/-111 xx,-
Px(X,(X0)
(34)
P (x0)
Vi
v,ul-1
x Xt
Xt-1
u
x0
(x Xo)
Vc-1
Ve-1
Since Dirac delta function has another form of
5(x: = +00, =
(35)
0, +
and xx 0, (x-x(xxx > O, have
p(x--1 Xt., xo) pr (x xo) Pt- (x, (x)
(x X0)
>0
Pt (x, (x0)
Vi
Vi
+ 00 x
X
X1-1 +
11
Xu
(x (x))
V
4-1
Vi
Viul-1
0,
Xt x
X1-1 +
X0
Vi-1
V
Vi-1
Ur
(36)
+ 00, Xt-1
Xt +
11
X0
V1
V₁
Vi-1 x
11V1-1
X1-1 #
11:1
X0
Vi
Vi
Vi-1 X₁
u.Ve-1
X1-1
U1-1
X0
Vi
V
lim N
Vi-1
Xx
+
u111
u.
xo.
o-0
Vt
Vi
C,3
Deterministic Optimization Objective
We first include the derivation of the variational bound for diffusion models provided by Sohl- Dickstein et al. 2015). The probability the generative model assigns to the data is:
p(xo) = / )dx
/ p(xoer) pr (x1:T (xo) dx1:
pr xx)
/ (r(xxrr(x0) (x))
p(Xo:T)
(37)
dx
pr (x1:1 (Co)
= / pr(x1T|x0)p(x
T
pl p(x ((x)) dxi:T.
Pr. (x, xx-1)
17

---

Markovian Diffusion Process
Deternjnistic Diffusion. Process
Matching
Figure 4: We demonstrate the trajectory differences among Markovian Diffusion Process, Determin- istic Diffusion and Flow Matching.
Training amounts to minimizing the negative log likelihood:
L= -- / p(xo) log p(xo)xxx
=- / p(xo) log
/
p(x+-1 (xx))) dxi:r
dxo
pe (xx (x
S - / pr (Xo:r) log p(XT)
p 1 (x:)
dxpe
Pt [x1 X x
T
Pr Xx (1=
(Xoer)
- -logp(x))
log
p( pix,-11x0 xx+)
(x log (ogg(xx/bb (x) + p(XT)
T
log p(x:- Xt: Xo)
-Epr
log
p(x,-1 (2)
2=2
Epr
(x
log pogg((xlle xx)
DRL (p(xp Xx, xo) |p 11(X)-1 (x,))
Le-1
where LT is usually ignored as constant and p(x; -1 (x,) is parameterized with a neural network (x(x to approximate the conditioned probability distributions in the reverse process. Since p(x-11xx, o
lim N Ye-i X + (4-1- - uvv- X1) I'I). the parameterized po (xx--(/Xx)
can take the same form (Xg(Xx, (), I because the Dirac delta function is special case of Gaussian distribution and the KI. divergence of two Gaussians can be simplified. Finally, the training objective for the deterministic diffusion process is divided as:
Cr constant
Co log (xo - xx(x1, 1))
L =
L 1 cl|xo xo(x1, 1)1 +
lim log 0t 0
(38)
a
2
U1
207 u,
V
where the simplified version [xo (xe, 112 is the same DDPMs but with different coefficients.
18

---

D Different Diffusion Trajectories
We illustrate the trajectories different diffusion processes in Figure generation for the Markovian diffusion process is:
The forward and reverse
X = v + v1 aiE;
Voa
at))
Xu + Var az-
X
1- αe
0
X
(39)
α4 [[ a1
v1 a4
E1-1.
The deterministic diffusion process:
X; = Vāxxo
VI ŌGE
V /a C2 1--1
(arii
Xt-1
VI α1
X0
(40)
v1
04-1
X1.
V1 Ctt
The deterministic flow matching with optimal transport:
1 (1
xo T
(41)
Xy-1 -xo + t t-
Xx.
t
E
Details of the Function G
Equation
13 defines the function G(x, c) as the inversion of coefficient function.
Flow Matching
The coefficient u, = - 1/T, where = x (1 u.) Therefore,
G(xo, E) = to T Uty) = T / (1+ J(c,J) f(c.I)
(42)
f(x0, I) f(x0, J)
Diffusion
The coefficient for Variance Exploding Vi = 0o or
where = x
log dn log log
log (66.))f((e.2 - -log do
G(Xo, E) = to Vta = = log V, log To
f(xa. (xxo.T)
log oT log T x
(43)
log log o
For Variance Preserving, the function G(xo, e) is more difficult to calculate since u, ar. where I' ag 02 3r, and B, is also influenced by noise schedulers. This makes G(xo, e) hard to calculate. Fortunately, we can bypass this function and provide the corresponding pseudo code.
F
Details of the Training Objective
The rescaled vector field is calculated as:
dx;
dž, dr dr di
dl
T x G(Xo.E)
u' (xo, T) xo + v' v' T)E
(44)
T r G(xo,E) E) u
19

---

Considering the expectation form of it there is:
Ex, [X, [xo)] = Σ p p(x/|xx0) (5 xxo)
Σ p(x/|x0)) u' (xo.
T x G(xo, E)
(x0, 7) xo + v' v' 5)E
(45)
Σ p(xz|200) u' (x0. 7) xo + v'xo 5)E
u' (xa,
E p(x xx) xo)xo| v' (x0, T)
ü, [x, (xo)).
Therefore, the training objective El ū 2 ce El xol1?, where e the coefficient.
G Code Implementations
Our framework is module constructed on current diffusion models. We demonstrate our kernel part rescale diffusion trajectory with pseudo python code as below:
def rescale diffusion_traaectioiooo (x_0, epsilon, embedding,
labels, alphas_cumprod, timesteps, mode):
#embedding: embedding matrix, f(x,,)=(embeddding * x) [i]
#labels: I
cumpoos list of all u
#timesteps:
#mode: noising or denoising
#1. get f(x,i):
self_dot torch. sum (embedding embedding, dim=-1)
f_x_i [al__do[[labell]]]] labels labels[.... None]
None]
#2. get f(x,j) and f(eps.j):
embedding embedding pereddinggpermuteeeee permute 0)
f_x_j = torch. matmul (x_0, embedding) f_eps_j torch matmul (epsilon, embedding)
#3. get f(x,1) f(x.)): (usually >=0; smaller closer) #filter out f(x,1)-f(xx) with a large positive number 100
fxi mii_minu_fxi (f_x_i (tatterra labels, 100)
#4. get f(eps, i) and f(eps.j) f(eps, 1): (larger more noise)
f_eps_i = torch. @a_tee(ff_es___ -1, labels)
#filter out
with a large negative number -100 sipsj_minus_fepi (f_eps_j - f_eps_i) scatter labels, -100)
#5. get fraction and u_t_0
#mask results outside the support set info_mask =
0) I (fxi miximminss_fx 0)
fraction fix_minus_fjx
@jeps_minus_fit
fraction [info info_mask] = 100
min frac.
(antiion
minimum
#Diffusion Variance Preserving eq. (9)
u_t_ = torch.sqrts s1 / (1 fra_ 2)) None]
#6. rescale timesteps sqrt_alphas_ccmprod = torch torch.sqrta cauphsssccccc)
20

---

###!! important trick!!!##t
#ile do not need to calculate the function G(x_0,t) (eq. (12)).
#Timesteps diffusion processes are discrete and we just iterate over and compare with all coefficient functions. #Besides, function is easy to calculate for Flow Matching.
index torch. sum c qurt_alphas_cumpro dim=-1)
#T is the maximum timestep, for example T=2000. #confactor is the confidency factor #tau is the rescaled timestep
#delta_tau is the rescaled decoding velocity if mode 'noising':
tau = (timesteps + index -
(((timesteps 1)
index)) long clamp (0, T)
tau (confactor * tau. f)oat +
(1.0 confactor) timesteps. float ()) long () clamp (0,
return tau elif mode 'denoising':
delta_tau (T index) T
delta_tau (confactor delta_ tau +
(1 confactor) * 1.0). clamp 1) return delta_tau
21

---

Table 6: FID of difference sampling strategies.
Gaussian
Deterministic
BINARY CODING
13.39
3.86
FIXED EMBEDDING
12.21
9.15
TRAINABLE EMBEDDING
22.24
10.99
H
Analysis
Gaussian Sampling Our framework is com- patible with the Gaussian sampling in DDPM, where random noises can be added into each
Algorithm 3 Gaussian Sampling
1: == T
2: X ~ IO(.)
I/ Initialing Σ>==
iteration step. Algorithm El demonstrates the Gaussian sampling procedure. Compared with algorithm 2 Gaussian noise z (O, I) with decreasing variance is injected to the estimated next state X,- This noise z will be mapped changing the initial sampling Xr through the trajectory alteration step. We illus- trate the deterministic and Gaussian sampling for our model on CIFAR- 10 T where the deterministic sampling can achieve much better performance of FID. We assume this is because our coefficient functions u(xo: () and v (Xo, t) are dynamically calculated to rescale the deterministic trajectory in the training stage. In the inference stage, X6 is replaced by (x,, t), where errors will accumulate if the predicted pseudo target changes frequently. Moreover, Gaussian sampling will further introduce random noises at each reverse step, making our rescaled timestep far away from the training situation. Therefore, errors in the calculations of trajectory scaling will explode over iterations.
3: for A/ == -A1,..., Al. do
4:
II Gaussian Noise II Pseudo Target ? = T ((x;)]] I/ Trajectory Alteration
N=(00,00))
5:
Xn = (x,,)
6:
7:
Ta = -T(t At, G(xo. 2))
1/ eq-
25
8:
u(Xo, TA )x0 v(80; TATTT) ) + // Updating
9: I = 1 Al, A 7A
10: end for
11: == (x t)
1/x1 + X0
return Xo
I Limitations
Our framework is proposed to migrate the powerful continuous diffusion models to discrete problems. There another technical route that directly designs the diffusion process on the discrete state space and our method is not useful for this scenario. However, we believe the continuous diffusion models can be general framework for generative modeling and our effort can advance this target.
We prefer X, as the training target because we highly depend on the reliability of the predicted Xo during inference. Although it is possible to use other targets, the modeling effect will decrease in practical use, which limits the flexibility of diffusion modeling. For example, predicting the é and recovering Xy with eq. (23) is inefficient, because small error in predicting é will be amplified by eq. (23) and lead to the collapse of G(oo, é).
Our approach requires extra computational cost. But they are acceptable since rescaling process a series of parallel matrix computations. Considering that our approach is compatible with the Seff-Conditioning [Chen al. 20236 our overhead negligible when it is used.
J Other Experimental Details
For language modeling, we utilize the model configuration transformer-bwsti-dees in FAIRSBQ framework (Ott et al. 2019 for IWSLT14 DE-EN, which has 6 transformer layers, 4 attention heads, 512 hidden dimensions, and 1024 feed forward layer dimensions. For other datasets, the configuration is monsformer-base which has 6 transformer layers, attention heads, 512 hidden dimensions, and 2048 feed forward layer dimensions. The embedding dimension 128. The beam size is length prediction beam x 5 generation beam, since the length prediction is unstable for diffusion language models. For reranking, we take length prediction beam x 3 generation beam as Difformer let the transformer choose the best one.
22

---

(A) Bit Diffusion repro (FID 10.37)
(B) Ours (FID 3.86)
Figure 5: Generated BINARY CODING images of reproduced Bit Diffusion and Ours on CIFAR-
For image generation, set the scaling factor 0.5 for training. Besides, find that smaller factor for inference is sometime useful. We set = 0.45 on binary coding and r = 0.2 on fixed embedding during inference. When the pixel embedding is learnable, the scaling factor is 0.5, which is the same as training.
Our experiments performed with Nvidia 80G A100. Each language result requires about 2 days on one single A100. Each image result requires about a week on one single A100.
K
Impact Statements
This paper presents work whose goal is advance the field of Deep Learning. The datasets we used has been widely deployed for many years and has basically no negative impact. Our approach is framework that migrates existing diffusion models discrete problems, which does not provide large preitrained model that can be used to generate fake contents.
L
Case Study
Generated sentences on IWSIT14 DE-EN and GIGAWORD are illustrated in Table and Table Generated images on CIFAR- 10 are depicted in Figure 5 6 and 7
23

---

Table 7: Cases of translation IWSLT14 DE-EN.
Target: ENGLISH
Source: GERMAN
Difformer
Ours
Golden
ich möchte ihnen
want to tell you about this
want to tell you how we ve figured that out.
want tell you how we found that out.
erzählen, wie wir das herausgefunden haben. da gingen ganz schön viele verrückte dinge vor sich
lots of crazy things
there were quite lot of crazy things going
there was a whole lot of crazy going on in there
man macht etwas, das eigentlich ein wenig anders ist.
you do something a little different.
you re doing some- thing that actu- ally a little bit differ- ent.
you do something that actually lit- tle different.
und die welt in der wir lebten sah so aus
and the world we lived like this
and the world we lived looked like this
and the world we used to live in looked like this they expect one bil- lion more gamers in the next decade.
man erwartet eine zusätzliche milliarde spieler im nächsten jahrzehnt
you expect an
you expect an extra billion players in the next decade.
next billion players
hat diese vorteile und risiken. was wollen sie tun ?
b has risks. what do you want do ?
b has these benefits and risks, what do you want to do
has these benefits and these risks what do you want to do ?
wir haben also so eine situation, wo je weiter unsere wissenschaft fortschreitet, wir uns mehr eingestehen müssen, dass diese kategorien, die wir für stabile anatomische kategorien gehalten hatten, welche sehr einfache zuordnungen herstellten um dauerhafte
so we have this situation where the continuing our science continues need to admit the more that these categories that we thought were stable anatomical categories. which made a very simple collaborations to create permanent identity ories are much unsharers than we ve as- sumed.
we have a situa- tion where as the science
so what we have is a sort of situation where the farther our science goes the more have to admit to ourselves that these categories that we thought of as stable anatomi- cal categories that mapped very sim- ply to stable identity categories are lot more fuzzy than we thought
further goes on we have to admit in terms the more that these categories that thought of be a sta- ble anatomical cate- gories, which made very simple assa- ments to create per- manent identity cat- egories are much more blanky than we ve accepted.
identiäätskategorien zu schaffen. viel unschärfer sind, als wir angenommen haben..
24

---

Table 8: Cases of summarization GIGAWORD.
Target
Source
Difformer
Ours
Golden
the asian swimming record tumbled again at the seven-day olympic test event here friday.
asian swim- ming record
asian
swim- tumble again
asian swimming
ming
record tumbles
falls again
again china 's
olympic event
test
olympic trials
truck carrying illegal north african immigrants flipped over in northeastern spain, killing ## and injuring six others, police said monday.
truck carrying illegal immi- grants crashes in spain killing ##
## illegal im- migrants killed in truck acci- dent in north- eastern spain new zealand shares close #.## percent lower
## immigrants killed in road ac- cident in spain
new zealand share prices closed # percent lower wednesday after investors took their lead from further weakness in overseas markets. dealers said
new zealand shares fall #.## percent
new shares down #.## percent
zealand close
the sudanese opposition said here thursday it had killed more than ### government soldiers in an ambush in the east of the country.
sudanese opposition
sudanese opposition claims ### soldiers killed in ambush cox news ser- vice sports bud- get
sudanese op- position says #### government
claims over
### soldiers killed
troops killed in
ambush
these sports stories for release tuesday september ## #Hif are moving today to clients of the new york times news service.
thursday
cox news ser- vice tuesday
sports budget
sports budget
bangladesh and india signed a deal here thursday giving green signal to resumption passenger train service between the neighboring countries after ## years.
bangladesh in- dia sign agree- ment on train service
bangladesh in- dia sign agree- ment to resume train service
bangladesh india sign agreement for resumption of train service after ## years
(A) Bit Diffusion repro (FID 12.96)
(B) Ours (FID 9.15)
Figure 6: Generated FIXED EMBEDDING images of reproduced Bit Diffusion and Ours CIFAR-10.
25

---

(A) Bit Diffusion repro (FID 19.26)
(B) Ours (FID 10.99)
Figure 7: Generated TRAINABLE EMBEDDING images of reproduced Bit Diffusion and Ours on CIFAR-10. 10.
26

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Contributions and scope are in abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Discussed (7) Conclusion and (H) Limitations sections.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
27

---

Justification:
From sections (A) to (E) in appendices.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced
statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We use algorithms I and to to demonstrate how to reproduce our algorithm. We provide a paragraph of Experimental Setup in (4) Language Modeling and (5) Discrete Image Generation sections. Other details are in section (I) and pseudo code of our kernel process (F).
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
28

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The pseudo code of our kernel process demonstrated in (F) and will public our code on github.com.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr cc/
Sublic/guides http//ooosuuiiisiooolo.........e
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed reproduce the results. See the NeurlPS code and data submission guidelines (https
cttpss.cccpubblci
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide a paragraph Experimental Setup in (4) Language Modeling and (5) Discrete Image Generation sections. Other details are in section (I), We provide ablation studies on the hyperparameters.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Error bars
reported because it would be too computationally expensive. Each result in the experiment table needs to be run on an 80G A100 for at least 2 days. The huge overhead required to obtain statistically significant error bar makes it impossible for us to achieve it.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
29

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Details are in section (I).
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Discussed in section (J).
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
30

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our approach is framework involves algorithms but not pre-traee pre-trained models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We provide the link or citation of each asset, where licenses are in the link.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
31

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
No new assets introduced.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: No crowdsoureing.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No crowdsourcing.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
32


Reviews:
[{"review_id": "c6PKS0wCDE", "reviewer": "Reviewer_ooCU", "summary": "The paper discusses a novel approach to discrete modeling using boundary conditions and diffusion processes. The primary contributions include:\n- Development of a framework for discrete image generation that incorporates binary coding and pixel embedding.\n- Introduction of an intermediate state to illustrate the correlation between discreteness and modeling difficulty.\n- Extensive experimental evaluation on datasets like CIFAR-10, demonstrating competitive results compared to state-of-the-art models.\n- Proposal of a new method for stochastic processes that improve model performance, especially in cases of categorical pixels.", "questions": "See weakness.", "limitations": "Limitation is mentioned in one sentence in the conclusion section, but not comprehensively discussed.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Originality: The paper introduces a unique method for addressing discrete modeling through a combination of binary coding and pixel embedding, which is innovative in the context of diffusion models.\n- Quality: The experimental setup is robust, with detailed evaluation metrics and comparisons to existing methods. The results show significant improvements, indicating the effectiveness of the proposed approach.\n- Clarity: The paper is well-structured, with clear explanations of the methodology and the underlying mathematical formulations. Figures and tables are used effectively to illustrate the results.\n- Significance: The approach has broad applicability in various fields requiring discrete data generation, such as image and language modeling. The improvements in FID scores and other metrics highlight the potential impact of this work on the research community.", "weaknesses": "1. The approach is primarily tested on specific datasets like CIFAR-10. Expanding the evaluation to a wider range of datasets on high-resolution images could provide a more comprehensive assessment of the model's generalizability.\n2. Language model experiments are mainly on translation tasks. Could you provide experiments on language modeling tasks, which is more common for benchmarking diffusion LMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your meticulous review and the valuable comments. Please kindly find our response below.\n\n**Q: Expanding datasets for images and languages**\n\n1) For both language and image tasks, our framework is constructed on our baselines, Difformer and BitDiffusion, with the same configurations. We strictly follow their benchmarks and datasets to ensure a fair comparison.\n\n2) Our experiments across languages and images may have reflected the generalizability of our framework.\n\n3) Since training the image model takes a week and training the language model takes at least two days (lines 719-720), we do not have enough computing resources to scale to a larger and more complex benchmark.\nWe sincerely accept your suggestions and will try to expand our method to larger benchmarks when we have enough computing resources.\n\n4) We try to conduct the experiment on ImageNet 64, but our computing resources are currently insufficient to complete it. We show the temporary results of the first 100K steps as follows.\n| Binary Coding | FiD |\n|  ----  | ----  |\n| BitDiffusion  | 47.82 |\n| Ours  | 31.68 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gWdHCd1M5h", "reviewer": "Reviewer_m5hg", "summary": "A discrepancy exists when using continuous diffusion models to model discrete data, which has not been sufficiently addressed in past methods. This paper redesigns the forward and backward processes of the diffusion model to eliminate this issue. Experiments on translation tasks and CIFAR-10 effectively validate the proposed method's effectiveness.", "questions": "1. In Sec. 4, the authors present an untrainable $p(W|x_0)$ given the embedding layer. If we employ a trainable $p_{\\theta}(W|x_0)$ (e.g., Diffusion-LM [1]), will the discrepancy between discrete data and continuous modeling still exist? If it does, will your proposed method still bring improvements in this scenario?\n\n2. This paper presents the results of translation tasks. I am curious to know whether the proposed method would be effective for models like Plaid [2], which can compute text likelihood. The discrepancy between discrete data and continuous modeling mainly occurs during the step of decoding the embedding into text, while the likelihood calculation does not require this decoding step.\n\n3. In Table 3, the authors report FIDs of 51.27 and 30.97 for D3PM. However, the best generative FID on CIFAR-10 in the original D3PM paper is 7.34. Why didn't the authors report this result as a baseline? This requires further explanation.\n\n[1] Diffusion-LM Improves Controllable Text Generation\n\n[2] Likelihood-Based Diffusion Language Models", "limitations": "yes", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The writing is clear and concise.\n\n2. This paper aims to address an important and interesting question.\n\n3. Experiments demonstrate that the proposed method surpasses all baselines, effectively validating its effectiveness.", "weaknesses": "Please see the Questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your meticulous review and the valuable comments. Please kindly find our response below.\n\n**Q1: The embedding layer**\n\n1) We want to clarify that the embedding layer we use is actually trainable, as illustrated in Lines 200 and 208. The $\\theta$ is only used to denote the diffusion parameters in our paper. \n\n2) The discrepancy between discrete data and continuous modeling exists for both fixed and trainable embedding layers.\n\n3) Besides, we are fully aware that it has been demonstrated in Diffusion-LM that a trainable embedding layer achieves better performance than the fixed one. \n\n**Q2: Application scenarios of our framework**\n\n1) There seems to be an embedding layer in Plaid as well (Section 3.1 in Plaid), so our framework is theoretically applicable. Likelihood calculation is also a similarity function that has the same effect as the dot product in our work, which is compatible with the $f(\\mathbf{x},\\mathcal{I})$ framework we defined.\n\n2) Besides, as illustrated in Lines 28-30, our framework is specifically designed for continuous diffuison models, where the discrete diffusion process with the discrete state space will not face the problem in Figure 1A and Lines 38-41. Therefore, our framework will extend the generality of the widely used continuous diffusion processes, but does no help for discrete diffusion processes.\n\n**Q3: FiD of D3PM**\n\nWe want to clarify that we have reported the original D3PM in Table 3 with the FiD of 7.34, first line (D3PM GAUSS) in the part of Discrete Ordinal Pixels.\n\nAs demonstrated in Table 3 and Section 5, we use different type of image embeddings. Different embedding type possesses different level of continuous information. The D3PMs of 51.27 and 30.97 FiDs are the UNIFORM and ABSORBING versions with Categorical Pixels.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "v7czXFft9L", "reviewer": "Reviewer_uQEt", "summary": "In this study, the authors propose a framework to address the discrepancy between discrete data and continuous modeling in diffusion processes. The approach involves a two-step forward process that first estimates the boundary as a prior distribution, then rescales the forward trajectory to build a boundary conditional diffusion model. The reverse process is proportionally adjusted to ensure the learned contours yield more accurate discrete data. Experimental results show the method's strong performance in language modeling and discrete image generation tasks, outperforming previous models in several areas and establishing a new state-of-the-art for categorical image generation on the Cifar-10 dataset.", "questions": "- Is there a possibility that the boundaries of the discrete areas of two distinct data points might intersect? If so, why would pushing the noise to the boundary be beneficial? Could certain perturbations potentially lead to inaccuracies in the final result?\n- According to Equation (14), shouldn't the input of $\\Psi^{-1}([\\cdot,\\cdot])$ be the boundary point and the \"crossing boundary time\"? If so, why does line 8 of Algorithm 2 suggest that a sequence of $\\tau$ ranging from $T$ to $r\\times t_0$ can serve as the input of $\\Psi^{-1}([\\cdot,\\cdot])$?\n- also please refer to weaknesses.\n\nIf these questions are addressed, I would consider revising the score.", "limitations": "The authors do acknowledge the limitations of their work, which is commendable. Furthermore, I recommend conducting theoretical analysis, such as exploring the convergence properties of their new framework.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The authors have conducted comprehensive experiments, providing thorough and complete details in their exploration. \n- The novelty of the study lies in its discussion of the concept of \"discrete area\", presenting a fresh perspective in the field.", "weaknesses": "- The explanation as to why this new framework can enhance performance is not clearly articulated in the study. \n- From my understanding, Algorithm 1 appears to be equivalent to training original diffusion models with $t\\sim U[t_0,T]$ as opposed to $t\\sim U[1,T]$.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your meticulous review and the valuable comments. Please kindly find our response below.\n\n**Q1: Why can our framework enhance performance**\n\n1) **From the perspective of Motivation**\n\nAs illustrated in Introduction and Figure 1, we find that the continuous diffusion processes oversimplify the objective of discrete modeling. The discrete data are treated as single points, while they are actually areas in the diffusion space. This means diffusion models learned on the oversimplified objective generate the sub-optimal probability density contour, which is inconsistent with the discrete priors (Lines 38-41 and Figure 1A). Therefore, **our method takes the discrete prior into consideration and design a more appropriate diffusion trajectory for discrete modeling problems** (Lines 42-65 and Figure 1B).\n\n2) **From the perspective of Methodology**\n\nWe theoretically derive the diffusion process conditioned on the discrete priors and obtain the sampling distribution of the corresponding forward process $\\tilde{p}_t(\\mathbf{x}|\\mathbf{x}_0)$ (Equations 21 and 22). It is easy to calculate that the $\\tilde{p}_0(\\mathbf{x}|\\mathbf{x}_0)$ is exactly the discrete boundary and $\\tilde{p}_T(\\mathbf{x}|\\mathbf{x}_0)$ is a Gaussian distribution. During the inference stage, i.e., reversing the time step from $T$ to $0$, the density probability will gradually be consistent with the contour when approaching the boundary, which is inline with Lines 46-51 and Figure 1B. Hence, **our method will precisely guide the random noise into the discrete area.** Figure 2 demonstrates sample points exactly calculated by our proposed method (Equation 22) and the probability contours calculated by our method can be easily adapted to different discrete boundaries and noising trajectories, which are inline with our expectations in Figure 1 of the Introduction Section.\n\n3. **From the perspective of Experiment**\n\nOur framework is constructed on Difformer and BitDiffusion with the same configurations for both language and image generation tasks. As in Table 1 and 3, **our method demonstrates significant improvements over them**.\nBesides, ablations in table 4 reveal that, with the increase of the confidence factor $r$, i.e., increasing the influence of the discrete prior, the discrete modeling performance improves in all aspects.  \n\n**Q2: Algorithm 1**\n\nYou can consider $t\\sim U[t_0,T]$. However, $t_0$ is not a constant but a random variable $t_0=G(\\mathbf{x}_0,\\boldsymbol{\\epsilon})$ as illustrated in equation 12.\n\n**Sampling the $t$ from $U[1,T]$ and then transform it to $\\tau$ with $\\mathcal{T}(t,G(\\mathbf{x}_0,\\boldsymbol{\\epsilon}))$ is equivalent to directly sampling $\\tau$ from $U[G(\\mathbf{x}_0,\\boldsymbol{\\epsilon}), T]$.** We choose the former procedure because it is easy to perform in parallel. Algorithm 1 is just a commonly used element substitution technique in sampling.\n\n**Q3: Boundaries of discrete areas**\n\n*Due to the character limit of the rebuttal, this is only an outline of our answer. Please refer to the comment for more details.*\n\n1. **The discrete area of two distinct data points will not intersect.**\n\nAs illustrated in Lines 34-37 and Equation 6, points in the discrete area will not be attributed to any other area by definition.\n\n2. **Boundaries of two areas can overlap with the confidence factor $r=1$, but this is impossible when $r<1$.**\n\nWhen the confidence factor $r=1$, as illustrated in Equations 7 and 8, the boundary is where the probabilities of points belonging to the neighbour areas equal. When the confidence factor $r$ decreases, the range of each discrete area is gradually reduced and there is no overlap between boundaries.\n\n3. **Pushing the noise to the boundary is beneficial.**\n\n- **Empiricaly**, previous work like Difformer and Dinoiser and our Table 1 have revealed that **increasing the minimal noise scale can benefit the performance** (Lines 217-219). And it is obvious that **pushing the noise to the boundary is precisely what increases the minimal noise scale**.\n\n- **Theoretically**, probability density inside the discrete area can be ignored.\nBased on the observation of the discrete area that any point inside this area will be attributed to the corresponding discrete data (Lines 34-37), we can easily get the derivation: **we don't need to model the probability density inside the discrete area because our diffusion model only needs to guide a noise to get into this area, where the accurate end position doesn't matter.** Therefore, pushing the noise to the boundary does not cause theoretical damage to the probability modeling. And the dffusion model can be less disrupted by invalid data, i.e., points inside the area, during the training stage.\n\n\n4. **Certain perturbations can lead to inaccuracies in the final result without our confidence factor.**\n\nWe use the confidence factor $r$ (Lines 137-147) to control the influence of perturbations. When we set a smaller $r$, the learned reverse process will be robust to perturbations.\n\n**Q4: Input of function $\\Psi^{-1}([\\cdot,\\cdot])$**\n\n**The input of $\\Psi^{-1}([\\cdot,\\cdot])$ is valid for any arbitrary pair of point $\\tilde{\\mathbf{x}}_t$ and time $\\tau$.**\nFunctions $\\Psi$ and $\\Psi^{-1}$ in Equation 14 are directly extended from Equations 4 and 5, which construct the invertible relationship between any point-time pair ($\\tilde{\\mathbf{x}}_t$-$\\tau$) and their initial noise $\\boldsymbol{\\epsilon}$. \n\nIn Equation 14 of Section 3.1, we take the boundary point $\\mathbf{x}_{t_0}$ and the crossing boundary time $t_0$ pair as input because we just want to estimate this boundary. However, this does not mean the input of $\\Psi^{-1}$ can only be the pair of boundary point and crossing boundary time, while any other point-time pairs are still valid. This is why we use $\\Psi^{-1}$ to update the trajectory in Line 8 of Algorithm 2.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sELXraQ8iy", "reviewer": "Reviewer_1VFK", "summary": "The authors propose a discrete diffusion model that operates in the embedding space of vectors that represent discrete tokens. They propose a method that takes into account the fact that whole regions of the space will decode to the same discrete token because during decoding, the closest embedding vector is selected. The method is that at training time, instead of interpolating between $x_0$ and $x_1$, they interpolate from $x_{t_0}$ to $x_1$ where $x_{t_0}$ is the point at which the line between $x_0$ and $x_1$ intersects the 'discrete boundary' of $x_0$ which is the region of embedding space that all decodes to $x_0$. The authors test their method on language tasks as well as CIFAR-10 image modeling.", "questions": "Here I have listed some more minor questions for the authors that can help improve the exposition. My main concerns are given in the previous discussion.\n\n- How does your formulation handle the fact that for some $x\\_0$ and $\\epsilon$ samples, there will never be a boundary crossing because $x\\_0$ may be on the outside of the space or $\\epsilon$ is sampled within $x\\_0$'s discrete boudary.\n- On L187 (24) is already deterministic because its an ODE so I don't understand why you then propose a deterministic alternative.\n- In Table 2 it is unclear what ablation you are actually doing, what does it mean to only rescale the forward process versus rescaling the forward and backward?", "limitations": "I don't believe the authors adequately describe the limitations of their method when r is near 1, they say that it can be unstable but provide no further details or explanation regarding this failure case.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The idea of trying to exploit the fact that the embedding space has a very specific structure consisting of volumes that all decode to the same discrete token is interesting and novel.\n\nThe experimental results look quite promising with good performance compared to other discrete diffusion approaches on machine translation and text summarization.\n\nThe authors are working on a significant problem as finding diffusion methods that can perform on par with autoregressive models for discrete modeling would be very impactful.", "weaknesses": "The paper's main weakness is lack of clarity and precision in the description of the method. I really struggle to understand what the authors are doing exactly and how it is justified.\n\n- It is not clear to me that your training objective and sampling regime are justified. The flow matching paradigm is to construct a $p_t(x_t | x_0)$ probability path and then find a $u_t(x_t | x_0)$ vector field that generates this probability path through the transform defined by ODE integrating along the $u_t(x_t | x_0)$ vector field. The final step is to say that a vector field that generates the unconditional probability path $p_t(x_t)$ is equal to $\\mathbb{E}\\_{p(x_0 | x_t)} [ u_t(x_t | x_0) ]$ and this can be learned tractably with an objective of $\\mathbb{E}\\_{p(x_0) p(x_t | x_0)} [ || v_t^\\theta(x_t) - u_t(x_t | x_0) ||^2 ]$. However, I don't see how your method fits in this framework. This is for two reasons: 1) at no point do you define the conditional vector field $\\tilde{u}\\_t(\\tilde{x}\\_t | x\\_0)$ that generates your rescaled probability path $\\tilde{p}\\_t(\\tilde{x}\\_t | x\\_0)$. How are you able to generate along these probability paths without defining $\\tilde{u}\\_t(\\tilde{x}\\_t | x\\_0)$ ? 2) You sample using $u\\_t(x\\_t | x\\_0)$ for the original $p\\_t(x\\_t | x\\_0)$ but with the $\\hat{x}\\_0$ prediction from the neural network. This is valid in the original flow matching case because $u\\_t(x\\_t | x\\_0)$ is linear and so the $\\mathbb{E}\\_{p(x\\_0) p(x\\_t | x\\_0)} [ || v\\_t^\\theta(x\\_t) - u\\_t(x\\_t | x\\_0) ||^2 ]$ objective can be re-arranged into $x\\_0$ prediction $\\mathbb{E}\\_{p(x\\_0) p(x\\_t | x\\_0)} [ || x\\_\\theta(x\\_t , t) - x\\_0 ||^2 ]$. However we don't necessarily know if $\\tilde{u}\\_t(\\tilde{x}\\_t | x\\_0)$ that generates $\\tilde{p}\\_t(\\tilde{x}\\_t | x\\_0)$ is linear so we can't be sure it is valid to re-arrange to $x\\_0$ prediction.\n- I think after the method is explained in section 3, the authors should have text that revisits the probability contour intuition they provide during the introduction. I think this is a nice mental model to understand the method but it is very unclear how exactly the method presented in section 3 gives you the desired probability contours that are promised in the introduction. For example, if a contour is very long in embedding space, then definitely the far away points on the contour with have low probability under $\\tilde{p}\\_t(\\tilde{x}\\_t | x\\_0)$ so therefore probability is not constant along the discrete boundary and therefore the discrete boundary does not line up with the probability contour. These kinds of doubts would be alleviated with a clearer presentation linking back to the original probability contour view and some more precise statements.\n- The discussion relating to the confidence factor is very unclear. Firstly, I'm unsure why it is even called a confidence factor, we can be fully confident that our data is discrete so why is there uncertainty about this? It is also concerning that the method can oscillate and mode collapse when r=1 which is your full method. This sounds like a major flaw of the approach if it cannot work at all when your new method is used in isolation. This warrants more discussion regarding this failure case. Furthermore L195 should be explained more as it is very unclear what \"fixing the initial path\" means and it sounds very important if it is required for your method to work.\n\n\nRegarding the experiments:\n- It seems unfair to allow your method to be re-ranked in Table 1 if you don't also re-rank the transformer baseline\n- In Table 3, it is hard to get a good sense of how your method compares to other discrete diffusion approaches. Since they all use different embedding strategies it seems that it is only a good comparison to compare methods within the same embedding strategy. In this case, you are really only comparing to bit diffusion but your reproduction seems to perform much worse than the originally stated results in the bit diffusion paper. It is therefore difficult to place your method with confidence amongst the other discrete diffusion methods.\n\n\n\nMy score is given on the basis of the very unclear presentation meaning I cannot understand the method the author's are proposing. If this can be cleared up in the rebuttal period, I am happy to raise my score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your meticulous review and the valuable comments. Please kindly find our response below.\n\n**Q1: The rescaled vector field and probability path**\n\n*Due to the character limit, this is only an outline. Please refer to comments for more details.*\n\n1. Our neural network predicts $\\mathbf{x}_0$ not $\\tilde{u}_t(\\tilde{\\mathbf{x}}_t|\\mathbf{x}_0)$.\n\n2. $\\tilde{u}_t(\\tilde{\\mathbf{x}}_t|\\mathbf{x}_0)$ is tractable but we did not derive it because it is not used.\n\n3. We generate the probability path with the deterministic reverse diffusion process (Equation 25).\n\n4. The training objective $||\\mathbf{x}_0-\\mathbf{x} _\\theta||$ is valid. Derivations in Comments.\n\n**Q2: Method and the probability contour**\n\n*Due to the character limit, this is only an outline. Please refer to comments for more details.*\n\n1. We have demonstrated the rescaled probability contours of $\\tilde{p}_t(\\tilde{\\mathbf{x}}_t|\\mathbf{x}_0)$ in Figure 2, where sample points are exactly calculated by our proposed method (Equation 22).\n\n2. $\\tilde{p}_t(\\mathbf{x}|\\mathbf{x}_0)$ is a series of functions, where the consistency to the boundary contour is inversely proportional to the subscript $t$.\n\n**Q3: Confidence factor**\n\n*Due to the character limit, this is only an outline. Please refer to comments for more details.*\n\n1. The confidence factor is named after some thoughts of sampling, such as the confidence interval. Our data is discrete and the uncertainty comes from the learned diffusion model.\n\n2. The mode collapse when $r=1$ comes from two part: One is the uncertainty of the learned diffusion model as described above. The other involves a traditional problem of diffusion processes.\n\n3. Fixing the initial path means the Trajectory Alteration (Line 8 in Algorithm2) is optional in practice. This is a negligible detail of our model.\n\n**Q4: Re-ranking**\n\nThis is a fair comparision because beam searching the generated results of the transformer is re-ranking with transformer. \n\nRe-ranking is a traditional strategy widely used in Non-autoregressive translation[7,8]. We use the generated results of our method re-ranked by the transformer to demonstrate their upper bound. Since the generation of diffusion LMs are highly random, their own probability scores often cannot reflect the upperbound of their performance. Re-ranking the generated results with an auto-regressive model can better demonstrate the potential capability of the diffusion model.\nBesides, re-ranking the transformer baseline just gets what it generates. Because **beam search is exactly re-ranking the generated contents with the transformer's probabilities themselves.**\n\n**Q5: Table 3**\n\n1. **We believe it is a reasonable comparision because we strictly follows the setting of BitDiffusion[9] as in the Table 1 of their paper.**\nWe want to clarify that the embedding strategy is only valid for continuous diffusion models, where there is not embedding stage for discrete diffusion models.\nWe compare the continuous diffusion models and the discrete diffusion models under the same level of continuous information, Continuous Pixels, Discrete Ordinal Pixels, and Categorical Pixels, where the latter the less continuous information models can obtain.\n\n2. **We have clarified in Lines 285-287 that since they only provide the model code without a training script with detailed hyperparameters or a checkpoint, we have to reproduce with exactly the same configuration based on their code and paper.** We have done our best effort to reproduce the results and we promise to release our code and reproduction. Besides, we achieve the best reproduction result in the open source community.  We are confident in the reliability of our experimental results and hope to address the reviewer's concerns.\n\n3. **Even ignoring the above issues, our method still shows effectiveness.** Considering the gap introduced by the continuous information that the original BitDiffusion (Fid 3.48) can not beat DDPM (Fid 3.17) under the Gaussian sampling, our method (Fid 3.86) achieves improvement over DDIM (FiD 4.04), which are both deterministic diffusion processes. This means that our approach can achieve better results with less continuous information.\n\n**Q6: Samples in the boundary**\n\n1. **It is possible that a random noise $\\boldsymbol{\\epsilon}$ is sampled within $\\mathbf{x}_0$'s discrete boundary.** However, the probability of this happening in a high-dimensional representation space is very low. According to our statistics during the training phase, this probability is less than 0.1%.\n\n2. **Our solution is demonstrated in the pseudo code (Line 639, Appendix F), where we mask out this situation when $f(\\boldsymbol{\\epsilon},\\mathcal{I})>f(\\boldsymbol{\\epsilon},\\mathcal{J})$.** This means that random noise $\\boldsymbol{\\epsilon}$ sampled within $\\mathbf{x}_0$'s discrete boundary will not be rescaled and its trajectory is the same as original DDPM.\n\n**Q7: Line 187**\n\nSimilar to 3.2 in Q1, as demonstrated in Lines 186-189, we provide an alternative for the ODE decoding, which is a deterministic reverse process. **The word 'deterministic' is used to reflect the difference between our approach and traditional Gaussian process in DDPM**, because both of them model the reverse process.\n\n**Q8: Ablation in Table 2**\n\n1. **First three lines in Table 2 reveal where the performance improvement of our method mainly comes from.**\nOnly rescale the forward process means we just train the diffusion model with the rescaled trajectory and reverse the trajectory as in the DDPM. Rescaling the forward and backward means we use the rescaled trajectory for both training and inference. The results in Table 2 shows that training with the rescaled trajectories makes a major contribution to the final performance of our method.\n\n2. **Last line in Table 2 reveals that our method is robust for different trajectory types that optimal transport trajectory used in Flow Matching can sometimes achieve better results.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The authors present a model that adapts continuous diffusion processes to discrete modeling. The proposed method is focused on injecting guidance for discrete boundaries. The approach is validated on vision and language tasks. 

The reviewers agreed on the novelty of the presented approach compared to reference methods. Two of them raised some theoretical concerns, but the authors addressed them in detail in the rebuttal stage. There are also some concerns regarding the experimental part of the work, especially for images where Cifar10 is used. However, the results for language applications and the novelty of the work are satisfactory enough, and the paper should be accepted.
Decision: Accept (poster)