Title: Discrete Modeling via Boundary Conditional Diffusion Processes
Venue: NeurIPS 2024
Accepted: yes
Keywords: Diffusion Models

Discrete Modeling

Language Generation

Discrete Image Generation
Abstract: We present an novel framework for efficiently and effectively extending the powerful continuous diffusion processes to discrete modeling.
Previous approaches have suffered from the discrepancy between discrete data and continuous modeling.
Our study reveals that the absence of guidance from discrete boundaries in learning probability contours is one of the main reasons.
To address this issue, we propose a two-step forward process that first estimates the boundary as a prior distribution and then rescales the forward trajectory to construct a boundary conditional diffusion model.
The reverse process is proportionally adjusted to guarantee that the learned contours yield more precise discrete data.
Experimental results indicate that our approach achieves strong performance in both language modeling and discrete image generation tasks.
In language modeling, our approach surpasses previous state-of-the-art continuous diffusion language models in three translation tasks and a summarization task, while also demonstrating competitive performance compared to auto-regressive transformers. Moreover, our method achieves comparable results to continuous diffusion models when using discrete ordinal pixels and establishes a new state-of-the-art for categorical image generation on the Cifar-10 dataset.

Discrete Modeling via Boundary Conditional Diffusion Processes

Abstract

We present an novel framework for efficiently and effectively extending the powerful continuous diffusion processes to discrete modeling. Previous approaches have suffered from the discrepancy between discrete data and continuous modeling. Our study reveals that the absence of guidance from discrete boundaries in learning probability contours is one of the main reasons. To address this issue, we propose a two-step forward process that first estimates the boundary as a prior distribution and then rescales the forward trajectory to construct a boundary conditional diffusion model. The reverse process is proportionally adjusted to guarantee that the learned contours yield more precise discrete data. Experimental results indicate that our approach achieves strong performance in both language modeling and discrete image generation tasks. In language modeling, our approach surpasses previous state-of-the-art continuous diffusion language models in three translation tasks and a summarization task, while also demonstrating competitive performance compared to auto-regressive transformers. Moreover, our method achieves comparable results to continuous diffusion models when using discrete ordinal pixels and establishes a new state-of-the-art for categorical image generation on the CIFAR-10 dataset.

1 Introduction

Discrete modeling is essential due to the natural prevalence of discreteness in numerous domains, including proteins [Madani et al., 2020, 2023], images [Parmar et al., 2018, Dosovitskiy et al., 2021], and natural language [Sutskever et al., 2014, Brown et al., 2020]. Recent dominant framework for discrete modeling is the Transformer [Vaswani et al., 2017] with an autoregressive manner. While achieving impressive performance, it does suffer from a slow step-by-step generation process, especially for long sequences. Continuous Diffusion models [Sohl-Dickstein et al., 2015, Ho et al., 2020], on the contrary, exhibit the ability to recover high-dimensional data from noise in parallel with limited iteration steps. Although proved to be effective in continuous data generation [Rombach et al., 2022, Kong et al., 2021], they continue to encounter challenges in discrete modeling [Austin et al., 2021, Chen et al., 2023b, Li et al., 2022, Gong et al., 2023b].

In this paper, we reveal a significant discrepancy pertaining to the modeling of discrete data using continuous diffusion models. Current approaches represent a discrete sample with a vector point in the continuous space. The diffusion process learns a neural network to model the probability distributions that recovers this continuous point from Gaussian noise. However, the discrete data actually corresponds to an area in the continuous space rather than a single point, where the oversimplified assumption leads to a mismatch between learned probability contours and the boundary of the discrete area. Take language generation as an example, a word is represented with an embedding vector in the embedding space. To generate this word, it is impractical to strictly enforce the predicted vector to be an exact match to the embedding. On the contrary, vectors around this embedding can also generate

the same word, thereby defining the collective area they encompass as a discrete area of this word. As illustrated in Figure 1A, suppose the learned probability density function is p θ ( x ) and two points x i and x o are sampled in the same density contour where p θ ( x i ) = p θ ( x o ) . It is obvious that x i lies in the discrete area and is able to recover the discrete data while x o can not. This means that the diffusion model only learns a simplified scenario that does not match the real probability distribution.

⇐

(

A

)

(

B

)

⇐

x

0

⇐

∇

p

(

x

|

x

0

)

∇

p

θ

(

x

)

x

i

x

o

x t , we utilize the Ordinary Differential Equations (ODEs) to describe the forward trajectory.

Discrete Area x 0 To address the issues above, we proposed to take the boundaries of discrete areas as priors, as shown in Figure 1B, where boundary curves are regarded as oracle contours. As it gradually approaches the discrete boundary, the learned density contours of diffusion models are expected to transform from Gaussian distributions to the boundary distribution. Therefore, we propose to divide the forward process into two steps. First is the boundary estimation where we precisely calculate the stopping time t 0 and position x t 0 at which the forward trajectory cross the boundary. Then we rescale the trajectory for both training and inference stages to make the sampling probability of noisy point x t conditioned on the boundary. To make the boundary estimation tractable (appendix A) and eliminate randomness in conditional state transitions x t 0 →

Our approach is experimented in both language modeling and discrete image generation. On three machine translation datasets (IWSLT14 DE-EN [Cettolo et al., 2012], WMT14 EN-DE, WMT16 EN-RO) and a text summarization dataset (GIGAWORD [Rush et al., 2015]) for language modeling, our proposed approach not only significantly improves existing diffusion models to at most 7 . 8% but also achieves competitive performance to autoregressive transformers. For image generation on CIFAR-10 [Krizhevsky et al., 2009], our model realizes a comparable result to continuous diffusion models with discrete ordinal pixels and establishes a new state-of-the-art for categorical pixels.

2 Preliminaries

Diffusion Models To model a real distribution q ( x 0 ) , diffusion models utilize a forward process p t ( x | x 0 ) with T steps to gradually add Gaussian noise π ( x ) = N ( 0 , I ) into the data distribution, where p T ( x | x 0 ) = π ( x ) . There are different architectures for the forward process. A common approach [Ho et al., 2020] considers the forward process as the Markovian process, where p t ( x | x 0 ) = ∏ t s =1 p s ( x s | x s -1 ) combines a series of Gaussian distributions. Thus the forward process follows a Gaussian distribution that p t ( x | x 0 ) = N ( √ ¯ α t x 0 , (1 -¯ α t ) I ) (Variance Preserving) or p t ( x | x 0 ) = N ( x 0 , σ 2 t I ) (Variance Exploding) [Song et al., 2021b], where noise scheduler ¯ α t monotonically decreases from 1 to 0 and σ t increases from sufficiently small to the maximum pairwise distance between all training data points. To recover data from noise, diffusion processes train neural networks x θ ( x t , t ) to predict x 0 (other equivalent targets include ϵ and ∇ log p ( x t ) ) from x t ∼ p t ( x | x 0 ) :

Samples are generated with a series of reverse state transition p ( x t -1 | x t , x θ ( x t , t )) .

Flow Matching Another architecture [Lipman et al., 2023] utilizes the ODEs and defines a time-dependent flow function ϕ t ( x ) = σ t ( x 0 ) x + µ t ( x 0 ) that maps p t ( x | x 0 ) = [ ϕ t ] ∗ π ( x ) = π ( ϕ glyph[axisshort] 1 t ( x )) ∣ ∣ ∣ det d ϕ glyph[axisshort] 1 t ( x ) d x ∣ ∣ ∣ = N ( µ t ( x 0 ) , σ 2 t ( x 0 ) I ) , where µ t and σ t can be the same as in diffusion

1

σ

1

σ

x

r

0

x

b

0

t

= 0

=

⇐ ⇒

=

1

σ

1

σ

x

r

0

x

b

0

t

=

T/

2

=

⇐ ⇒

=

1

σ

1

σ

x

r

0

x

b

0

t

=

T

(A) Rescaled Probability Contours

(B) Rescaled Forward Trajectory

Discrete Area of

x

0

x

T

/similarequal

/epsilon1

→

˜

x

T

x

τ

→

˜

x

t

x

t

0

→

˜

x

1

x

0

models or a more straightforward form that µ t = (1 -t T ) x 0 and σ t = t T . Recovering data from noises relies on the vector field u t ( x | x 0 ) that generates the probability path with the ODE d ϕ T -t ( x ) = u T -t ( ϕ T -t ( x ) | x 0 )d t, t : 0 → T . Neural networks u θ ( x , t ) are trained to estimate the vector field u t ( x | x 0 ) via the following objective:

Besides, the vector field is proved to have the form:

3 Methodology

As illustrated in Figure 2, our objective is to refine the probability density contours of p t ( x | x 0 ) so that they better fit the boundaries of discrete samples while still allowing for the ease of sampling. Let x 0 denote the samples from a real distribution q ( x 0 ) . Obtaining a boundary-aware corresponding noisy data x at time t ∈ [1 , T ] is p t ( x | x 0 ) = ∫ p t ( x , x t 0 , t 0 | x 0 )d x t 0 d t 0 , where t 0 is a random variable distributed according to when the diffusion trajectory and the discrete boundary intersect, and x t 0 is the corresponding sample point at t 0 . Then the forward process is rescaled in two steps:

where the latter term is to calculate the discrete boundaries and the former term is to rescale the forward trajectory. In order to make the equation tractable and ensure that x and x t 0 are on the same trajectory, we model the forward process with flow functions ϕ t ( x ) and extend the notation as:

where u ( · ) and v ( · ) are coefficient functions and sampling x t from p t ( x | x 0 ) equals to

3.1 Estimate Discrete Boundaries

Before figuring out the joint distribution p ( x t 0 , t 0 | x 0 ) , let's start by discussing how to verify whether an arbitrary point x in the continuous space belongs to the discrete area of x 0 . Suppose x 0 , which exists in the continuous space S , is the representation vector of a discrete random variable I in a discrete space with K states. Besides, J is another discrete random variable i.i.d. with I . We define the discrete area of x 0 in the continuous space S as:

where f ( x , I ) is a function assessing the likelihood of an arbitrary continuous point x inside the discrete area of x 0 . For instance, in language modeling, K is the vocabulary size. I , J ∈ K n are two different sequences of n tokens and x 0 ∈ R [ n,m ] is a sequence of m -dimensional vector embeddings for I . f ( x , I ) is the dot similarity function. C I collects all vectors in the embedding space that will be decoded to generate I and excludes vectors associated with any other token sequences J .

Given a noisy point x t 0 locating at the boundary between C I and C J , we can get | f ( x t 0 , I ) -f ( x t 0 , J ) | = 0 based on previous definition. Replacing x t 0 with eqs. (5) and (6), there is:

In language modeling and categorical images, f ( · ) is a linear projection function that:

Further simplification of this equation can not be universally applied to all arbitrary forms of u t 0 and v t 0 . Therefore, we calculate separately for several commonly occurring special cases.

Diffusion Process For variance preserving, there is u 2 t + v 2 t = 1 and we have:

For variance exploding, there are u t = 1 and v t = σ t . We can obtain:

Flow Matching For optimal transport, there is u t + v t = 1 and similarly we get:

As a result, t 0 can be directly derived by inverting the coefficient function u t or v t , which depends on the choice of noise scheduling strategies. Since their differences do not affect our results, we omit the detailed calculation (appendix E) and denote this process with a function G ( · ) :

It's worth noting that t 0 is not a scalar but a vector, where the dimension is the number of elements in x 0 . If x 0 is a sequence of n tokens, t 0 ∈ [1 , T ] n . If x 0 is a RGB image with 3 -channel × h -height × w -width of pixels, t 0 ∈ [1 , T ] 3 × h × w . Furthermore, the corresponding noisy sample x t 0 is derived as:

which is a time-independent function of the Gaussian noise ϵ . It's worth mentioning that both p ( t 0 | x 0 ) and p ( x t 0 | x 0 ) are intractable, since G ( x 0 , ϵ ) and ψ G ( x 0 , ϵ ) ( ϵ ) are not invertible to ϵ . Different ϵ s can be mapped to a same t 0 or x t 0 . Fortunately, there is an one-to-one mapping between ϵ and the [ x t 0 ; t 0 ] pair. We denote the boundary flow function and the corresponding inversion as

and the joint boundary distribution is calculated as

The support set of x t 0 is restricted to the boundary contour, while other regions in the space are assigned a probability of 0 . To obtain the complete boundary, it is necessary to iterate over all possible choices of J and perform pairwise comparisons with I . The complexity is O ( n × K ) , where n elements in x 0 is independently iterated. In practical implementation, obtaining the tightest boundary only requires one step of parallel calculation and an extra min( · ) function over all t 0 candidates.

Confidence Factor The discrete area defined by eq. (7) represents an ideal scenario in which the confidence of the boundary is insufficiently reliable for practical application. Due to the intractability of obtaining the probability density function across the entire discrete area and calculating its confidence interval, we employ an empirical strategy. This approach involves utilizing a confidence factor, denoted as r , ranging from 0 to 1 , which is multiplied by t 0 to strike a balance between confidence and discreteness. Therefore, r = 0 implies the exclusion of discrete priors, causing the discrete area to collapse into a single point, which is the original diffusion process. As the value of r increases, the modeling of discrete boundaries improves at the expense of reliability. Empirically, when the model is conditioned with good guidance, setting a larger value for r allows us to obtain better discrete priors. However, in the case of unconditional modeling, maintaining reliability becomes more crucial to prevent oscillations and even collapses during training.

3.2 Rescale the Forward Trajectory

In this section, we introduce how to formulate the forward trajectory conditioned on discrete boundaries and derive the rescaled noisy sampling distribution. We start with the boundary-independent forward process p t ( x | x 0 ) . Let x t denote a noisy point at time t sampled from p t ( x | x 0 ) , there is ϵ t = ( x t -u ( x 0 , t ) x 0 ) / v ( x 0 , t ) given eq. (5). Equations (13) and (14) provide the corresponding [ x t 0 ; t 0 ] pair on the same trajectory, which is deterministically calculated with no randomness:

To model the transition probability p t ( x t 0 , t 0 | x t , x 0 ) , we utilize the Dirac delta function δ ( x ) ≃ lim σ → 0 N ( 0 , σ 2 I ) , which can be loosely thought of as aggregating all probability densities toward the origin, assigning an infinite density at the origin and zero densities elsewhere. Therefore, we have p t ( x t 0 , t 0 | x t , x 0 ) = δ ([ x t 0 ; t 0 ] -Ψ( ϵ t )) . Then the forward process, conditioned on the discrete boundary, is simply derived via Bayes' rule:

̸

Since p t ( x t | x 0 ) > 0 and p ( x t 0 , t 0 | x 0 ) > 0 , p t ( x t | x t 0 , t 0 , x 0 ) is also a delta function that

Based on the translation property of the Dirac delta function, i.e. ∫ f ( x ) δ ( x -a )d x = f ( a ) , the original forward process p t ( x t | x 0 ) = [ ψ t ◦ Ψ glyph[axisshort] 1 ◦ Ψ] ∗ π ( x t ) = [ ψ t ] ∗ π ( x t ) naturally ignores the influence of discrete boundaries, even if the boundary information is explicitly added as a condition.

To enable the discrete priors, we propose a simple and intuitive approach: rescale the forward trajectory. As shown in Figure 2B, the original forward process flows from x 0 to a random noise ϵ , and we reset the starting point to x t 0 . Accordingly, the intermediate noisy points x t , t ∈ [1 , T ] will be proportionally mapped on this new path, which is

Similar to eq. (19), the rescaled conditional forward process is a Dirac delta function:

˜ p t (˜ x t | x t 0 , t 0 , x 0 ) = δ ( ˜ x t -u ( x 0 , T ( t, t 0 )) x 0 -v ( x 0 , T ( t, t 0 ))Ψ glyph[axisshort] 1 ([ x t 0 ; t 0 ]) ) . (21) However, ˜ p t (˜ x t | x 0 ) faces the same problem of irreversibility as in eq. (14) and we derive it as:

Obtaining the probability density function requires gathering together the probability densities of the same location ˜ x t with different τ , which is intractable. Fortunately, we only need to sample noiy points from this probability distribution ˜ x t ∼ ˜ p t (˜ x t | x 0 ) , which is easy to implement:

3.3 Recover Data from Noise

Training Objective Theoretically, the diffusion neural networks can be trained as in eq. (2), where the rescaled vector field is derived as ˜ u t = d˜ x t d t = d˜ x t d τ d τ d t . However, since a low error estimation on x 0 is of significant importance to our trajectory rescaling method, according to eqs. (10) to (13), we convert the objective to an upper bound of the eq. (2) (See appendix F for more details) and train a neural network x θ (˜ x t , t ) to predict x 0 directly:

The training procedure is demonstrated in algorithm 1 and key steps are summarized in the line 4.

Reverse Process A direct approach that follows the flow matching is to solve the ODE of d ψ T -t ( x ) = ˜ u T -t ( ψ T -t ( x ) | x 0 )d t, ψ T ( x ) ∼ π ( x ) . This form of transformation is inefficient with x 0 -prediction during inference because we have to solve the equation of τ = T ( t, G ( x θ , ˜ x t -u ( x θ ,τ ) x θ v ( x θ ,τ ) )) to get the τ with respect to the change of ˜ x t and x θ in real time. Therefore, we provide a deterministic reverse process as an alternative, which is a special case of DDIM [Song et al., 2021a] or the ODE with discrete timesteps. Given the time intervals ∆ t ∈ [∆ t 1 , . . . ∆ t s ] , ∑ ∆ t = T , we general-

ize the boundary conditions [ x t 0 ; t 0 ] in ˜ p t (˜ x t | x t 0 , t 0 , x 0 ) of eq. (21) and Ψ glyph[axisshort] 1 ([ x t 0 ; t 0 ]) of eq. (15) to any arbitrary condition pairs [˜ x t ; τ ] and obtain the reverse process:

where ˆ x 0 = x θ (˜ x t , t ) and τ ∆ is the previous timestep of τ on the same rescaled trajectory.

Sampling from the reverse process is illustrated in algorithm 2. Similar to the sampling process of DDIM [Song et al., 2021a], it starts from the Gaussian noise, iteratively predicts the pseudo target ˆ x 0 , and updates the reverse trajectory. However, since the τ and ˆ ϵ are mutually conditioned, we have to keep track of the t , τ , ˜ x t , and ˆ ϵ during each iteration and split the update of ˆ ϵ into an asynchronous step (line 8). Because reverse trajectory keeps changing due to different pseudo targets ˆ x 0 predicted by learned neural networks, which brings severe instability, sometimes simply fixing the initial path (removing the line 8) exhibits better performance in experiments.

4 Language Modeling

Recent diffusion language models [Li et al., 2022, Gong et al., 2023b] inherit the embedding-rounding framework that a sentence with n discrete tokens W = [ w 1 , . . . , w n ] is embedded to a continuous space via a trainable embedding layer EMB ( W ) = [ EMB ( w 1 ) , . . . , EMB ( w n )] . The vocabulary set is K that ∀ w n ∈ K . Besides, the token embeddings are used as the target points x 0 = [ x 1 0 , . . . , x n 0 ] , x n 0 = EMB ( w n ) , for continuous diffusion trajectories. Hence, generating tokens from embeddings is:

where f ( x , j ) = EMB ( j ) · x is the dot production distance. It's also the function assessing the likelihood of point x inside the discrete area of j . The coefficient functions follow the DDPM [Ho et al., 2020], which are u ( x 0 , t ) = √ ¯ α t and v ( x 0 , t ) = √ 1 -¯ α t . Besides, the objectives are

and an additional rounding objective, which is commonly used in language modeling,

The final training target is given by L = L θ + L r , where the x 0 of the same token sequence W keeps changing because the embedding layer EMB is trainable, which makes the model hard to be trained. Since previous work does not model discrete areas, a large number of noisy samples inside this area will make L r too small to guide the training of the embedding layer, leading to a mode collapse.

Experimental Setup Datasets used for experiments include three translation tasks (IWSLT14 DE-EN [Cettolo et al., 2012], WMT14 EN-DE, and WMT16 EN-RO 1 ) and one text summarization task (GIGAWORD [Rush et al., 2015]). We mainly follow the setting of Gao et al. [2022], which is inherited from previous non-auto-regressive text generation works [Gu et al., 2018, 2019, Ghazvininejad et al., 2019], where translation datasets are distilled [Kim and Rush, 2016]. Baselines are mainly continuous diffusion language models. DiffuSeq [Gong et al., 2023b] and SeqDiffuSeq [Yuan et al., 2022] are derived from Diffusion-LM [Li et al., 2022]. Difformer [Gao et al., 2022] and Dinoiser [Ye et al., 2023] are recent empirical studies highlighting that scaling up the noise is beneficial for language modeling. We also compare with discrete diffusion language models, including D3PM [Austin et al., 2021] and SEDD [Lou et al., 2023]. Since SEDD is a pre-trained language model, we configure its framework and train it from scratch specifically for our tasks. In addition, auto-regressive transformer [Vaswani et al., 2017] is still one of the most powerful architectures for language generation.

Our boundary conditional diffusion language model is constructed from Difformer [Gao et al., 2022], where the model configuration is transformer-iwslt-de-en in FAIRSEQ framework [Ott et al., 2019] for IWSLT14 DE-EN and transformer-base for other datasets. Sentences are tokenized with Byte-Pair Encoding [Sennrich et al., 2016] and evaluated by detokenized BLEU [Papineni et al., 2002] for machine translation and ROUGE [Lin, 2004] for summarization. During training, the diffusion step is T = 2000 and the confidence factor r = 1 for translation tasks since they have strong conditions, while r = 0 . 5 for summarization. Sentences are generated deterministically with 20 steps.

Results Performances are demonstrated in Table 1. Our approach achieves the state-of-the-art compared with continuous diffusion language models and outperforms the two discrete baselines on three machine translation and one text summarization tasks. Our method shows advantages, with a 73 . 6% significant improvement at most on WMT14 EN-DE, over DiffuSeq [Gong et al., 2023b] and SeqDiffuSeq [Yuan et al., 2022], which are two basic methods directly applying diffusion process to language modeling. Compared with recent strong diffusion language models like Difformer [Gao et al., 2022] and Dinoiser [Ye et al., 2023], which have deployed various effective noise scheduling strategies on diffusion processes from the empirical perspective, our model is still superior with at most 3 . 07 advancement of BLEU score on WMT16 EN-RO. This implies the effectiveness of modeling discrete priors. In addition, we illustrate the performance of auto-regressive modeling, where we use the transformer [Vaswani et al., 2017] to rerank the generated sentence candidates ( 7

L

length beam × 3 sentence beams) of our model. The reranked performance can even outperform transformers on IWSLT14 DE-EN and WMT16 EN-RO.

Ablation Our approach is a general framework applicable to almost all continuous diffusion models, providing them with discrete boundaries as priors. We choose Difformer [Gao et al., 2022] as the base model and follow the configurations. As proved in eq. (19), the original forward process will ignore the discrete pri-

ors although explicitly demonstrated. We conduct ablation experiments on the rescaling module. As illustrated in Table 2, our approach rescales the trajectory of both forward and reverse processes on Difformer. Only rescaling the forward trajectory is also effective but sub-optimal due to the inconsistent distribution during inference. Due to computational cost and fair comparison, our method leaves room for improvement. For example, replacing the forward trajectory with optimal transport in Flow Matching, u ( x 0 , t ) = 1 -t/T and v ( x 0 , t ) = t/T , achieves better performance on WMT16.

Analysis Our training objective, eq. (24), is an upper bound of the eq. (2). We demonstrate the influence of this approximation in Table 3 on IWSLT14 DE-EN to reveal the thought of our formula. On the one hand, L x 0 brings theoretical errors at a constant scale. On the other hand, L x 0 mitigates some experimental errors from the neural networks. The first row L x 0 is the objective we used in eq. (24) and the second row L ˜ u t = E { t, x 0 , ˜ x t } [ ∥ ˜ u t (˜ x t | x θ (˜ x t , t )) -d˜ x t d t ∥ 2 ] is directly derived from the eq. (2). The first two columns represent the error expectations of x 0 and ˜ u t on the test set. It is easy to observe that, with the dynamic coefficient d τ d t = T -r × G ( x 0 , ϵ ) T (appendix F), the value of x 0 's error ( 8 . 44 ) is much larger than the ˜ u t 's error ( 1 . 56 ). Therefore, L x 0 is beneficial for reducing the impact of the prediction error from the neural network. The third column in Table 3 illustrates the one-step accuracy of predicting x 0 and the fourth column is the BLEU score on the test set. Experimental results show that optimizing the upper bound has a negligible impact on the final performance (only a 0 . 2% drop of the BLEU score), while can improve the efficiency of the loss calculation during the training phase.

5 Discrete Image Generation

Image pixels are usually treated as real numbers in continuous space since adjacent pixel values exhibit linear continuity.They are essentially discrete and quantized data with a finite state space, such as 256 states in RGB format. We utilize two discrete image representations. One is binary coding provided by Bit Diffusion [Chen et al., 2023b] that converts a sub-pixel with 256 integers to a 8 -bit binary code. It is more efficient as it stores ordinal relationships, but the representation space it constructs will be sparse. Another is pixel embedding, which is a more discrete form of representation because the relationships between pixels are thoroughly broken down and reconstructed by learning the embedding representation. Each pixel is regarded as a one-hot vector and transformed with an embedding layer EMB as used in language. Furthermore, we design an intermediate state to demonstrate the correlation between discreteness and modeling difficulty, which is initializing a fixed embedding with binary coding. The optimization target for binary coding is the MSE loss, and pixel embeddings take the same objective as in language.

Experimental Setup We use CIFAR-10 [Krizhevsky et al., 2009] for discrete image generation. The evaluation metric is FID [Heusel et al., 2017], which compares 50K generated samples with the training set. Our image generation model is constructed on Bit Diffusion [Chen et al., 2023b], where the architecture is U-Net [Ronneberger et al., 2015] with 3 stages, 256 channels and 3 residual blocks

per stage. Diffusion steps are T = 1000 for both the training and inference stages. The model is trained for 1 . 5 M steps with the learning rate of 1 e glyph[axisshort] 4 and batch size of 128 . Since the training script and detailed hyperparameters of Bit Diffusion are not available, we have to reproduce it by ourselves and our boundary conditional diffusion model shares exactly the same configuration. Our confidence factors are r = 0 . 5 for all three settings. Other baselines include D3PM [Austin et al., 2021] and τ LDR [Campbell et al., 2022] which are discrete diffusion models. SDDM [Sun et al., 2023] utilizes vector quantization from VQGAN[Esser et al., 2021] as a continuous space for discrete data. We also compare with DDPM [Ho et al., 2020] and DDIM [Song et al., 2021a] on continuous pixels.

Results For binary coding, as shown in Table 4, our approach outperforms the reproduced Bit Diffusion and attains competitive results to state-of-the-art models. For pixel embedding where ordinal information is deconstructed and reconstituted, our method exhibits a notable improvement of 3 . 81 FID score over replicated Bit Diffusion. Moreover, in the case of categorical pixels, this advantage increases to 8 . 25 , positioning our approach with trainable embedding as a new state-of-the-art solution. Additionally, as deterministic diffusion processes, our model with binary coding can slightly exceed the performance of DDIM, where the generated samples are in Figure 3.

Analysis We analyze the influence of the confidence factor r in Table 5. The factor r is selected from

[0 , 0 . 2 , 0 . 3 , 0 . 5] , where r = 0 is the reproduced Bit Diffusion that discards the discrete priors. As the confidence factor increases, the impact of discreteness gradually improves, simultaneously enhancing the model's performance across all three settings. Since there is no guidance for unconditional image generation, we do not use a larger factor to prevent mode collapses.

6 Related Work

Discrete Modeling Auto-regressive models have demonstrated a domination over discrete modeling, especially for text generation [Vaswani et al., 2017, Brown et al., 2020, Achiam et al., 2023]. However, the computation

cost increases drastically as the size of sentence length or the image resolution increases. Diffusion models [Sohl-Dickstein et al., 2015, Ho et al., 2020, Dhariwal and Nichol, 2021, Saharia et al., 2022] can generate data in parallel, but are tailored for continuous problems. To generalize diffusion models for discrete data, the most straightforward methods define discrete processes in discrete spaces [Sohl-Dickstein et al., 2015, Hoogeboom et al., 2021b, Austin et al., 2021, Campbell et al., 2022, Zhang et al., 2023, Sun et al., 2023, Lou et al., 2023], which will be bothered by large number of discrete status. Besides, a simplified version of discrete diffusion processes is recently used in language modeling [He et al., 2023, Chen et al., 2023a]. Approaches in another line argue to located discrete data in continuous spaces, which is more flexible and efficient, with the mapping functions including binary bits [Chen et al., 2023b] and embeddings [Li et al., 2022, Gong et al., 2023b,a, Yuan et al., 2022, Gulrajani and Hashimoto, 2023, Han et al., 2023]. Other generative models adapted for discrete modeling includes Variational Autoencoders [Kingma and Welling, 2014], Generative Adversarial Networks [Hjelm et al., 2018, Fedus et al., 2018], and Normalizing Flows [Lindt and Hoogeboom, 2021, Hoogeboom et al., 2021a, Tan et al., 2022].

Diffusion Models with Deterministic Trajectory Deterministic diffusion process is usually used in the inference stage to speed up sampling, where DDIM [Song et al., 2021a] derives a serial of non-Markovian diffusion processes and the deterministic one is a special case from this implicit perspective. Additionally, deterministic diffusion processes can be converted to ordinary differential equations [Song et al., 2021b], which is utilized by recent sampling acceleration approaches such as DEIS [Zhang and Chen, 2023] and DPM-Solvers [Lu et al., 2022b,a, Zheng et al., 2023]. Our approach requires a deterministic forward trajectory to eliminate the randomness between the boundary point and sampled point. Flow matching [Liu, 2022, Lipman et al., 2023, Albergo and Vanden-Eijnden, 2023, Liu et al., 2023] is a collection of generative models that employ ordinary differential equations to facilitate both forward and reverse processes. They can be regarded as generally equivalent to Diffusion models. Therefore, we extend the framework of flow matching for our method.

7 Conclusion

We studied the gap between discrete modeling and continuous spaces, focusing on the inconsistency between probability density contours learned by continuous diffusion models and discrete boundaries. We have proposed a novel and general approach to address this issue by enabling continuous diffusion models to be conditioned on discrete priors, which is achieved via discrete boundary estimation and trajectory rescaling. An important limitation is that our method is designed for continuous diffusion models, where discrete diffusion models constructed specially on the discrete state space would not encounter the problem. However, discrete diffusion models also possess their own shortcomings, and the practical applications of continuous diffusion models are more extensive. We believe that our method has the potential to advance the development of unified and general diffusion models. By bridging the gap between discrete and continuous modeling, we hope to inspire new possibilities for modeling complex systems and phenomena.

References

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 , 2023.

Michael Samuel Albergo and Eric Vanden-Eijnden. Building normalizing flows with stochastic interpolants. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=li7qeBbCR1t .

Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Structured denoising diffusion models in discrete state-spaces. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , 2021.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 1877-1901. Curran Associates, Inc., 2020.

Andrew Campbell, Joe Benton, Valentin De Bortoli, Tom Rainforth, George Deligiannidis, and Arnaud Doucet. A continuous time framework for discrete denoising models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022.

Mauro Cettolo, Christian Girardi, and Marcello Federico. WIT3: Web inventory of transcribed and translated talks. In Mauro Cettolo, Marcello Federico, Lucia Specia, and Andy Way, editors, Proceedings of the 16th Annual Conference of the European Association for Machine Translation , pages 261-268, Trento, Italy, May 28-30 2012. European Association for Machine Translation.

Jiaao Chen, Aston Zhang, Mu Li, Alex Smola, and Diyi Yang. A cheaper and better diffusion language model with soft-masked noise. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages 4765-4775, Singapore, December 2023a. Association for Computational Linguistics.

Ting Chen, Ruixiang Zhang, and Geoffrey Hinton. Analog bits: Generating discrete data using diffusion models with self-conditioning. In The Eleventh International Conference on Learning Representations , 2023b.

Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 8780-8794. Curran Associates, Inc., 2021.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations , 2021. URL https://openreview. net/forum?id=YicbFdNTTy .

Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 12873-12883, 2021.

William Fedus, Ian Goodfellow, and Andrew M. Dai. Maskgan: Better text generation via filling in the _. In International Conference on Learning Representations , 2018.

Zhujin Gao, Junliang Guo, Xu Tan, Yongxin Zhu, Fang Zhang, Jiang Bian, and Linli Xu. Difformer: Empowering diffusion model on embedding space for text generation. arXiv preprint arXiv:2212.09412 , 2022.

Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. Mask-predict: Parallel decoding of conditional masked language models. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pages 6112-6121, Hong Kong, China, November 2019. Association for Computational Linguistics.

Shansan Gong, Mukai Li, Jiangtao Feng, Zhiyong Wu, and Lingpeng Kong. DiffuSeq-v2: Bridging discrete and continuous text spaces for accelerated Seq2Seq diffusion models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023 , pages 9868-9875, Singapore, December 2023a. Association for Computational Linguistics.

Shansan Gong, Mukai Li, Jiangtao Feng, Zhiyong Wu, and Lingpeng Kong. Diffuseq: Sequence to sequence text generation with diffusion models. In The Eleventh International Conference on Learning Representations , 2023b.

Jiatao Gu, James Bradbury, Caiming Xiong, Victor O.K. Li, and Richard Socher. Non-autoregressive neural machine translation. In International Conference on Learning Representations , 2018.

Jiatao Gu, Changhan Wang, and Junbo Zhao. Levenshtein transformer. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.

Ishaan Gulrajani and Tatsunori Hashimoto. Likelihood-based diffusion language models. In Thirtyseventh Conference on Neural Information Processing Systems , 2023.

Xiaochuang Han, Sachin Kumar, and Yulia Tsvetkov. SSD-LM: Semi-autoregressive simplex-based diffusion language model for text generation and modular control. In Anna Rogers, Jordan BoydGraber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 11575-11596, Toronto, Canada, July 2023. Association for Computational Linguistics.

Zhengfu He, Tianxiang Sun, Qiong Tang, Kuanning Wang, Xuanjing Huang, and Xipeng Qiu. DiffusionBERT: Improving generative masked language models with diffusion models. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 4521-4534, Toronto, Canada, July 2023. Association for Computational Linguistics.

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems , 30, 2017.

R Devon Hjelm, Athul Paul Jacob, Adam Trischler, Gerry Che, Kyunghyun Cho, and Yoshua Bengio. Boundary seeking gans. In International Conference on Learning Representations , 2018.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 6840-6851. Curran Associates, Inc., 2020.

Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling. Argmax flows and multinomial diffusion: Learning categorical distributions. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 12454-12465. Curran Associates, Inc., 2021a.

Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling. Argmax flows and multinomial diffusion: Learning categorical distributions. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , 2021b.

Yoon Kim and Alexander M. Rush. Sequence-level knowledge distillation. In Jian Su, Kevin Duh, and Xavier Carreras, editors, Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing , pages 1317-1327, Austin, Texas, November 2016. Association for Computational Linguistics.

Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In International Conference on Learning Representations , 2014.

Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. In International Conference on Learning Representations , 2021.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori Hashimoto. DiffusionLM improves controllable text generation. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022.

Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out , pages 74-81, Barcelona, Spain, July 2004. Association for Computational Linguistics.

Alexandra Lindt and Emiel Hoogeboom. Discrete denoising flows. In ICML Workshop on Invertible Neural Networks, Normalizing Flows, and Explicit Likelihood Models , 2021.

Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In The Eleventh International Conference on Learning Representations , 2023.

Qiang Liu. Rectified flow: A marginal preserving approach to optimal transport. arXiv preprint arXiv:2209.14577 , 2022.

Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. In International conference on learning representations (ICLR) , 2023.

Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion language modeling by estimating the ratios of the data distribution. arXiv preprint arXiv:2310.16834 , 2023.

Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arXiv:2211.01095 , 2022a.

Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. DPM-solver: A fast ODE solver for diffusion probabilistic model sampling in around 10 steps. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022b.

Ali Madani, Bryan McCann, Nikhil Naik, Nitish Shirish Keskar, Namrata Anand, Raphael R Eguchi, Po-Ssu Huang, and Richard Socher. Progen: Language modeling for protein generation. arXiv preprint arXiv:2004.03497 , 2020.

Ali Madani, Ben Krause, Eric R Greene, Subu Subramanian, Benjamin P Mohr, James M Holton, Jose Luis Olmos, Caiming Xiong, Zachary Z Sun, Richard Socher, et al. Large language models generate functional protein sequences across diverse families. Nature Biotechnology , 41(8): 1099-1106, 2023.

Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations , 2019.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Pierre Isabelle, Eugene Charniak, and Dekang Lin, editors, Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics , pages 311-318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguistics.

Niki J. Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In International Conference on Machine Learning (ICML) , 2018. URL http://proceedings.mlr.press/v80/parmar18a.html .

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Highresolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 10684-10695, June 2022.

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention-MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18 , pages 234-241. Springer, 2015.

Alexander M. Rush, Sumit Chopra, and Jason Weston. A neural attention model for abstractive sentence summarization. In Lluís Màrquez, Chris Callison-Burch, and Jian Su, editors, Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing , pages 379-389, Lisbon, Portugal, September 2015. Association for Computational Linguistics.

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 36479-36494. Curran Associates, Inc., 2022.

Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Katrin Erk and Noah A. Smith, editors, Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 1715-1725, Berlin, Germany, August 2016. Association for Computational Linguistics.

Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference on Machine Learning , volume 37 of Proceedings of Machine Learning Research , pages 2256-2265, Lille, France, 07-09 Jul 2015. PMLR.

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations , 2021a.

Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations , 2021b.

Haoran Sun, Lijun Yu, Bo Dai, Dale Schuurmans, and Hanjun Dai. Score-based continuous-time discrete diffusion models. In The Eleventh International Conference on Learning Representations , 2023.

Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. Advances in neural information processing systems , 27, 2014.

Shawn Tan, Chin-Wei Huang, Alessandro Sordoni, and Aaron Courville. Learning to dequantise with truncated flows. In International Conference on Learning Representations , 2022.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017.

Jiasheng Ye, Zaixiang Zheng, Yu Bao, Lihua Qian, and Mingxuan Wang. Dinoiser: Diffused conditional sequence learning by manipulating noises. arXiv preprint arXiv:2302.10025 , 2023.

Hongyi Yuan, Zheng Yuan, Chuanqi Tan, Fei Huang, and Songfang Huang. Seqdiffuseq: Text diffusion with encoder-decoder transformers. ArXiv , abs/2212.10325, 2022.

Pengze Zhang, Hubery Yin, Chen Li, and Xiaohua Xie. Formulating discrete probability flow through optimal transport. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

Qinsheng Zhang and Yongxin Chen. Fast sampling of diffusion models with exponential integrator. In The Eleventh International Conference on Learning Representations , 2023.

Kaiwen Zheng, Cheng Lu, Jianfei Chen, and Jun Zhu. Dpm-solver-v3: Improved diffusion ode solver with empirical model statistics. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

A Stopping Time for Forward Process

The forward diffusion process X = { x n , n ≥ 0 } is a markovian stochastic process with a transition probability p ( x i | x i -1 ) = N ( x i ; √ α i x i -1 , (1 -α i ) I ) . And a stopping time t 0 with respect to X is a random time such that for each n ≥ 0 , the event { t 0 = n } is completely determined by the total information known up to time n , { x 0 , . . . , x n } . Suppose the random variables { x n } are in a one-dimensional space and the forward process starts with x 0 = 0 . Besides, let A, x 0 ∈ A be the discrete area belonging to x 0 that for each points in area A will be regarded as x 0 during data generation. Our expected stopping time is defined as:

which represents the first time x n leaves area A . We can write the probability of stopping time as:

Since the diffusion process is established in continuous space, calculating the probability of the stopping time requires integrating over each intermediate state x 1: n -1 , rather than a simple state transfer as in the discrete space. Hence, directly obtain the stopping time is intractable. Additionally, even if we are able to get probability of the stopping time, we can only get a distribution over the time dimension, without knowing the exact time of x n leaving area A . Therefore, we need to eliminate randomness from the state transition x i -1 → x i and find a deterministic forward trajectory to estimate the stopping time.

B Properties of Dirac Delta Function

There are several useful properties of Dirac delta function:

Symmetry Property: δ ( - x ) = δ ( x )

Scaling Property: δ ( ax ) = δ ( x ) | a |

Translation Property: ∫ f ( x ) δ ( x -a )d x = f ( a )

C Bridging Flow Matching and DDPM

In this work, we utilizes the framework of Flow Matching to model the diffusion processes, where the forward process is defined by flow functions in eq. (5). Although having different mathematical forms, it is essentially equivalent to traditional diffusion processes. Here, we provide an alternative form from the perspective of state transfer p t ( x t | x t -1 ) .

C.1 Deterministic Forward Process

Equation (5) gives the definition p t ( x t | x 0 ) = [ ψ t ] ∗ π ( x ) , where ψ t ( x ) = u t x 0 + v t x . Here we provide the equivalent derivation of p t ( x t | x 0 ) from the perspective of diffusion processes:

where p ( x 1 | x 0 ) = N ( u 1 x 0 , v 2 1 I ) is the first step of the forward process at which the global noise is introduced into the forward trajectory. The state transfer probability of forward process p s ( x s | x s -1 , x 0 ) = δ ( x s -u s x 0 -v s ψ glyph[axisshort] 1 s -1 ( x s -1 )) is a Dirac delta function. Therefore,

where we denote the integral of x 1 as Q 1 . Based on

(Symmetry Property of Dirac Delta Function)

and the Translation Property of the Dirac delta function, we can calculate Q 1 as:

Then we can continue the deviation of p t ( x t | x 0 ) as:

Therefore, the probability distribution of x t conditioned on x 0 follows a Gaussian distribution N ( u t x 0 , v 2 t I ) , which is the same as in original DDPMs when the coefficient functions are defined as u t = √ ¯ α t and v t = √ 1 -¯ α t . This provides an important benefit that the Flow Matching and diffusion models share the same training procedure.

C.2 Deterministic Reverse Process

The reverse tranfer probability follows Bayes' rule:

Since Dirac delta function has another form of

and p t ( x t | x 0 ) > 0 , p t -1 ( x t -1 | x t ) > 0 , we have

̸

C.3 Deterministic Optimization Objective

We first include the derivation of the variational bound for diffusion models provided by SohlDickstein et al. [2015]. The probability the generative model assigns to the data is:

̸

̸

Deterministic Diffusion Process

Flow Matching

Training amounts to minimizing the negative log likelihood:

where L T is usually ignored as a constant and p ( x t -1 | x t ) is parameterized with a neural network p θ ( x t -1 | x t ) to approximate the conditioned probability distributions in the reverse process. Since p ( x t -1 | x t , x 0 ) = lim σ → 0 N ( v t -1 v t x t + ( u t -1 -u t v t -1 v t x 0 ) , σ 2 I ) , the parameterized p θ ( x t -1 | x t ) can take the same form N ( µ θ ( x t , t ) , σ 2 t I ) because the Dirac delta function is a special case of Gaussian distribution and the KL divergence of two Gaussians can be simplified. Finally, the training objective for the deterministic diffusion process is divided as:

where the simplified version ∥ x 0 -x θ ( x t , t ) ∥ 2 is the same as DDPMs but with different coefficients.

D Different Diffusion Trajectories

We illustrate the trajectories of different diffusion processes in Figure 4. The forward and reverse generation for the Markovian diffusion process is:

The deterministic diffusion process:

The deterministic flow matching with optimal transport:

E Details of the Function G

Equation (13) defines the function G ( x , ϵ ) as the inversion of coefficient function.

Flow Matching The coefficient is u t = 1 -t/T , where t = T × (1 -u t ) . Therefore,

Diffusion The coefficient for Variance Exploding is v T = σ 0 ( σ T σ 0 ) t T , where t = T × log v t -log σ 0 log σ T -log σ 0 .

For Variance Preserving, the function G ( x 0 , ϵ ) is more difficult to calculate since u t = √ ¯ α t , where ¯ α = ∏ t i =1 α i , α t = 1 -β t , and β t is also influenced by noise schedulers. This makes G ( x 0 , ϵ ) hard to calculate. Fortunately, we can bypass this function and provide the corresponding pseudo code.

F Details of the Training Objective

The rescaled vector field is calculated as:

Considering the expectation form of ˜ u t , there is:

Therefore, the training objective E ∥ ˜ u t -˜ u θ ∥ 2 ≤ c E ∥ x 0 -x θ ∥ 2 , where c is the coefficient.

G Code Implementations

Our framework is a module constructed on current diffusion models. We demonstrate our kernel part rescale diffusion trajectory with pseudo python code as below:

H Analysis

Gaussian Sampling Our framework is compatible with the Gaussian sampling in DDPM, where random noises can be added into each iteration step. Algorithm 3 demonstrates the Gaussian sampling procedure. Compared with algorithm 2, a Gaussian noise z ∼ N ( 0 , σ 2 t I ) with a decreasing variance σ t is injected to the estimated next state ˜ x t . This noise z will be mapped as changing the initial sampling ˜ x T through the trajectory alteration step. We illustrate the deterministic and Gaussian sampling for our model on CIFAR-10 in Table 6, where the deterministic sampling can achieve a much better performance of FID. We assume this is because our coefficient functions u ( x 0 , t ) and

Algorithm 3 Gaussian Sampling

1:

t

:=

T

,

τ

:=

T

2:

˜

x

t

∼ N

(

0

,

I

)

// Initialing

3:

for

∆

t

:= ∆

t

1

, . . . ,

∆

t

s

do

//

∑

∆

t

=

T

4:

z

∼ N

(

0

, σ

2

t

I

)

5:

ˆ

x

0

:=

x

θ

(˜

x

t

, t

)

// Pseudo Target

6:

ˆ

ϵ

:= Ψ

glyph[axisshort]

1

x

t

;

τ

])

// Trajectory Alteration

7:

τ

∆

:=

T

(

t

-

∆

t, G

(ˆ

x

0

,

ˆ

ϵ

))

// eq. (25)

8:

˜

x

t

:=

u

(ˆ

x

0

, τ

∆

)ˆ

x

0

+

v

(ˆ

x

0

, τ

∆

)ˆ

ϵ

+

z

9:

t

:=

t

-

∆

t

,

τ

:=

τ

∆

// Updating

10:

end for

11:

x

0

:=

x

θ

(˜

x

t

, t

)

//

x

1

→

x

0

12:

return

x

0

v ( x 0 , t ) are dynamically calculated to rescale the deterministic trajectory in the training stage. In the inference stage, x 0 is replaced by x θ ( x t , t ) , where errors will accumulate if the predicted pseudo target changes frequently. Moreover, Gaussian sampling will further introduce random noises at each reverse step, making our rescaled timestep τ far away from the training situation. Therefore, errors in the calculations of trajectory scaling will explode over iterations.

I Limitations

Our framework is proposed to migrate the powerful continuous diffusion models to discrete problems. There is another technical route that directly designs the diffusion process on the discrete state space and our method is not useful for this scenario. However, we believe the continuous diffusion models can be a general framework for generative modeling and our effort can advance this target.

We prefer x 0 as the training target because we highly depend on the reliability of the predicted ˆ x 0 during inference. Although it is possible to use other targets, the modeling effect will decrease in practical use, which limits the flexibility of diffusion modeling. For example, predicting the ˆ ϵ and recovering ˆ x 0 with eq. (23) is inefficient, because a small error in predicting ˆ ϵ will be amplified by eq. (23) and lead to the collapse of G (ˆ x 0 , ˆ ϵ ) .

Our approach requires extra computational cost. But they are acceptable since our rescaling process is a series of parallel matrix computations. Considering that our approach is compatible with the Self-Conditioning [Chen et al., 2023b], our overhead is negligible when it is used.

J Other Experimental Details

For language modeling, we utilize the model configuration transformer-iwslt-de-en in FAIRSEQ framework [Ott et al., 2019] for IWSLT14 DE-EN, which has 6 transformer layers, 4 attention heads, 512 hidden dimensions, and 1024 feed forward layer dimensions. For other datasets, the configuration is transformer-base , which has 6 transformer layers, 8 attention heads, 512 hidden dimensions, and 2048 feed forward layer dimensions. The embedding dimension is 128 . The beam size is 1 length prediction beam × 5 generation beam, since the length prediction is unstable for diffusion language models. For reranking, we take 7 length prediction beam × 3 generation beam as Difformer to let the transformer choose the best one.

(A) Bit Diffusion repro

(FID

10

.

37

)

(B) Ours (FID

3

.

86

)

For image generation, we set the scaling factor r = 0 . 5 for training. Besides, we find that a smaller factor for inference is sometime useful. We set r = 0 . 45 on binary coding and r = 0 . 2 on fixed embedding during inference. When the pixel embedding is learnable, the scaling factor is r = 0 . 5 , which is the same as training.

Our experiments are performed with Nvidia 80G A100. Each language result requires about 2 days on one single A100. Each image result requires about a week on one single A100.

K Impact Statements

This paper presents work whose goal is to advance the field of Deep Learning. The datasets we used has been widely deployed for many years and has basically no negative impact. Our approach is a framework that migrates existing diffusion models to discrete problems, which does not provide a large pre-trained model that can be used to generate fake contents.

L Case Study

Generated sentences on IWSLT14 DE-EN and GIGAWORD are illustrated in Table 7 and Table 8. Generated images on CIFAR-10 are depicted in Figure 5, 6, and 7.

(A) Bit Diffusion repro (FID 12 . 96 )

(B) Ours (FID 9 . 15 )

repro

19

.

26

(B) Ours (FID 10 . 99 )

(A) Bit Diffusion

(FID

)

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Contributions and scope are in abstract and introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Discussed in (7) Conclusion and (H) Limitations sections.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: From sections (A) to (E) in appendices.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We use algorithms 1 and 2 to demonstrate how to reproduce our algorithm. We provide a paragraph of Experimental Setup in (4) Language Modeling and (5) Discrete Image Generation sections. Other details are in section (I) and pseudo code of our kernel process is in (F).

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The pseudo code of our kernel process is demonstrated in (F) and we will public our code on github.com.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide a paragraph of Experimental Setup in (4) Language Modeling and (5) Discrete Image Generation sections. Other details are in section (I). We provide ablation studies on the hyperparameters.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Error bars are not reported because it would be too computationally expensive. Each result in the experiment table needs to be run on an 80G A100 for at least 2 days. The huge overhead required to obtain a statistically significant error bar makes it impossible for us to achieve it.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Details are in section (I).

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Discussed in section (J).

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our approach is a framework involves algorithms but not pre-trained models. Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We provide the link or citation of each asset, where licenses are in the link.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: No new assets introduced.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: No crowdsourcing.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: No crowdsourcing.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a discrete diffusion model that operates in the embedding space of vectors that represent discrete tokens. They propose a method that takes into account the fact that whole regions of the space will decode to the same discrete token because during decoding, the closest embedding vector is selected. The method is that at training time, instead of interpolating between $x_0$ and $x_1$, they interpolate from $x_{t_0}$ to $x_1$ where $x_{t_0}$ is the point at which the line between $x_0$ and $x_1$ intersects the 'discrete boundary' of $x_0$ which is the region of embedding space that all decodes to $x_0$. The authors test their method on language tasks as well as CIFAR-10 image modeling.\n\nSTRENGTHS:\nThe idea of trying to exploit the fact that the embedding space has a very specific structure consisting of volumes that all decode to the same discrete token is interesting and novel.\n\nThe experimental results look quite promising with good performance compared to other discrete diffusion approaches on machine translation and text summarization.\n\nThe authors are working on a significant problem as finding diffusion methods that can perform on par with autoregressive models for discrete modeling would be very impactful.\n\nWEAKNESSES:\nThe paper's main weakness is lack of clarity and precision in the description of the method. I really struggle to understand what the authors are doing exactly and how it is justified.\n\n- It is not clear to me that your training objective and sampling regime are justified. The flow matching paradigm is to construct a $p_t(x_t | x_0)$ probability path and then find a $u_t(x_t | x_0)$ vector field that generates this probability path through the transform defined by ODE integrating along the $u_t(x_t | x_0)$ vector field. The final step is to say that a vector field that generates the unconditional probability path $p_t(x_t)$ is equal to $\\mathbb{E}\\_{p(x_0 | x_t)} [ u_t(x_t | x_0) ]$ and this can be learned tractably with an objective of $\\mathbb{E}\\_{p(x_0) p(x_t | x_0)} [ || v_t^\\theta(x_t) - u_t(x_t | x_0) ||^2 ]$. However, I don't see how your method fits in this framework. This is for two reasons: 1) at no point do you define the conditional vector field $\\tilde{u}\\_t(\\tilde{x}\\_t | x\\_0)$ that generates your rescaled probability path $\\tilde{p}\\_t(\\tilde{x}\\_t | x\\_0)$. How are you able to generate along these probability paths without defining $\\tilde{u}\\_t(\\tilde{x}\\_t | x\\_0)$ ? 2) You sample using $u\\_t(x\\_t | x\\_0)$ for the original $p\\_t(x\\_t | x\\_0)$ but with the $\\hat{x}\\_0$ prediction from the neural network. This is valid in the original flow matching case because $u\\_t(x\\_t | x\\_0)$ is linear and so the $\\mathbb{E}\\_{p(x\\_0) p(x\\_t | x\\_0)} [ || v\\_t^\\theta(x\\_t) - u\\_t(x\\_t | x\\_0) ||^2 ]$ objective can be re-arranged into $x\\_0$ prediction $\\mathbb{E}\\_{p(x\\_0) p(x\\_t | x\\_0)} [ || x\\_\\theta(x\\_t , t) - x\\_0 ||^2 ]$. However we don't necessarily know if $\\tilde{u}\\_t(\\tilde{x}\\_t | x\\_0)$ that generates $\\tilde{p}\\_t(\\tilde{x}\\_t | x\\_0)$ is linear so we can't be sure it is valid to re-arrange to $x\\_0$ prediction.\n- I think after the method is explained in section 3, the authors should have text that revisits the probability contour intuition they provide during the introduction. I think this is a nice mental model to understand the method but it is very unclear how exactly the method presented in section 3 gives you the desired probability contours that are promised in the introduction. For example, if a contour is very long in embedding space, then definitely the far away points on the contour with have low probability under $\\tilde{p}\\_t(\\tilde{x}\\_t | x\\_0)$ so therefore probability is not constant along the discrete boundary and therefore the discrete boundary does not line up with the probability contour. These kinds of doubts would be alleviated with a clearer presentation linking back to the original probability contour view and some more precise statements.\n- The discussion relating to the confidence factor is very unclear. Firstly, I'm unsure why it is even called a confidence factor, we can be fully confident that our data is discrete so why is there uncertainty about this? It is also concerning that the method can oscillate and mode collapse when r=1 which is your full method. This sounds like a major flaw of the approach if it cannot work at all when your new method is used in isolation. This warrants more discussion regarding this failure case. Furthermore L195 should be explained more as it is very unclear what \"fixing the initial path\" means and it sounds very important if it is required for your method to work.\n\n\nRegarding the experiments:\n- It seems unfair to allow your method to be re-ranked in Table 1 if you don't also re-rank the transformer baseline\n- In Table 3, it is hard to get a good sense of how your method compares to other discrete diffusion approaches. Since they all use different embedding strategies it seems that it is only a good comparison to compare methods within the same embedding strategy. In this case, you are really only comparing to bit diffusion but your reproduction seems to perform much worse than the originally stated results in the bit diffusion paper. It is therefore difficult to place your method with confidence amongst the other discrete diffusion methods.\n\n\n\nMy score is given on the basis of the very unclear presentation meaning I cannot understand the method the author's are proposing. If this can be cleared up in the rebuttal period, I am happy to raise my score.\n\nQUESTIONS:\nHere I have listed some more minor questions for the authors that can help improve the exposition. My main concerns are given in the previous discussion.\n\n- How does your formulation handle the fact that for some $x\\_0$ and $\\epsilon$ samples, there will never be a boundary crossing because $x\\_0$ may be on the outside of the space or $\\epsilon$ is sampled within $x\\_0$'s discrete boudary.\n- On L187 (24) is already deterministic because its an ODE so I don't understand why you then propose a deterministic alternative.\n- In Table 2 it is unclear what ablation you are actually doing, what does it mean to only rescale the forward process versus rescaling the forward and backward?\n\nLIMITATIONS:\nI don't believe the authors adequately describe the limitations of their method when r is near 1, they say that it can be unstable but provide no further details or explanation regarding this failure case.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this study, the authors propose a framework to address the discrepancy between discrete data and continuous modeling in diffusion processes. The approach involves a two-step forward process that first estimates the boundary as a prior distribution, then rescales the forward trajectory to build a boundary conditional diffusion model. The reverse process is proportionally adjusted to ensure the learned contours yield more accurate discrete data. Experimental results show the method's strong performance in language modeling and discrete image generation tasks, outperforming previous models in several areas and establishing a new state-of-the-art for categorical image generation on the Cifar-10 dataset.\n\nSTRENGTHS:\n- The authors have conducted comprehensive experiments, providing thorough and complete details in their exploration. \n- The novelty of the study lies in its discussion of the concept of \"discrete area\", presenting a fresh perspective in the field.\n\nWEAKNESSES:\n- The explanation as to why this new framework can enhance performance is not clearly articulated in the study. \n- From my understanding, Algorithm 1 appears to be equivalent to training original diffusion models with $t\\sim U[t_0,T]$ as opposed to $t\\sim U[1,T]$.\n\nQUESTIONS:\n- Is there a possibility that the boundaries of the discrete areas of two distinct data points might intersect? If so, why would pushing the noise to the boundary be beneficial? Could certain perturbations potentially lead to inaccuracies in the final result?\n- According to Equation (14), shouldn't the input of $\\Psi^{-1}([\\cdot,\\cdot])$ be the boundary point and the \"crossing boundary time\"? If so, why does line 8 of Algorithm 2 suggest that a sequence of $\\tau$ ranging from $T$ to $r\\times t_0$ can serve as the input of $\\Psi^{-1}([\\cdot,\\cdot])$?\n- also please refer to weaknesses.\n\nIf these questions are addressed, I would consider revising the score.\n\nLIMITATIONS:\nThe authors do acknowledge the limitations of their work, which is commendable. Furthermore, I recommend conducting theoretical analysis, such as exploring the convergence properties of their new framework.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-09"}

comments: SUMMARY:
A discrepancy exists when using continuous diffusion models to model discrete data, which has not been sufficiently addressed in past methods. This paper redesigns the forward and backward processes of the diffusion model to eliminate this issue. Experiments on translation tasks and CIFAR-10 effectively validate the proposed method's effectiveness.

STRENGTHS:
1. The writing is clear and concise.

2. This paper aims to address an important and interesting question.

3. Experiments demonstrate that the proposed method surpasses all baselines, effectively validating its effectiveness.

WEAKNESSES:
Please see the Questions.

QUESTIONS:
1. In Sec. 4, the authors present an untrainable $p(W|x_0)$ given the embedding layer. If we employ a trainable $p_{\theta}(W|x_0)$ (e.g., Diffusion-LM [1]), will the discrepancy between discrete data and continuous modeling still exist? If it does, will your proposed method still bring improvements in this scenario?

2. This paper presents the results of translation tasks. I am curious to know whether the proposed method would be effective for models like Plaid [2], which can compute text likelihood. The discrepancy between discrete data and continuous modeling mainly occurs during the step of decoding the embedding into text, while the likelihood calculation does not require this decoding step.

3. In Table 3, the authors report FIDs of 51.27 and 30.97 for D3PM. However, the best generative FID on CIFAR-10 in the original D3PM paper is 7.34. Why didn't the authors report this result as a baseline? This requires further explanation.

[1] Diffusion-LM Improves Controllable Text Generation

[2] Likelihood-Based Diffusion Language Models

LIMITATIONS:
yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper discusses a novel approach to discrete modeling using boundary conditions and diffusion processes. The primary contributions include:\n- Development of a framework for discrete image generation that incorporates binary coding and pixel embedding.\n- Introduction of an intermediate state to illustrate the correlation between discreteness and modeling difficulty.\n- Extensive experimental evaluation on datasets like CIFAR-10, demonstrating competitive results compared to state-of-the-art models.\n- Proposal of a new method for stochastic processes that improve model performance, especially in cases of categorical pixels.\n\nSTRENGTHS:\n- Originality: The paper introduces a unique method for addressing discrete modeling through a combination of binary coding and pixel embedding, which is innovative in the context of diffusion models.\n- Quality: The experimental setup is robust, with detailed evaluation metrics and comparisons to existing methods. The results show significant improvements, indicating the effectiveness of the proposed approach.\n- Clarity: The paper is well-structured, with clear explanations of the methodology and the underlying mathematical formulations. Figures and tables are used effectively to illustrate the results.\n- Significance: The approach has broad applicability in various fields requiring discrete data generation, such as image and language modeling. The improvements in FID scores and other metrics highlight the potential impact of this work on the research community.\n\nWEAKNESSES:\n1. The approach is primarily tested on specific datasets like CIFAR-10. Expanding the evaluation to a wider range of datasets on high-resolution images could provide a more comprehensive assessment of the model's generalizability.\n2. Language model experiments are mainly on translation tasks. Could you provide experiments on language modeling tasks, which is more common for benchmarking diffusion LMs.\n\nQUESTIONS:\nSee weakness.\n\nLIMITATIONS:\nLimitation is mentioned in one sentence in the conclusion section, but not comprehensively discussed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-18"}