Discrete Copula Diffusion
Authors: Anji Liu
Oliver Broadrick
Mathias Niepert
Guy Van den Broeck
Venue: ICLR 2025 Poster (2025)
Keywords: discrete diffusion models
copula
Primary area: generative models

Abstract: Discrete diffusion models have recently shown significant progress in modeling complex data, such as natural languages and DNA sequences. However, unlike diffusion models for continuous data, which can generate high-quality samples in just a few denoising steps, modern discrete diffusion models still require hundreds or even thousands of denoising steps to perform well. In this paper, we identify a fundamental limitation that prevents discrete diffusion models from achieving strong performance with fewer steps -- they fail to capture dependencies between output variables at each denoising step. To address this issue, we provide a formal explanation and introduce a general approach to supplement the missing dependency information by incorporating another deep generative model, termed the copula model. Our method does not require fine-tuning either the diffusion model or the copula model, yet it enables high-quality sample generation with significantly fewer denoising steps. When we apply this approach to autoregressive copula models, the combined model outperforms both models individually in unconditional and conditional text generation. Specifically, the hybrid model achieves better (un)conditional text generation using 8 to 32 times fewer denoising steps than the diffusion model alone. In addition to presenting an effective discrete diffusion generation algorithm, this paper emphasizes the importance of modeling inter-variable dependencies in discrete diffusion.

Published as a conference paper at ICLR 2025
DISCRETE COPULA DIFFUSION
Anji Liu Oliver Broadrick', Mathias Niepert Guy Van den Broeck
'Department of Computer Science, University of California, Los Angeles, USA Institute for Artificial Intelligence, University of Stuttgart, Germany
{liuanji, obroadrick, guyvdb |ecs ucla. edu ma mathias niepert uni-stuttgart.....
ABSTRACT
Discrete diffusion models have recently shown significant progress in modeling complex data, such as natural languages and DNA sequences. However, unlike diffusion models for continuous data, which can generate high-quality samples in just a few denoising steps, modern discrete diffusion models still require hundreds even thousands of denoising steps to perform well. In this paper, we identify a fundamental limitation that prevents discrete diffusion models from achieving strong performance with fewer steps they fail to capture dependencies between output variables at each denoising step. To address this issue, we provide a for- mal explanation and introduce a general approach to supplement the missing de- pendency information by incorporating another deep generative model, termed the copula model. Our method does require fine-tuning either the diffusion model or the copula model, yet it enables high-quality sample generation with significantly fewer denoising steps. When we apply this approach to autoregres- sive copula models, the combined model outperforms both models individually in unconditional and conditional text generation. Specifically, the hybrid model achieves better (un)conditional text generation using 8 to 32 times fewer denois- ing steps than the diffusion model alone. In addition to presenting an effective discrete diffusion generation algorithm, this paper emphasizes the importance of modeling inter-variable dependencies in discrete diffusion
INTRODUCTION
Discrete diffusion models have recently achieved sigmificant progress in modeling complex data such as natural languages (Campbell et al.) [2022] Sahoo et al.] (024).. protein sequences (Gruver et al.)
2023 Morehead et al. 2023), and graphs (Vignac et al. 2022 Huang al.] (023). In particular, recent discrete diffusion models for text generation (Lou et al. 2024; Sahoo et al. 2024 Shi et al.)
2024) have matched or even surpassed the performance of autoregressive models at the scale of
GPT-2 (Radford et al. 2019). Additionally, discrete diffusion models offer improved inference-time controllability using guidance from auxiliary models such as classifiers (Dhariwal Nichol 202T).
making them suitable for controlled generation tasks (Li et al.
2022 Han et al. 2023).
Despite these promising results, discrete diffusiio models still require hundreds to thousands of denoising steps to produce high-quality samples (Austin et al., 2021: Sahoo al. 2024), signifi- cantly affecting their efficiency. In this paper, we identify a fundamental limitation in most discrete diffusion models that hinders their ability to generate high-quality samples in just few steps.
We illustrate the problem in Figure [ At each denoising step, partially completed sample shown denoising model, which predicts the univariate
in the top-left is fed into marginal distributions for each masked token independently. A new output sequence is then sampled based on these univariate marginals before proceeding the next denoising step. The key issue with this process is that when multiple "edits" (i.e., replacing masked tokens with data tokens) are made simultaneously, the model does not account for the joint probability of these changes occurring together. As result, the generated samples often lack coherence, as shown in the bottom-
Code available at http:///gghaaaccoom/mmaaiiiimmla

---

Published as a conference paper at ICLR 2025
The
<MASK>
dog
<MASK> the neighbors.
Inffernnce
Discrete DifTusion Model
0.18
Copula Model
The
puppy
0.11
dog
scared
0.18
the neighbors.
0.11
0:09
0.12
0.79
0.07
0.8
025
Note: this is computed
barking
0.07
frightendd
0.15
black
0.05
calmed
0.07
Univariate marginals
0.07
X
0.47
0.49
004
0.22
0.20
004
implicitly in practice.
0.08
0.44
0.10
0
I Sumple independently
+
jolntty 0
The puppy dog scared the neighbors. 1- likely
The puppy dog calmed the neighbors. 1 likely The barking dog scared the neighbors. 2 likely
The puppy dog frightened the neighbors.
2-d likely
Figure Discrete Copula Diffusion (DCD). At each denoising step, a partially completed se- quence is given as input (top-left). The diffusion model independently predicts the univariate marginals for each masked token, which leads to the samples in the bottom-left. DCD introduces an additional copula model (top-right) to capture the inter-variable dependencies, thereby supple- menting the information missed by the diffusion model. By combining outputs from both models in principled way, DCD achieves better performance than either model individually (see improved samples in the bottom-rrght), enabling few-step discrete diffusion generation.
left of Figure 1 This problem is exacerbated in few-step generation, where many tokens must be edited simultancously. We formally demonstrate that if the diffusion model predicts each variable independently. an irredueible term (in addition to the data entropy) remains in the negative evidence lower bound (ELBO), preventing the model from perfectly capturing the data distribution.
We propose using a generative model, which we refer to as the copula model, to compensate for the missing dependency information between output variables at each denoising step. Our method operates only inference time and can be adapted to any discrete diffusion model and a wide range of copula models. As illustrated on the right side of Figure [ the input sequence is also fed into a copula model that (implicitly) produces information on inter-variable dependencies. This information is combined with the univariate marginals predicted by the diffusion model to produce a more accurate distribution, resulting in higher-quality samples, shown in the bottom-righ corner.
We formally show that the univariate marginals from the diffusion model and the dependencies cap- tured by the copula model can be combined in a principled way, leading to a better approximation of the true denoising distribution under mild assumptions. Further, finding this combined distribution reduces to solving convex optimization problem that can be efficiently approximated in practice.
By instantiating the copula model an autoregressive deep generative model such as GPT (Radford) jet al., (211)), we propose an algorithm that combines any pretrained discrete diffusion model with an autoregressive model to form hybrid model called Discrete Copula Diffusion (DCD). This model is capable of producing high-quality (un)connitional samples with only a few denoising steps. Empirical results on text and antibody generation show that DCD significantly outperforms both of its base models. Moreover, DCD achieves comparable or better performance using 8 to 32 times fewer denoising steps compared to the base discrete diffusion model. In addition proposing a discrete diffusion model capable of few-step generation, we emphasize the importance of modeling inter-variable dependencies in discrete diffusion models.
PRELIMINARIES
We aim to model the joint distribution of variables Xo, set of categorical variables with C cate- gories. Discrete diffusion models (Austin et al. 202I) learn to sample from p(Xo) by modeling the reversal of the following noising process involving X, and set of auxiliary variables (x,)
(1) where Ca; p) refers to the Categorical distribution over æ with class probabilities p. and Q is a C'C transition matrix that is applied independently to every variable =;_; (denote xf_1 as the äth variable of x;-1) to get the corresponding categorical distribution of z]. Specifically, each variable xi_1 is treated as a one-hot vector of size Cx1,, 1, which is then multiplied by Q. to compute the class probabilities of zς. The noising process is designed such that p(xr) follows a simple distribution regardless of the data distribution.
Vt E (L. q(æ, 1)--) == Cat(xx Qu-x;-1),
2

---

Published as a conference paper at ICLR 2025
Instead of using fixed number of predefined time steps, we treat t as continuous variable within the range [), The noising process is now defined by the rate of change p(æ;) w.r.. t: dp(as)
Q Q-p(2:), where QERCxC is transition rate matrix. For any 0<s<t<T, we have
g == Cat( exp((tes)s---pppppee xx.),
where exp(-) denotes the matrix exponential.
Discrete diffusion models represent the reverse diffusion process as Markov chain from ør to Zo. effectively reversing the noising process. Specifically, the reverse diffusion modeled as:
po( :- p(xT) II t=0
pa (=x+1).
In the discrete-time framework, the model trained by maximizing the ELBO, which is defined by the forward joint distribution (0o1:r//=)pp(a))) and the reverse joint distribution (pa (zo:r)) )) (Ho
cct al., (020). the continuous-ting time framework, we can either adopt an extended ELBO objective
(Zhao al. 2024) to learn the likelihood ratios {pl (o(x:() kr/)/p( (z:)) ) amy, allowing for the recovery
of p( Ft) t) in an indirect manner (Lou et al., 2024 Meng et al. 2022 Sun et al. 2022
Following the reverse diffusion process, sampling from a diffusion model involves first sampling
from the prior p(xT) and then recursively sampling æT-1 2o following {ps (2e (2)
CHALLENGE OF MODELING VARIABLE DEPENDENCIES
Unlike continuous diffusion models, which can produce high-quality samples with just few steps (e.g., -.../Song& et al. (2023): Zhou et al. ), discrete diffusion models exhibit strong positive correlation between sample quality and the number of denoising steps. For instance, to generate 1024 text tokens, recent discrete diffusion model SEDD (Lou al. 2024) requires 1024 steps to reach around 35 perplexity (PPL), while with 32 denoising steps the PPL is only around 130.
We argue that the need for large number of sampling steps in discrete diffusion models stems from their inability to capture inter-variable dependence among the outputs. Specifically, at each time step discrete diffusion models independently sample each variable from æ, conditioned # i.e, (1+1)))++)==== == II (x14)) As result, when changing multiple variables from 2+41 to Øt the model fails account for the joint probability of these modifications happening together. In the following, we first quantitatively analyze the performance degradation caused by this independent denoising assumption. We then discuss approaches mitigate this issue.
Quantifying the Performance Drop. The total correlation of distribution p(X) is the KL- divergence between itself and the product of its univariate marginals:
Dre(p (X)(((X)) (re(e(x))=<((e) log (p(z)/ II(as))
x
The following result demonstrates that, under the independent denoising assumption, there is an irreducible component the ELBO that directly stems from ignoring inter-variable dependencies.
Proposition 1. Assume the denoising distributions {po(a,, a 11)1 are fully factorized. Let H(r(X)) denote the entropy of p(X). For any choice of denoising distributions (or equivalently, any parameterization 0), the negative ELBO of the diffusion model is lower bounded by
H(p(Xo)) Σ Dro(o(Xr--(xx)) where Dro(p(Y(X)) :- 1E (To(p-P)))(((
The first term represents the entropy of the data distribution and is irreducible. The second term additionally depends on the noising process and the chosen noise levels, which set an upper limit on the performance of discrete diffusion models that use the independent denoising assumption. Note that although Dro ( X,-1)) is zero according to the definition of the noising process, Dro(q(X, (x(,, Xx)) is not unless the data distribution is fully factorized.
Closing the Performance Gap. While increasing the number denoising steps can improve sample quality, it also introduces significant computational overhead during inference. Our goal is to
3

---

Published as a conference paper at ICLR 2025
use fewer denoising steps while maintaining good sample quality, As shown in Proposition given fixed noising strategy and the number of denoising steps, the only way to reduce the negative ELBO lower bound in Equation is to relax the independent denoising assumption. That is, in addition to modeling the univariate marginals, we must also account for dependencies between variables.
The challenge of capturing inter-variable dependencies during each denoising step can be addressed through adjustments during either training or inference. A direct approach involves modeling both the univariate marginals and the inter-variable dependencies within the diffusion model. However, this requires improving existing sequence-to-sequeee dependencies between output variables directly, which not very well studied in the literature.
architectures (e.g.. Devlin (2018) to capture
Instead, we propose an inference-time time solution that complements the information missed by the pre- trained discrete diffusion model. Specifically, we aim to combine the univariate marginals produced by the diffusion model with the vater-variable dependencies learned by another (possibly smaller) deep generative model, which we refer to as the copula model. The term "copula" traditionally refers to the dependencies between random variables in statistics (Nelsen 2006).
4 MODELING VARIABLE DEPENDENCIES WITH COPULA MODELS
As motivated the previous section, our main goal is to combine the univariate marginals produced by the diffusion model with the inter-variable dependencies captured by copula model. In this section, we first formalize the concept of "combining" " two such distributions in a general context (Sec. 4.1 We then specialize the formulation to the case of diffusion models (Sec. 4.2
4.1 COMBINING UNIVARIATE MARGINALS WITH INTER- VARIABLE DEPENDENCIES
In this section, we discuss how to best inject vater-variable dependence using copula models given target distribution Ptar over X. Assume we have access to Ptar through two sources: (i) the set of all univariate marginal distributions (Ptar(X,) h. and (ii) an estimate Post of the target distribution coming from the copula model, which is also generative model. Our goal is to combine these two estimates to construct ş that is "closer" to the true distribution Ptar than either estimate individually.
We construct P the distribution that (i) matches the set of univariate marginals {Ptar {Pt, J. and (ii) minimizes the KL divergence to Pest- The intuition is that by ensuring P has the correct univariate marginals,
achieve good approximation of Ptar even if Post is biased. To formalize this, first define information projection (I-projection).
Definition 1. The I-projection of a distribution (X) onto a set of distributions P over X is
p" arg min DiL (p q)-
pEP
Let Piar denote the set of distributions over X that share the same univariate marginals as p. We define p the I-projection of Pest onto PP The following proposition shows that regardless of the initial estimate Pest of Ptar. the 1-projection e will be an improved estimate of Ptar in KL-divergence.
Proposition 2. If there exists and IT s.t. Pear (x:)
( # (x.), then DKL (Ptar || p) < DKL DRL(Ptee) Pest..
Having now seen that á is an improved estimate of ptar, we next explore whether it is feasible to compute á given [Prar ( [x)), and Pest- We start by showing that ş has a simple form. Proposition 3. Assume Va, Prax (x)) > 0 and Pest (x)) > 0. Then P exists and has the form
p(x) = (as((a:)
II (x,))
where o; is positive function that depends on x;.
Assume X consists of N categorical variables, each with C categories, can represent the factors } using a matrix VERNV XXC, Under this representation, the combined distribution is
p(x) = Pest (=)) II cxp(V[i,
where V[i denotes the element the ith row and jth column of V and Vii, log (zx). Determining the true matrix V corresponding to p. which is the I-projection of Pest onto PPiee can be reformulated as solving the following convex optimization problem.
4

---

Published as a conference paper at ICLR 2025
General prompt: Let's do ouidoor sports! How about <X>> <>>?
Given additional suffix: in Switzerland
Probability table
Marginals copula:
Use I-projection to adjust p(Y)
Marginals
alpine
Pso 40
skiing
P4400.48
skiing
plum0.90
skiing
diving
equivalln
scuba
diving
52
diving
PP100.07
depentency
alpine
101-00.02
Most peobable planses
while keeping the copula unchanged
scuba
P
Copula
copala,XX Y)
PiuPuu
123
saculee diving
alpine skiing
Sample from
joint
Hew shout alpine skiing?
Figure 2: Illustration of the decomposition of a distribution into univariate marginals and a copula.
Theorem 1. V minimizes the following convex objective function, then the corresponding
defined by Equation
is the I-projection of Pest onto PP
L(V; Ptacs Pest)
Σ Pest (2)) II expp(((i,
Σ
V(i (tar(
(4)
We proceed to explain why I-projecting Pest leads to a better estimate of Plar as suggested by Propo- sition 2 In general, joint distribution can be viewed as combining two independent pieces of information: (i) a set of univariate marginal distributions and (ii) a copula describing the association or dependence among the variables. By the classical work of Sklar (955)). for continuous variables the copula can take the form of a joint distribution with uniform margins and can be combined quite simply with univariate marginal distributions recover the full joint distribution, a fact heavily ex- ploited in statistics (Nelsen 2006). While the discrete case is somewhat less straightforward, recent work of Geenens (2020) has developed the fundamental notions of discrete copula modeling as well, where the information of a copula can be parameterized by odds ratios.
Figure 2 shows an example consisting of two binary variables X and Y. The probability table on the left can be equivalently expressed using univariate marginals (i.e., Po-, p-o. p.1) and the odds Intuitively, w 125 indicates that the phrases "alpine skiing" and "scuba diving" are more likely than others (e.g.. "alpine diving") " and the marginals decide which of the two phrases appears more frequently. The idea of representing the copula with odds ratios generalizes to the multivariate case and is presented in Appendix C
ratio (i.e., copula) w: P5oP11 as shown in the middle of Figure 2 Poo
The following result demonstrates that, under its functional form in Equation onto PPetar only improves the univariate marginals and leaves the copula unchanged regardless of V.
I- I-projecting Pest
Proposition 4. For a positive distribution p and any V E RNC, the distribution g(æ) p(æ)-
II. exp(VVi, vr.)) has the same copula as p.
In general, Proposition 4 holds because scaling factors (e.g., exp( (Vii, xxl)) cancel in odds ratios.
For example, in the 2 x case in Figure 2 scaling the top row of the probability table by would
result the odds ratio w 4po00P11
PsoPII
apo1Pro
Po1P10
4.2 MODELING DEPENDENCE IN DISCRETE DIFFUSION MODELS
Recall from Section S that our goal is to capture inter-variable dependencies between the output variables at each denoising step (e.g., sampling I; from q(x, [x++))). Similar to the general case shown in Section 4.1 we first have a set of univariate marginals {Pam (X;) [x]=+++)}; from the dif- fusion model. Notably, these univariate marginals are fairly accurate since for both discrete-time and continuous- diffusion models, if their respective training losses are minimized, the model recovers the true univariate marginals. This is formally justified in Appendix D
Alongside the univariate marginals, we assume access to a copula model that encodes a distribution over X, Following Section 4.. combining the copula model's distribution with the univariate marginals from the diffusion model will lead to an improved estimate of (X, x++1) (Prop. 2
The performance of the augmented diffusion model hinges two key questions: (i) how well can the copula model capture the inter- variable dependencies in (X, (x+1) (defined by the data distri- bution and the noising process): (ii) given a good copula distribution, how to effectively combine with the univariate marginals obtained from the diffusion model, i.e., how to solve Equation (4).
Equation
closely resembles the matrix scaling problem Idel 2016 See Appendix for details.
5

---

Published as a conference paper at ICLR 2025
5 AUTOREGRESSIVE MODELS AS COPULA MODELS
This section answers the two questions above tailored to the case where the copula model is an autoregressive model such as GPT (Radford al. 2019) and State Space Models (Dao & Gu 2024). Specifically, Section 5.1 discusses how to approximate (X+,(<((+1)) using an autoregressive model trained on the clean data distribution p(Xp) under certain noising processes. Section 5.2 explores the process of performing I-projection from the (autoregressive) copula distribution onto the set of distributions with univariate marginals (pam (X; 1 Finally, Section 5.33 summarizes the sampling procedure with a discrete diffusion model and an autoregressive copula model.
5.1 EXTRACTING COPULA DISTRIBUTIONS FROM AUTOREGRESSIVE MODELS
At step t, to sample z conditioned on T1+1, we need a copula distribution Pcopula (X,) that closely approximates q(x,, 11)+ While this might suggest that the copula model should also be trained with a diffusion model objective, which brings us back to the problem of modeling inter-variable dependencies, show that any model trained the clean data distribution can serve as copula model that indirectly approximates q( (x,] ((+1) under the absorbing mask forward noising process.
The absorbing mask noising process gradually converts data tokens in por-p(22 (Xo) a new category denoted <MASK> through the sequence 2,...., Zr. Specifically, each token in Do is independently converted to <MASK> with probabilities <<a1 <... < aa = in æ1..... ZT. respectively. This is a widely used noising strategy for discrete diffusion models. Since this process only transforms data tokens into the mask token, preserves the dependencies between the remaining unmasked tokens. Therefore, we can decompose q( gX,,| (x++1) as (xt+1) Σx, ((x ( ( .I+1), where g (0++1) is inuitively capturing the joint distribution of generating all currently masked tokens, and g( [ā., xt++)) captures only the choice of which currently masked tokens will actually be generated. Formally, define I as the set of variables i such that 2 <MASK> and J as its complement. The auxiliary distributions have the following form.
Proposition 5. Assume p(Xa) is the clean data distribution and {q(X,) (xX follows the ab- sorbing mask noising process. Let oz, be the probability of conversion to the musk state from X to X3 (Vi). Define X, set of auxiliary variables such that
p(X. = XT x(41) 1[ x/41..
(5)
Then, the distribution q(X, [ 11+1) is the following: q(X,) (0 x++1) = II, a( [z]] x1+1)
-For iE T, q(2)) [i,, m)41) equals 10/0441 xf <MASK> and equals 1 - 04/ /0111 if
-For j. q (it, oi )) 11 if and only if x 2
Since (( x++1) is fully factorized, the copula model only needs account for vaterlvariable transform Peopula (Xo), which timates the clean data distribution, into Pcopula (X, xx+1) that approximates q ((x+(=(+1) by con-
dependencies in q (X, [ Following Equation (5)
ditioning it on the unmasked tokens in Ø++1 (i.e., 2f41). Specifically, for autoregressive copula
models (i.e., Peopula (a) == II. Peoputa(x;; )), we construct Pcopula ( (x2+1) by conditioning each variable on the corresponding preceding tokens in 4//11 while enforcing 7/- =1(41 (VjEJ):
Peopula (wea (1++1)
II sE7
Propula (X- x
1 JEJ
(6)
This copula distribution is biased even if the autoregressive model perfectly captures the data distri- bution since it cannot condition on subsequent unmasked tokens in #t+1. In contrast, while being able to condition on all unmasked tokens, diffusion models cannot capture dependence between variables. Combining the two estimates in proper way will lead to better empirical performance.
Continuing with the example in Figure 2 we assume an autoregressive copula model encodes the probability table on the left. As shown on the right, when provided with the suffix prompt "in Switzerland", the copula model alone cannot adjust its probabilities, as it can only condition on prefix prompts. However, a diffusion model that captures the strong dependence between "Switzer- land" and Y = "skingg can, through I-projection, set the correct marginal probabilities of Y, while keeping the copula unchanged. This allows the model to reliably generate "how about alpine skiing."

---

Published as a conference paper at ICLR 2025
Algorithm Draw samples from a discrete diffusion model with the help of copula model
1: Inputs: a diffusion model pam. copula model Propula. number of time steps '
2: Outputs: a sample do from the discrete diffusion model augmented by the copula model
Initialize: Sample ær from the prior noise distribution p (Xx.)
4: for t == T- to 0
5: Compute {pam (Xi (x.+1)) } and {pam (x:1 ) using the diffusion model
6: Compute Vii, 31] = log Pam (81 m - Pam Pam ( <4 (Vi, 85) following Equation 10
7: Sample ãē from (( [x++1] Pcopula (2)+1) 11, exp V[i, (Pcopula is defined by Equation 6
8:
Sample at from g(x, Te, 2++1) (defined in Proposition 5
Lastly, we need the univariate marginals of gXx, (x+1)), which can be derived by renormalizing (x(x/(0++1)) to zero out the probability of the mask state according to the following result.
Proposition 6. For each i and data category of <MASK>, gf X clax++) ((x clare+1).
As a result, for each i, the distribution Pam (X; (1)44) can be similarly obtained by renormalizing Pam (X) xx-1), which is directly obtained from the denoising model, to exclude the mask state.
5.2 APPROXIMATE I- I-PROJECTION WITH AUTOREGRESSIVE MODELS
Given univariate marginals {Pam [Ma [x+11) and an autoregressive copula distribution Propula (X, (x) both of which estimate the target distribution q(,, [1)+))), our goal is to com- bine them following the T-projection procedure described in Section 4.1 Specifically, this involves solving the convex optimization problem in Equation { which is specialized to the following:
Σ Peopula (ampulaaa:,/a000
Il (1I,
ΣΣ
Vii pPm (atm(a//aaa (#(())+11-
Following Theorem if V minimizes Equation 7 then the distribution defined by p( (2(+1) Pcopula (2) 1 - II exp( Vit, E:l) is the I-projection of Peopula (Ø. [4++1) onto the set of distribu- tions with the univariate marginals {pam (8/(æ141));. which is the desired combined distribution.
Consider initializing all coefficients in V to zero, i.e., (x+1) Peopula (Ze [++1). For each row i, if we only optimize the values Vii, and fix the rest to zero, the optimal coefficients are
Ye, cli log Pam (X; c(ax,+1) log Peopula (X; c|
(8)
We approximate the solution to Equation 7 by applying the above update (Eq. (8) to each row in
V independently, as it strikes proper balance between efficiency and empirical performance.
While the first term on the right-hand side of Equation (8) can be acquired from the diffusion model, the second term is not accessible through the copula model. Plug in the definition in Equation (6)
the required marginal probabilities can be written as (for E J, Peopula ( (i/ [x+11) iff 2 = x 1+1
ViEl, Peopula (5) 11+ Peopula (X; Xx, = where K; - jJEJ and j <i). (9)
The above probabilities cannot be computed from the autoregressive model since we need to "marginalize out" preceding tokens that are not in K; (i.e., those not given evidence in æ++1))
However, these terms can be estimated using the diffusion model. Assume both the diffusion model and the autoregressive model perfectly encode the data distribution. According to Propo- sition 6 the diffusion model computes pam [(x])+))=(({{p)))))))
Comparing it to Equation (9). which gives Peopula(X]] (X/| [x++)) ((x (x we only need to additionally restrict the diffusion model to only condition on preceding unmasked tokens in I++1, since K; is the intersection of J and {j : j < i). Therefore, if both models well-approximate the data distribution, Peopula
have [ ( (1))))))) R Pam ( where the equality holds since all <i but not in x/; are <MASK>, and does not "contribute to" the distribution of X
(8+](=(41) (
values in x141 according to Proposition 5). Correspondingly, we update V following
Vi,c, Vli. c] log Pam (X} c/2 log Pam (X) c (41)))
(10)
7

---

Published as a conference paper at ICLR 2025
For denoising neural networks that are implemented with bidirectional Transformers, we simply apply causal attention masks to the self-attention layers to obtain {pam (X; x x141))-
5.3 THE OVERALL DIFFUSION SAMPLING PROCESS
Given a diffusion model Pau and an autoregressive copula model Pcopula: the sampling, procedure is outlined in Algorithm | First, we sample æt from the prior noise distribution p(Xr) (line 3). During each denoising step i, we compute the univariate marginals {pam (X/] and
{pam (X; [1111 based on the previously obtained I++1 (line 5). These marginals are then used
to compute the entries in V (line 6), which approximates the I-projection of Peopula (Xe( onto the set of distributions with univariate marginals {pam (Xi x+++1) }) (cf. Sec. 5.2).
Afterwards, sample ã; from the combined distribution pXx, x (line 7). Specifically, follow- ing Equation 6 we sample autoregressively following p [2 11. 0 [2++1 2,²)), where
p P (141, Poopula (X, = - = -τ¹) exp (i. 1 - =
Finally, we sample 2, from gX,, (e, xt+1) (line 8) as defined in Proposition 5 To improve the algorithm's efficiency, we introduce a variant that unmasks tokens in autoregressive manner. Specifically, at step I, all tokens except the first (T ()/T portion the tokens in og are converted to <MASK>. Since á is sampled autoregressively. this allows us to techniques such KV- caching for autoregressive Transformers (Pope et al.) 2023) to significantly reduce computation cost introduced by the copula model. See Appendix E for details and the concrete algorithm.
EXPERIMENTS
We empirically validate the proposed method, Discrete Copula Diffusion (DCD), on language mod- [..c cling tasks (Sec. 6.1 and 6.2) and antibody sequence infilling tasks (Sec. 6.3). For all tasks, we eval- uate whether DCD effectively reduce the number of diffusion steps while maintaining strong performance. Specifically, since DCD combines two pretrained models: a discrete diffusion model and autoregressive copula model, we examine whether DCD outperforms each individual model.
6.1 UNCONDITIONAL TEXT GENERATION
We first compare the quality of unconditional samples generated by models trained on either Web- Text (Radford et al. 2019)
Wex (Gokaslan & Cohen 2019). which contain web content extracted from URLs shared on Reddit with a minimum number of upvotes. We adopt the medium- sized SEDD model (Lou al. 2024) (SEDD&) since it is SoTA discrete diffusion model for text generation. The GPT-2-small model RRadford et al. 2019) (GPT-2g) serves as the copula model.
We generate samples of 128 tokens each. Following Han et al. (2023): ; Dieleman et al. (2022), we evaluate sample quality using their generative perplexity, which is the perplexity of the samples when evaluated with the GPT-2-large model. Since previous studies have observed that this metric can be affected by distribution annealing methods such as nucleus sampling, we always sample directly from the models. SEDDy, is evaluated with to 256 diffusion steps and DCD (i.c., SEDD, with GPT-22, as the copula model) is run with diffusion steps ranging from 2 to 32. We adopt the log-linear noise schedule suggested by the SEDD paper. See Appendix G.T for more details.
For each configuration, we draw 10,000 samples and report the average perplexity in Figure First, when fixing the number of denoising steps between 2 to 32, we observe that DCD outperforms both SEDD,, with the same number of denoising steps and GPT-2;. This provides empirical validation of the effectiveness of the I-projection procedure for modeling interlvariable dependencies.
Additionally, DCD with just 4 denoising steps achieves performance comparable to SEDD,; with 128 steps, representing a 32x reduction in the number of denoising steps. This result not only demonstrates the efficiency of DCD but also underscores the importance of modeling inter-variable dependencies in discrete diffusion models, particularly in few-step generation settings.
Finally, shown in Figure 4 SEDD fails generate fluent and meaningful sentences given only few diffusion steps, as too many tokens have to be generated in each step. In contrast, by modeling the inter-variable dependencies, DCD generates smooth sentences with only 4 denoising steps.
8

---

Published as a conference paper at ICLR 2025
SEDD (4 steps)
2560
SEDDmn
interesting that the A+N using enforzope thewhich Cookbook using
Generative Parplexity
DCD (SEDDm GPT-2s)
made antimidesis stull'(he grow and juiges 7" And "age guods
640
SEDD (256 sleps)
Singh, who served as chief minister in charge and prime ministee in change of the L.K, had asked Russian PM to attend the leeland meeting of 2005.
160
DCD (4 steps)
He added the United States should continue "dosble in channel media
discussions" but stressed the importance of agrocment bssed on the purpose of dialoguie Putin said Moscow had envisaged sending ravy ships from
40
DCD (16 steps)
8
16
32
64
128
256
Among the dozens of layoflk Detroit inflicted last week in September fell to lnyotl of 243,000 worliers, or just percent of the city's 3.2 million population.
# denoising steps
Figure 3: Generative perplexity (4) with
Figure 4: Generated text from SEDD and DCD with different number of steps. See Appendix I for more.
different numbers of denoising steps.
Table 1: Evaluation of text infilling performance using the MAUVE score (1) with 5 prompt masks. Scores of DCD are all better than (i) SEDD with the same denoising steps, and (ii) GPT-23-
Prompt ranges
SSD-LM
GPT-2;
SEDD,
DCD (ours)
(remainder is masked)
100
500
N/A
2
4
8
16
2
4
8
16
32
[0.1.0.2] & (0.5.0.7]
0.057 0.083
0.079
0.013
0.051 0.122 0.152 0.201
0.158
0.187 0.185
0.195 0.211
[0.25.0.75]
0.072 0.108
0,188
0.027
0.110 0.226 0.237 0.278
0,249
0.251 0.257 0.314 0.298
10.00.111 & (0.4.066) [0.9.1.0]
0.333 0.681
0,928
0.827 0.940 0.972 0.980 0.979
0,962
0.976 0,979 0.982 0.983
[0.4.0.5] & (0.811.0
0.436 0.565
0,914
0.896 0.944 0.978 0.978 0.980
0.963
0.975 0.975 0.976 0.981
[0.2.0.] & [06,088]
0.041 0.054
0.069
0.016 0.056 0.128 0.207 0.215
0.171 0.178 0,215 0.217 0.403
Efficiency. We compare the sample time and the generative perplexity of DCD against competitive
baselines in Figure 5 We additionally adopt another recent discrete diffusion baseline MDLM
2024). We adopt the autoregressive version of DCD as described in Section [5. and Appendix E Compared to the baselines, DCD consistently achieves better generative perplexity
(Shhoo et al.
given a fixed runtime constraint. It also requires less time to reach a desired perplexity value. We
defer comprehensive study of DCD's efficiency to Appendix F
6.2 CONDITIONAL TEXT GENERATION
We now move on to conditional text generation, where certain tokens are provided in advance, and the task is to generate the remaining tokens. As shown in the first column of Table I five mask strategies, where tokens in specific prompt ranges are given (we use a sequence length of 128). We adopt the MAUVE score (Pillutla et al. (02TT) with the default settings to compare the difference between the generated and original texts. See Appendix G.2 for further details.
For all methods, we the same set of 2,000 text sequences from the validation set of WikiText-- 103 (Merity et al. (2022). After applying the prompt mask, we generate 5 samples for each prompt, resulting in total number of 10,000 samples.
In addition to SEDDy and GPT-2s, we compare against SSD-LM (Han et al. 2023), which is a semi- autoregressive diffusion model designed for text infilling. We adopt the autoregressive unmasking variant of DCD described in the last paragraph of Section 5.3
Results are presented in Table First, DCD outperforms all three baselines in all five tasks. Ad- ditionally, when fixing the number of denoising steps between 2 and 32, DCD surpasses both of its base models. Notably, while both GPT-2, and the 2-step SEDD,, performs poorly on the first, the second, and the fifth tasks, combining them in a principled way allows DCD to achieve significantly better performance using only two denoising steps.
6.3 ANTIBODY SEQUENCE INFILLING
We consider the task of unguided antibody infilling, where certain complementarity determining regions (CDRs) of antibodies (i.c., sequences of amino acids) are missing and to be generated by the model. We adopt NOC-D (Gruver et al. 2023), which is discrete diffusion model trained on 104K

---

Published as a conference paper at ICLR 2025
Generative Perplexity
300
SEDDm (320M)
Figure 6: Antibody sequence infilling perfor- mance measured by sequence recovery rate (t). We compare DCD against its two base models in two tasks, where amino acids at different loca- tions are masked. DCD outperforms both base- lines with only 4 denoising steps.
DCD (SEDDm GPT2s; 405M)
200
GPT2s (85M)
MDLM (90M)
100
Task
Method
steps
10
15
HCDR[1+2+3]
(H+L)CDRI
Runtime (s/sample)
GPT NOS-D DCD
N/A
57.21
90.28
Figure 5: Sampling time generative perplex- ity (the autoregressive version of DCD is used).
64
51.56
88.82
58.28
91.58
antibody sequences from the Observed Antibody Space dataset (Ruffolo al. 2023 We further
train a GPT model on the same dataset as the copula model. See Appendix G.3 for training details.
We follow Gruver al. (2023) to select the same 10 antibody seed sequences from paired OAS Olsen et al. 2022). We consider two infilling tasks: (i) three CDRs (HCDRI, HCDR2, HCDR3} masked, and (fi) two CDRs (HCDRI, LCDRI) are masked. We follow the original paper and run 64 diffusion steps for NOS-D. For DCD (i.e., combining NOS-D with the trained GPT model as the copula model), we use 4 denoising steps. We measure the sequence recovery rate, i.e., the accuracy of the infilled sequences given the ground truth sequence.
As shown in Figure 6 by combining the univariate marginals from NOS-D and the dependencies captured by the GPT model, DCD can also perform well in antibody sequence infilling tasks.
7 RELATED WORK AND CONCLUSION
Diffusion models have been widely applied to model discrete data such as text and DNA sequences. Encouraged by the successes of continuous diffusion models (e.g.. Ho et al. (2020); [Snn et al. (020 ) initial attempts convert discrete data into continuous embeddings with either predetined or learned mappings. This enables the use of continuous diffusion models for discrete data (Chen ct al. 2022 Dieleman et al. 2022 Li et al.. 2022 Lovelace al.. 2023). However, due the need for ad-hoc mappings between the discrete data space and the continuous embedding space, which have to be pre-defined or pre-trained, continuous diffusion models are not as effective for modeling
discrete distributions (Strudel al. 2022 Li ct al. 2022 Dieleman et al. (022).
Austin et al. 202T proposed the first diffusion model designed directly for discrete data. Later works further improved discrete diffusion models from various aspects such as better loss func- tions/learning objectives (Campbell et al. 2022] Meng et al.) 2022 Lou et al. 2024 Benton et al.)
2022). better model architectures (Sun et al. (022), better sampling algorithms (Chen et al. (023))
and unifying and scaling up existing techniques (Sahoo et al. 2024 Shi et al. 2024)..
Despite the recent breakthroughs of discrete diffusion models, few papers address the challenge of sampling in a few denoising steps. Some works attribute the failure to perform high-quality few-step generation to a scaling problem of the model. However, we show that the fundamental problem lies in the assumption made by discrete diffusion models that each variable is denoised independently each step. In addition identifying this problem, we propose general solution Discrete Copula Diffusion that combines a discrete diffusion model with copula model at inference time to obtain a better estimate of the denoising distribution at each step. Concurrently, Guo al. (2024) show that energy-based models can also be used as copula models to capture inter-variable dependencies.
There are a few limitations of DCD. First, in addition to a discrete diffusion model, it requires another copula model, which may require additional training for certain applications. Second, although the I-projected distribution is guaranteed as a better estimate of the target distribution, the I-projection step often needs to be approximated in practice. Finally, although DCD requires fewer denoising steps, the computation cost of each step is higher than in discrete diffusion mod- els. Therefore, DCD may not always provide a notable speedup. However, DCD points out the inter-dependency modeling problem and opens up the possibility of combining different types of generative models for better overall performance.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
This work was funded in part by the DARPA ANSR program under award FA8750-23-2-0004 the DARPA CODORD program under award HR00112590089, the Deutsche Forschungsgemein- schaft (DFG, German Research Foundation) under Germany's 's Excellence Strategy EXC 2075 390740016, NSF grant #IIS- 1943641, and gifts from Adobe Research and Amazon. We acknowl- edge the support of the Stuttgart Center for Simulation Science (SimTech). MN thanks IMPRS-IS (International Max Planck Research School for Intelligent Systems) for the support.
REFERENCES
Zeyuan Allen-Zhu, Yuanzhi Li, Rafael Oliveira, and Avi Wigderson. Much faster algorithms for matrix scaling. In 2017 IEEE 58th Annual Symposium on Foundations of Computer Science (FOCS), pp. 890-901. IEEE, 2017.
Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffussion models in discrete state-spaces. Advances in Neural Information Processing Systems, 34: 7981-17993, 2021.
Joe Benton, Yuyang Shi, Valentin De Bortoli, George Deligiannidis, and Arnaud Doucet. From denoising diffusions to denoising markov models. arXiv preprint arXtv:2211.03595, 2022.
Andrew Campbell, Joe Benton, Valentin De Bortoli, Tom Rainforth, George Deligiannidis, and Arnaud Doucet. A continuous time framework for discrete denoising models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, pp. 28266 28279, 2022.
Boyuan Chen, Diego Marti Monso, Yilun Du Max Simchowitz, Russ Tedrake, and Vincent Sitz- mann. Diffusion forcing: Next-token prediction meets full-sequence diffusion. arXiv preprint arXiv:2407.01392, 2024.
Ting Chen, Ruixiang ZHANG, and Geoffrey Hinton. Analog bits: Generating discrete data using diffusion models with self-conditioning. In The Eleventh International Conference on Learning Representations, 2022.
Zixiang Chen, Huizhuo Yuan, Yongqian Li, Yiwen Kou, Junkai Zhang, and Quanquan Gu. Fast sampling via de-randomization for discrete diffusion models. arXiv preprint arXiv:2312.09193, 2023.
Tri Dao and Albert Gu. Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality. In International Conference on Machine Learning (ICML), 2024.
Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 1810.88005. 2018
Prafulla Dhariwal and Alex Nichol. Diffusion models beat gans on image synthesis. In Proceedings of the 35th International Conference on Neural Information Processing Systems, pp. 8780-8794, 2021.
Sander Dieleman, Laurent Sartran, Arman Roshannai, Nikolay Savinov, Yaroslav Ganin, Pierre H Richemond, Arnaud Doucet, Robin Strudel, Chris Dyer, Conor Durkan, et al. Continuous diffu- sion for categorical data. arXiv preprint arXiv:2211. 2022.
Gery Geenens. Copula modeling for discrete random vectors. Dependence 2020.
Aaron Gokaslan and Vanya Cohen. Openwebtext corpus. http://SSky1io0007.gtthi. (peenWebTextCorpus 2019.
Nate Gruver, Samuel Stanton, Nathan Frey, Tim GJ Rudner, Isidro Hotzel, Julien Lafrance-Vannee Arvind Rajpal, Kyunghyun Cho, and Andrew Gordon Wilson. Protein design with guided discrete diffusion. In Proceedings of the 37th International Conference on Neural Information Processing Systems, pp. 12489-12517, 2023.
11

---

Published as a conference paper at ICLR 2025
Wei Guo, Yuchen Zhu, Molei Tao, and Yongxin Chen. Plug-and-play controllable generation for discrete masked models. arXiv preprint arXiv:2410.02143, 2024.
Xiaochuang Han, Sachin Kumar, and Yulia Tsvetkov. Ssd-Im: Semi- semi-autoregressivve Simplex-base" diffusion language model for text generation and modular control. In Proceedings of the 6lst Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp- 11575-11596, 2023.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Han Huang, Leilei Sun, Bowen Du, and Weifeng Lv. Conditional diffusion based discrete graph structures for molecular graph generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 4302-4311, 2023.
Martin Idel. A review of matrix scaling and sinkhorn's 's normal form for matrices and positive maps. arXiv preprint arXiv: /609.06660000000 (609,06349, 2016.
Bahman Kalantari and Leonid Khachiyan. On the rate of convergence of deterministic and random- ized ras matrix scaling algorithms. Operations research letters, 14(5):237-244, 1993.
Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori Hashimoto. Diffusion- Im improves controllable text generation. Advances in Neural Information Processing Systems, 2022.
Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution. arXiv preprint arXiv:2310.16834 2024.
Justin Lovelace, Varsha Kishore, Chao Wan, Eliot Shekhtman, and Kilian Q Weinberger. Latent diffusion for language generation. In Proceedings the 37th International Conference on Neural Information Processing Systems, pp. (6998-57022 2023.
Chenlin Meng, Kristy Choi, Jiaming Song, and Stefano Ermon. Concrete score matching: General- ized score matching for discrete data. Advances in Neural Information Processing Systems, 35: 34532-34545, 2022.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference on Learning Representations, 2022.
Alex Morehead, Jeffrey Ruffolo, Aadyot Bhatnagar, and Ali Madani. Towards joint sequence- structure generation of nucleic acid and protein complexes with se (3)-discrete diffusion. arXiv preprint arXiv:2401.06155 2023.
Roger B Nelsen. An introduction copulas, 2006.
Tobias H Olsen, Fergus Boyles, and Charlotte M Deane. Observed antibody space: A diverse database of cleaned, annotated, and translated unpaired and paired antibody sequences. Protein Science, 31(1):141-146, 2022.
Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. Mauve: measuring the gap between neural text and human text using di- vergence frontiers. In Proceedings of the 35th International Conference on Neural Information Processing Systems, pp. 1816-4828 2021.
Reiner Pope. Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Jonathan Heck. Kefan Xiao, Shivani Agrawal, and Jeff Dean. Efficiently scaling transformer inference. Proceedings of Machine Learning and Systems, 5:606-624, 2023.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Tamás Rudas. Lectures on categorical data analysis. Springer, 2018.
12

---

Published as a conference paper at ICLR 2025
Jeffrey A Ruffolo, Lee-Shin Chu, Sai Pooja Mahajan, and Jeffrey J Gray. Fast, accurate antibody structure prediction from deep learning on massive set of natural antibodies. Nature communica- tions, 14(1):2389, 2023.
Ludger Ruschendorf. Convergence of the iterative proportional fitting procedure. The Annals of Statistics, pp. 1 1160-1174, 1995.
Subham Sekhar Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin T Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and effective masked diffusion language models. arXiv preprint MXXiv:2406.07524 2024.
Jiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis K. Titsias. Simplified and gener- alized masked diffusion for discrete data. Advances in Neural Information Processing Systems, 2024.
Richard Sinkhorn. A relationship between arbitrary positive matrices and doubly stochastic matri- ces. The annals of mathematical statistics, 35(2):876-879, 1964.
M Sklar. Fonctions de répartition à dimensions et leurs marges. In Annules de I'ISUP, volume 8, pp. 229-231, 1959.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamies. In International conference on machine learn- ing, pp. 2256-2265. PMLR, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Interna- tional Conference on Learning Representations, 2020.
Yang Song, Prafulla Dhariwal, Mark Chen, and Ila Sutskever. Consistency models. International Conference Machine Learning, pp. 3221 122252.. PMLR, 2023.
Robin Strudel, Corentin Tallec, Florent Altché, Yilun Du, Yaroslav Ganin, Arthur Mensch, Will Grathwohl, Nikolay Savinov, Sander Dieleman, Laurent Sifre, et al. Self-conditioned embedding diffusion for text generation. arXiv preprint arXiv:2211.044 1.02366 2022.
Haoran Sun. Lijun Yu, Bo Dai, Dale Schuurmans, and Hanjun Dai. Score-based continuous-time discrete diffusion models. In The Eleventh International Conference on Learning Representa- tions, 2022.
Clement Vignac, Igor Krawezuk, Antoine Siraudin, Bohan Wang, Volkan Cevher, and Pascal Frossard. Digress: Discrete denoising diffusion for graph generation. In The Eleventh Inter- national Conference on Learning Representations, 2022.
Tong Wu, Zhihao Fan, Xiao Liu, Hai-Tao Zheng, Yeyun Gong, Jian Jiao, Juntao Li, Jian Guo, Nan Duan, Weizhu Chen, al. Ar-diffusion: Auto-regressive diffusion model for text generation. In Teirth- Conference on Neural Information Processing Systems, 2023.
Lingxiao Zhao, Xueying Ding, Lijun Yu, and Leman Akoglu. Improving and unifying discrtte&comminuous-tinee discrete denoising diffusion. arXiv preprint arXiv:2402.0377 037011 2024.
Mingyuan Zhou, Huangjie Zheng, Zhendong Wang, Mingzhang Yin, and Hai Huang. Score identity distillation: Exponentially fast distillation of pretrained diffusion models for one-step generation. In International Conference on Machine Learning, 2024.
13

---

Published as a conference paper at ICLR 2025
A PROOF OF THE THEORETICAL RESULTS
Proof of Proposition y Following
(Ho et al. 2020
et al. 2015), the negative ELBO
can be decomposed as follows:
Pe 5 (F x22)
L = Eg
log p(rr)
log
(( xx1-1)
Pe Po(xx=1 (2a) ((0
Egg
log p(r))
log
( (24)) (24))
log p
pe (a (t))
E
log
log logp())))
((xT))
q ( xt.)
T
(KL(T(TTT) p(xT)) B.g
Σ
pa (a( (x1(0(0)-111r))
H(xo))
(11)
The first term equals 0 as we assume the noise distribution p(X)) is consistent in the noising and the denoising processes. Given the independent denoising assumption, when the denoising distribution are optimal, we have
Vt E (1,..., T), po(2x-11ax;) = II g(
Plug in Equation 11 and using the definition of total correlation, we have:
L (KL(T(xT) || p(xT)) + Eg
Σ
DkL(g(x1-11ss))
lI II 9 [(<(L11121) H(xo)
1=1
DKL(g(IT) || p(xT)) + Σ Dro(((Xx-1Xx.)) H(p(Xo))
T 2 H(p(Xo)) + Σ Dre(((X(-1Xx))
Proof of Proposition According to Pythagoras' triangle-inequality theorem, if ý is the I- projection of Pest onto Pitee. and Pite convex (this can be shown by applying the definition of convex set), the following holds for any p' E PPiar:
D Peat) 2 DRL(W 11)) + DKL (D Pest).
(12)
Choosing p/ -Paar we have
DRL. (ptar ()) S (par || Pest) - DKL(D Pest) < DKL. (ptar Pest),
where the last inequality holds since DKL (f pest)> if the set of univariate marginals of Pest and Ptar are different (as assumed the proposition).
Proof of Proposition 3 Following the definition of P. we write down the constrained optimization problem as follows
minimize (kL(p' || Pest)
s.t. Vi (1..... N). E (l...., C}. Σ 21) = Prar(x;).
14

---

Published as a conference paper at ICLR 2025
To incorporate the constraints, use the method of Lagrange multipliers. The Lagrangian for this problem is
c(p, (WDEE) ('x( log
p'(x)
N
(aa)))
Σ p'( Peac((ss)
Post (x)
=12
where the Lagrange multipliers { enforce the univariate marginal constraints.
To minimize the Lagrangian with respect to p' (æ), we take the partial derivative of L(y', (A,) with respect to (zx and it to 0:
oL(y', (1)) log 0p((z)
p((x)) +1+ (as)) (e))
Simplifying this equation gives
p'(x)= Post (x) exp (11 (-1-E
Defining cxpp)))))) - 1/N) gives p'(x)) = Pest (x: (z,s).
Existence of the solution follows from the fact that (i) the objective function is convex and bounded (since probability values are in [0, IJ, and (ii) the set of constraints is feasible (e.g., p'(x)) II, (r,) or (xx))= Ptarr
Proof of Theorem 7 We show that for any V that minimizes the objective function L(V; Pears Pest), the corresponding p' defined by p' p' (2) II, cxp(VVii belongs to the set Pirar- Specifically, for any V that minimizes the objective, the partial derivative of L(V Pears Post) with respect any V[i should be 0:
aL(V: Pear:Pest exp( ([i, V[
Σ Pest (20 F.)) II
exp((V Ptar 0.
jtt
Plug in the definition of p', we have
Er' (z)) - Prar((x)) = p((xz) Pear (zi).
(13)
Since Equation 13 holds for all (i) pairs, we have that every minimizer of L(V; Ptars Pest corresponds to distribution p' in DPw converse: if distribution p' with the above-defined form belongs to PPtar then the corresponding V is a minimizer of L(V; Ptar: Pest).
mar Since L(V; Ptar Pest) convex, we can also argue the
According to Proposition 3 the solution to the following I-projection exists and its solution و has the same form as y/.
arg min DKL (p' Peat). N'EPE...
Since á has the same form as p' (by Prop. and belongs to Pleee it is the a minimizer of L(V; Ptars P.st).
Proof of Proposition The copula of is shown to invariant under rescalings of the form ((x) oc
p(x) II, exp( (i, 2.)) for any V N by using the parameterization of discrete copula by conditional odds ratios (Definition 2p. The scaling factors cancel in the ratios as shown, e.g. by
Rudas (2018) Theorem 12.3).
15

---

Published as a conference paper at ICLR 2025
Proof of Proposition
We start by writing the probability ((01/22441 using the Bayes' rule:
g(xx))
g ( g(x(+1/201)
g(x1+1))
= =Σ 20p
a(0 (x)) g( (z0) p(æo),
(14)
((x2+1)
where the last equality follows from g( - Emo (20) ( p(zo)- Recall from the proposition that
I is defined the set of variables i such that 2441 <MASK> and J is the complement of I.
First, we must have -/ = for JE E since for any other value of X we have q(x++1 (x+) = in
Equation (14). As result, (xx,(<1++1 is also zero.
We then move our attention to the variables in I. We first consider the probability g(Xi <MASK> xx+1) for any I. Following Equation [4 we have
(X; = <MASK> ((+1) = =ΣΣ
(( (2x) (((,:/20) p(20),
((+++1)
20
1
2(<2++1 (2;)) ((x.),
q (
g(tt+ = <MASK> X; = <MASK> g(X; = <MASK>)
(X (+1 <MASK>)
x(; <MASK>)
O2
(15)
((;+) = <MASK>)
a++1
We then focus on X/ x/, where none of the value in x is <MASK>, Note that we also need to have X/- = =4t11.
(( xx+1) x
Σ ((0++1)0)) ( ((x,/=))))))))))))))))))))())( g(xo),
20
(a)
q 12x()
(Xo xz),
11
0++1 0q
q(xo xt))
0tt
(Xo =x;),
(16)
where p( (Xo) is the data distribution; (a) follows from the fact that no value in æ; is <MASK>, hence 20 Rzi 01-1-a; is the probability of transitioning into the mask state from time / to time (+) 1.
Denote X; as set of variables with the same configuration and semantics as X,, with the only dif- ference that the category <MASK> is excluded. By following Equation (16) and apply normalization, we conclude that
q [+++) = p(X क/ x m/+1) 1|2 m/41l
(17)
This matches the definition Equation
5
Finally, we verify the correctness of the distribution q(Xe [2, x141) defined in the proposition by verifying the following for any 2t
q(<e|<++1) Σ g( q( x.+1).
(18)
Denote K the of variables such that æ; = <MASK> and its complement. First, if L C J (i.e., 1 C K), then both the left-hand side (LHS) and the right-hand sides (RHS) are zero Specifi- cally, the RHS is zero since according to the definition, Vie & K, we have q( x/ =0
Next, if K C I, we can decompose q( (x(+1) as follows
((0//2++1) = q((,"" K (x(1)) II
q(af)
II g(x) (241). JEJ
(19)
iEK
16

---

Published as a conference paper at ICLR 2025
For any jE z/ + #2/1 then both the LHS and the RHS of Equation 18)
Otherwise
always have q(>// [++4)) =1. Therefore, Equation 19) can be further simplified as
(<m,)<++1) q( K (++1)) II
(x))) 21+1).
(20)
KEK
We then proceed to simplify the RHS of Equation (18)
Σ
q 11
x(+1)
q( Tt. x(41),
[K]
14||
Σ
[
a++1 ag
a:41
a++1
[K|-|
q , at
[e(+1)
a,+1
II a(<)/aat1)
0/++1
#
REK
|||-|K|
1 I.K [xe+1)
06-1 -0
0+++1
EEK
x2 p(x
P.K P.K
I.K X = )
II
g(+))=++))
iEK
(0) X (X
IiK
L (x411)
II KEK
(21)
where (a) follows from Equation
15 (6) applies the definition in Equation 17 and (c) is result
of applying Equation 16 to the case where a (क. / ax.) are not <MASK>.
By combining Equations 21 and (20p, we conclude that the LHS and the RHS of Equation (8)
are proportional to each other. Since they are both normalized distributions, they must also match exactly.
Proof of Proposition We first state a more detailed version of the proposition: for each variable and data category c (c <MASK>), we have
1 Z q(x: clæ,41), where Z
qX; c|
Σ
9(X: c(x
CX
According to the proof of Proposition 5 Equation (8) holds for all m4 Therefore, we have that for each i and each data category xt + <MASK>,
Σ (( qaa//ãr, x(+1).
(22)
If i E J, then both the LHS of the above equation and q( [ m4+1) equals one if and only if xt ==:44---- Therefore, the result holds trivially.
Next, if if denote I,:= ((t), Equation
22 is simplified to
<00 x++1)
Σ
q x (0 x.41),
q _t [m++1) g 1 4(41)
# Av
a(} = 2((011) = at)
xt x(11)11)
act1 -4
14+1
17

---

Published as a conference paper at ICLR 2025
Therefore, have
1 g(X; 21(x++1)= z q(x; xx)0411), where =
Σ
g(x; =x/1x441)
B RELATION BETWEEN L(V; Ptar, Post) AND MATRIX SCALING
The matrix scaling problem gives matrix A as input and asks for diagonal 'scaling" matrices X and such that XAY is doubly stochastic (its row and column sums are all one). More gener- ally, target and column solvability of this problem for positive matrices was established by [Sinkhorn (1964), and its algo- rithms (sometimes called iterative proportional fitting), generalizations, and numerous applications have been studied thoroughly KKlantari & Khachiyan 1993; Ruschendorf, 1995 Allen-Zhu et al., 2017]; see (Idel 2016) for review. Taking the mulndimensional generalization of the problem and interpreting the tensor as (unnormalized) probability distribution yields the connection to our problem, with the target sums being the univariate marginal distributions.
vectors and
provided and need contain only ones. The
C PARAMETERIZING DISCRETE COPULAS BY ODDS RATIOS
We start by formally defining odds ratios.
Definition 2 (Rudas (2018)). Let be distribution over variables X each taking values in {0, 1}. For partition of into sets and B, the conditional odds ratio of variables conditioned on the assignment B is
Ilo. b)
(A C
IIee p(a,
where is the assignments A whose parity is the same as the number of variables in A, and d is the set of assignments whose parity is different.
In the case of more than two categories per variable, COR b) can generalized further to be a set similarly defined ratios (see, e.g., Rudas (2018). Together the set of all conditional odds ra- tios COR, (A) B b) for partitions of X into sets A and B with |A| 2 completely specifies the association among the variables the joint distribution p. as established by the following thcorem.
Theorem 2 (Rudas (2118)). Let q and r be positive probability distributions on a the set of variables X each taking values {0, kk) Then there exists a unique probability distribution p such that p has the same univariate marginal distributions as that is, for all
p(x.) = q(x:),
and p has the same copula q. that is for all partitions of X into sets A and B with [A] 2 2,
COR = b) = COR,(AR))) = b).
Proof. This follows from (Rudas 2018 Theorem 10.2) by taking the descending set to contain the empty set and all singletons (and the ascending set, its complement).
Theorem 2 shows how any distribution can be viewed as combining independent marginal distri-
butions (i.e., from r) and odds ratios (i.e., from q). Such combination has desirable properties. For example, in the case two variables with possibly many categories, has been shown that among all distributions with the same margins as the distribution minimizes the KL-divergence to q (Geenens) 2020) Theorem 6.2), i.c. that is the information projection of q onto the set of distributions with the margins of r.
18

---

Published as a conference paper at ICLR 2025
D UNBIASED UNIVARIATE MARGINALS FROM DISCRETE DIFFUSION MODELS
In this section, show that when their respective training losses are minimized, discrete-time and continuous-time discrete diffusion models recover the true univariate marginals.
Discrete-Time Diffusion Models. Discrete-time diffusion models (Austin et al. 2021 trained to maximize the ELBO between the forward joint distribution p( )g 1:7 (D), where p p(2o) is the
data distribution, and the reverse joint distribution (.o.))). The ELBO can be simplified to
p(aTT) q(TT)
po (x (2x)) q( x.a)
By
log
Σ
log
+ log p(1ro)
Assume that po (x) encodes fully-factorized distribution, the above objective can be simplified as
T ΣΣ
|24)) log po (xt) + Eyg log p(ærr q(TT)
+ log p(zzo)
q( xx()
where the second term is independent pe- From the first term of the above formula, conclude that the ELBO objective is maximized when po(ak_1l<e) = q(x;_1l22) (2x)) for every / and every i.
Continuous- Time Diffusion Models. As described in Section 2
many continuous-time diffusion
models learn approximate the likelihood ratio (defined se (Ze; a/i; t)) at all noise levels & E [0, T):
q X: = =
so s0 x'iit)
((X: x2,)
Specifically, Lou et al. (2024); Meng et al. (2022) directly parameterize a neural network to ap-
proximate the likelihood ratios, and Sun et al. (2022) approximates the likelihood ratios with the
conditional distributions Pa [x; (x)" (vi,t) t).
For each æt, since there exponentially many possible æ/, is infeasible have neural network to directly model the likelihood ratio for all pairs of (2,,,/) Instead, they focus on (æa, ax)) pairs where at, and æe, are only different in single variable, i.e., their Hamming distance is one. For example, in Lou et al. (2024), they represent sæ as sæ sæ(2,, vř; t, i), which computes the likelihood ".r"".i).
ratio between a, and ar' = (x))" vi)- se is trained by minimizing the following objective:
Wh (0( q(X, (x) wil) log so (z., slit, ] (X: xx)
Σ
E, (x,
vi
where {w,} h. are positive weights. When the above objective is minimized, recovers the correct likelihood ratios:
qX, = (x) M wiH)
Vi, i, so v/: (,
(23)
q( (X; a.))
At inference time, continuous-time discrete diffusion models select a list of time steps 0 < to < ... f4 = to sample from: first sample from the prior p p(X.,) and then sample recursively from (po( H where Po(x,,,, (zt,) ) is obtained from so(w,, vf;t, i) in an indirect manner.
<
dp(m,) Specifically, assume dt =Q Q.x(x)) we have
q(0,, (x,,) = g( 1xt,..)) (
g(x,,)
= g( |20,
Σ eeeppp Expp(-At-)) (z),
((, = x)) <(x, It:))
"This argument largely follows Theorem 4.1 Lou et al.
2024
We include it for the sake of completeness.
19

---

Published as a conference paper at ICLR 2025
1000
SEDDm
Runtime (ms)
DCD (SEDDm GPT-2s)
100
10
8
16
32
64
128
# denoising steps
Figure 7: Sampling time of DCD and its two base models with 2 to 128 denoising steps.
where At: - 4441 and exp( - (A x) denotes the product of At (pp(-At:)(aa the aY__,, -th row and wh column of exp(-Ai-Q).
Plug in Equation 23
compute the marginal of s/__. (i.c., pe po(x (x,,)) following
(X) y/x,,) x g( ((,-))
Σ eep((AAx (2 :tte)))))))))) vit..))
exp((tt-Q)
Σ eexp(-AI-Q)(. ( vl: of
Therefore, if se perfectly learns the likelihood ratios between inputs with Hamming distance at most one, then the correct marginals q( {z (Z,,) can be computed using sg.
E IMPLEMENTATION DETAILS OF DCD
We describe details about the "autoregressive" version of DCD introduced in Section 5.3
According
to Section 5.3 the first (T-t-1)/TT portion of the tokens in #4+11 are unmasked. At step t, we only need additionally unmask the tokens spanning the (T-11) to (T-1)/T fraction of the sequence We do this by caching the keys and values generated by the attention layers of tokens generated in previous denoising steps. So at step we will have the KV-caches of the first ( 1)/T fraction of tokens. As a result, the computational cost for running the autoregressive Transformer is independent the number of denoising steps.
Algorithm DCD with Autoregressive Copula Models and Using Autoregressive Sampling
1: Inputs: a diffusion model Pam, an autoregressive model Pars number time steps T, sequence length
2: Outputs: a sample as from the discrete diffusion model augmented by the autoregressive model Initialize: Sample ay from the prior noise distribution p(Xr)
4: for T-
5: imin, imax (T (T)-t) (w.lo.gg assume L is divisible by 2')
6: Compute {pam (X: (x)+))+)) and (pam (X) (1( for each i E [imin fmax) using the diffusion model
Compute V(a, a;) = log Pam (#2 (m++1) log Pam (m4))) (Vi [mmin fmax). 2()
8:
æ: t- 1+++1
9:
for i Amin imax
10: L Sample x² from p(xz)) Par (aike I. exp( and store it to
Additional Runtime Analysis. Figure 7 displays the generation time per sample for SEDD GPT-2g, and DCD. When the number of denoising steps is small, the computation cost of running GPT-2; dominates the total runtime of DCD. However, as the number of denoising steps increases, this cost is amortized because, with K KV-caching, the total computation cost for running GPT-2; stays constant.
20

---

Published as a conference paper at ICLR 2025
Sequence length 128
Sequence length 128
1000
1000
SFDDm
SEDDm
Generative Perplexity
800
DCD (SEDDm + GPT2s)
Generalive Perplexity
800
DCD (SEDDm + GPT2s)
GPT2s
GPT2s
MDLM
MDLM
600
600
400
400
200
200
3.5
3.75
4.0
4.25
4.5
0
0.2
0.4
0.6
0.8
Entropy
Runtime (s/sample)
Sequence length 1024
Sequence length 1024
400
400
SEDDm
SEDDm
Generative Perplexity
DCD (SEDDm + GPT2s)
Generative Perplexity
DCD (SEDDm GPT2s)
300
GPT2s
300
GPT2s
MDLM
MDLM
200
200
100
100
+.
5.2
5.4
5.6
5.8
5
10
15
Entropy
Runtime (s/sample)
Figure 8: Comparison between generative perplexity (L), diversity (measured by sentence entropy; t), and runtime (4) of DCD with baselines.
F ADDITIONAL UNCONDITIONAL GENERATION EXPERIMENTS
To better understand the relation between quality (measured by generative perplexity). diversity (measured by sentence entropy* and speed for DCD and its baselines. Specifically, more efficient version of DCD described in the last paragraph of Section 5.3 and Appendix B to generate text sequences of lengths 128 and 1024. In addition models used by DCD, we compare them with MDLM (Sahoo et al. 2024), more recent discrete diffusion model that is more efficient than SEDD. Note that DCD can use any discrete diffusion model as its base model.
the
SEDD and GPT2, the two base
First, we compare the sample time and the generative perplexity (the second and the fourth sub-plot in Figure R). Compared to SEDD, GPT, and MDLM, DCD consistently achieves better generative perplexity given a fixed runtime constraint. It also requires less time achieve a desired perplexity value.
Additionally, we compare the perplexity and diversity of the generated text sequences. Following community standards, adopt the sentence entropy to measure the diversity of generated text. Specifically, the entropy of each text sequence the entropy of its token frequency distribution, and the final sentence entropy is the average entropy over all generated sequences. The desired behavior to have low generative perplexity and high sentence entropy (which means high diversity). Results shown in the table below and Figure 8 s first and third sub-plot. Compared the two discrete diffusion models (SEDD and MDLM), DCD achieves better generative perplexity under the same
'The sentence entropy of sequence is the entropy of its token frequency distribution. The reported number averaged across all samples.
21

---

Published as a conference paper at ICLR 2025
entropy, which offers better perplexity-diveniee tradeoff. Compared to the autoregressive GPT model, although the entropy of DCD is lower, it achieves better generative perplexity with slightly worse entropy.
G ADDITIONAL EXPERIMENTAL DETAILS
This section provides additional details of the experiments.
G.1 UNCONDITIONAL TEXT GENERATION
We adopt the SEDD-medium model with 320M non-embedding parameters trained on WebnWetText. The model is accessed through HuggingFace: ftttps://buugingeeet
SEDD.
co/ Louaaron/ seedd/sed--aeee sed///edddddmmmmmmmdme We follow the original paper (Lou al., 2024) and use the log- linear noise schedule o(t) = log(1 - (1 (1-ct)), which leads to the forward transition probabilities
(0<s<t<T):
( (..) == Cat exp(a(t - s) Q) x.).
The absorbing mask forward noising process is used. The corresponding transition rate matrix
0 0
: 7
0
-1
0
where the last category is <MASK>.
GPT. The GPT-2-small model is obtained from HuggingFace:
co/
DCD. We implement DCD by combining SEDD, and GPT-2s following the steps in Algorithm In line 8, instead of masking tokens independently, we group chunks of 8 tokens together and mask/unmask them with the same probability given the noise schedule (i.e., az/a44 shown in Prop. 5).
G.2 CONDITIONAL TEXT GENERATION
MAUVE Score. We adopt the MAUVE implementation available in the Python package evaluate. We use the default hyperparameters established by the original paper (Pillutla et al., 2021), which is also the default used by the package. We found that the number of samples and the number of samples given fixed prompt influenced the score. Therefore, we randomly selected the 2,000 prompts and generated samples for each prompt for all methods.
Detailed Runtime Analysis. As shown in Algorithm in each denoising step of DCD, we need to run the discrete diffusion model twice: first to compute {p(X;|1x44)>> and next to compute {
} by applying causal attention masks to the same denoising neural network given that based on the Transformer architecture. Next, as discussed Appendix the total runtime consumed by the autoregressive model remains constant across different numbers of denoising steps thanks to the KV-caching mechanism. Therefore, the runtime of DCD will be dominated by the computation cost of the autoregressive model with only few denoising steps. As the number of denoising steps increases, the runtime the autoregressive model will be amortized and the total computation cost will be dominated by the cost to evaluate the diffusion model.
SSD-LM. SSD-LM (Han al. 2023) is semm-auttregresssive model that uses techniques from dis- crete diffusion models to prediet/denoise chunks of sequences in autoregressive manner. Specifi- cally, given predefined chunk size, SSD-LM diffuses tokens in each chunk one by one conditioned on all previous chunks. As result, the model is semi-autoregressive and cannot see suffix prompts.
While the official implementation on GitHub (https: /ttp:://g/tttbbbt com/ xhan77 sad- only allows conditioning tokens in previous prompts, improved their code to also allow ditioning on tokens in the current chunk that is being diffused. Specifically, we replace the diffusion model's 's input corresponding the prompt tokens with the ground truth token embeddings.
22

---

Published as a conference paper at ICLR 2025
We followed the original paper to choose chunk size of 32 and use top-p sampling with p= 0095.. The remaining hyperparameters are kept as default.
G.3 ANTIBODY SEQUENCE INFILLING
Detailed Task Description. The adopted antibodies with an immunoglobulin G (lgG) format, which comprises a heavy (H) chain and light (L) chain. Each chain has three complementarity determining regions (CDRs) that are crucial toward the binding affinity to the target antigen.
Training NOS-D. We use the training script as well as the dataset provided in the official GitHub repo of NOS-D /eaetps://gitee using the default settings (e.g., learning rate and its schedule).
com/ ngruvgr/ (rruver/NO)) The model is trained with 50 epochs
Training GPT. We use the same dataset provided the repository of NOS-D and use the GPT implementation from GPT model has 6 layers, an embedding size of 512, and 16 attention heads. The model is trained for 10 epochs with the default settings in the nanoGPT repository.
com/ ka arppahyy ntw./annnn/////// ree/ master The
DCD. When implementing DCD for the antibody sequence infilling task, we add an additional scaling factor to the coefficients in V. That is, V is updated in line 6 of Algorithm I following
Vi, Zt. Vli. = = 3 (log
Pdm Pam (o) xx+1) log Pam (#11=(11)
where we set 0.1 for this task. We note that 3= works well for the language modeling tasks. The need to choose smaller 3 in this task may be caused by the fact that the dataset and the models are much smaller and are more prone to overfitting.
H ADDITIONAL RELATED WORK
We briefly review a class of related works that perform beemi-Jautoregressive diffusion, which is weakly related our work since we also "combine" discrete diffusion models with autoregressive models. Specifically, Wu et al. (2023); Chen et al. (2024) perform diffusion in an autoregressive manner by allowing the noising schedule be variable-dependent. Variables the beginning are kept unchanged at small ts and are corrupted only when t is close to T"; variables at the end are corrupted in the first few time steps. During sampling where t moves from T O, initial vari- ables/tokens are first denoised, followed by later tokens. This allows the diffusion model to perform "autoregressive denoising".
ADDITIONAL TEXT SAMPLES
We provide randomly selected unconditional samples in Figure and and conditional samples in
Figure [[ and [2]
23

---

Published as a conference paper at ICLR 2025
.... DHHS, Dion Todd of Detroit, Detroit Tigers team players Marcus Johnson of Detroit Lions, Minnesota Vikings team players Troy Polamalu of Detroit Tigers, mellow lines for opposing teams, but they do not share a common mentality.
"At the end of the day, if you're just grouping me but you also come for the tram stop, everybody plays by the rules," Kalim said of Johnson. "If you start to play offense, you 'e going to make mistakes. We don't just give them time because we don 't think they're going to win. We don't just teach them how to take care
Both 1-2 range panels are available for hours of ongoing use (25 burning power consumption plus periodic gas combustion cycle). Also included moisture protection from agricultural inscets and biological spills of greenhouse gases duc to different costs of livestock sit idle in the energy cycle of so many nations.
The high yield yields from large, agricultural biofuels rely entirely on larger projections placed together by country by government into 2020. Such predictions assume an billion tonne increase in capacity to 5 billion tonnes per year six billion years from now then consumption for nearly all full-time, undertaking- Stage + 2.0 exponential growth.
acquisition jack. Add this to the negotiations, and the place starts going down. Veteran grocers like State Farm have halted stocking their own toll booths and warnings because they fear getting squeezed out by major retailers. Meanwhile, stores like Wal-Mart have reduced shelf space by as much as 8.6 pereent. Wal-Mart Stores Canada has been the most profitable Wal-Mart store chain in Canada (green grocer WalMart Stores Canadian now accounts for nearly a third of sales, up from just pereent in 1996). Mcanwhile Wal-Mart continues to aggressively sell Canadian goods. Since 1995, Wal-Mart Stores Canada has more than tripled the volume and
I find myself divided. Like I was growing up in this world. I saw my dad constantly being obsessed about faith, constantly being remembered in my mind his name. My dad thought that I wasn't going to grow up to say myself, if tried to make my family happy they wouldn't believe me anymore. Every time looked in my eyes thought was crazy, but thought I was my brother. I was worried I would always feel jealous of my father and slowly, started to think about family. found a family where everyone ruled mc alone towards the end, in hell. Family was always there, it was
flap: Now you can lay your hands on a wavy pattern without touching the nerves, or you fancy you can lay your arms on an occluded specialty? Any hypothesis relied from those vainly generalized action.[303] Reconeiling the patient's single hairs with multiple llings stipulated that a single story could be shortened half, but in faet lengthened in less flexible forms. If, however, little succeeded at the scholarly step, that compromise was subtlety in his notes, provided he lacked enough hairs to lie straight down. He could even weave rings for his harp especially wheat but there was
Figure 9: Randomly selected unconditional samples from DCD (SEDD GPT-2g) with 4 denoising steps.
24

---

Published as a conference paper at ICLR 2025
March if they can overcome federal complaints.
Farmers say they have acquiesced to pressure from U.S. agribusiness giants to cut back on pesticides, while environmentalists say regulators allege lack of oversight by officials.
"We won't tolerate anything bad," " said Rutko Guerra, spokesman for Cornell University Extension. "It's clear conflict of interest."
Comell University Extension estimates about 400 pesticides are sold through the city each year in violation of the Public Health Act and other state laws, prompting over $49 million fines - the largest ever levied by public university.
Each time components are created a buffer is created. This buffer holds the components and should be updated whenever they touch on the screen. The buffer foundation passes every buffer value from the component to the component's buffer.
After the component has been created markers is placed inside the marker stating which colors are used in the new paint direction. During the paint direction there are three modes: launch to draw pixels, around draw so baccarat will appear misaligned, and lowdraw so baccarat will appear perfectly aligned.
... in on their autobiographical stories of realizing happiness. The Stimulant Prophecy was an important spiritual awakening that occurred during the 1950s, 1960s and 1970s. awakened believers to cultivate spiritual fortitude as well as resolve conflicts and lead to more successful relationships.Acoom
to Tages Jephzei (The Stimulant Prophecy), this event ultimately caused Muslims to develop compassion for one another by their communal experiences.Ir also fueled support for incongruity in mainline Muslim societies and gave hope for physical cleansings.In 1976, Tages Jephzei published his book The Stimulant Prophecy: Understanding Muslims...
...would proper address regulations affecting this country." " Hc recalled how he supported Deputy Minority Leader Nancy Pelosi's (D-Calif.) efforts explore Russian meddling in the 2 election: "When Nancy Pelosi [D-Calif.] spoke to me, he wanted to consider Russian interference in election." Kennedy also added that he believes new law requiring commercial polluters to disclose their emissions during emissions tests will help clean up the air: "[I] hope that the EPA will follow through with two years of programs that are going to reduce [during gas] emissions in some form or another," " Kennedy told reporters.
Figure 10: Randomly selected unconditional samples from DCD (SEDD, + GPT-2g) with 32 de- noising steps.
25

---

Published as a conference paper at ICLR 2025
are hear the exhortations "We ask of everyone to speak the voice God" and "we ask to be loved for the game and what they wanted to do next for the series." " In fact, PS3 announcement executives confirmed this month that Sony Pictures Entertainment plans to release PS Vita versions of Sony PlayStation Classics PlayStationGS, The Last of Us, Ratchet, Square did not come up with the revolutionary " idca that would warrant a new entry for the PlayStation 3. Speaking in an interview, it was revealed that Square Enix felt Square Enix could not offer a few new plugins without secing Square Enix make Square Enix's visionary Battle...
in the American version, and Warner Bros. added Nobuo Ukiura of Miyazaki Animation to directing on character design. A large team of writers handled the script. The game's story was developed by Kouki Watanabe in Chouki no Namco Europe, and Fujikyo Pictures Entertainment released it theatrically on May 25, 1999 in North America, followed with an expanded edition in November of that year. was also adapted into manga and an original video animation series. Due low sales, Warner Bros. suffered widespread cancellation due to lack of revenue. Warner Bros. also shut it down due to its failure
They save many enemies you through Chrono Trigger Online missions, unlock quests in missions missions, unlock special quests, having a higher difficulty than those found in the rest of the game. These include boss & combat objectives. Chrono Trigger Online contains one Chrono Trigger EP with unlockable Chrono Trigger ARC girls.
This the first patch which implements the PTZ system, is carried over directly from Valkyira Chronicles. During missions, players select each unit using top @-left position. They determine their unit type, which determines their ability and the size of their field of vision. They can only activate...
The Final Fantasia in Japanese / Media. Vision for the PlayStation Portable. Released in January 2011 in Japan, it debuted novel while hold in the North East Stand. It garnered reviews for its breathtaking narrative, disappointing plot, and creepy characters. A fourth graphic novel is in the first game and follows the Nameless ", a penal military unit serving the nation of Gallia during the Second Europan War.
scenery was was composed in short animation. When the game ended early on Mikami Sakura was drawn by Makoto Masui.. A large team of writers handled the script. The game's soundtrack lasted around hours and Sugiyama Shogarashi, Makoto Masuyama and Kyoko Takamura included Takme Ibara. The music theme was originally released in 2009, with an expanded edition in November of that year. It was also adapted into manga and an original video animation series. Due to low sales, the game release was delayed to three weeks in spring 2011. Following its final release on May 23rd 2013.
Figure 11: Randomly selected conditional samples from DCD (SEDD, GPT-2g) with 4 denoising steps. Prompt texts are bolded and in blue.
26

---

pool order Of the 155 young Published as a conference paper at ICLR 2025
every character has. You can choose combat situation best suit the character. To learn Battle Potentials, each character has a unique skills makes them invaluable. One of Potentials best suit the character is Point squirrel on the map, the character Leda can use skills like Star Wars Matchmaker", the character Jaden can activate Direct Command and move around the battlefield without depleting his Action Point gauge, the character Reila can shift to melee objects to send morgues (so more reliable), Mira can change her story situation to battle nweire battle to ward strategise, a Command Pointcher
this State building institution. We could be build defensive system to the United States Arsenal in this city LLittle Rock) ). This system seem really feasible and good. The name of the City that would to this scenario go by the Rocky Mountain Sound as the Academy as well is MADISON FIREWRIGHT NASHA ROCK. -John M Harrel Telegram, January 31, 1861
The item was intended simply piece of news, but it also served an "opportunity" for the U.S. Fortresses on Little Rock. Setting aside the exception of the basisicks' menansi slogan, it was all
the air the end of those years run in early 1923. An original design for the society called The Darling of the American for Being Unnamed put forth on the air at the end of those years held early 1925. The unnamed pursuits of the American were previously documented by History Magazine called St. Luke's Society for the Propagation of the Gospel in its November 1919 exhibition.
Religious @-@ themed books include The Red Book, The Hidden Voyage, an opera which was written on behalf of John Ford and produced under the contract of the Protestant revival organization, The Evangelical Fund (without Contemplation)...
letter city Evans noted in more details reminded Christine Barker to supervise the household, and give both her mother and sister complete authority their development. (See Evansdone & Sullivan) 79. Christine Barker continued her adult life but when she reached ends of age, during which time her big sister Ruth had died unexpectedly of heart attack. Barker was unable to pursuc her art to any significant extent following her sister's death, as all of her parents perished and she lacked the discipline, learning needed to as a professional age. (See Evanssic) 80. Although moving art was a lifctime profession for Christine Barker, bcar...
Pool: At Mumbai airport Shivaji Park. Women Technical girls under @-@@ 17 women's team competed in Confederation of Asian Football's premier youth competition. 2011 year-13 results :U 13 medal NW15 qualification women, five girls had to be narrowed from an initial pool of 49 young women. Two girls from the SOS Children Village Bakoteh were chosen for the USC and two girls from the Meijer 's Village Bakoteh. The remaining Meijer girl was selected for the opening ceremony. A After the AU's teams
Figure 12: Randomly selected conditional samples from DCD (SEDD, - -GPT-2g) with 32 denoising steps. Prompt texts are bolded and in blue.
27


Reviews:
[{"review_id": "QpENasyqkb", "reviewer": "Reviewer_nyBv", "summary": "This paper studies the dependency issue of multivariables in each step of discrete diffusion model.  The paper claims that discrete diffusion model requires many steps because that each step of the diffusion model cannot fully capture the required inter-variable dependency. To model the dependency among these variables, the author proposes an interesting setting, that given both accurate estimation of marginal distribution and approximated estimation of full distribution, how to obtain a much accurate estimation of the full distribution. An optimization algorithm,  I-projection, is proposed to solve the problem. \nTo inject inter-variable dependency for each diffusion step, the author proposes to estimate the full distribution with an fixed-ordering autoregressive model, and then approximate each step's target distribution of certain marginalization of the autoregressive model. Many approximations are proposed to achieve the final \"correction\" step.", "questions": "Except these weaknesses, I have the following question to inference and experiment. \n\n1. While the proposed method requires few diffusion steps, it does contain many additional steps (line 5 6 7) shown in Algorithm 1. Can the author give a fair comparison of inference time instead of diffusion steps, given that each diffusion step has different cost?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The proposed issue of inter-variable dependency of generated elements within diffusion model is an interesting observation and a valid issue. \n\n2. The proposed mathematical problem that finding a better estimation of full distribution given both marginal distributions and inaccurately estimated full distribution is a very interesting problem that can combines two type of methods together. The author shows that diffusion model and autoregressive model can be combined under the proposed setting. \n\n3. The author proposed a good realization of the proposed combination: how to estimate an approximated target distribution with inter-variable dependency using autoregressive model, and how to find the I-project's solution to derive the much accurate estimation of the target distribution. (Under absorbing \n\n4. In experiment, the author shows that the method can achieve better performance than all baselines even given a few diffusion steps.", "weaknesses": "1. Lack an detailed explanation of why the inter-dependency issue is only for discrete diffusion model but not for continuous diffusion model, which can affect the soundness. \n2. Section 2 line 121 mentioned that continuous-time case the objective is optimizing likelihood ratio. This is not fully correct, given [1] shows that the SEDD's objective can be exactly derived from ELBO setting. The author also lacks reference for its continuous extension. \n3. Lack a clear definition of \"copula\", while it appears many times in many different ways. \n4. Section 5.1 needs a major revision: it is hard to read while the described equations are not that hard. Some suggestions: define the each variable's meaning before putting them into equations. like X-tilde, which I don't understand before read equation (5) in detail. Also, give an illustration figure to describe the setting of each variable shown in equation (5) under absorbing state case. Most importantly, describing your easy-to-understand intuition and main thought before throwing all equations that hide your thoughts deeply. \n5. Need certain consistency, for example, equation (5) and (6) 's last part: one use element case while the other use combined case. Notation/equation should be consistent. I also don't like the notation of I and J which does not have the t+1 subscript but only represent the case for t+1 time. These notation/equation issues make the paper hard to follow. \n6. The proposed analysis has many approximations that can introduce huge approximation error, in both copula distribution of I-projection. I personally think the error within the copula distribution can be huge, given that the autoregressive method is based on the inherent ordering of the sequence (like text), while the conditioned part of equation (6) clearly needs more unmasked tokens. Ideally, one would like to train an \"autoregressive diffusion\" model proposed by Emiel Hoogeboom [2], which can make the ordering issue less problematic. The ideal copula model should work for any combination of J and I ordering for any time step t, which is a huge requirement. I suggest the author think about how to quantify the influence of these approximations in copula model and the I-projection stage.\n7. Another limitation is that the current copula model only works for absorbing state diffusion model. Which is not ideal and not widely used for many other domains such as graphs given it does not address many symmetry problems. \n\n\n\n\n\n\n\n[1] Zhao, L., Ding, X., Yu, L., & Akoglu, L. (2024). Improving and Unifying Discrete&Continuous-time Discrete Denoising Diffusion. arXiv preprint arXiv:2402.03701.\n[2] Hoogeboom, E., Gritsenko, A. A., Bastings, J., Poole, B., Berg, R. V. D., & Salimans, T. (2021). Autoregressive diffusion models. arXiv preprint arXiv:2110.02037.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BoWmFoYUdh", "reviewer": "Reviewer_bAgJ", "summary": "This paper proposes the Discrete Copula Diffusion (DCD) method. It addresses the inability of discrete diffusion models to capture inter-variable dependencies during denoising, which hampers their few-step generation performance. DCD combines a discrete diffusion model with an autoregressive copula model at inference. By integrating the univariate marginals from the diffusion model and the dependencies captured by the copula model, it approximates the true denoising distribution. Experiments in unconditional and conditional text generation and antibody sequence infilling demonstrate that DCD outperforms individual models, achieving better results with significantly fewer denoising steps, thus emphasizing the importance of modeling inter-variable dependencies in discrete diffusion.", "questions": "1. Is the combined distribution $\\hat{p}$ (Eq 3) a normalized distribution? Otherwise, the notation should be proportionate.\n\n2. Could we compute the likelihood (or bound) for DCD model? Is so, it will be good to have a perplexity metric which is also a common benchmark; if not, what's the challenge for it?", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Originality: the paper proposes an interesting approach of combining a discrete diffusion model with a copula model to address the issue of capturing inter-variable dependencies in discrete diffusion, which is a relatively new direction.\n\n2. Quality: the paper provides a formal analysis of the problem in discrete diffusion models, showing the limitations of the independent denoising assumption and how to improve it with the copula model.\n\n3. Clarity: The paper is well-structured, and explanations of concepts like copula models, I-projection, and the noising process in discrete diffusion are detailed and understandable, making it easy to follow the flow of ideas.", "weaknesses": "My main concerns are about the experiment setup and results.\n\n1. Reasonability of assumption: from Sec 4, the motivation is that the diffusion model learns better the marginal distributions, while copula (which is AR model) captures correlation but may be more biased in marginals. I think it's better to show empirical evidence for this assumption.\n\n2. Computational cost: this is my main concern. Could the author provide a simple complexity analysis for sampling time? According to Alg 1, I think line 5 requires twice forward computation with bidirectional and causal attention, and line 7 (eq 6) even needs another computation. As shown in Fig, this process is extremely slow.\n\n3. I suggest having another plot like Fig 3 but making the x-axis the wall clock time, so it shows efficiency directly. As indicated by Fig 5, I guess the practical acceleration is limited.\n\n4. An important baseline is missing. I suggest also comparing DCD against MDLM [1] where the cache sampler can also significantly reduce timestep and accelerate the sampling speed. \n\n[1] Sahoo, Subham Sekhar, et al. \"Simple and Effective Masked Diffusion Language Models.\"\n\n5. Evaluation protocols are too synthetic. The paper just studies 128-length text generation and shows the significance, but typical studies are all done on longer sequences such as 1024 length. Besides, it's critical to also report diversity scores for the generations. Generative PPL is known to suffer from mode collapse generation without consideration for diversity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gNROIf6h33", "reviewer": "Reviewer_YU9j", "summary": "Traditional Discrete Diffusion models suffer from severe degradation in sampling quality when the number of sampling steps is reduced. In contrast, continuous Diffusion Models don't have this issue. This paper first points out that this is because discrete diffusion models cannot capture the dependencies between output variables at each denoising step. To address this problem, the paper proposes incorporating another deep generative model, named the copula model. The copula model can be trained independently and only needs to be incorporated during the sampling process of the Discrete diffusion model. Experiments demonstrate that the proposed method can significantly improve the sampling speed of discrete diffusion models.", "questions": "1. I have questions about the results in Figure 4. The figure gives me the impression that if we further increase the number of denoising steps, SEDD's performance could still improve, while DCD's results can't improve anymore. I would like to know how the performance comparison between the two would look if the number of denoising steps were further increased.\n\n2. Since I'm not very familiar with synthetic text data, I'm unsure whether perplexity is an appropriate metric for measuring the quality of generated data (especially the ability to restore the ground-truth data distribution). Are there any other metrics to measure if the dependencies between different variables are correctly captured? For other types of discrete data, such as tabular data, there are more metrics available to measure the quality of generated data (such as metrics for correlation between different dimensions). I would like to know how the method proposed in this paper performs on these types of data.\n\n3. Can the authors provide a more detailed comparison of the computational costs and training requirements between their proposed method and existing approaches?  As demonstrated in Figure 5, your methods look much slower.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. This paper addresses a very important issue: how to improve the sampling efficiency of diffusion models (reducing sampling steps while maintaining sampling quality) is a crucial topic. While this problem has been well solved in continuous diffusion models, it remains unresolved in discrete diffusion models.\n\n2. This paper is well-written, and I particularly appreciate its insightful illustrations. For example, Figure 1 provides an excellent clarification of the existing problems in traditional Discrete Diffusion models.\n\n3. The paper provides solid theoretical analysis for the proposed method.\n\n4. The proposed method doesn't require modifications to the diffusion model during training, thus offering excellent scalability.", "weaknesses": "1. Since this paper requires training an additional (new) generative model, the training requirements of the proposed method might be higher. Considering that Autoregressive methods are already very well-suited for generative modeling of discrete data, I am uncertain if this paper feels like using a complex model to improve a relatively simple model.\n\n2. Using perplexity metrics alone is not sufficient. For example, we expect the generated text to be sufficiently diverse, but perplexity metrics struggle to reflect the diversity of generated samples.\n\n3. Most experiments were conducted with a generation length of 128, which does not reflect real-world scenarios, e.g., with a length of 1024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rDjy4twIYn", "reviewer": "Reviewer_5yGc", "summary": "The paper proposes using copulas to enhance the approximation accuracy of the backward process in masked diffusion models. Specifically, it shows that the true posterior $q(x_t|x_{t+1})$ can be decomposed into a latent auxiliary variable model $q(x_t|x_{t+1}) = \\sum_{\\tilde{x}_t} q(\\tilde{x}_t | x_{t+1}) q(x_t|\\tilde{x}_t, x_{t+1})$, in which $q(x_t|\\tilde{x}_t, x_{t+1})$ has a closed form and $q(\\tilde{x}_t | x_{t+1})$ can be approximated using a pre-trained auto-regressive model. To further enhance modelling capabilities, the paper suggests learning an I-projection of $q(\\tilde{x}_t | x_{t+1})$, so that the univariate marginals align with the true data marginals. Experimental results demonstrate that this approach reduces the number of sampling steps.", "questions": "- In line with weakness 1, could you add a plot in Figure 3 showing the relationship between running time and perplexity, and include the running time for GPT-2 models as well?\n- In addressing weakness 2, could you consider a variant model that solely uses Equation 5 for denoising, without applying the copula? This would provide insight into how well the autoregressive model approximates the conditional distribution of the data.\n- Moreover, one interesting idea to address weakness 2, could be to use generative marginal models [1] to learn p(x_0), and the conditional in eq.5 has close form. Specifically,  p(x^I | x^J) could be computed using Bayes’ rule: p(x^I , x^J) / p(x^J). The trade-off here is that while enerative marginal models may not be as expressive as autoregressive models, they provide exact computations, offering a different balance between model power and precision.\n- Proposition 1 is confusing. There is no model distribution in Equation 2, so it's not obvious how relaxing the independent denoising assumption would reduce the negative ELBO. Additionally, in line 727 of the appendix, could you explain in more detail how the inequality holds?\n- Could you also apply the proposed discrete copula diffusion to MDLM [2,3] and present results for running time versus perplexity? This would help address concerns about the method’s broader applicability across all masked diffusion models. Additionally, it would be valuable to explore whether improvements in the autoregressive model lead to better performance in the proposed method, which raises an interesting question worth investigating.\n- Some suggestion in tmers of writing\n    - On line 293, instead of stating \"it can be decomposed...,\" it would be clearer to explicitly write out: $q(x_t|x_{t+1}) = \\sum_{\\tilde{x_t}} q(\\tilde{x_t} | x_{t+1})q(x_t|\\tilde{x_t}, x_{t+1})$\n    - Could you add a paragraph in the main text discussing the limitations of the proposed method and potential directions for future work? Including this would help motivate readers to explore further improvements and applications.\n\n[1] Liu, Sulin, Peter J. Ramadge, and Ryan P. Adams. \"Generative Marginalization Models.\" *arXiv preprint arXiv:2310.12920* (2023).\n\n[2] Shi, Jiaxin, et al. \"Simplified and Generalized Masked Diffusion for Discrete Data.\" *arXiv preprint arXiv:2406.04329* (2024).\n\n[3] Sahoo, Subham Sekhar, et al. \"Simple and Effective Masked Diffusion Language Models.\" *arXiv preprint arXiv:2406.07524* (2024).", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-motivated and  proposes a theorectically sounded framework to improve masked discrete diffusion models. Decomposing the true posterior into a latent auxiliary variable model is a very interesting observation.", "weaknesses": "I have two main concerns:\n\n1. Although the proposed method reduces the number of sampling steps, the number of function evaluations (NFE) appears to be quite large. Unlike discrete diffusion, which requires only one NFE per step, each step in the proposed method involves evaluating the autoregressive model, the diffusion model, and solving a convex optimization problem. This suggests that the computational cost is significantly higher than that of discrete diffusion and autoregressive models.\n2.  As shown in Eq.6, the paper uses a pre-trained auto-regressive model to approximate $q(\\tilde{x}_t | x_{t+1})$. However this is a biased approximation. This bias can impact the overall fidelity of the backward process, potentially leading to discrepancies in the modeled distribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)