Discrete Flow Matching
Authors: Itai Gat
Tal Remez
Neta Shaul
Felix Kreuk
Ricky T. Q. Chen
Gabriel Synnaeve
Yossi Adi
Yaron Lipman
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Language modeling
Flow matching
Diffusion models
Primary area: diffusion_based_models
TL;DR: Flow Matching for discrete data with applications to language modeling

Abstract: Despite Flow Matching and diffusion models having emerged as powerful generative paradigms for continuous variables such as images and videos, their application to high-dimensional discrete data, such as language, is still limited.  In this work, we present Discrete Flow Matching, a novel discrete flow paradigm designed specifically for generating discrete data.  Discrete Flow Matching offers several key contributions:  (i) it works with a general family of probability paths interpolating between source and target distributions; (ii) it allows for a generic formula for sampling from these probability paths using learned posteriors such as the probability denoiser ($x$-prediction) and noise-prediction ($\epsilon$-prediction); (iii) practically, focusing on specific probability paths defined with different schedulers improves generative perplexity compared to previous discrete diffusion and flow models; and (iv) by scaling Discrete Flow Matching models up to 1.7B parameters, we reach 6.7% Pass@1 and 13.4% Pass@10 on HumanEval and 6.7% Pass@1 and 20.6% Pass@10 on 1-shot MBPP coding benchmarks. Our approach is capable of generating high-quality discrete data in a non-autoregressive fashion, significantly closing the gap between autoregressive models and discrete flow models.

Discrete Flow Matching
Itai Gat
Tal Remez'
Neta Shaul
Felix Kreuk'
Ricky T. Q. Chen'
Gabriel Synnaeve'
Yossi Adi!
Yaron Lipman'
Meta FAIR
Weizmann Institute of Science
Abstract
Despite Flow Matching and diffusion models having emerged as powerful genera- tive paradigms for continuous variables such as images and videos, their application high-dimensional discrete data, such as language, is still limited. In this work, we present Discrete Flow Matching, a novel discrete flow paradigm designed specifically for generating discrete data. Discrete Flow Matching offers several key contributions: (i) it works with general family of probability paths interpolating between source and target distributions; (ūi) it allows for a generic formula for sampling from these probability paths using learned posteriors such as the proba- bility denoiser (x-prediction) and noise-prediction (<-prediction); (ii)) practically, focusing specific probability paths defined with different schedulers improves generative perplexity compared to previous discrete diffusion and flow models; and (iv) by scaling Discrete Flow Matching models up to 1.7B parameters, we reach 6.7% Pass @ and 13.4% Pass on HumanEval and 6.7% Pass @ and 20.6% Pass @1 10 I-shor MBPP coding benchmarks. Our approach is capable of generating high-quality discrete data in a non-autore con-autoreeesssvve fashion, significantly closing the gap between autoregressive models and discrete flow models.
Introduction
Despite the remarkable success of diffusion and flow models generating continuous spatial signals such as images (Ho et al. 20200] Rombach et al. 2022 Esser et al. (024) and videos (Singer et al. 2022; Blattmann et al., 2023). their performance still falters when applied discrete sequential data compared to autoregressive models. Recent progress in adapting diffusion and flow models to the discrete setting has been made via mostly two approaches: embedding the discrete data continuous space and applying continuous diffusion (Dieleman et al.) 20222, Stark et al.] 2024) or designing
diffusion or flow processes over discrete state spaces (Austin al. 2021a Campbell al. 2022).
In this paper, we pursue the discrete flow approach of Camphell et al. (0024) and introduce Discrete Flow Matching, theoretical framework and algorithmic methodology for discrete flow models that yields state-of-the-art discrete non-autoregressive generative approach. Surprisingly. Discrete Flow Matching exhibits similarities with the continuous Flow Matching (Lipman et al.) (022) approach proposed for continuous signals. Notably, its generating probability velocity, employed in the sampling algorithm, is identical in form its continuous counterpart. Additionally, Discrete Flow Matching offers the following advancements and simplifications over prior methods: It encompasses more comprehensive family probability paths transforming source (noise) distributions into target (data) distributions. accommodating arbitrary source-target couplings and dependent schedulers. Furthermore, provides a unified formulation for the generating probability velocity directly expressed in terms of the learned posteriors and schedulers, along with unified and general theory and algorithm for corrector sampling and iterations. In practice, we observe that path and corrector schedulers are pivotal, and their proper tuning leads to substantial improvements in 38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

int):
def find_pocition_ varlauorr x): def sinary_eearch(aarr,, x):
"*eRur n-th Psbenacek
start
number.
high len(arz)
end len(arr)-1
>>> fsb(10)
(Sb(1)
shile high low: mid (high lou) 11 2
# While perforning binary search shile start and:
I f(b(8)
mid (start end) //
21
# If is greater if [mid] x: lox = mid + if smalter
if arr [nid] start mid swaller
greater
if return 0
return
return fib(n-1) + fib(n-))
elif arr [mid] x:
elif are [mid] > x:
end mld
high mid
else:
else:
return mid
return mid
raturn
return
Figure 1: Code generation examples using Discrete Flow Matching. Code condition is marked in gray model generation is marked in yellow Left sub-figure presents the standard left-to-right prompting: Middle and Right sub-figures, presents complex infilling setup.
generation quality. We have trained 1.7B parameter Discrete Flow Matching model on the same data mix in Llama-2 (Touvron et al., (023) and Code (Roziere et al.) 2023) achieving 6.7%PPas and 13.4% Pass TO on Humankvvl and 6.7% Pass@) and 20.6% Pass 10 on I-shot MBPP coding benchmarks; Figure | shows some code generation examples. In conditional text generation our model produces texts with generative perplexity score of 9.7 as measured by the Llama-3 8B model, surpassing 1.7B autoregressive model that achieves 22.3 and not far from the Llama-2 7B model that achieves 8.3 in generative perplexity score. We strongly believe that Discrete Flow Matching represents significant step in bridging the performance gap between discrete diffusion and autoregressive models, and that further enhancements are possible by exploring the vast design space that Discrete Flow Matching has offer.
2 Discrete Flow Matching
2.1 Setup and notations
In discrete sequence modeling, we denote sequence as an array of N elements (x* x,2 .......... x.) Each element, or token, within this sequence is selected from a vocabulary of size d. Consequently. the entire set of possible sequences is given by = [d], where [d] fl..... d}. A random variable taking values in the space ' is denoted by X and its corresponding probability mass function (PMF) P(X z). For simplicity, throughout the paper, we sometimes omit the random variable and use p(a:) to denote the PMF.
To describe marginalization properties, we denote p(x²) the x² marginal of p. i.e., p(x') p(x( where x² (..., _=11, .f+1 ....) E [d]a-= are all the arguments excluding i. Similarly, p(x)) Ex p(z), and xt E [d]. A useful PMF is the delta function, dy. D, which is defined by
p(x).
(x(x) = II yy (a"), where (x)) = {1 ==1
x² = y
(I)
10
x² + y²
With the marginal notation d,(x') (x+) and 8,(x²) (x)) = 11, 0gr (x²) which simplifies
notation.
2.2 Source and target distributions
In discrete generative models goal is to transform source samples Xo p to target samples X1 ~ q. Our training data, consist of pairs X, and X, that are sampled from a joint distribution (z, y), satisfying the marginals constraints p(x) = EyED (Ix, = =EeD (2, y), i.e.,
(1) (1).
(2)
2

---

In the simplest case, the training pairs Xo and X1 are sampled independently from the source and target distributions respectively,
(Xo; X1) ~ p (()))q(X1)
Example: source and couplings. Common instantiations of source distribution p are: (i) adding a special token value often referred to 'mask" or "dummy' token, denoted here by m, and setting the source distribution be all-mask sequences, i.e., p(x) = (x)); and (ii) using uniform distribution over D. which is equivalent to drawing each x' independently to be some value in [d] with equal probability, denoted p(a)) = (z)).. In this paper we focus mainly on (i). We further consider two choices of couplings Independent coupling, which we call unconditional coupling (U-coupling), π (Zo, 21) = (x1)) A random sample that realizes this choice have the form
(Xo, X1) ((m...., m). m), X1).
where X1 (X1) is random sample from the training The second choice of coupling (20, 21} = (1) which we find improves conditional sampling, partially masks inputs with samples of the form
(1 (1) -
(m, ())
where X1 ~ (X2,) and E {0, 1] N is random variable indicating the conditioning, O denotes the entry-wise product, and 1 E RV is the vector of all ones. We call this conditional coupling (C-coupling).
2.3 Probability paths
We follow the Flow Matching approach (Lipman al. 2022 Liu al. 2022 Albergo and Vanden- Eijnden 2022) that uses a predefined probability path pt interpolating p and I.e.,
(6) to train the generative model taking source sample Xo ~ to a target sample X1 ~ 9. We use arbitrary coupling of source and target (Pooladian al. 2023 Tong et al. 2023), (zg. 21), and the symmetric Flow Matching path (Albergo and Vanden-Fijndee 2022) define the marginal probability path,
Pu p and
N pa (z [20 21),
Pc(x) = Σ Pe(x) x1 (1), where pa (2|20, x1) = II x=1
(7)
TDDDIED
and (z (ro, .1) is time- dime-dependent probability on the space of tokens [d] conditioned on the pair
To F. and satisfying Po(x") [zo, x 620 (x') and pi (x* =0. = 8,,, (x*). If the conditional path
p:(=")ar x1) satisfies these boundary conditions then the marginal path pr(x)) satisfies equation 6
In developing the framework, paths that are also tractable to learn within the Flow Matching framework. We consider conditional probability paths as a convex sum of m conditional probabilities w 21)
would like consider as general as possible set of probability
m
Pe(a' ((10'120,00) 21)
Σ
3i Kig
u (s* (10,
where E, ni = and N².3 2 0 are collectively called the scheduler. Note that the scheduler can be defined independently for each location in the sequence i E [N] or uniformly for all tokens, R... n/.
simple yet useful instance of these conditional paths is reminiscent of the continuous Flow Matching paths formulated as convex interpolants,
Pr (z [2oo .1) = =(1 - Kz) kx) (x²) + Kr82,
where the scheduler K2 satisfies Kg = O, 1y = 1, and monotonically increasing in t. Another interesting instantiation of equation 8 is adding uniform noise with some probability depending on
(x (mo, σ1) = ni dw, (x') + pa')(aa) + x2 k260,, (x'),
(10)
where xJ 0, x3 1, Ng N² 0 (remembering that E3 xi i.j 1 and
R.C" 3.j 2 0).
3

---

(x
(X,)
u (x X()
(X,,
X
Flow R-V
Flow [d]
div in RM
div in [d]
Figure 2: Discrete flow in D [d] with d 4, (middle-left) versus continuous flow in
Rh, . = (left). The rate of change of probability of a state (gray disk) is given by the divergence operator shown in the continuous case (middle right) and the discrete case (right).
2.4 Generating Probability Velocities
Continuous generating velocity. Sampling in continuous Flow Matching is performed by updating the current (continuous) sample X, E RRN / [0. 1), according to a learned generating velocity field (x())), E [N]. Euler sampling follows the (deterministic) rule
Xi+h x; (X,)
(1I)
where > user-defined time step. Note that equation is updating separately each of the sample coordinates, X;, E [N], see e.g., Figure 2 left. The velocity (X,)) can be either directly modeled with a neural network, or parameterized via the denoiser (a.k.a. z--rediction) or noise-prediction (a.k.a. E-prediction), see left column in Table If. for all E [0, 1), starting at
X, ~ Pt. and sampling with equation] provides X14h ~ Pt+hh + t then we say that u, generates pr.
Generating probability velocity. For defining Flow Matching in the discrete setting, we follow Campbell et al. (2024) and consider a Continuous-Time discrete Markov Chain (CTMC) paradigm, namely the sample X, is jumping between states DD depending on continuous time value t E [0, 1]. Similar to the continuous Flow Matching setting described above, we focus on model that predicts the rate of probability change of the current sample X, in each of its N tokens, see Figure 2 middle-left. Then, each token of the sample X, Pt is updated independently by
X ~ xx; (-) + huf(,,
(12)
where call de the probability velocity as reminiscent of the velocity field in continuous Flow Matching, and as in the continuous case, we define:
Definition 1. Probability velocity u, generales the probability path pe if, for all / [0, 1) and given a sample X, Pt. the sample X defined in equation 12 satisfies Xeth Pt+h + oh)).
Algorithm I formulates basic sampling algorithm given generating probability velocity u4- In order for the r.h.s. equation 12 to define a proper PMF for sufficiently small h > is necessary and sufficient that the probability velocity satisfies the conditions
ui(a" 2) 0, and ut("", 2) for all E [N] and a' # =
Σ
(13)
Now the main question is how to find probabil- ity velocity de that generates the probability path
Algorithm I Flow Matching sampling.
defined in equations 7 and 88 A key insight in
Require: velocity us, sample ~ p. step size h for ł 0, h, 2h, ....... ...... 1- do X 6x-()) huil,, X). for [N] & cq. 24
Flow Matching (Lipman et al. (002) that ot can be constructed as marginalization of con- ditional probability velocities, uff (x 2/00, TT).
end for return X
generating the corresponding conditional probability paths (r (zo, (="|,x0 æ1). This can also be shown to hold in the discrete CTMC setting (Campbell et al. 2024), where a reformulation our context and notation is as follows.
'The o(h) notation means function going faster than /e as h 0, i.e.. o(h5) h 0.

---

Theorem 2. Given conditional probability velocity w/(x", 22/,,, generating conditional probability path pr(z(zo. 21). the marginal velocity defined by
(x(x
Σ
u/ 2 2/ an )pr 21
(14)
R0 ED
generates the marginal probability path pe(a), where by Bayes' rule
(r(a zo. 21 (1))) 21)
Pe x1 x1(2)
(15)
(z(2)
The proof, similar to the continuous Flow Matching case, shows that un and satisfy the (discrete version of the) Continuity Equation.
For completeness we provide simple proof of this theorem in Appendix E.2
The Continuity Equation. To provide the mathematical tool for showing that a probability velocity We does indeed generate the probability path pt, and also further highlight the similarities to the continuous case, we next formulate the Kolmogorov Equations, which describe the state probability rate Pe (z). æ D, in CTMC Continuity Equation (CE). The Continuity Equation, similarly to Kolmogorov Equations, describes (z)), E R in the continuous case, and is formulated as the Partial Differential Equation (PDE)
Pe (x) div= = 0,
(16)
where the divergence operator (v) applied vector field v: RN RN defined by
div = Σ 2
(17)
am
and intuitively means the total flux leaving T, see Figure 2 (middle-right). This gives an intuitive explanation to the Continuity Equation: the rate of the probability (.()) of a state x E RV equals the total incoming probability flux, pru,, at In the discrete case (CTMC) the Continuity Equation (equation 16) holds as is, once the discrete divergence operator is properly defined, i.c., to measure the outgoing flux from discrete state. In more detail, given some vector field, which in the discrete case is scalar-valued function over pairs of states, V : D x D -> R, the discrete divergence
div (v) = Σ (u(2,z) - u(x
(18)
RED
where 2(2,z) represents the flux x and v(x 2) represent the opposite flux 2 2; Figure 2 right. Now, in our case (see Figure 2 middle-left), the probability flux at a state x E ' involves all sequences with most one token difference from æ. i.e., the probability flux pru; at takes the form v(zr, =) Pr (=) z) and v(2,x) = (2*) for = and æ that differ only in the i-th token, v(z x) pt (x) Di= uf(=", .), and v(x, = 0 for all other (z,x) E D D. A direct calculation now shows (see Appendix E.I.
N
div.. =- Σ P((2)
Σ 5z (a
(19)
sED
Checking that a probability velocity u4 generates a probability path Ps (in the sense of Definition T amounts to verifying the Continuity Equation (equation I6). Indeed, using arguments from Campbell al. (2024) and the discrete divergence operator, the PMF of X1+A defined by sampling according
to equation! 12
N
Ex. II 6x. (x*) huf(a* Xx)] Exx 8x. (I) + Σ 8x. (x (ut(" Xx)
+o(h)
(20)
1G
= p<(x)= hdiv o(h)
Pc(x) - hip (x) + ohh) = Pt+h + o(h),
where we assume X, ~ Pt, the first equality uses the identity II. [a' + hb* II.
ET a')b' + o(h), the second equality uses equation 19 and the previous-to- to-last equal- ity uses the Continuity Equation (equation (6). This shows that the Continuity Equation holds then We generates Pt in the sense of Definition]
5

---

Table 1: Generating (marginal) velocity fields have identical form for the continuous and discrete Flow Matching when using denoiser/noise-preddeee prediction parameterization; (1){(2) = Ex (x)=)((1 X1 is
the standard continuous denoiser (a.k.a. z-prediction) and (z) = Ex (-1=) Xo is the standard noise-prediction (a.k.a. e-prediction).
Continuous Flow Matching
Discrete Flow Matching
Marginal prob.
Pe(2)) Eno., Il (1a (1) 21)
Conditional prob. (x 6N,0, (11 k,00 (x²)
(1 m1) neda, (x") + (1 k2)620
VF-Denoiser ((((
1-6t (Xi) x:] (((X,)
At
"ila', 1-ke (pqp((='(x)) 8x, (x)) tie Ke [6x, (x*) - Poojele' [XX)]
VF-Noise-pred
ds [xi (x,)
xx( =
Conditional and marginal generating velocities. We provide the probability velocities generating
the conditional probability paths (c 2o, 21) defined in equations 7 and 8 Then, using the marginal-
ization formula in equation [4 we end up with a closed-form marginal velocity for the probability paths Pt (z). In Appendix E.3 we show
Theorem 3 (Probability velocity of conditional paths). A generating probability velocity for the
conditional paths pe (x x1) defined in equations 7 Randd
2g w (x') + (x)) ua(a", =lw0, Σ 3=1
(21)
with ab = kg =k( -
k/t /k/ and by = k/ /x/ where == arg min x2 2/5
/x(
Now, computing the marginal probability velocity using equation 4 applied to the conditional probability velocity in equation 21 gives
(
Σ
o0 (x b1 (,"),
(22)
3=1
where the posteriors w/ of w (that are later shown to be tractable to learn) are defined by
(x),2)
Σ w (x' 21)p (zo, 21
(23)
EDD
where (T 21l²) (defined in equation 15p is the posterior probability of øo. 21 conditioned on the
current state X, 2. A useful instantiation of the general velocity in equation 22 is when considering
the path family in equation for which w [20|2x0,00 æ1) = 8z, (a*), w² (x* 220 x1 8zo (x') Kr
i.1
Nt, K3 1 Kg, A₁ (i.e.. monotonically non-decreaingg in t) and this case equation 22 reads
as
u/ 2) = Rs
1 6
(24)
where we use the notation Pije Pije (z]()))))= = Exo,,, Ez (p (10, for the probability denoiser.
Sampling backward in time. We also sample backwards in time by following the sampling rule
Xi ~ (x;(-) (. X,). In this case -u((=", 2)) should satisfy equation 13 A (backward-tings generating probability velocity can then be achieved from equation 22 with the simple change to the
coefficients a. and 61d, see Appendix E.4 For defined with equation the generating velocity is
htt [8_(x") - Pojo(x"|=)]
ua(aa, 2)
(25)
Ki
where in this case poje (20()) = Exo &ap (e)o)) x1/2) is the probability noise noise-prediction

---

Remarkably, the generating velocity fields in 24 and 25 take the exact same form as the generating (a.k.a. marginal) velocity fields continuous flow matching when parameterized via the denoiser or noise- poise-prediction parameterizations and using the same schedulers, see V Appendix F.9
for explanation of the continuous case. Appendix F.4 we provide the backward-time version of Theorem 3
Corrector sampling. tion 25 i.e.,
Combining the forward-time ün equuatonn 24p and backward- de ( (equa-
dt( 2) (, 2) 3, 8, (z),
(26)
provides valid forward-time probability velocity field (i.e., satisfies equation 13 for & E (0, 1) as long as dir, 3r > O. This velocity field can used for two types corrector sampling: (i) When
Oz B, = sampling with ü, leads to corrector sampling where intuitively each step moves + 3 forward in time and 3, backwards, which allows reintroducing noise into the sampling process; and (ii) when a2 3, 0 sampling with in when fixing E (0, 1) leads to corrector iterations where limit samples distribute according to Pt. In Appendix E.6 we prove:
Theorem 4. For perfecily trained posteriors and or., .r O, L. (0,1), de in equation 26 is
probability velocity, i.e, satisfies equation [3 and: (i) For oz - Br = 1, dt provides a probability velocity generating Púi (ūi) For Ot Be
repeatedly sampling with ūy at fixed E (0,1) and sufficiently small h is guaranteed to converge to sample from pt-
One simplification to equation 26 can be done in the case of paths constructed with conditional as in
equation independent coupling (1r, .11) = ((x)))((11), and i.i.d. source p(xu) II p(a))) e.g. p(z)) is uniform over [d] or (1)()) In this case, the backward-time formula in equation 25
take
an equivalent simpler form
Rtt [5_(x') p(x')),
"/
(27)
Kt
which does not require estimation of the posterior Poj+ See Appendix E.5 for the derivation.
Training. Equation 22 shows that for generating samples from a probabilty path (x()) require the posteriors di (x' X2). Training such posteriors can be done by minimizing the loss
((0) =
Σ X,(.x,,)))x.,;;; log of (Y)]Xx;;) JE(M],
(28)
where / is sampled according to some distribution in [0, 1] (we used uniform), (Xo, X1) ((o, Xi). X, ~ Pr (X, X1), and Y ~ w ( Xo, X1);
Rp denotes the learnable parameters. In the common case we use in this paper of learning single posterior, i.e. the probability denoiser Pije. the loss takes the form [())= EiEiNI XX).)..., log Pije (Xj X2). In Appendix E.7
we prove:
Proposition 5. The minimizer of L (equation 28 is wi (X.) (equation 23
3 Related work
In the section we cover the most related work to ours; Appendix A we cover other related work.
Discrete Flows (Campbell et al. (024) is probably the most related work to ours. We build upon their CTMC framework and offer the following generalizations and simplifications: We consider arbitrary couplings (Xo, X1), and offer novel and rather general family of probability paths (equation 8) for which we provide the generating probability velocities in a unified closed-form formula (cquations
22 25). These in particular recreate the same formulas as the continuous Flow Matching counterpart (Table We furthermore develop a general corrector velocity (equation 26) that unifies both corrector iterations (Song et al. 2020 Campbell et al. 2022) and stochastic sampling of Campbell et al. (2024). We show that particular choices of noise schedulers K₂ (Kg = t reproduces Campbell et al. (2024)) and corrector schedulers provide boost in results. Lastly, we opted for the term probability velocity for ui (x' X2) as it is not precisely a rate matrix in the state space D x D used in CTMC since u²(x', 2) for all E [N] define multiple self-edges z z.
Masked modeling (Gaavvininejad et al., 2019: Chang et al. 2022). In case of a masked model, i.e., when the source distribution p(x) 0m (2)). we achieve an interesting connection with MaskGit
7

---

Table Generative perplexity on unconditional text generation compared to prior work. All models are sampled without the use of temperature or corrector steps. Double precision sampling results are reported in 5
METHOD
NFE
LLAMA-24
LLAMA-34
GPT2,
ENTROPY
Data
7.0
9.4
14.7
7.7
Autoregressive
1024
31.4
54.8
45.3
7.1
Savinov
2021
200
29.5
45.1
34.7
5.2
Austin et al. 2021a
1000
697.6
768.8
837.8
7.6
Han al. (202
>10000
73.3
203.1
99.2
4.8
Lou al.
2023
256/512/ 1024
38.6/33.7/27.2
69.2/58.6/43.9
64.3/53.4/40.5
7.8/7.7/7.6
Campbell al. K2024
256/512/ 1024
38.5/33.5/28.7
69.0/56.5/46.5
65.2/53.3/43.0
7.8/7.7/7.6
FM (equation
256/512/103 1024
58.5/48.8/33.8 30.0/27.5/22.3
34.2/30.0/22.5
54.2/43.5/29.3
7.717.6/7.2
FM (equation
256/512/1024
48.2/43.5/31.9
47.7/418/28.1
7.6/7.5/7.1
Table 3: Generative perplexity on conditional text generation.
METHOD
MODEL SIZE
NFE
LLAMA-21
LLAMA-3L
ENTROPY
Llama-3 (Reference)
8B
512
6.4
7.3
6.8
Llama-2 (Reference)
7B
512
5.3
8.3
7.1
Autoregressive
1.7B
512
14.3
22.3
7.2
Savinov al. (2021)
1.7B
200
10.8
15.4
4.7
FM (U-coupling)
1.7B
256/512
10.7/9.5
11.2/10.3
6.7/6.7
FM (C-coupling)
1.7B
256/512
10.2/8.9
10.0/9.7
6.8/6.7
showing it is actually instance of Discrete Flow Matching with small yet crucial change its sampling algorithm. First, in Appendix E.8 we prove that in the masked setting, the probability denoiser Pije is -ndepenndenn
Proposition 6. For paths defined by equations] 7 and with source p(x) = 6m((x) the posterior Pe(Zo, 21 01|2) p(ao, 21 22122) is time-independent. Consequently, the probability denoiser (1)e(=²)2) pi (1(1))) also fime-independent.
This shows that the probability denoiser can be learned with no time dependence, similar to the unmasking probabilities in MaskGit. During sampling however, there are two main differences between sampling and MaskGit sampling. First, unmasking of tokens in algorithm is done according to the probability 5x, (x') + hut (x' X,) independently for each token x', E [N]. This procedure is justified as it samples from the correct probability asymptotically via the derivation of the Continuity Equation 20 This is in contrast to MaskGit that prioritizes the token to be unmasked according some confidence. In the experiments section show that MaskGit's prioritization, although has some benefit the very low NFE regime, actually introducing strong bias in the sampling procedure and leads to inferior overall results. Secondly, using corrector sampling allows for reintroducing masks to already unmasked tokens in way that is still guaranteed to produce samples from pt, Theorem we find this have significant positive effect the generation quality.
Discrete diffusion. D3PM (Austin al. 202 la and Argmax flows (Hoogeboom al.) 2021
introduced diffusion in discrete spaces by proposing a corruption process for categorical data. A later work by Campbell et al. (2022) introduced discrete diffusion models with continuous time, and Lou ett al. 2023) proposed learning probability ratios, extending score matching (Song and Ermon to discrete spaces.
2019
Experiments
We evaluate our method on tasks of language modeling, code generation, and image generation. For language modeling, we compare the proposed method against prior work considering the widely used generative perplexity metrie. We scale the models billion parameters and present results on coding tasks, i.c., HumanEval (Chen al. 202
MBPP (Austin et al., 2021b), demonstrating

---

Table 4: Execution based code generation evaluation.
HUMANEVALT
MBPP (1-SHOT)
METHOD
DATA
Pass@
Pass@ 10
Pass@25
Pass @
Pass 10
Pass@25
Autoregressive
Text
1.2
3.1
4.8
0.2
1.7
3.3
Code
14.3
21.3
27.8
17.0
34.3
44.1
FM
Text
1.2
2.6
4.0
0.4
1.1
3.6
Code
6.7
13.4
18.0
6.7
20.6
26.5
FM (Oracle length)
Code
11.6
18.3
20.6
13.1
28.4
34.2
the most promising results to date in a non-autorregressive context. In image generation, we present results for fully discrete CIFARIO (Krizhevsky al. 2009). Further details of the experimental setup for each model are provided in Appendix
Experimental setup. In our experiments we used the masked source, i.e., P 6m. and trained with both unconditional coupling (U-coupling, equation 4 and conditional couplings (C-coupling, equation (5) with the probability path defined in equations 7 and in one case I0] We trained probability denoiser (loss in equation (8) and sampled using the generating velocity in equation 24 and Algorithm[] [ We used partieular choice of probability path scheduler Ki, well corrector steps defined by scheduler a and temperature annealing. We found the choice these schedulers to be pivotal for the model's performance. In Appendix we perform an ablation study, evaluating various scheduler choices.
4.1 Language modeling
We experimented with our method in three settings: (i) Small model (150M parameters) comparison to other non-autoregressive baselines in unconditional text generation; (ii) Large model (1.7B parameters) comparison to autoregressive models in conditional text generation; and (iii) Large model (1.7B parameters) conditional code generation. As computing exact likelihood for non- autoregressive model is a challenge, for (i)
the generative perplexity metric (Appendix G measured with GPT2 (Radford et al. 2019). Llama-2 (Touvron et al.) [202)), and Llama-3, and we also monitor the sentence entropy (Appendix G to measure diversity of tokens and flag repetitive sequences, which typically yield low perplexity. Throughout our experiments we noticed entropy 2 usually corresponds diverse texts. For (iii) evaluated using the success rate of coding tasks.
Evaluation against prior work. We evaluate our method against prior work on vone modeling. For a fair comparison, all methods are trained on I 150M parameters models using the OpenWeTText (Gokaslan and Cohen 2019) dataset. We also fix all sampling hyperparameters to the most basic settings, 1.c., temperature. top probability, corrector steps. etc. For method tried two paths defined by equations and IO, Results are reported in Table 2 where our method outperforms all baselines in generative perplexity for all numbers of function evaluations (NFE).
Conditional text generation. In this experiment, we train both C-coupling and U-coupling 1.7B parameters FM models with paths defined by equation on a large scale data mix (Touvron et al.) 2023). Table 3 presents the generative perplexity of conditional generations from our method; the conditions we used are the prefixes of the first 1000 samples in OpenWeb dataset. We also compare to existing state-of-the-art autoregressive models. Our results demonstrate that our model effectively narrows the gap in generative perplexity with autoregressive models, while maintaining an entropy comparable to the recent Llama-3 8B model. Furthermore, we note the C-coupling trained model produces slightly better perplexity in conditional tasks than the U-coupling model. In Appendix [ present qualitative conditional samples produced by our U-coupling model.
we
Code generation. Here we trained our basic setting of 1.7B parameters FM model with U-coupling and path in equation on a code-focused data mix (Roziere al. (023). Table presenns results
on HumanEval and MBPP (1-shot) for pass @ {1, 10. 25/. In Table 4 Oracle length' evaluates
performance our model when conditioning on the length of the solution. This done by
inserting an 'end of text' token in the same position of the ground truth solution. Our method achieves non-trivial results on both tasks, which to the best of our knowledge is the first instance of non-autoregressive method being capable of non-trivial coding tasks. In Appendix C we analyze the

---

12
10
Inception Score
8
MaskGIT
4
MaskGIT
FM Linear
FM Linear
FM Cubic
FM- Cubic
32
128
512
2048
32
128
512
2048
NFE
NFE
(a) FID.
(b) Inception Score.
Figure 3: FID and Inception scores vs. number of function evaluations (NFE).
proposed method for code infilling, which can achieved our model allows non-autoregressive generation, Lastly, in Appendix we show qualitative examples of success and failure cases produced by our model on the coding tasks, and Appendix H.3 we show examples of code infilling.
4.2 Image generation
We performed fully discrete image generation, without using any metric neighboring information
between color values. We trained an FM model with U-coupling and path equation on CIFARIO
to predict discrete color value for tokens, i.e., d 256, with sequence length of N 32 x 32 x 3. For generative quality we evaluate the Fréchet Inception Distance (FID) (Heusel et 2017). Ablations
for the probability path schedulers are provided in Figure 8 in the Appendix G In Figure 3a we compare our method with: (i) MaskGIT (Chang et al. (022); and (ii) (Campbell al. 2024 which
coincides with our method for linear scheduler. More details in Appendix G As can be seen in Figure 3a our method outperforms both baselines, achieving 3.63 FID T024 NFE. In fig. 3bb observe similar trend when evaluating Inception score. As discussed above, MaskGit sampling performs better for low NFE but quickly deteriorates for higher NFE. We attribute this to bias introduced in the sampling process via the confidence mechanism.
5 Conclusions and future work
We introduce Discrete Flow Matching, a generalization continuous flow matching and discrete flows that provides a large design space of discrete -on-autoregressive generative models. Searching within this space we were able to train large scale language models that produce generated text with an improved generative perplexity compared to current non-autoregressive methods and able solve coding tasks at rates not achievable before with non-autoregressive models, as far as we are aware. While reducing the number of network evaluations required to generate discrete sample compared to autoregressive models, Discrete Flow Matching still does not achieve the level of sampling efficiency achieved by its continuous counterpart, flagging an interesting future work direction. Another interesting direction is to explore the space of probability paths in equation 8 (or generalization of which) beyond what we have done in this paper. We believe discrete non-autoregressive models have the potential to close the gap and even surpass autoregressive models well as unlock novel applications and use cases. As our work introduces an alternative modeling paradigm to discrete sequential data such language and code, we feel does not introduce significant societal risks beyond those that already exist with previous large language models.
10

---

References
Janice Ahn, Rishu Verma, Renze Lou, Di Liu, Rui Zhang, and Wenpeng Yin. Large language models for mathematical reasoning: Progresses and challenges. arXiv preprint arXiv:2402.00137, 2024.
Michael S Albergo and Eric Vanden-Fijnden Building normalizing flows with stochastic interpolants. arXiv preprint arXiv:2209. 155711 2022.
Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in Neural Information Processing Systems, 34:1 19:111991-99999 2021a.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry. Quoe Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732 2021b.
Victor Besnier and Mickael Chen. pytorch reproduction of masked generative image transformer, 2023.
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Andrew Campbell, Joe Benton, Valentin De Bortoli, Thomas Rainforth, George Deligiannidis, and Arnaud Doucet. A continuous time framework for discrete denoising models. Advances in Neural Information Processing Systems, 35:28266-2822 28279, 2022.
Andrew Campbell, Jason Yim, Regina Barzilay, Tom Rainforth, and Tommi Jaakkola. Generative flows on discrete state-spaces: Enabling multimodal flows with applications to protein co-design. arXiv preprint arXiv:2402.04999 2024.
Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1 1135--11 1325, 2022.
Huiwen Chang. Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming- Ming-Hsuan Yang, Kevin Murphy, William Freeman, Michael Rubinstein, et Muse: Text-to-image generation via masked generative transformers. arXiv preprint arXiv:2301.000 200040 2023.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained code. arXiv preprint vrXiv:2107.03374 2021.
Ting Chen, Ruixiang Zhang, and Geoffrey E. Hinton. Analog bits: Generating discrete data using diffusion models with self-conditioning. ArXiv, 2022.
Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexandre Défossez. Simple and controllable music generation. Advances Neural Information Processing Systems, 36, 2024.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in neural information processing systems, 34:8780-8799 2021.
Sander Dieleman, Laurent Sartran, Arman Roshannai, Nikolay Savinov, Yaroslav Ganin, Pierre H Richemond, Arnaud Doucet, Robin Strudel, Chris Dyer, Conor Durkan, et al. Continuous diffusion for categorical data. arXiv preprint arXiv:2211.15066 2022.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et Scaling rectified flow transformers for high-resolution image synthesis. arXiv preprint 2003::::0000032266 2024.

---

Noelia Ferruz and Birte Höcker. Controllable protein design with language models. Nature Machine Intelligence, 4(6):521-532. 2022.
Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. Mask-predict: Parallel decoding of conditional masked language models. arXiv preprint 1rX1v:1904.09322, 2019.
Aaron Gokaslan and Vanya Cohen. Openwebtext corpus. OpenWebTextCCrpus 2019.
http://88kylion007.gitzz
Xiaochuang Han, Sachin Kumar, and Yulia Tsvetkov. Ssd-Im: Semi-autoregressivee simplex-based diffusion language model for text generation and modular control. arXiv preprint arXiv:2210. 10.1:2210.17432. 2022.
Michael Hassid, Tal Remez, Tu Anh Nguyen, Itai Gat, Alexis Conneau, Felix Kreuk, Jade Copet, Alexandre Defossez, Gabriel Synnaeve, Emmanuel Dupoux, et al. Textually pretrained speech language models. Advances in Neural Information Processing Systems, 36, 2024.
Zhengfu He, Tianxiang Sun, Kuan Wang, Xuanjing Huang. and Xipeng Qiu. Diffusionbert: Improving generative masked language models with diffusion models. In Annual Meeting of the Association for Computational Linguistics, 2022.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge a local nash equilibrium. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
Jonathan Ho. Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-685// 2020.
Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling, Argmax flows and multinomial diffusion: Learning categorical distributions. Advances in Neural Information Processing Systems, 14:12454-12455, 2021.
Shima Imani, Liang Du, and Harsh Shrivastava. Mathprompter: Mathematical reasoning using large language models. In Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics ( (Volume 5: Industry Track), pages 37-42, 2023.
Felix Kreuk, Gabriel Synnaeve, Adam Polyak, Uriel Singer, Alexandre Défossez, Jade Copet, Devi Parikh, Yaniv Taigman, and Yossi Adi. Audiogen: Textually guided audio generation. arXiv preprint arXiv:2209. 15352, 2022.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. arXiv, 2009.
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Mare Marone, Christopher Akiki, Jia Li, Jenny Chim, al. Starcoder: may the source be with you! arXiv preprint arXiv:2305.06/60 2023.
Xiang Lisa Li, John Thickstun, Ishaan Guirajani. Percy Liang, and Tatsunori Hashimoto. Diffusion-hh Im improves controllable text generation. ArXiv, 2022.
Zheng- Wen Lin, Yeyun Gong, Yelong Shen, Tong Wu, Zhihao Fan, Chen Lin, Weizhu Chen, and Nan Duan. Genie Large scale pre-training for generation with diffusion model. In ArXiv, 2022.
Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamm Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprini arXiv:2210.027477 2022.
Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning generate and transfer data with rectified flow. arXiv preprint arXiv:2209.0300) 2022.
Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion language modeling by estimating the ratios of the data distribution. arXiv preprint arXiv:2310.16834, 2023.
Justin Lovelace, Varsha Kishore, Chao gang Wan, Eliot Shekhtman, and Kilian Q. Weinberger. Latent diffusion for language generation. ArXiv, 2022.
12

---

Ali Madani, Ben Krause, Eric R Greene, Subu Subramanian, Benjamin P Mohr, James M Holton, Jose Luis Olmos, Caiming Xiong, Zachary Z Sun, Richard Socher, et Large language models generate functional protein sequences across diverse families. Nature Biotechnology, 41(8): 1099-1106, 2023.
James R Norris. Markov chains. Number 2. Cambridge university press, 1998.
William Peebles and Saining Xie. Scalable diffusion models with transformers. arXiv preprint arXiv:2212.09778, 2022.
Aram-Alexante Pooladian, Heli Ben-Hamu, Carles Domingo-Enrich, Brandon Amos, Yaron Lipman, and Ricky TQ Chen. Multisample flow matching: Straightening flows with minibatch couplings. arXiv preprint arXiv:2304. 14772, 2023.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019. httpps:/aap..semanttcccoooooooo semanticscholl..
org/
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pages 10684-10695, 2022.
Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M Pawan Kumar, Emilien Dupont, Francisco JR Ruiz, Jordan S Ellenberg, Pengming Wang, Omar Fawzi, et al. Mathematical discoveries from program search with large language models. Nature, 625(7995)4468 475, 2024.
Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. Code Ilama: Open foundation models for code. arXiv preprint arXiv:2308. 0..3308..25000 /2950, 2023.
Nikolay Savinov, Junyoung Chung, Mikolaj Binkowski, Erich Elsen, and Aaron van den Oord. Step-unrolled denoising autoencoders for text generation. arXiv preprint arXiv:2112.06749, 2021.
Uriel Singer, Adam Polyak. Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209. 2022.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. basee generative modeling through stochastic differential equations. arXiv preprint 1tXiv:20111113456 2020.
Hannes Stark, Bowen Jing, Chenyu Wang, Gabriele Corso, Bonnie Berger, Regina Barzilay, and Tommi Jaakkola. Dirichlet flow matching with applications to dna sequence design. arXiv preprint arXiv:2402.05841 2024.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568: 127063, 2024.
Alexander Tong, Nikolay Malkin, Guillaume Huguet, Yanlei Zhang, Jarrid Rector- Brooks, Kilian Fatras, Guy Wolf, and Yoshua Bengio. Improving and generalizing flow-based generative models with minibatch optimal transport. arXiv preprint arXiv:2302.00482. 2023.
Rector-Brooks,
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
Qiang Zhang, Keyang Ding, Tianwen Lyv, Xinda Wang, Qingyu Yin, Yiwen Zhang, Jing Yu, Yuhao Wang, Xiaotong Li, Zhuoyi Xiang, et al. Scientific large language models: survey on biological & chemical domains. arXiv preprint arXiv:2401.14656 2024.
13

---

Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. survey of large language models. arXiv preprint arXiv:2303. 02.3003332,, 2023.
Kaiwen Zheng, Yongxin Chen, Hanzi Mao, Ming- Yu Liu, Jun Zhu, and Qinsheng Zhang. Masked diffusion models are secretly agnesago masked models and exploit inaccurate categorical sampling. arXiv preprint arXiv:2409. 02908, 2024.
Alon Ziv, Itai Gat, Gael Le Lan, Tal Remez, Felix Kreuk, Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. Masked audio generation using a single non-autoregressive transformer. arXiv preprint 2rXhv:241.00577, 2024.
14

---

A Related works, continuation
We provide here some more details on relevant related works.
Continuous diffusion and flows. Another line of works has been exploring the use of continuous space diffusion for discrete data, typically operating in the logits space (Dieleman et al. 2022 Li
et al. 2022 Han et al. 2022 Lin et al. 2022 Chen et al. 2022). An additional body of work has
been focusing on the adoption of latent diffusion-like modeling (Lovelace et al. 2022 He et al.
2022 Stark al. (2024) proposed learn continuous Flow Matching on the probability simplex
with Dirichlet paths.
Autoregressive modeling. Autoregressive models have been significant area of focus in recent
years, particularly in the context of natural language processing and machine learning (Zhao et al.
2023). Autoregressive modeling, in its most fundamental form, utilizes the chain rule to learn the joint sequence probability by breaking it down into next-token conditional probabilities. GPT-2 (Radford et al. 2019), showcased the power of autoregressive language models in generating coherent and
contextually relevant text over long passages. Its successor, GPT-3 (Brown et al., 2020), further
pushed the boundaries, demonstrating impressive performance across range of tasks without task-
specific training data. Later models were adapted to other domains such as, code (Roziere et al. [023
Li et al. 2023 Chen et al. 2021), biology (Zhang al., 2024; Ferruz and Höcker 2022 Madani
et al. 2023). math (Romera- Paredes et al. [024;] Imani et al. [003] 2223]1Ahn
Ahn et al. 2024). audio (Kreuk
al.
2022 Copet et al. 2024 Hassid et al. 2024) and more.
Masked generative modeling. Masked generative modeling proposes to mask variable portion of the input sequence and training a model to predict this masked section. (haavvinneejad et al. (2019) proposed MaskePredict, a masked language modeling with parallel decoding. [Savinov et al. (202T) extended the mask-modeling approach by employing an additional loss term that incorporates rolling model predictions. MaskGIT (Chang et al.) 2022) followed a similar path, for the task of class-conditioned image synthesis, Chang et al. 2023) extended this approach to high-quality textually guided image generation over low-resolution images followed by a super-resolution module. Recently, Ziv et al. (2024) proposed text-to-music method, which relies on the MaskGIT foundations while observing that span masking boosts the quality of the generated sequence significantly.
B Further implementation details
Safe sampling. When sampling according AlgorithmmTusnn the generating probability velocity in equation 22 an arbitrary step size h >> O can make some probabilities in 5x; (-) + hu X,) negative and consequently require clamping and injecting further error into the sampling process that can in turn accumulate a non-negligible global sampling error. simple fix that guarantees valid probability distribution while keeping the o(h) sampling етrоr at the relatively manageable price of potentially more function evaluations using the following adaptive step size Algorithm I t T [0, 1) use
at time
Nndapiive = min h, min
(29)
As can be verified with the general probability velocity formula in equation 22] the above choice for halniiive guarantees 0x²() + huf(( X,) is a valid PMF, As mostly used in this paper, for the probability denoiser parameterization (equation 24) the adaptive step is
audapiive
min
h,
Rit
(30)
With the corrector sampling (equations
26 and
51
we have the adaptive step:
Beki
halagive min h,
(31)
1 Rit
15

---

Conditioning. In our unconditional coupling (U-coupling), see equation 5 we define the condition- ing pattern based on prefixes of random length No N. i...
(I....,
During the training phase, we sample No ~ (0( N) and adjust the input sequence accordance with the mask 1.
During conditional sampling with Algorithm I we replace, after each update step, the relevant tokens with the conditioned ones, i.e., X 1 Y - @ X, where X is the current sample, Y the condition, and is the condition's 's mask.
NFE bound. For mask modeling, i.e., p 5m. have seen that the probability denoiser is time-independent (see Proposition 6). Consequently, when sampling with Algorithm I and uy from equation 24 without corrector sampling one is not required recompute the forward pass Pije(-|X.) X, is identical to X,-h (i.e., no im has been unmasked). This means that the NFE Algorithm in this case bounded by the number of tokens N.
Post training scheduler change. For a trained posterior vh ( [z) of a conditional probability path in equation 9 with scheduler Hie, the velocity is given by equations 24 or 25 where ( is either Pije(2") ] or Pojx(>²(z) respectively. In this case, we can apply the velocities in equations 24
and for sampling with any scheduler kig, using the change scheduler formula for posteriors,
<x(("/12) vrr (x*
(32)
where (x*)2a, is the posterior of the scheduler rig, t' Kig and x- is the inverse of K. The scheduler change formula in equation 32 is proved in Proposition 8 We note that by Proposition 6
for mask modeling, i.e., 5m. the posterior wi (x*|z) is time independent. Hence, in that case, the posterior is not affected by scheduler change.
C Code infilling
100
87.19
We additionally evaluate the proposed method considering the task code infilling. In which, we are provided with an input prompt that con- tains various spans of masked tokens, and our goal is to predict them based on the unmasked ones. See Figure [ (middle and right sub-figut sub-figures) for a visual example. Notice, this evaluation setup is the most similar to the training process.
80
60
40
11.59
compiles@1
20
For that, we randomly mask tokens with respect several masking rations, p E (0.0, 0.1, 0.2.... 1.03. from HumanEval and report both pass@ 1 and compiles@) I metrics. For the purpose of this analysis, we provide the oracle length for each masked span. In other words, the model predicts the masked tokens for already given maks length. Results for the 1.5B parameters models can be seen in Figure 4 As expected, both pass@1 and compiles@ @ keep improving we decrease the level of input masking. Interestingly, when considering the fully masked sequence, providing the oracle prediction length significantly improves the pass scores (6.7 vs. 11.6).
pass01
0.0
0.5
1.0
Mask probability
Figure 4: Pass@ I and compiles@* @ scores for the 1.5B parameter models as a function of the input masking rations on HumanEval.
D Ablations
Train and sampling path scheduler choice (15²). We study how the choice of the probability path scheduler affects the model performance. For that, consider parametric family of cubic
16

---

1.0
1.0
18
120
1L.1
0.8
0.6
8.6
1825. 61
B.4.G
0.0
63
1623
3-4 02
0.0
0.2
0.4
0.6
0.0
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
(a) Path scheduler, cubic poly.
(b) Corrector scheduler (1 1)"
(c) Temperature scheduler (1 6).
polynomial with parameters a,
Kg 2f + 3t² + a(t 2t + b(t t²
(33)
Note that = and = 0 and a and are setting the derivative of Ke at t 0 and = 1,
respectively. We visualize this di with choices of a, E (0, in Figure 5a
Temperature 0.8
Temperature 0.9
Temperature 1.0
30.0
100
27.5
14.32
(2.5
0,00
(3.11
160
12
12.00
15.3
GPT2
25.0
01.01
19.21
1..
03
140
12.13
05.2
22.5
120
20.0
100
15.0
12.5
10.3
Steps
Steps
Steps
Figure 6: Path scheduler choice during training using various of constant temperature values.
To test the effect of path schedulers in training we have trained 150M parameters models for all choices of a, b {0, 2, 3). We then generate 1000 samples from each model. The samples computed using A with the path scheduler the model was trained on, and with temperature levels {0.8, 0.9, 1}, where temperature is applied via
(x* (x.) = - log (1jo X,).
(34)
We then evaluate the generative perplexity of these samples with GPT-2. Figure ( shows the results. The graphs indicate that, in the context of text modality, the cubic polynomial scheduler with 0, = 2 (equivalent to square function) achieves the highest performance. Consequently, exclusively used this scheduler for the language models.
Corrector scheduler. In our experiments we only applied corrector sampling to our large models (U-coupling and C-coupling; 1.7B parameters). We used the optimal path schedulers from previous section and considered the following parametric family of schedulers for the corrector sampling:
1 at (1 1)%
(35)
where, we set B a2 and generate 1000 samples using Algorithm I with parameter values a. E (0, 0.25, 0.53 and a E {10, 15, 20). We then evaluated generative perplexity for these samples with Llama-2, showing results in Figure These plots indicate that smaller values of a and b result in lower perplexity values, albeit with somewhat reduced entropy. We therefore opted for setting b 0.25 that strikes good balance between perplexity and entropy.
Temperature scheduling.
For temperature sampling, we consider the following scheduler:
7(1 2
(36)
17

---

Model: Couplingg, Temperatur:: 10.0
Temperatare: 0.8. Alpha: 15
Model: Coupling, Temperature: 20.0
Entropy
Entropy
Entragy
Motel L-Coupting Tempenetuee Alphic 10.0
U-Coup ing. Temperatire: D.N, Alohac groplin
Motel: LxCoupling, Hempencure: Atahe: 20,0
10-
16
17
7.32
786
1.78
1:30
1.30
7.22
6.38
7.00
7.22
7.56
7.06
9.54
136
536
1.00
7.52
7:54
105
7/00
Entropy
Entroy
Entrogy
Mode:: C-Coubling Tempersture: d.n. Alshe: 10.0
Mode:: E-Coupling, Temperal aw: S.6, Alphic
Model: E-Coualing, Temperatiee: Alshe: 20,0
Entropy
Entropy graedin
Entrogy
Coupling, Temperature:
Coupling
Alpha:
Model Coupling, Temperatae
20
Nepplally
2
7.56
6.66
6.46
736
6.00 6,425
7:05
Entmpy
Entragy
Figure 7: Corrector scheduler ablation.
E Theory and proofs
E.1
Computation of the discrete divergence
We present the computation of the discrete divergence in equation 18 i.e.,
N
div div,(p.))) = Σ P(())
Σ dz (z (a)));(at
(37)
ZED
Computing the discrete divergence (equation 18 of the flux Pett state amounts to adding outgoing flux from and subtracting the incoming flux into Using the fact that 6z(z²) 1 if and only if = x or z differs from x only at the i-th token, gives:
div (p
Σ> (2)) (p(())u(((f,=)) (c(2)u((z',
2ED
Pc(x)
Σ )2
Σ
(x') (=())(())
ZED
= =p((a)
Σ (x)
equation 13
SED
(= (p
2ED
that gives equation 37 after noting that 6z (37) 8, (x²).
18

---

E.2 Conditional velocities lead to marginal velocities
We provide simple proof for Theorem 2 originally proved in Campbell et al. (2024):
Theorem
Given conditional probability velocity uf(a", X, (T generating conditional probability path 21), the marginal velocity defined by
Xi)
Xi)
Σ ED
ug(a", X₂ (ro Ii x1
(38)
generates the marginal probability path pt (2), where by Bayes' rule
Ps (r xı (x)) Pe(X, xn (1) x1)
(39)
(x
We start by taking the time derivative of the marginal probability path, pc(2) Exg,2 Pe ( I1 (x0, as follows,
Proof (Theorem 2.
po(x) Σ ia (1
21
N
Σ
Σ Pe(2/20,.21 x1
Σ
8 (x )uj("" 2/20. 21) x (<xu x1)
D. Continuity Equation
16
20.21
wl(x" 2|20, x1 pd(2/2o, 21 (x0, x1)
Σ
pr(z)
(a))
Σ mo
pc(=)
Σ Pc(2)
Σ 5. (x")))((((
-div.
Now since u.(x
(x', =)) is a convex combinations of u/(a", 2/w0, 21) and these satisfy equation then
also wf(x", Xx) satisfies equation 13
E.3
Probability velocities generating conditional probability paths
Equation with the coefficients a and bj are provided below,
6
w/(x², xe(oo,
m Σ
Rig
Rit
Rit z
w
(x (oo x2) x1)+ +
Rip
6x,
(x*),
(40)
j=1
Kip
where
8 = t(i,t) arg min
b. /
(41)
[m(m)
Theorem (Probability velocity of conditional paths). conditional paths pe (x) defined in equations and
generating probability velocity for the is
ui(a", Xe(ro, x11 = Σ
dd w (a' Jz0, x1) + 1,8x, (x'),
(42)
1=1
with a.d RR.. Kit Ri 1 and bj R6. /x// where e arg min 1/ //(())
19

---

Proof (Theorem 3/. X, D, and
First, let us show that equation 40 satisfies the conditions in equation 13 Fix
m
u/(x Σ
X xo
Kig
bid
Pig
w
(ro, 21) +
6x, (x)))
Rip
2
==1
R4
Σ
K4
2,6
m
m Σ
Ai Σ
1, and Σ k =0
5.8
K
B
j=1
K2
j=1
j
and for x² X have
ui(o", Xe 100 x1)
Ri f.E
2.5 w (101200, x1) 2
(43)
j=1
Kg
Rig
since k. 2 0, di, (z*12) 0. and that satisfies the Continuity Equation (equation 16p. To that end we write equation Sas as
0 since arg men
Second, we show
(1' x1)
1
pt (c (z0 22)
Σ jyhe
nit w (a* (10
(44)
ie Kig
where / arg min) J
Now by differentiating (00 we get
pe(x)aro, x1)
II (a (a)
N
pe(1/20, x1) = Σ pe(a (ro, xi (e
N
Σ pe(2 (mo,
Σ
kis n (x" (20, x11)
1=1
NN
Σ
pz 110,
Σ
d w (I'12o x1) + ni w² "'0" Io x1)
i=1
jite
m
pr (10,
Rig
Nog
u (x (00100,200 x1) + Pe (x 10o, x1)
equation 44
Kig
=pe (wx (m0)001)
NN
(x (p((0,0))
w (z0 21) + by
Σ (2(2)))_(2))))(()22oo
i=1
j=1
m
[m((= 211 Σ
id w (x'( x31) b(tm((*)
> -=))====0 = 62(x'), (2)) (x)
a
1=1
div div x1)ur(-/20 x1)),
20

---

as required.
E.4 Backward-time generating probability velocity.
Here we prove the equivalent of Theorem 3 for backward-time generating probability field. But first, let us justify the backward sampling formula,
Xi-n 6x;(+) Xe)(
(45)
Similar to equation 20 we have
Ex, (x+) hug((' Xx)] =Ex, (x, - h Σ 8x, (ut",, Xx) + o(h)
- (p( + A.div (pru:: + o(h)
16
pr(x) - (x)) + o(h) - pt (x) + o(h)-
Therefore if the Continuity equation holds and -u satisfies the conditions in equation 13 then given X; ~ pt. equation 45 provides an approximation X,-h Pt-h + o(h). The change the generating probability velocity in equation 22 to accommodate reverse time sampling is to replace the argmin in equation 41 with argmax,
f(i,t) A arg max k4 /k))
(46)
JE(m]
An analogous result to Theorem for backward sampling is therefore,
Theorem 7 (Probability velocity of conditional paths, backward time). The probability velocity - utt where ut defined in equation 21 with é arg maxjelmi (4 /x(()) is backward-tine time generating
probability velocity for the conditional paths pa (1 x1) defined in equations 7 Aand 8
Proof (Theorem 7. arbitrary X, D.
We follow the proof of Theorem 3 and indicate the relevant changes. First, for
Eut(r' X,) O,
(47)
exactly using the same arguments as the forward-time case. Now, for x' * Xi we have
m
"a(x", X,| x1)
wo (x (z)) 21) S
(48)
63 25.
due to l being now the argmax of the proof of the Continuity Equation follows through exactly the same also in this case.
Therefore - satisfies equation
13 Lastly, we notice that
E.5 Backward-time generating velocity for i.i.d. source p(un) and simple paths
Here we consider the case of probability paths defined via the conditionals in equation with independent coupling x ( F1 ((x)))(x1)) and i.i.d. source distribution p(rr) - ((
where p(z() is some PMF over id]. In this case one can simplify the backwad sampling formula
in equation 25 by using the following one which is equivalent (i.e., their difference is divergence free and consequently generate the same probability path p.):
Rit
0,(x" Xx)
(xxx p(x'))
(49)
The benefit in this equation is that it does not require the posterior Pojs, which needs to be learned in general cases.
To show that equation 49 is indeed generating probability velocity it is enough to show that
div [pr (ū, ū,)] 0,
(50)
21

---

where पn is the probability velocity in equation 25
Let us verify using equation 19
8xn (x*) Pr (r(2 x (p(r))(((1))
div [pe (ie ū;}] = Σ (x())
plat)
D x (z) p((o))((a))
Pt(z)
1,2
Σ
(xx") p(z))
Σ
8za (x" (p(ee))()) 21122220222))p000) 21)p(20)(((1)
[p(x*) - 600 (x0) Σ (p(2))))) 22) pl p(ao)g(ar)
Σ
1,20:11
Σ
[p(x") - 820 (x0)) pe Ioo in (p(e))((s)gl(e
=
1,10.21
Σ (((t)))(()) 60 (x (x'))(())) pi pe( on (1 (p(()))((1)))
1,
0,
where in the second to last equality we used the fact that the paths we are considering have the form: Pe(a* (mo, .1) = [[JENNI [Neden (I+) + (1 k20 (ar)] and therefore do not depend on the i-th source token, 2/--
E.6
Corrector steps
Theorem For perfectly trained posteriors and az Br > 0, t E (0,1), ūn in equation 26 is probability velocity, i.e., satisfies equation 13 and: (i) For a - 3g ūn provides probability
velocity generating Pe; (ii) For Br O, repeatedly sampling with āt at fixed E (0, 1) and sufficiently small is guaranteed to converge to sample from Pe-
Proof (Theorem Z.. First let us write explicitly ün from equation 26
a/ X.) = arū Xx) - (rū
ankt
Behr Pun (x (x),
a Ki
Behi
Pije (ax (x,) + Ki
+ Kit
8x, (x*).
(51)
Ktt
Since equation 51 is sum of PMFs with coefficients that up to zero the first condition in equation 13 i.e., Σn. ui(x' X2) = 0 holds. The second condition equation holds since for t EE (0. 1) we have atie I-Ke'
Bi 2 0. Now.
me
divz andiv 3,div = -xfp(()) + (x)((x)
linearity of div
D Equation 16
-aa (z))
(52)
For (i): Using equation 52 with 0+ Bg we get that
div divz(p,ūr) (.)
ie., ūn satisfies the continuity equation and therefore generates Pt.
For (ii): Setting Br 0 in equation 52 we get divz = 0 and therefore similar to
equation 20 we have
(x( = px(x) - hdiv.
Ex, (x) h 8x. (x)) (x)
(p(x)=)p(>)).
(53)
22

---

where using equation 51
we have
N
Peki dz (x Pun (xf1z)
p(z|=) - h
de 62 (x"))2 +
Σ
Kt.
Kx
i=1
Beiie
+ 1
Oz (x(2).
For sufficiently small h therefore p(x|2) is a convex combination of PMFs (in red) and consequently is itself a PMF in that is p]]]=) is a probability transition matrix, and p.(a)) is its stationary distribution, i.c., is an eigenvector p(æ[z) with eigenvalue 1, which is maximal. To prove convergence of the iterations in equation 53 we are left with showing that p(z[z) is irreducible and a-periodic, see Norris 1998) (Theorem 1.8.).. Irreducibly p(x|=) can be shown by connecting each two states 2, by changing positive (which is usually the case since as at-least one them is defined as soft-max finite logits); pepriodicityy is proved by showing p(æ[2) 0 which is true the coefficient of 6x(x) is greater than zero for sufficiently small h >> 0. Lastly, note that the iteration equation 53 changes one token time. An approximation this sampling can be achieved using our standard parallel sampling via equation
token at time, and assuming that Pije or Paje strictly
12 justified by equation 20
E.7 Training
Proposition
The minimizer of C (equation 28 is wi (x* (X,) (equation 23).
Proof (Proposition 5).
It is enough to prove the claim for m = 1, with single w wx x1).
((0) =
E
Σ
log we (y"12; (w (x xn (1)p:(2/z0))) xn (x x1)
N
Zo.41, ",,!
Po(2|or, xn )
(20, 21) pc(2)
E,
(z(z)
log vi (1'12;0) Σ y
Σ
N
-E,
Σ log dh (x)
that amounts to minimizing the Cross Entropy loss between (.('' (x)0) and (x()')X for all i [N], the minimizer of which satisfies ( X,; 0) úe (x).
E.8 Time-independent posterior for masked modeling
Proposition
6 For paths defined by equations and 9 with source p(x) = 6m(x) the posterior
Pe (o Z1lz) p(zo, 21|2) is time- ind-indeenddet. Consequently, the probability denoiser p1)((=1)2)
pi (x*1z) is also time-independent.
Proof (Proposition 0y.
First,
1(1 Re)
po(="loo (1 ne))m((²)) Kedi,
(i0de (2*)
2² #
and therefore
pr(2/00, =
II
(1 Kc)
Ke
II (zt
1:2f ofam
i:2f oten
23

---

The posterior now gives
I Pe(= to, ( x1) x ( x1)
Pr(2|20. xi ( 21)
(z(
Eso
[ Pr(22) [To 81) x (āo. T1)
Hiigt NZt Tiiz pm Kit [1iz dz (2') π (220 x1)
Σ Ego.es
I (1=11) mz) j: ston Ki
[ Mi oz (20) i (0o x1)
=((x0, T1 11(2))
showing that the posterior is time- -imeeindependent for dummy source distributions and convex paths. Consequently also the probability denoiser,
Σ dzn (z+t) pr pr (zxt) (r(, x1) pz(z)
(1)x(='|2)=
Σ
8z, (x"))(z0, 0112),
20.21
20.21
time-independent.
E.9
Continuous Flow Matching
For completeness we provide the formulas for denoiser (z-prediction) and noise-prediction (e- prediction) parameterizations of the generating velocity field [0, x RN - RV appearing
T
In Continuous Flow Matching one can chose several ways to define the probability paths (Lipman
al. 2022 et 2022, Albergo and Vanden-Fijnden, 2022 Pooladian al. 2023 Tong et
2023
px(x)= =
pe x 11) (zo, x ddxodæ
(54)
(11()))119(()) )dz
(55)
Poje (z)
(56)
Denoiser parameterization.
The conditional generating velocity field (x(x/r)) for (=(())))) i.c.
satisfy the Continuity Equation T6 takes the form (Lipman al. 2022)
R4
(I1
x),
(57)
1
and the marginal generating velocity field is therefore given by the marginalization with the posterior (2())
/
Re (x) xx) P1jo (alo(v/l)))g( (g(x1) dæ Pa(x))
(x))
1 Fit
hr
Ke [21]((x) -x]
where
(11(2//219((11) dr = Eximp((la) (x)
i (1)
(58)
This shows the continuous Flow Matching denoiser parameterization of the generating velocity field Table
Noise-prediction parameterization. the form
The conditional generating velocity field for (x)22) takes
Ue (x)(20)) = Ry (xr - Io);
(59)

---

and the marginal generating velocity field in this case is given by marginalization with the posterior (z(o))))
Ri
Poje Poje(2/<ro)p(or) dro
(x)
(x: xo)
Kit
pc(x)
Re x - @ope(2))
where
Poje Poje(z|2o)p((ng) dæ = (xorpe( (x(
(0)((x)
20
(60)
This shows the continuous Flow Matching noise- pesseppreiittion parameterization of the generating velocity field in Table
E.10 Scheduler change formula
Proposition 8, Assume a conditional probability path as in equation 9 then for any two schedulers Kit; Kly and w. (x))),) (d")aa their corresponding posteriors in equation 23
= =0'(2')2),
(61)
where t' = kzd and K 1 is the inverse of k.
Proof (Proposition g) For conditional probability path as in equation 9
Pe (1)) =
(z (20.
(62)
1=1
[1 Ke )50 (x²) + Kvda (xt)
(63)
NN
II [1 - x²)02o(z²) + K100a (a¹)
(64)
N
(1 2x),
(65)
1=1
(= x1),
(66)
where in the 3rd equality we used By nr. Thus, also for the marginal probability path as in equation 7
(e(x)=
Σ
pe (1
(67)
ED
Σ
21)
(68)
(x),
(69)
25

---

where in the 2nd equality we used pr (r æ1) = -/( a posterior as defined in equation 23
Finally the change of scheduler for
We (x)1=) = Σ
(0 xi)pe (zo, x1|2)
(70)
20.01ED
12 21) Pe pr (z|20, on )
Σ
w<x2|20
(71)
Pe (2)
20: x1) pl((2) (2 3 (F0,
Σ
w(x0
(72)
pb(2)
Σ
(m(x1100 21
(2o.
(73)
S0.01ED
= (x)12)
(74)
where in the 3rd equality we used both pe (2| xI) = p1(2/20 x1) and pr = pl(z).
L
F Inference time
One potential benefit of non-autoregressive decoding is improved latency due to a significantly lower number of decoding steps. To demonstrate that, measure the average latency of the proposed method compared with the autoregressive alternative using a single A100 GPU with 80 GB of RAM. We calculate the average latency time the HumanEval benchmark using batch size of When considering 256 NFEs, the proposed method was found to be ~2.5x faster than the autoregressive model (19.97 vs. 50.94 seconds on average per example). However, when considering 512 NFEs, both methods reach roughly the same latency. These results make sense as the number of tokens in most of the examples HumanEval are below 512. Notice, that these results analyze latency and not model throughput. Due to the kv-caching mechanism following the autoregressive approach will result in significantly better throughput compared to the proposed approach Ziv et al. (2024). We leave the construction of kv-cache mechanism to the proposed approach for future research.
G Experimental setup
G.1 Text
Data. We use three splits of data. First is OpenWebTer: (Gokaslan and Cohen 2019). Second is the same mix used in Llama-2 (Touvron et al.] (023),, including textual and code data. For the code-focused models we use the same split used CodeLlama (Roziere al., 2023). For the small models, we use Open Text. For the big models we use the 1lama-2 and Codel.lama mixes.
Models. We train two sizes of models: small (150M parameters) and large (1.7B parameters). For the small model we used a DiT transformer architecture (Peebles and Xic 2022) with 12 layers, 12 attention heads, and hidden dimension of 768. We also used GPT2 tokenizer. The small models were trained on Open WebText. For the large model, we use also used a DiT transformer architecture but with 48 layers, 24 attention heads, and hidden dimension of 1536 (Peebles and Xie 2022). For these models we used a tiktoken tokenizer. The large models were trained on the Llama-2 2 mix and the CodeLlama mix. For both models we used ROPE (Su et al.) 2024 embedding with 0 = 10000. Models are trained with Adam optimizer with 3 = 0.9 and B2 0.999. We use dropout rate of 0. Models are trained with warm-up of 2500 steps, with a peak learning rate of 3e-4. We train the big models with batch size of 4096 for 1.3 million iterations and the big models with batch size of 512 for 400 thousand iterations.
Entropy metric. We report the entropy tokens within a sequence, averaged over all generated sequences. This intuitively quantifies the diversity of tokens within a given sequence. It's important note that when computing sequence entropy, tokens not present in the sequence are excluded from consideration.
26

---

Generative perplexity metric. The generative perplexity metric is the average likelihood of generated text evaluated with second (usually stronger) model. We report the generative perplexity when averaged over 1000 samples.
Double precision sampling. Zheng et al. (2024) demonstrated that sampling from a high- dimensional distribution with full precision lead similar affect as sampling with temperature. We evaluate our model using a categorical sampler in double precision. T presents the results of baselines compared our method.
Table 5: Double precision sampling. Generative perplexity on unconditional text generation compared to prior work. All models are sampled without the use of temperature corrector steps.
МATHOD
NFE
LLAMA-2
LLA
GPT2,
ENTROPY
7.0
14.7
Autoregressive
1024
31.4
54.8
45.3
7.1
Savinov et al. (202)
200
29.5
45.1
34.7
Austin ct (2021a
1000
697.6
768.8
837.8
Han et
2022
>10000
73.3
203.1
99.2
Lou et
2023
256/512/102
56.6/54.0/56.1
122.1/115.7/117.7
115.0/ 15..0/107.8/LU0 109.5
8.1/8.1/8.1
Campbell et al. (2024
256/512/108 1024
52.0/54.6/50.9
102.6/ 107.1/ 103.4
8.0/8.1/8.0
FM (equation] 9
256/512/1024
51.3/53.3/50.1
104.0/115.0/101.
100.8/107 4/97.5
8.0/8.1/8.0
FM (equation 10
256/512/1024
51.9/52.7/50.0
104.7/113.9/10055
99.2/105.1/95.8
8.0/8.1/8.0
G.2
Image
Models. For all our experiments on CIFARIO we use the U-Net architecture as in Dhariwal and Nichol (202I), with following three changes make fully discrete and time independent (as we used mask modeling): (i) We replace the first layer with an embedding table of size 257 x 96, and we stack the channel features such that the input the U-Net is of shape 288 32 x 32. (ii) We enlarge the size of the final layer to output tensor of shape 3 x 32 x 32 x 257. (iii) We remove the time dependency from architecture. The hyper-parameted of the architecture: channels 96, depth 5, channels multiple [3,4.4], heads channels 64, attention resolution 16, dropout 0.4, which gives total parameters count of 113M. We optimize the network using Adam optimizer with B1 = 0.9 and 32 = 0.999, learning rate of le-4. We trained with effective batch size pf 512 for roughly 300K iterations.
Scheduler ablation. Figure 8 shows FID of our method with four different schedulers: Linear, Quadratic, Cubic, Cosine. both for training and evaluation. That is, for each scheduler we trained model and evaluate FID with all four schedulers. We observe a high variance in FID between different schedulers, with the Cubic scheduler generally performing the best on both training and evaluation.
Comparison with baselines. In the following, provide implementation details for produc- ing Figure 3a that compares our schedulers and sampling algorithm with those employed by previous works.
Cubic Scheduler (Ours). For the Cubic scheduler we set the corrector scheduler above to,
a 1 at - t)b,
3 a
(75)
and search over the parameters a, E {0, 0.25, 0.5, 2,2, and {6,8, 10, 12, 14). Additionally, we search over the temperature [1, 0.8}. We find that a 2, 0.25, a = 12 give best FID.
Linear Scheduler (Campbell et al. 2024). For the linear scheduler we search over two additional hyper-parameters the method: (i) For corrector scheduler as in equation 26 we set a: + ln)
3 = - where is the stochasticity parameter as in Campbell et al. (2024), and search over 7) E 1,2,, 10, 15). (ii) We search over temperature in (1,0.9, 0.8). Finally, we find that the best
FID is achieved by 7 10 and temperature 0.9.
27

---

Linear
9.75
7.02
6.93
7.12
Quadratic
8.03
5.72
5.72
5.80
Score
Train
7.37
5.04
5.07
5.15
Cosine
7.79
5.29
5.14
5.31
Cubic
Eval
Figure 8: Comparison of FID on CIFARIO with four schedulers: Linear, Quadratic, Cubic, Cosine, for both train and evaluation. Corrector sampling not used in this experiment (i.e., dg = 0 in equation 26 and temperature is set to 1.
MaskGIT (Chang et al., 2022). For the MaskGIT we train and sample with the Cosine scheduler
ktt) = cos t) which reported achieve best results by Chang et al. (2022). For sampling we adjust the code from the re- e--mmllementation of Besnier and Chen (2023). In addition, we also search over the temperature in (1, 0.9, 0.8, 0.7, 0.6, 0.5}, and we find the best FID is achieved by temperature 1.
28

---

H
Code generation - qualitative examples
H.1
Success cases
from typing import List
def @elow_zero (operations: List [int]) -> bool:
You're given list of deposit and we vithraaaal operations
bank account that starts with zero balance. Your task to detect if at any point balance of account fallls below zero, and at that point function should return True. Otherwise it should return False. (elow_zerr((i) 2, 3])
False
>>> below, ([1, 2, -4, 5])
True
balance
for operation in operations:
if balance operation 0:
return True
balance balance operation return False
def encode_shift(e str):
returns encoded string by shifting every character by the alphabet.
return "..join(cchr(((rr)) 5 ord("a")) % 26) ord("a")) for in s])
def Sacode_shiftie str): ann
takes as input string encoded with sncode_shift function. Returns decoded string.
decoded 0
for ch in s: decoded. append ( (((ord(ch) ord("a")) 26) ord("a")))
return "", (ddecoodd)
29

---

from typing import List, Any
def filter integers (values: (vales:
List [Any]) >> List [int]:
Filter given list of any python values only for integers >>> Silter_integera(C [5]
3.14, 5))
>>> filter_ inter_integers(L. 2, 3, 'abc', O, (II)
[1, 2, 3]
return [value for value in values if isinstance (value, int)]
def common (11: list, list):
"""Return sorted unique common el elements for two lists.
>>> common([1, 4, 3, 34, 653, 2, 5], [5, 7, 1, 5, 9, 653, 121))
[1, 5, 653]
>>> common ([5, 3, 2, 81. [3, 21)
[2, 3]
return sorted(set(11 (11) set (12))
def fibfib(n: int):
"""The FibFib number sequence is sequence similar to the Pibbonaces sequnece that 's defined follows: fibféb(0)
f(bffb(1)
ftbfeb(2) =-
fibfeb(n) == fébffb(n-1) fabfab(n-2) fabfib (n-3).
Please write function efficiently compute the n-th element of the fibfib number sequence.
>>> fibfib(1)
>>> fobffb((5
>>> fobfsb(8)
24
if n == 0:
return 0
elif == 1:
return 0
elif n 2:
return 1
else:
return fibfib(n-1) fibfii(n-2) fibfib(n-3)
30

---

H.2 Failure cases
def solve(s):
"""You given string
if s(t] letter, reverse its case from lower upper versa, otherwise keep it as it is.
If the string contains no letters, reverse the string. The function should return the resulted sträng. Emamples
solve ("1234") "4321"
solve ("ab" ") "AB"
solve ( (*#a8C") "#A8c"
if 8 is None: return if len(s) -- 0:
return s
result string
res s)r()
to lower case
s.lower ()
for range (1en
ch = s[i] if ch. islooee))))
res. append
ch. isupper()
res. append (owweerr)) ())
else:
res append
from typing import List, Tuple
def rolling max (numbers: List[int]) -> List[int]: u*. Prom a given list of integers, generate list of rolling mazémun element found until given moment in the sequence.
>>> rolling. mam ([1, 2, 3, 2, 3, 4, 2])
[1, 2, 3, 3, 3, 4, 43
max_ ___mmmeerr 0
result
for number in numbers:
max _ex_numberr>.. max (result)
if number max max_number: result.c result.append (number)
else:
result [maxxnumber]]
return result
31

---

def truncate_ number (number: float) -> float:
cuncatee_umberreee
unn Given positive floating point number, at be decomposed into and integer part (largest integer smaller than given oumber) and decimals (leftover part always amaller than 1).
Return the decimal part of the number. >>> pruncate_number(3.. 0.5
return math. floor (number)
from typing import List, Tuple
def sum_product (numbers: List[int]) Tuplelint, int]:
ann For a given list of integers, return a tuple consisting of a sum and
product of all the integers in list. Empty sum should be equal to 0 and empty product should be equal to 1.
>>> sum.product(())
(0, 1)
>>> sum cum_prooduc((I) ([1, 2, 3, 41)
(10, 24)
if len (numbers) == or numbers [o] == 0: return sum product
for num in numbers:
if abs (num) 0:
product product num sum num return sum, product
from typing import List
def factorize (n: int) -> List[int]:
ann Return list of prime factors of given integer in the order from smallest factors should listed number of times corresponding
largest. Each of to how many times it appeares in factorization. Input number should be equal
to the product of all factors.
>>> factorize(8)
12, 2, 2]
>>> factorize(25)
[6, 5]
>>> factorize (70)
[2, 5, 7)
ann
factors []
factor 1
for in range(2, n): factor *=
while factor == 0: factors append (factor)) factor +=
return factors
32

---

from typing import List, Optional
def longest (string:: List[str]) -> Optional [str]:
Out of list strings, return the longest one. Return the first one in case of multiple strings of the length. Return Jone in the input list is empty.
>>> longest([])
>>> (('''st(''a', 'b', 'c'))
'a'
>>> longest(('a', 'bb', 'eee '])
'eee'
long
for string in strings:
if len(string) > len(non));
long string
return long
33

---

H.3 Infilling
(arr)
n lan(arr)
optimize code, so if the array is already sorted, it doesn 't need to go through the entire process
Traverse through all array elements for i in range(n-1):
range(n) also work but outer loop will repeat one time more than needed.
Last elements are already in place
swapped False
for j in range(0, n-i-1):
traverse the array from to n-i-1 Swap if element found is greater than the nent element
if arr [j] arr [j + 1]:
swapped True
ajr[j], arr + 1] arr 1], arr
if not swapped:
# if we haven't 't needed to make single swap, we can just exit the main loop. return
Function to perform Breadth First Search on graph represented using adjacency list
def bfa (adjLisst,, source visited):
Create queue for BFS q d)que()
Hark the current node visited and enqueue it visited[ source ] True
q aappend( source
Iterate over the queue while q:
Dequeue a vertew from queue and print it
currentNode q popleft() print ( currentNode. end=" ")
Get all adjacent vertices of the dequeued vertez If an adjacent has not been visited, then mark visited and enqueue for adjacent in adjList[ currentNodd ]:
if not visited[ adjacent ]
visited adjacent True q. append adjacent )
34

---

Returns indem of in if present, else def binary low, high x):
Check base case if high >" low
mid ( (igh low) ) // 2
If element is present at the middle itself if arr mid ]
mid
If element is smaller than mid, then it can only be present in left subarray
elif arr mid]
return
low mid 1, x)
Else the element can only be present in right subarray
else:
return binary,
high x)
else:
Element is not present in the array
return
35

---

Python program for Digkstra's single source shortest path algoritha. The program is for adjacency matriz representation of the graph class Graph():
..init__. vertices):
self V = vertices
self. graph [[O for column in range (vertices)]
for
range (vertices)]
def printsolution dist): print print("Vertes: ("Vertex for node in range(self.W).
Distance from Source")
print (node,
dist [node]]
utility function to find the vertez with monimum distance value, from the set of vertices not yet included in shortest path tree def (inDis dist, sptSet):
Instialize minimum distance for nezt mode min 1e7
Search mearest vertez not in the shortest path tree
for in range(self. v): if dist [v] min and sptSet [v] == False: dist[v]
min_index
return min_index
Function implements Dijkstra's single source shortest path algorithm for a graph represented using adjacency matria representation def dijkstra(selff, src):
dist [1e7] self.V
dist [src] 0
processed [False] self.V
for cout in range
Pick the minimum distance vertez from
the set of vertices not yet processed. u is always equal to sre in first iteration
uv = self (diiis, processed)
Put the minimum distance vertez the shortest path tree processed uv True
Update distance value of the adjacent vertices # of the picked vertez only if the current distance is greater than distance and the shortest path tree for v in (aage(selff vV)
the vertez in
if (self. graph [ uv ][v] > and processed [ uv ] == False and dist [uv dist [ cout ] self graph[ uv ][v :); dist [uv dist cout self self.graphl uv ][v] self, printSollution(distt))
36

---

I Textual generations
We present below example generations for the proposed method together with several baseline methods. We provide both conditional and unconditional generations. For the conditional generations, we mark the prompt in gray.
1.1 Conditional generation
The United States on Wednesday asked the UN Security Council to slap an oil embargo on North Korea and freeze the assets of leader Kim Jong-un, in response Pyongyang's
response to the revelations it had restarted its nuclear work in March. "We will continue working to use maximum international pressure on North Korea agree the suspension of its nuclear program and reinstate sanctions," said John Bolton, who served as National Security Advisor and Secretary of State under US President Bill Clinton. "Here is North Korea's ' response to our sanctions," Bolton wrote in a letter to House Minority Leader Nancy Pelosi. "We want you to know that the international community is seriously monitoring North Korea at this time. North Korea is still complying with our requests from the past few days." " Bolton said Monday. "We have been working through the United Nations to providc the information that they gave us." Asked whether any international pressure will be put in place for North Korea to give up its nuclcar weapons, Bolton said the United States can use maximum pressure to get North Korca to abandon its nuclcar weapons if it wants. "We've 'We've been working to use maximum pressure North Korea, through the Security Council, and we will continue to do so," " said White House Deputy Press Secretary Sarah Huckabee Sanders in Washington. "We're re committed taking any steps necessary to help North Korea pursue its only option for peace, including this period," she added. The United States did not plan to produce any more oil at this time last year and had not planned to do so this year. "We believe that the North Korea approach is misguided in moving forward with its nuclear program endanger peace and security in its homeland and neighbors in Asia," said Bolton, adding that the US supplies its nuclear weapons. "We don't 't want them sell their nuclear weapons other nations," he said. Bolton said the US would look for pressure on North Korea, which has been known to use nuclear weapons, as leverage to negotiations with the US. "I will reiterate what have said before. So, the US has to put pressure on North Korea. But who else is going to hold the cards? Somebody else has to hold the cards," Mr Bolton said. Bolton described what the United States is prepared to do get North Korea to agree to give up its weapons and asks for sanctions. "As far know. we have to use the pressure the reason for putting sanctions on North Korea," he said, adding that the US does not plan to ask the UN Security Council for sanctions alone.
The defender is available for the Maribor first leg but his club believed he should be suspended. SNS Group Celtic made an administrative blunder which saw Efe Ambrose left behind
in the midfield in the Maribor department and has given him a potential three- three-match ban today. Although Efe Ambrose will be suspended next Friday, according to reports in Scottish media, the Celtic defender will still be fit for the Champions League first leg at Celtic Stadium in the middle of August. However, the Celtic club wrongly thought that Efe should only receive threc-match ban hecause he is available for the first leg. Although Ele Ambrose may receive a three-match ban next Friday. Efe Ambrose was part of the Celtic squad for the last match against Liverpool. However, says SNS Group Celtic he was making a tactical error and was left behind in midfield. It is understood that Efe Ambrose did not make the final squad and only played 11 games for the club this scason. Efe Ambrose made his professional debut for Celtic in 2008 but spent nine months looking for new club return to. With a career-high high 72 Celtic appearances, Efe is among Celtic's 's most capped players ever.
37

---

Carl Jara aka Grain Damaged is an award-winning professional sand sculptor from Cleveland, Ohio. Jara says he has known since high- school that he wanted to be artist. After studying
English and Art History at the Northeastern University. Jara says one semester he started carving custom sculpture into sand molds, but didn't 't know how do it. With the help of an instructor, he found out and learned how to use rubber molds to make art. Later, he made the decision learn how to use sand addition to how he makes his own sculptures, Jara says he does special events for comics companies such as Stan Lee and also for institutions like local community colleges and universities. In November of this year, he won the WWHS, The Very Art Of The Very Things Cleveland competition. Afterward, he will continue carving for clients in the comics industry and looks forward sand sculpting in Cleveland. The Artist is professional sculptor who has been making art, for over 25 years, various shapes and sizes. The artist says art all about relationships and the best way to go into the heart is to create art. The artist has taught in various high schools in the Cleveland area and has taught full time Honors Studio for High School students in grades 6, 7, and 8 time for over 20 years. Since Art a personal form of artistic expression, he works individually allows the student that work engages their imagination and presents ideas ways that inform challenge their own paths. Miguel Romano is a professional artist who worked in 3D modeling and animation the areas of weh design and production. The artist currently works as a digital artist in the ad and mass communication industries. In coming to Concrete Cleveland, he is excited to apply the 3D development and production skills he have his work. The artist has BFA in sculpture from Ohio University. along with MFA in sculpture. We look forward to seeing his work very soon! Designed and installed by Concrete This Week. He is guy originally from Cleveland, Ohio where he pursued a career as a nurse. He then moved to the Atlanta, GA area where he returned to school with a BSN and a BS in nursing and is a licensed nurse. Hc is a proud sorority brother and still has extraccurricular, well taking music lessons and the occasional play. Hc is lovely asset Concrete Cleveland and looks forward sceing concrete
sculpt himself. In
way that
38

---

1.2 Unconditional generation
Here's how that strategy works for your job:
1) You now plan upon what you accomplish to fulfill your goals.
Management cannot plan what happens to you. This may not be your ultimate personal decision, but it's perfectly fine to look it. You just need to make sure you want to achieve this. Now, because you' 'e worked at goal. you don't have to talk about your status tomorrow, after all. you have to do your job and take care of yourself.
Next steps, there may be some work to do. There is a company down the road you right literally millions of things that would have to be done. But of course it would have different outlook. If you're 're going do something, the customer might not be able to tell you.
2) Between those two steps are the plan in step so that your actions will be executed.
Then you have taken other steps (usually few less important changes), like delivery. If you already know what that means, and you're re having stay up and take action you can make sure you don't 't have to point out in the moment to plan the action.
With business goals, it is not easy to pick up what appears best for us. We have to see what really is. What we do. We can't It make a plan on the floor and come back up with exactly what you're doing. If you want to work every step, then you need to diffcrentiate from the action and what the next step represents.
Eventually, you'll be less motivated to focus this step and the previous onc. Unfortunately if you don't 't change your main thing, you may be able to lose your motivation work on "pivot." Unless that's possible, and you don't I change something, then the task may not at the right time. Instead of doing something, is just in advance of your ultimate goal.
Although you might make mistake with every single day to day plan, it still is great opportunity to correct your mistake, become new and commit to working extra hours and meeting your goals promptly. The truth is, everything goes right for you no matter how quick a decision you become. The customer will never allow you to make the worst decisions. Otherwise, you make the very first decision.
3) Take timing as part of action. If you don't 't feel like you can keep it, a plan without help of timing stops you from doing. When it's like your plan in action can lead to something such as this: Now that you know what to do. For example, you might live in place in the building that serves every customer, has 3 employces per team, and 3 clients on one. You will get things done the next day. Change your performance the first step towards greater success, for example. Your team, at this point in the Customer department. will know how the customer deal with single employee, the level chain, and more. Make sure you take action now that you change it. As company, it won't t be hard but you will have lots of work to do when you change. Make sure it's your night.
Watch the humour but also the humanity hehind the work we' 'e doing. The truth is something very tragic and delicate in the middle of very fractured world. It's the one thing that makes me proud. I feel like singular individual has had to come together with this story. There's lot of people who I've worked with for the very beginning, because have got people, you't 've got people who have just had these eyes on this story, and this sense of what we are, that run through our final movie.
It's the very beginning we're re at. The very beginning, we're not there, we will get there but we won't 't need. This is story and these amazing actors, these fantastic violence, violence, that was just are elements and complex world of conflict. When something like that set build this narrative and you're 'e're directing the world these characters based around their individual needs it's very, extremely confusing, very heartbreaking quite intense-this was all built within it, and what it is, it's a 35-year old period that was slavery and still was very strong, these guys were operating to the edge and going to the point where ended up setting up big narrative, OK, that's good, it's okay, in some ways heroism is noble imperative that we are fighting against, and recognize maturity as the mercurial nature and these are all human and we've got to clean it up so that that stuff is there and we've ve got to restore it. And the project we're re looking at here is our common goal is that anything can be done to make that happen and everybody can do whatever their want to do and do it as they please. That's 's the spirit of it. That's 's the movie I've made with Steven Wright in writing.
it's really
39

---

NeurIPS Paper Checklist
Claims
Do the main claims made in the abstract and introduction accurately reflect paper's contributions and scope?
Question:
Answer: [Yes]
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No] As each experiment is highly resource intensive we did report error bars. However, we did report results on a wide range of tasks and setups.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [No]
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes] See Discussion section.
11. Safeguards
40

---

Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
41


Reviews:
[{"review_id": "KDNKkMQFZy", "reviewer": "Reviewer_oYBb", "summary": "The paper presents a discrete flow matching method for modeling discrete data with discrete state space. The paper presents unified frameworks for training and sampling from the discrete probabilistic model. Importantly, the paper also studied scaling up the model to 1.7B parameters and tested the model on code generation tasks. Comprehensive experiments shows that the proposed method outperform existing methods and also close the gap to autoregressive models.", "questions": "See weakness.", "limitations": "I didn't find limitations specifically discussed in the paper, though the author marked it discussed in the checklist. Please correct me if I missed it.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Impressive empirical studies with 1.8B scaling up model. To my knowledge, this is the first discrete diffusion that scaled up to this scale.\n\n2. The paper provides a principled view of the discrete diffusion model, and proposes several novel techniques such as backward sampling in conditional generation scenarios.\n\n3. The paper is very clearly written and well organized.", "weaknesses": "The paper lacks some experiments on common benchmarks used in existing discrete diffusion models, e.g., LM1B and OWT. I understand the scaling-up experiments on HumanEval are more challenging, but I believe a comparison on the common benchmark would be important to justify the effectiveness over existing methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Comment: The paper lacks some experiments on common benchmarks used in existing discrete diffusion models, e.g., LM1B and OWT.** First, please note that our experimental setup already includes the OWT dataset (see Table 2 and lines 268-273). Second, per the reviewer’s suggestion, we trained and evaluated our and baselines models on the LM1B dataset, please see Table 1 in the main rebuttal reply. \n\n_Experimental setup._ For the reviewer’s convenience, we would like to detail all the evaluations done in this work. We experiment with two data modalities, text and images on small and large scale datasets.\n\n| Modality | Scale | Datasets | Compared to | Metrics | Comments |\n|---|---|---|---|---|---|\n| Text | Small | Open web text (OWT) | State-of-the-art prior works, autoregressive modeling | Generative perplexity (using Llama 2, 3, and GPT2), Entropy, NFE | All models are evaluated without temperature annealing.  |\n| Image | Small | CIFAR10 | Campbel et al., Maskgit | FID, added inception score in this rebuttal, NFE |  |\n| Code | Large | Large scale code mix | Autoregressive (note: no other discrete diffusion/flow worked on this tasks before) | Pass@1, Pass@10, Pass@25 on Humaneval and MBPP |  |\n| Text | Large | Large scale text mix | Autoregressive, Savinov et al. | Generative perplexity (using Llama 2, 3, and GPT2), Entropy, NFE |  |\n\n**Comment: I didn't find limitations specifically discussed in the paper.** In the conclusions (Section 5), we mention the following limitations: (1) Discrete flow matching requires a higher number of function evaluations compared to its (deterministic) continuous counterpart. We attribute this to its stochastic sampling, similar to sampling by approximating a solution of an SDE which typically possesses a lower convergence order than their ODE solvers counterparts (see e.g., [1]). (2) There remains a performance gap between autoregressive modeling and our proposed approach.\n\n[1] Sauer, T., 2011. Numerical solution of stochastic differential equations in finance. In Handbook of computational finance (pp. 529-550). Berlin, Heidelberg: Springer Berlin Heidelberg.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "J69mVJc2i5", "reviewer": "Reviewer_ppDj", "summary": "The paper introduces a novel approach called Discrete Flow Matching, which adapts continuous flow models to discrete sequential data. It extends prior work by integrating discrete state spaces and time-dependent schedulers into a unified framework for non-autoregressive generative modeling. Methodologically, Discrete FM employs generating probability velocities derived from learned posteriors and schedulers, enabling efficient sampling and correction processes. This approach facilitates the transformation of noise distributions into target data distributions with enhanced flexibility and performance. Experimental evaluations across language modeling, code generation, and image generation tasks demonstrate significant improvements over existing methods. Specifically, the model achieves state-of-the-art results on various benchmarks, including HumanEval and MBPP coding tasks, showcasing its efficacy in generating high-quality outputs.", "questions": "- I am a bit confused with Eq 10 especially the scheduler terms. Please explain or tell if there are mistakes.\n\n- What are the main reasons for discrete flow matching to require significantly large number of evaluations?", "limitations": "The authors acknowledged their limitations such as the number of evaluation is high compared to continuous flow matching.", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The methodology design is solid with formal analysis and mathematical proofs. \n\n- The paper conducted a set of experiments in language modeling, code generation, and image generation tasks, showcasing the most promising results to date in a non-autoregressive context.\n\n- Technical details about the methodologies are rich. And the experimental setup includes detailed descriptions of the methodologies used, such as masked source training, conditional couplings, probability path schedulers, and corrector steps, which are pivotal for the model's performance.", "weaknesses": "- The method is heavily based on the Continuous-Time discrete Markov Chain (CTMC) paradigm from Campbell et al., although this paper proposed theoretical and empirical improvements such as the unified formulation for more general probabilistic paths and velocity as well as the scheduler designs.\n\n- There are still performance gap in code generation evaluation when compared with autoregressive language models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Comment: I am a bit confused with Eq 10 especially the scheduler terms.** Equation 10 proposes a second  instantiation of a conditional probability path $p_t(\\cdot|x_0,x_1)$ where given some pair $(x^i_0,x^i_1)$ of source and target tokens, the token at time $t$ is: $x^i_1$ with probability $\\kappa^1_t$; $x^i_0$ with probability $\\kappa^3_t$; and a uniformly distributed random token with probability $\\kappa^2_t$. This probability path is reminiscent of the brownian bridge in the continuous diffusion world. Lastly, note that the conditions on the schedulers $\\kappa^j_t$ that guarantee this conditional path interpolates between $\\delta_{x_0}(x^i)$ and $\\delta_{x_1}(x^i)$  and stays a proper PMF for all $t\\in[0,1]$ are the ones described in lines 98 and 91. We will clarify this in the revised version.\n\n\n**Question: What are the main reasons for discrete flow matching to require a significantly large number of evaluations?** Discrete flow matching requires a higher number of function evaluations compared to its (deterministic) continuous counterpart. We attribute this to its stochastic sampling, similar to sampling by approximating a solution of an SDE which typically possesses a lower strong convergence order than their ODE solvers counterparts. In particular, for the Euler sampling case the deterministic Euler method has strong convergence order of $1$ while the non-deterministic Euler (Euler-Maruyama) has strong convergence order of $\\frac{1}{2}$, see e.g., [1]. While this gives some intuition we do agree an analysis of global convergence of discrete sampling is interesting and defer it to future work; in this paper we only show local convergence error (see $o(h)=O(h^2)$ term in equation (22)). \n\n[1] Sauer, T., 2011. Numerical solution of stochastic differential equations in finance. In Handbook of computational finance (pp. 529-550). Berlin, Heidelberg: Springer Berlin Heidelberg.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WvCX8gpsbU", "reviewer": "Reviewer_vMd5", "summary": "The paper proposes an approach to generative modelling for discrete data, i.e. multidimensional distributions where the variable along every dimension can take value in a finite set. This is an alternative approach to autoregressive generative modelling for discrete data which is currently actively studied for language and code generation.\n\nThe philosophy of the proposed method is heavily inspired by the Flow Matching algorithm [1] and Continuous Time Markov Chains (CTMC), which were previously used in [2] to propose a similar model. That is, for a given CTMC, the authors define the vector field generating samples from this CTMC by local updates of the samples (independently along every dimension). Based on the PMF (Probability Mass Function) of the CTMC, the authors derive a formula for the vector field. Furthermore, they introduce the continuity equation analogous to the continuous case, which allows for an easy validation that the change of the density given by CTMC corresponds to the vector field.\n\nThe authors perform an ablation study of some of their design choices and extensive empirical studies for generation of code, language, and discrete-valued images. The proposed model outperforms the competitors bridging the gap between flow-based models and autoregressive models.\n\n[1] Lipman, Yaron, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. \"Flow matching for generative modeling.\" *arXiv preprint arXiv:2210.02747* (2022).\n\n[2] Campbell, Andrew, Jason Yim, Regina Barzilay, Tom Rainforth, and Tommi Jaakkola. \"Generative flows on discrete state-spaces: Enabling multimodal flows with applications to protein co-design.\" arXiv preprint arXiv:2402.04997 (2024).", "questions": "I would suggest adding a discussion of the variable-length generation. It is an important difference with autoregressive modelling and I’m wondering how the authors handle this issue given that their model has two important properties:\n\n1. Conditional generation for partially masked sequences.\n2. Independence of the conditional distributions between dimensions.\n\nThe questions I would like to have answered are:\n\n1. Does one have to define the length of the generated sequence before generation? If so how this can be decided?\n2. What’s the computational cost if one continues generating the sequence in the autoregressive way?", "limitations": "NeurIPS Paper Checklist does not follow the required format.\n\nThe paper adequately discusses the limitations of the proposed approach. The only thing that is not cover enough in the paper is the variable-length generation.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The paper presents a complete study of the actively studied topic in the field. It is clearly written, the developments presented in the paper are novel and are properly studied empirically. The paper is of great interest to the NeurIPS community.", "weaknesses": "The paper presents a complete study and its methodological part does not raise any major concerns. However, I would like to ask the authors to clarify the following question.\n\nWhy do posteriors in eq. 15 define a proper distribution? Indeed, according to eq. 8, the linear combination of the conditional distributions with the schedulers should define a correct distribution (sum up to $1$). If we simply try to sum over all possible $x^i$ in eq. 15, we won’t get $1$, hence eq. 15 does not define a posterior. This is an important detail when defining the objective in eq. 25, which is the cross entropy between distributions.\n\nThe minor\n\n- Descriptions of the schedulers on lines 96 and 98 are not complete, i.e. there are no conditions that schedulers have to define correct probability distributions in eqs. (9,10) and the scheduler $\\kappa^3$ is missing a description.\n- Figure 2. The plots presented in the figure have $d = 2$ unlike what is stated in the caption.\n- At the beginning of the paper, the authors introduce the notation of random variables as capital letters (e.g. $X_t$). This creates confusion in equations 13-17 because there the authors clearly mean $X_t$ to be the value of a random variable.\n- There is a typo in line 160 when describing the panel of Fig. 2.\n- There is a typo in line 163 when defining the vector field $v(x,z)$.\n- There is a type in line 582. Index $\\ell$ is not a function of $j$ according to eq. (36).\n- I haven’t checked thoroughly, but I think equations on the top of page 19 should have a summation over $j\\neq \\ell$ instead of the summation over all possible $j$.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Question: Why do posteriors in Equation 15 define a proper distribution?** Equation 15 is a proper distribution as follows: \n$$\\sum_{x^i} \\hat{w}^j_t(x^i|X_t) = \\sum_{x_0,x_1} \\overbrace{\\big( \\sum_{x^i} w^j(x^i|x_0,x_1)\\big )}^{=1} p_t(x_0,x_1|X_t) = \\sum_{x_0,x_1} p_t(x_0,x_1|X_t) = 1,$$\nwhere in the first equality we change the summation order. \n\n**Comment: Descriptions of the schedulers on lines 96 and 98 are not complete.** The conditions in lines 96 and 98 should be understood as _additional conditions_ to the general conditions presented in  Line 91, i.e., $\\sum_{j} \\kappa_{t}^{i,j}=1$ and $\\kappa_{t}^{i,j} \\ge 0$. Combining the conditions in Line 91 with the ones in Lines 96 and 98 guarantees proper distributions. We realize now this is confusing and will clarify this in the revised version of the paper - thank you. \n\n**Comment: Figure 2. The plots presented in the figure have $d=2$ unlike what is stated in the caption.** This is indeed a typo, but should be $d=4$ (and $N=2$), that is, the figure depicts the state space of two tokens $x=(x^1,x^2)$ where each token can get a value in a vocabulary of size $d=4$.\n\n**Comment: Random variable notation creates confusion.** We agree with the reviewer that this notation is confusing and we will use a lower case letter, e.g., $z$, instead in the revised paper. \n\n**Comments: Typos Lines 160, 163, and 582.** Thanks, we will fix it in the revised version.\n\n**Comment: Summation on page 19.** Please note that in the case of $j=\\ell$ the term in the summation equals zero, thus, summation over $j \\ne \\ell$ will yield the same result.\n\n**Question: Does one have to define the length of the generated sequence before generation? If so how this can be decided?** No, the length does not have to be defined before generation. We train the model using flattened data samples which are separated by an end-of-text (EOT) token. The model can predict EOT token in any location in the generated sequence $x^1,x^2,\\ldots,x^N$ and the EOT will indicate the end of the generated text (similar to autoregressive modeling). \n\n**Question: What is the computational cost if one continues generating the sequence in the autoregressive way?** The model generates a sequence of length $\\leq N$, where $<N$ length can happen if the EOT token is predicted. If it does not (some texts are longer than $N$ tokens) one can potentially continue to generate by conditioning on the last $K<N$ tokens and predict the next $N-K$ tokens, where $K$ is user defined: e.g., large $K$ will provide more context but shorter extension. The cost of predicting these $N-K$ new tokens will be equivalent to a full sequence generation with our model. It is very interesting to develop methods to accelerate the generation, but we leave it to future research.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aPqpGt66UG", "reviewer": "Reviewer_7VEG", "summary": "This paper presents Discrete Flow Matching, a new method for generating discrete data, such as language. The approach uses a general family of probability paths between source and target distributions. It offers a formula for sampling from these paths using learned posteriors. By focusing on specific probability paths, it improves generative perplexity compared to previous similar models. When scaled up, the method achieves notable performance on benchmarks. This approach bridges the gap between autoregressive models and discrete flow models.", "questions": "as above", "limitations": "as above", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The method is well-motivated for solving the discrete-state data in flow matching.", "weaknesses": "- In line 32, you mention one advantage of FM is its flexibility in handling non-Gaussian target distributions. Have you demonstrated this case?\n- The method's description is unclear and confusing. I cannot distinguish which part pertains to Campbell's methods and which part is yours.\n- In Table 2, it's unclear why the results from Austin et al. [2021a] are poor, as no explanation is provided.\n- In Line 214, I'm not entirely sure what the difference is. Perhaps a table could clarify it?\n- In Figure 3, the Inception Score (IS) needs to be included to demonstrate the diversity of image generation.\n- Except the perplexity, how about the performance on BLEU, bertscore?\n- Diversity is a significant concern for flow-based models. What about the result on diversity-related metrics?\n- In F.1, in the context of conditional generation, it's unclear what the source (src) and target (tgt) samples are. Are they conditional prompts, target prompts, or a combination of Gaussian noise and target prompts?\n- In section H, there is no qualitative analysis of unconditional generation.\n- In Section H, I did not see the color red. I only observed the background colors: grey and yellow.\n- It will be great that the code can be provided.\n- Related works are missing.  it's important to note that flow matching has been utilized in various domains to capture the reader's interest. E.g., boosting diffusion[1],image generation[5], depth estimation[2], motiom[3], even text generation[4].\n\n[1]. Boosting Latent Diffusion with Flow Matching\n\n[2]. DepthFM: Fast Monocular Depth Estimation with Flow Matching\n\n[3]. Motion Flow Matching for Human Motion Synthesis and Editing\n\n[4]. Flow Matching for Conditional Text Generation in a Few Sampling Steps\n\n[5], Latent Space Editing in Transformer-based Flow Matching", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Question: In line 32, you mention one advantage of FM is its flexibility in handling non-Gaussian target distributions. Have you demonstrated this case?** Yes, in fact all the probability paths we use in this paper are non-Gaussian. We worked with mask source distribution which corresponds to a delta function concentrated on a special “mask” token. \n\n\n**Comment: The method's description is unclear and confusing. I cannot distinguish which part pertains to Campbell's methods and which part is yours.** Thank you for the comment. We build upon Campbell’s work [1] but generalize it in ways that allow significant improvement in performance. In particular, we offer the following contributions: (i) we consider arbitrary data coupling $(X_0,X_1)$ and use it for conditioning; (ii) we offer a novel family of probability paths (equation 8) that generalizes the paths used in Campbell as particular cases; (iii) in particular, we show that incorporating polynomial schedulers $\\kappa_t$ considerably improves performance; (iv) we provide a unified and closed-form formula for marginal probability velocity (rate) in equations 14, 16, 17 and show it has the exact same form as in continuous Flow Matching, see Table 1. Campbell provided this rate as an expectation and resorted to compute it individually for the masked and uniform noise cases; (v) we develop a general yet closed form formula for corrector sampling with arbitrary schedulers (equation 23). This generalizes Campbell’s stochastic sampling constant $\\eta$ ($\\alpha_t = 1 + t\\eta$ and $\\beta_t = \\alpha_t - 1$), and we also note that Campbell’s stochastic sampling (Proposition 3.3 and equation 9 in [1]) incorporates the detailed balanced matrix in an implicit way and therefore requires a particular solution in each case; (vi) we show that particular polynomial correctors schedulers provide a further significant boost in results. \n\n[1] Andrew Campbell, Jason Yim, Regina Barzilay, Tom Rainforth, and Tommi Jaakkola. \"Generative flows on discrete state-spaces: Enabling multimodal flows with applications to protein co-design.\"\n\n**Question: Why are the results from Austin et al. [2021a] poor?** We use the Pytroch implementation from https://github.com/cloneofsimo/d3pm, with the same architecture (DiT), tokenizer (GPT2), and data (Open Web Text) as we use for our model. We will add these details to the revised version.\n\n**Comment: In Line 214, I'm not entirely sure what the difference is.** As mentioned above (see (v),(vi) in the above answer) compared to [1] we provide more general (arbitrary schedulers) and closed form corrector steps. The discrete and continuous diffusion works [2, 3] define only corrector *iterations* and not sampling  (i.e., do not progress in time, similar to case (ii) described in line 280 in our submission) and perform corrector iterations by incorporating equal forward and reverse rates for the diffusion probability paths they consider. In our case we develop a closed form corrector step incorporating arbitrary schedulers (see $\\alpha_t,\\beta_t$ in equation (23)) that includes both corrector **sampling** and corrector **iterations** as special cases. \n\n[2] Andrew Campbell, Joe Benton, Valentin De Bortoli, Thomas Rainforth, George Deligiannidis, and Arnaud Doucet. \"A continuous time framework for discrete denoising models.\"\n\n[3] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. “Score-based generative modeling through stochastic differential equations.”\n\n\n**Comment: In Figure 3, the Inception Score (IS) needs to be included to demonstrate the diversity of image generation.** Per the reviewer’s request, we added a corresponding inception score graph in the PDF attached to the main response of rebuttal. Note that we observed a similar trend to the FID demonstrated in the original submission.\n\n**Question: Except the perplexity, how about the performance on BLEU, bertscore?** BLEU and BERT-score are traditionally used for the “data-data” case (e.g., translation), that is when both source and target samples are data, and then generated samples are compared to target (test) data. We use “noise-data” (except the conditioning part) and therefore we couldn't find a reasonable way to apply these metrics in our case. \n\n**Comment: Diversity is a significant concern for flow-based models. What about the result on diversity-related metrics?** We are not aware that flow-based models have a diversity concern in general, maybe the reviewer refers to distilled flow models?  In any case, we didn’t encounter particular diversity issues with our model, and in Tables 2, 3, and Figure 7, with additional details provided in Appendix F.1, we present the entropy of tokens within generated sequences to illustrate the diversity of the model's predictions. Furthermore, our modeling allows for control over the diversity by adjusting the temperature during sampling from  $p_{1|t}$.\n\nTo further address the reviewer’s concerns, we attach in the rebuttal’s main reply several uncurated generations, sampled with the same prompt. These uncurated generations demonstrate the diversity of our model's output.\n\n**Comment: In F.1, in the context of conditional generation, it's unclear what the source (src) and target (tgt) samples are. Are they conditional prompts, target prompts, or a combination of Gaussian noise and target prompts?** In the context of conditional generation (Appendix F.1), the source-target pairs $(X_0,X_1)$ are as described in Equation 5 in the paper, i.e., $(X_0,X_1) = (\\mathbb{I} \\odot X_1  + (\\mathbf{1}-\\mathbb{I})\\odot (\\mathbb{m},\\ldots,\\mathbb{m}) , X_1)$, where $\\mathbb{I} \\in \\set{0,1}^{N}$ is indicating the conditioning mask. There are no target prompts, nor a combination of Gaussian noise/target prompts. We did our best to understand the reviewer’s question, if we got it wrong, please clarify.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We appreciate the reviewers' insightful feedback on our paper. We address each of the comments/questions raised by the reviewers in the specific threads below. We would be happy to address any remaining concerns during the discussion period.

Here we summarize the new experiments we performed during rebuttal period to address remaining concerns of reviewers: 
1. As suggested by reviewer oYBb, in Table 1, we report a new comparison to the baselines on the LM1B dataset.
| Method | NFE | Llama-2 | Llama-3 | GPT2 | Entropy |
|---|---|---|---|---|---|
| Data |  | 5.9 | 7.7 | 17.6 | 8.0 |
| Han et al. | >10000 | 67.8 | 97.6 | 150.1 | 8.0 |
| Lou et al. | 256/512/1024 | 27.9/26.1/23.7 | 41.7/39.2/35.0 | 129.3/120.9/104.2 | 8.1/8.1/8.2 |
| Campbell et al. | 256/512/1024 | 26.4/25.7/23.1 | 39.7/34.9/32.6 | 121.7/115.1/99.3 | 8.2/8.2/8.2 |
| FM (Ours) | 256/512/1024 | **24.2/23.4/20.8** | **35.2/33.2/29.2** | **110.6/106.6/86.7** | 8.2/8.2/8.1 |
2. As suggested by reviewer 7VEG, we have computed the inception score for both our proposed and baseline CIFAR10 models. The results are presented in the attached PDF, and demonstrate a similar trend as with the reported FID metric in the submission.
3. As suggested by reviewer 7VEG, we added unconditional samples of our 1.7B Discrete Flow Matching text model to the rebuttal’s PDF.
4. To address reviewer 7VEG concerns about diversity of predictions, we added below more **uncurated** qualitative samples, generated with the same prompt.

**Uncurated qualitative conditional samples**

**Einstein was a German-born theoretical physicist who is widely held to be one of the greatest and most influential scientists of all time. Best known for developing the** famous theory of relativity, Einstein was also well known for his advocating for women in the field of science. Find out everything to know about him and scientific theories about the universe.
Albert Einstein is widely held to be one of the most significant scientists of the 21st century. He is also acknowledged for the multibillion impact of his work.
Described as a mathematician, physicist and philosopher, he is mainly regarded for his theories on the universe. And so, he is also considered to have had an immense impact. It is estimated that somewhere around 10,000 people contributed towards the creation of his theories. As a result, the concepts, that he created, and in his case, that of his brilliant genius, and his theories that afforded him the Theory of Relativity, have all consolidated until this day. So, let’s learn everything to know about Einstein and all of his theories and works.


**Albert Einstein was a German-born theoretical physicist who is widely held to be one of the greatest and most influential scientists of all time. Best known for developing the theory of the** earth movement, the theory of relativity, and the principles of the theory of general physics and the theory of special relativity, which are precursors to the quantum theory.
He was the second leading theoretical physicist in the 20 history of science and a cosmopolitan statesman. He is among the most important and influential figures of all time in the development of best known for developing the theory of the earth movement, the theory of relativity and the principles of the theory of general physics and the theory of special physics. Einstein's theories of relativity, which emerged in the first decade of the twentieth century, describe the relationship between the two poles of the universe. His theories also summarize the relationship between gravity and electromagnetism, the speed of light (sometimes used as the "speed"), special effects including telekinesis (the illusion of self), and time travel, and a popular but flawed quantum theory.


**Albert Einstein was a German-born theoretical physicist who is widely held to be one of the greatest and most influential scientists of all time. Best known for developing the** laws of motion and the theory of general relativity, Einstein's contributions developed over time to create his final achievement in physics, relativity with its general theories and theory of gravity, sometimes called the special theory of relativity. He is widely considered as the father of modern quantum mechanics, wrote his Aclonomical Revolution (one could be called the greatest pedagogues of all time), revealed the Newtonian Square of suggestion, the Doppler effect for light and a gyromagnetic effect, all of which lead to the standard physics of quantum mechanics.

Meta-review: The paper introduces a novel variant of the flow-matching model for discrete data types. The authors provide solid theoretical foundations for their approach and conduct experiments in language modeling, code generation, and image generation tasks. The model can be seen as an alternative approach to autoregressive models and possibly opens new research directions in generative models for discrete data. 

The reviewers are pretty consistent in evaluating the presented approach's quality and novelty, and many concerns were solved during the rebuttal phase. The idea of discrete flow matching was initially introduced by Campbell et al., 2024, which may slightly lower the contribution. However, the authors clearly point out the differences and compare quantitatively with this approach considering the image generation task. Therefore, the paper should be accepted.
Decision: Accept (spotlight)