Discrete Flow Matching

Authors: itai  gat
tal  remez
neta  shaul
felix  kreuk
ricky t q chen
gabriel  synnaeve
yossi  adi
yaron  lipman
meta  fair
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 12:43:19

Abstract: Despite Flow Matching and diffusion models having emerged as powerful generative paradigms for continuous variables such as images and videos, their application to high-dimensional discrete data, such as language, is still limited.  In this work, we present Discrete Flow Matching, a novel discrete flow paradigm designed specifically for generating discrete data.  Discrete Flow Matching offers several key contributions:  (i) it works with a general family of probability paths interpolating between source and target distributions; (ii) it allows for a generic formula for sampling from these probability paths using learned posteriors such as the probability denoiser ($x$-prediction) and noise-prediction ($\epsilon$-prediction); (iii) practically, focusing on specific probability paths defined with different schedulers improves generative perplexity compared to previous discrete diffusion and flow models; and (iv) by scaling Discrete Flow Matching models up to 1.7B parameters, we reach 6.7% Pass@1 and 13.4% Pass@10 on HumanEval and 6.7% Pass@1 and 20.6% Pass@10 on 1-shot MBPP coding benchmarks. Our approach is capable of generating high-quality discrete data in a non-autoregressive fashion, significantly closing the gap between autoregressive models and discrete flow models.

Main paper: 
 introduction: 
 Despite the remarkable success of diffusion and flow models in generating continuous spatial signals such as images (Ho et al., 2020; Rombach et al., 2022; Esser et al., 2024) and videos (Singer et al., 2022; Blattmann et al., 2023) , their performance still falters when applied to discrete sequential data compared to autoregressive models. Recent progress in adapting diffusion and flow models to the discrete setting has been made via mostly two approaches: embedding the discrete data in continuous space and applying continuous diffusion (Dieleman et al., 2022; Stark et al., 2024) or designing diffusion or flow processes over discrete state spaces (Austin et al., 2021a; Campbell et al., 2022) .In this paper, we pursue the discrete flow approach of Campbell et al. (2024) and introduce Discrete Flow Matching, a theoretical framework and algorithmic methodology for discrete flow models that yields a state-of-the-art discrete non-autoregressive generative approach. Surprisingly, Discrete Flow Matching exhibits similarities with the continuous Flow Matching (Lipman et al., 2022) approach proposed for continuous signals. Notably, its generating probability velocity, employed in the sampling algorithm, is identical in form to its continuous counterpart. Additionally, Discrete Flow Matching offers the following advancements and simplifications over prior methods: It encompasses a more comprehensive family of probability paths transforming source (noise) distributions into target (data) distributions, accommodating arbitrary source-target couplings and time-dependent schedulers. Furthermore, it provides a unified formulation for the generating probability velocity directly expressed in terms of the learned posteriors and schedulers, along with a unified and general theory and algorithm for corrector sampling and iterations. In practice, we observe that path and corrector schedulers are pivotal, and their proper tuning leads to substantial improvements in def fib(n: int):"""Return n-th Fibonacci number. >>> fib (10) 55 >>> fib(1) 1 >>> fib (8) 21 """ if n < 1: return 0 if n < 2: return 1 return fib(n-1) + fib(n-2) generation quality. We have trained a 1.7B parameter Discrete Flow Matching model on the same data mix as in Llama-2 (Touvron et al., 2023) and CodeLlama (Roziere et al., 2023) , achieving 6.7% Pass@1 and 13.4% Pass@10 on HumanEval and 6.7% Pass@1 and 20.6% Pass@10 on 1-shot MBPP coding benchmarks; Figure 1 shows some code generation examples. In conditional text generation our model produces texts with a generative perplexity score of 9.7 as measured by the Llama-3 8B model, surpassing a 1.7B autoregressive model that achieves 22.3 and not far from the Llama-2 7B model that achieves 8.3 in generative perplexity score. We strongly believe that Discrete Flow Matching represents a significant step in bridging the performance gap between discrete diffusion and autoregressive models, and that further enhancements are possible by exploring the vast design space that Discrete Flow Matching has to offer. 
setup and notations: 
 In discrete sequence modeling, we denote a sequence x as an array of N elements (x 1 , x 2 , . . . , x N ). Each element, or token, within this sequence is selected from a vocabulary of size d. Consequently, the entire set of possible sequences is given by D = [d] N , where [d] = {1, . . . , d}. A random variable taking values in the space D is denoted by X and its corresponding probability mass function (PMF) is P (X = x). For simplicity, throughout the paper, we sometimes omit the random variable X and use p(x) to denote the PMF.To describe marginalization properties, we denote p(x i ) the x i marginal of p, i.e., p(x i ) = x ī p(x), where x ī = (. . . , x i-1 , x i+1 , . . .) ∈ [d] N -1 are all the arguments excluding i. Similarly, p(x ī) =x i p(x), and x i ∈ [d] . A useful PMF is the delta function, δ y , y ∈ D, which is defined byEQUATION)With the marginal notation δ y (x i ) = δ y i (x i ) and δ y (x ī) = δ y ī (x ī) = j̸ =i δ y j (x j ) which simplifies notation.In discrete generative models our goal is to transform source samples X 0 ∼ p to target samples X 1 ∼ q. Our training data, consist of pairs X 0 and X 1 that are sampled from a joint distribution π(x, y), satisfying the marginals constraints p(x) = y∈D π(x, y), q(y) = x∈D π(x, y), i.e., (X 0 , X 1 ) ∼ π(X 0 , X 1 ).(2)In the simplest case, the training pairs X 0 and X 1 are sampled independently from the source and target distributions respectively, (X 0 , X 1 ) ∼ p(X 0 )q(X 1 ).(3)Example: source and couplings. Common instantiations of source distribution p are: (i) adding a special token value often referred to as a 'mask' or 'dummy' token, denoted here by m, and setting the source distribution to be all-mask sequences, i.e., p(x) = δ m (x); and (ii) using uniform distribution over D, which is equivalent to drawing each x i independently to be some value in [d] with equal probability, denoted p(x) = p u (x). In this paper we focus mainly on (i). We further consider two choices of couplings π: Independent coupling, which we call unconditional coupling (U-coupling), π(x 0 , x 1 ) = p(x 0 )q(x 1 ). A random sample that realizes this choice have the formEQUATIONwhere X 1 ∼ q(X 1 ) is a random sample from the training set. The second choice of coupling π(x 0 , x 1 ) = p(x 0 |x 1 )q(x 1 ), which we find improves conditional sampling, partially masks inputs with samples of the form (X 0 , X 1 ) = (I ⊙ X 1 + (1 -I) ⊙ (m, . . . , m), X 1 ),where X 1 ∼ q(X 1 ) and I ∈ {0, 1} N is a random variable indicating the conditioning, ⊙ denotes the entry-wise product, and 1 ∈ R N is the vector of all ones. We call this conditional coupling (C-coupling).We follow the Flow Matching approach (Lipman et al., 2022; Liu et al., 2022; Albergo and Vanden-Eijnden, 2022 ) that uses a predefined probability path p t interpolating p and q, i.e., p 0 = p and p 1 = q (6)to train the generative model taking a source sample X 0 ∼ p to a target sample X 1 ∼ q. We use arbitrary coupling of source and target (Pooladian et al., 2023; Tong et al., 2023) , π(x 0 , x 1 ), and the symmetric Flow Matching path (Albergo and Vanden-Eijnden, 2022) to define the marginal probability path,p t (x) = x0,x1∈Dp t (x|x 0 , x 1 )π(x 0 , x 1 ), where p t (x|x 0 , x 1 ) = N i=1 p t (x i |x 0 , x 1 ), (7) and p t (x i |x 0 , x 1 ) is a time-dependent probability on the space of tokens [d] conditioned on the pair x 0 , x 1 , and satisfying p 0 (x i |x 0 , x 1 ) = δ x0 (x i ) and p 1 (x i |x 0 , x 1 ) = δ x1 (x i ). If the conditional path p t (x i |x 0 , x 1 ) satisfies these boundary conditions then the marginal path p t (x) satisfies equation 6.In developing the framework, we would like to consider as general as possible set of probability paths that are also tractable to learn within the Flow Matching framework. We consider conditional probability paths as a convex sum of m conditional probabilities w j (x i |x 0 , x 1 ), i.e.,EQUATIONwhere j κ i,j t = 1 and κ i,j t ≥ 0 are collectively called the scheduler. Note that the scheduler can be defined independently for each location in the sequence i ∈ [N ] or uniformly for all tokens, κ i,j t = κ j t . A simple yet useful instance of these conditional paths is reminiscent of the continuous Flow Matching paths formulated as convex interpolants,EQUATIONwhere the scheduler κ t satisfies κ 0 = 0, κ 1 = 1, and monotonically increasing in t. Another interesting instantiation of equation 8 is adding uniform noise with some probability depending on t,EQUATIONwhere κ 1 0 = 0, κ 1 1 = 1, κ 2 0 = κ 2 1 = 0 (remembering that j κ i,j t = 1 and κ i,j t ≥ 0).+ - - - - + - + - + - - - + + + Flow R N Flow [d] N div in R N div in [d] N Figure 2: Discrete flow in D = [d] N with d = 4, N = 2 (middle-left) versus continuous flow in R N , N = 2 (left).The rate of change of probability of a state (gray disk) is given by the divergence operator shown in the continuous case (middle right) and the discrete case (right).Continuous generating velocity. Sampling in continuous Flow Matching is performed by updating the current (continuous) sample X t ∈ R N , t ∈ [0, 1), according to a learned generating velocity fieldu i t (X t ), i ∈ [N ].Euler sampling follows the (deterministic) ruleEQUATIONwhere h > 0 is a user-defined time step. Note that equation 11 is updating separately each of the sample coordinates,X i t , i ∈ [N ], see e.g., Figure 2 , left. The velocity u i t (X t ) can be either directly modeled with a neural network, or parameterized via the denoiser (a.k.a. x-prediction) or noise-prediction (a.k.a. ε-prediction), see left column in Table 1 . If, for all t ∈ [0, 1), starting at X t ∼ p t and sampling with equation 11 provides X t+h ∼ p t+h + o(h) 1 then we say that u t generates p t .Generating probability velocity. For defining Flow Matching in the discrete setting, we follow Campbell et al. (2024) and consider a Continuous-Time discrete Markov Chain (CTMC) paradigm, namely the sample X t is jumping between states in D, depending on a continuous time value t ∈ [0, 1]. Similar to the continuous Flow Matching setting described above, we focus on a model that predicts the rate of probability change of the current sample X t in each of its N tokens, see Figure 2 , middle-left. Then, each token of the sample X t ∼ p t is updated independently byEQUATIONwhere we call u t the probability velocity as reminiscent of the velocity field in continuous Flow Matching, and as in the continuous case, we define: Definition 1. Probability velocity u t generates the probability path p t if, for all t ∈ [0, 1) and given a sample X t ∼ p t , the sample X t+h defined in equation 12 satisfies X t+h ∼ p t+h + o(h).Algorithm 1 formulates a basic sampling algorithm given a generating probability velocity u t . In order for the r.h.s. of equation 12 to define a proper PMF for sufficiently small h > 0, it is necessary and sufficient that the probability velocity satisfies the conditionsEQUATION)Algorithm 1 Flow Matching sampling.Require:velocity u t , sample X ∼ p, step size h = 1 n for t = 0, h, 2h, . . . , 1 -h do X i ∼ δ X i (•) + hu i t (•, X), for i ∈ [N ]▷ eq. 24 or 22 end for return X Now the main question is how to find a probability velocity u t that generates the probability path defined in equations 7 and 8? A key insight in Flow Matching (Lipman et al., 2022) is that u t can be constructed as a marginalization of conditional probability velocities, u i t (x i , z|x 0 , x 1 ), generating the corresponding conditional probability paths p t (x i |x 0 , x 1 ). This can also be shown to hold in the discrete CTMC setting (Campbell et al., 2024) , where a reformulation in our context and notation is as follows. 1 The o(h ℓ ) notation means a function going to zero faster than h ℓ as h → 0, i.e., o(h ℓ )h ℓ h→0 -→ 0.Theorem 2. Given a conditional probability velocity u i t (x i , z|x 0 , x 1 ) generating a conditional probability path p t (x|x 0 , x 1 ), the marginal velocity defined byu i t (x i , z) = x0,x1∈D u i t (x i , z|x 0 , x 1 )p t (x 0 , x 1 |z) (14)generates the marginal probability path p t (x), where by Bayes' ruleEQUATION)For completeness we provide a simple proof of this theorem in Appendix E.2. The proof, similar to the continuous Flow Matching case, shows that u t and p t satisfy the (discrete version of the) Continuity Equation The Continuity Equation. EQUATIONwhere the divergence operator div x (v) applied to a vector field v : R N → R N is defined byEQUATIONand intuitively means the total flux leaving x, see Figure 2 (middle-right). This gives an intuitive explanation to the Continuity Equation: the rate of the probability ṗt (x) of a state x ∈ R N equals the total incoming probability flux, p t u t , at x. In the discrete case (CTMC) the Continuity Equation (equation 16) holds as is, once the discrete divergence operator is properly defined, i.e., to measure the outgoing flux from a discrete state. In more detail, given some vector field, which in the discrete case is a scalar-valued function over pairs of states, v : D × D → R, the discrete divergence isEQUATIONwhere v(z, x) represents the flux x → z and v(x, z) represent the opposite flux z → x; see Figure 2 , right. Now, in our case (see Figure 2 , middle-left), the probability flux at a state x ∈ D involves all sequences with at most one token difference from x, i.e., the probability flux p t u t at x takes the form v(x, z) = p t (z)u i t (x i , z) and v(z, x) = p t (x)u i t (z i , x) for z and x that differ only in the i-th token, v(x, x) = p t (x) N i=1 u i t (x i , x), and v(x, z) = 0 for all other (z, x) ∈ D × D. A direct calculation now shows (see Appendix E.1):EQUATION)Checking that a probability velocity u t generates a probability path p t (in the sense of Definition 1) amounts to verifying the Continuity Equation (equation 16). Indeed, using arguments from Campbell et al. (2024) and the discrete divergence operator, the PMF of X t+h defined by sampling according to equation 12 isEQUATIONwhere we assume X t ∼ p t , the first equality uses the identity i a i + hb i = i a i + h i ( j̸ =i a j )b i + o(h), the second equality uses equation 19, and the previous-to-last equality uses the Continuity Equation (equation 16). This shows that if the Continuity Equation holds then u t generates p t in the sense of Definition 1.Table 1 : Generating (marginal) velocity fields have identical form for the continuous and discrete Flow Matching when using denoiser/noise-prediction parameterization; x1|t (z) = E X1∼pt(•|z) X 1 is the standard continuous denoiser (a.k.a. x-prediction) and x0|t (z) = E X0∼pt(•|z) X 0 is the standard noise-prediction (a.k.a. ϵ-prediction).Discrete Flow Matching Marginal prob.p t (x) = x0,x1 N i=1 p t (x i |x 0 , x 1 )π(x 0 , x 1 ) Conditional prob. p t (x i |x 0 , x 1 ) = δ κtx1+(1-κt)x0 (x i ) p t (x i |x 0 , x 1 ) = κ t δ x1 (x i ) + (1 -κ t )δ x0 (x i ) VF-Denoiser u i t (X t ) = κt 1-κt xi 1|t (X t ) -X i t u i t (x i , X t ) = κt 1-κt p 1|t (x i |X t ) -δ Xt (x i ) VF-Noise-pred u i t (X t ) = κt κt X i t -xi 0|t (X t ) u i t (x i , X t ) = κt κt δ Xt (x i ) -p 0|t (x i |X t )Conditional and marginal generating velocities. We provide the probability velocities generating the conditional probability paths p t (x|x 0 , x 1 ) defined in equations 7 and 8. Then, using the marginalization formula in equation 14 we end up with a closed-form marginal velocity for the probability paths p t (x). In Appendix E.3 we show Theorem 3 (Probability velocity of conditional paths). A generating probability velocity for the conditional paths p t (x|x 0 , x 1 ) defined in equations 7 and 8 isEQUATIONwith a i,j t = κi,j t -κ i,j t κi,ℓ t /κ i,ℓ t , and b i t = κi,ℓ t /κ i,ℓ t where ℓ = arg min j∈[m] κi,j t /κ i,j t . Now, computing the marginal probability velocity using equation 14 applied to the conditional probability velocity in equation 21 givesEQUATIONwhere the posteriors ŵj t of w j (that are later shown to be tractable to learn) are defined byEQUATIONwhere p t (x 0 , x 1 |z) (defined in equation 15) is the posterior probability of x 0 , x 1 conditioned on the current state X t = z. A useful instantiation of the general velocity in equation 22 is when considering the path family in equation 9, for whichw 1 (x i |x 0 , x 1 ) = δ x1 (x i ), w 2 (x i |x 0 , x 1 ) = δ x0 (x i ), κ i,1 t = κ t , κ i,2 t = 1 -κ t , κt ≥ 0 (i.e., monotonically non-decreasing in t) and in this case equation 22 reads asEQUATION)where we use the notation p 1|t (x i |z) = x0,x1 δ x1 (x i )p t (x 0 , x 1 |z) for the probability denoiser.Sampling backward in time. We can also sample backwards in time by following the sampling ruleX i t-h ∼ δ X i t (•) -hu i t (•, X t ).In this case -u i t (x i , z) should satisfy equation 13. A (backward-time) generating probability velocity can then be achieved from equation 22 with the simple change to the coefficients a i,j t and b i,j t , see Appendix E.4. For p t defined with equation 9 the generating velocity isEQUATIONwhere in this case p 0|t (x i |z) = x0,x1∈D δ x0 (x i )p t (x 0 , x 1 |z) is the probability noise-prediction.Remarkably, the generating velocity fields in 24 and 25 take the exact same form as the generating (a.k.a. marginal) velocity fields in continuous flow matching when parameterized via the denoiser or noise-prediction parameterizations and using the same schedulers, see Table 1 and Appendix E.9 for explanation of the continuous case. In Appendix E.4 we provide the backward-time version of Theorem 3.Corrector sampling. Combining the forward-time ût ( equation 24) and backward-time ǔt ( equation 25), i.e., ūi t (x i , z) = α t ûi t (x i , z) -β t ǔi t (x i , z), (26) provides a valid forward-time probability velocity field (i.e., satisfies equation 13) for t ∈ (0, 1) as long as α t , β t > 0. This velocity field can be used for two types of corrector sampling: (i) When α t -β t = 1 sampling with ūt leads to corrector sampling where intuitively each step moves 1 + β t forward in time and -β t backwards, which allows reintroducing noise into the sampling process; and (ii) when α t -β t = 0 sampling with ūt when fixing t ∈ (0, 1) leads to corrector iterations where limit samples distribute according to p t . In Appendix E.6 we prove: Theorem 4. For perfectly trained posteriors and α t , β t > 0, t ∈ (0, 1), ūt in equation 26 is a probability velocity, i.e., satisfies equation 13, and: (i) For α t -β t = 1, ūt provides a probability velocity generating p t ; (ii) For α t -β t = 0, repeatedly sampling with ūt at fixed t ∈ (0, 1) and sufficiently small h is guaranteed to converge to a sample from p t .One simplification to equation 26 can be done in the case of paths constructed with conditional as in equation 9, independent coupling π(x 0 , x 1 ) = p(x 0 )q(x 1 ), and i.i.d. source p(x 0 ) = N i=1 p(x i 0 ), e.g., p(x i 0 ) is uniform over [d] or δ m (x i 0 ). In this case, the backward-time formula in equation 25 take an equivalent simpler formEQUATIONwhich does not require estimation of the posterior p 0|t . See Appendix E.5 for the derivation.Training. Equation 22 shows that for generating samples from a probabilty path p t (x) we require the posteriors ŵj t (x i |X t ). Training such posteriors can be done by minimizing the lossEQUATION)where t is sampled according to some distribution in [0, 1] (we used uniform), (X 0 , X 1 ) ∼ π(X 0 , X 1 ), X t ∼ p t (X t |X 0 , X 1 ), and Y i j ∼ w j (Y i j |X 0 , X 1 ); θ ∈ R p denotes the learnable parameters. In the common case we use in this paper of learning a single posterior, i.e., the probability denoiser p 1|t , the loss takes the formL(θ) = -i∈[N ] E t,(X0,X1),Xt log p 1|t (X i 1 |X t ).In Appendix E.7 we prove: Proposition 5. The minimizer of L (equation 28) is ŵj t (x i |X t ) (equation 23). 
related work: 
 In the section we cover the most related work to ours; in Appendix A we cover other related work.Discrete Flows (Campbell et al., 2024) is probably the most related work to ours. We build upon their CTMC framework and offer the following generalizations and simplifications: We consider arbitrary couplings (X 0 , X 1 ), and offer a novel and rather general family of probability paths (equation 8) for which we provide the generating probability velocities in a unified closed-form formula (equations 22-25). These in particular recreate the same formulas as the continuous Flow Matching counterpart (Table 1 ). We furthermore develop a general corrector velocity (equation 26) that unifies both corrector iterations (Song et al., 2020; Campbell et al., 2022) and stochastic sampling of Campbell et al. (2024) . We show that particular choices of noise schedulers κ t (κ t = t reproduces Campbell et al. ( 2024)) and corrector schedulers provide a boost in results. Lastly, we opted for the term probability velocity for u i t (x i , X t ) as it is not precisely a rate matrix in the state space D × D used in CTMC since u i t (x i , z) for all i ∈ [N ] define multiple self-edges z → z. Masked modeling (Ghazvininejad et al., 2019; Chang et al., 2022) . In case of a masked model, i.e., when the source distribution is p(x) = δ m (x), we achieve an interesting connection with MaskGit 2024) 256 / 512 / 1024 38.5/33.5/28.7 69.0/56.5/ 46.5 65.2/53.3/43.0 7.8 / 7.7 / 7 showing it is actually an instance of Discrete Flow Matching with a small yet crucial change to its sampling algorithm. First, in Appendix E.8 we prove that in the masked setting, the probability denoiser p 1|t is time-independent: Proposition 6. For paths defined by equations 7 and 9 with source p(x) = δ m (x) the posterior p t (x 0 , x 1 |z) = p(x 0 , x 1 |z) is time-independent. Consequently, the probability denoiserp 1|t (x i |z) = p 1 (x i |z) is also time-independent.This shows that the probability denoiser can be learned with no time dependence, similar to the unmasking probabilities in MaskGit. During sampling however, there are two main differences between our sampling and MaskGit sampling. First, unmasking of tokens in our algorithm is done according to the probability δ Xt (x i ) + hu i t (x i , X t ) independently for each tokenx i , i ∈ [N ]. This procedure is justified as it samples from the correct probability asymptotically via the derivation of the Continuity Equation 20. This is in contrast to MaskGit that prioritizes the token to be unmasked according to some confidence. In the experiments section we show that MaskGit's prioritization, although has some benefit in the very low NFE regime, is actually introducing a strong bias in the sampling procedure and leads to inferior overall results. Secondly, using corrector sampling allows for reintroducing masks to already unmasked tokens in a way that is still guaranteed to produce samples from p t , see Theorem 4; we find this to have a significant positive effect on the generation quality.Discrete diffusion. D3PM (Austin et al., 2021a) and Argmax flows (Hoogeboom et al., 2021) introduced diffusion in discrete spaces by proposing a corruption process for categorical data. A later work by Campbell et al. (2022) introduced discrete diffusion models with continuous time, and Lou et al. (2023) proposed learning probability ratios, extending score matching (Song and Ermon, 2019) to discrete spaces. 
experiments: 
 We evaluate our method on the tasks of language modeling, code generation, and image generation. For language modeling, we compare the proposed method against prior work considering the widely used generative perplexity metric. We scale the models to 1.7 billion parameters and present results on coding tasks, i.e., HumanEval (Chen et al., 2021) , MBPP (Austin et al., 2021b) , demonstrating the most promising results to date in a non-autoregressive context. In image generation, we present results for a fully discrete CIFAR10 (Krizhevsky et al., 2009) . Further details of the experimental setup for each model are provided in Appendix G.Experimental setup. In our experiments we used the masked source, i.e., p = δ m , and trained with both unconditional coupling (U-coupling, equation 4) and conditional couplings (C-coupling, equation 5) with the probability path defined in equations 7, 9 and in one case 10. We trained a probability denoiser (loss in equation 28) and sampled using the generating velocity in equation 24 and Algorithm 1. We used a particular choice of probability path scheduler κ t , as well as corrector steps defined by a scheduler α t and temperature annealing. We found the choice of these schedulers to be pivotal for the model's performance. In Appendix D we perform an ablation study, evaluating various scheduler choices.We experimented with our method in three settings: (i) Small model (150M parameters) -comparison to other non-autoregressive baselines in unconditional text generation; (ii) Large model (1.7B parameters) -comparison to autoregressive models in conditional text generation; and (iii) Large model (1.7B parameters) -conditional code generation. As computing exact likelihood for nonautoregressive model is a challenge, for (i),(ii) we use the generative perplexity metric (Appendix G measured with GPT2 (Radford et al., 2019) , Llama-2 (Touvron et al., 2023) , and Llama-3, and we also monitor the sentence entropy (Appendix G) to measure diversity of tokens and flag repetitive sequences, which typically yield low perplexity. Throughout our experiments we noticed entropy ≥ 6 usually corresponds to diverse texts. For (iii) we evaluated using the success rate of coding tasks.Evaluation against prior work. We evaluate our method against prior work on non-autoregressive modeling. For a fair comparison, all methods are trained on a 150M parameters models using the OpenWebText (Gokaslan and Cohen, 2019) dataset. We also fix all sampling hyperparameters to the most basic settings, i.e., no temperature, top probability, corrector steps, etc. For our method we tried two paths defined by equations 9 and 10. Results are reported in Table 2 , where our method outperforms all baselines in generative perplexity for all numbers of function evaluations (NFE).Conditional text generation. In this experiment, we train both C-coupling and U-coupling 1.7B parameters FM models with paths defined by equation 9 on a large scale data mix (Touvron et al., 2023) . Table 3 presents the generative perplexity of conditional generations from our method; the conditions we used are the prefixes of the first 1000 samples in OpenWeb dataset. We also compare to existing state-of-the-art autoregressive models. Our results demonstrate that our model effectively narrows the gap in generative perplexity with autoregressive models, while maintaining an entropy comparable to the recent Llama-3 8B model. Furthermore, we note the C-coupling trained model produces slightly better perplexity in conditional tasks than the U-coupling model. In Appendix I we present qualitative conditional samples produced by our U-coupling model.Here we trained our basic setting of a 1.7B parameters FM model with U-coupling and path as in equation 9 on a code-focused data mix (Roziere et al., 2023) . Table 4 presents results on HumanEval and MBPP (1-shot) for pass@{1, 10, 25}. In Table 4 , 'Oracle length' evaluates the performance of our model when conditioning on the length of the solution. This is done by inserting an 'end of text' token in the same position of the ground truth solution. Our method achieves non-trivial results on both tasks, which to the best of our knowledge is the first instance of a non-autoregressive method being capable of non-trivial coding tasks. In Appendix C, we analyze the proposed method for code infilling, which can be achieved as our model allows non-autoregressive generation. Lastly, in Appendix H we show qualitative examples of success and failure cases produced by our model on the coding tasks, and in Appendix H. 3 we show examples of code infilling.We performed a fully discrete image generation, without using any metric or neighboring information between color values. We trained an FM model with U-coupling and path as in equation 9 on CIFAR10 to predict discrete color value for tokens, i.e., d = 256, with sequence length of N = 32 × 32 × 3. For generative quality we evaluate the Fréchet Inception Distance (FID) (Heusel et al., 2017) . Ablations for the probability path schedulers are provided in Figure 8 in the Appendix G. In Figure 3a we compare our method with: (i) MaskGIT (Chang et al., 2022) ; and (ii) (Campbell et al., 2024) which coincides with our method for a linear scheduler. More details in Appendix G. As can be seen in the Figure 3a , our method outperforms both baselines, achieving 3.63 FID at 1024 NFE. In fig. 3b we observe a similar trend when evaluating Inception score. As discussed above, MaskGit sampling performs better for low NFE but quickly deteriorates for higher NFE. We attribute this to a bias introduced in the sampling process via the confidence mechanism. We provide here some more details on relevant related works.Continuous diffusion and flows. Another line of works has been exploring the use of continuous space diffusion for discrete data, typically operating in the logits space (Dieleman et al., 2022; Li et al., 2022; Han et al., 2022; Lin et al., 2022; Chen et al., 2022 ). An additional body of work has been focusing on the adoption of latent diffusion-like modeling (Lovelace et al., 2022; He et al., 2022) . Stark et al. (2024) proposed to learn a continuous Flow Matching on the probability simplex with Dirichlet paths.Autoregressive modeling. Autoregressive models have been a significant area of focus in recent years, particularly in the context of natural language processing and machine learning (Zhao et al., 2023) . Autoregressive modeling, in its most fundamental form, utilizes the chain rule to learn the joint sequence probability by breaking it down into next-token conditional probabilities. GPT-2 (Radford et al., 2019) , showcased the power of autoregressive language models in generating coherent and contextually relevant text over long passages. Its successor, GPT-3 (Brown et al., 2020) , further pushed the boundaries, demonstrating impressive performance across a range of tasks without taskspecific training data. Later models were adapted to other domains such as, code (Roziere et al., 2023; Li et al., 2023; Chen et al., 2021) , biology (Zhang et al., 2024; Ferruz and Höcker, 2022; Madani et al., 2023) , math (Romera-Paredes et al., 2024; Imani et al., 2023; Ahn et al., 2024) , audio (Kreuk et al., 2022; Copet et al., 2024; Hassid et al., 2024) and more.Masked generative modeling. Masked generative modeling proposes to mask a variable portion of the input sequence and training a model to predict this masked section. Ghazvininejad et al. (2019) proposed Mask-Predict, a masked language modeling with parallel decoding. Savinov et al. (2021) extended the mask-modeling approach by employing an additional loss term that incorporates rolling model predictions. MaskGIT (Chang et al., 2022 ) followed a similar path, for the task of class-conditioned image synthesis, Chang et al. (2023) extended this approach to high-quality textually guided image generation over low-resolution images followed by a super-resolution module.Recently, Ziv et al. ( 2024) proposed a text-to-music method, which relies on the MaskGIT foundations while observing that span masking boosts the quality of the generated sequence significantly.Safe sampling. When sampling according to Algorithm 1 using the generating probability velocity in equation 22, an arbitrary step size h > 0 can make some probabilities in δ X i t (•) + hu i t (•, X t ) negative and consequently require clamping and injecting further error into the sampling process that can in turn accumulate to a non-negligible global sampling error. A simple fix that guarantees a valid probability distribution while keeping the o(h) sampling error at the relatively manageable price of potentially more function evaluations is using the following adaptive step size in Algorithm 1: at time t ∈ [0, 1) useEQUATION)As can be verified with the general probability velocity formula in equation 22, the above choice for h adaptive guarantees δ X i t (•) + hu i t (•, X t ) is a valid PMF. As mostly used in this paper, for the probability denoiser parameterization (equation 24) the adaptive step isEQUATION)With the corrector sampling (equations 26 and 51) we have the adaptive step:EQUATION)Conditioning. In our unconditional coupling (U-coupling), see equation 5, we define the conditioning pattern based on prefixes of random length N 0 < N , i.e.,I = ( N0 1, . . . , 1, N -N0 0, . . . , 0).During the training phase, we sample N 0 ∼ U(0, N ) and adjust the input sequence in accordance with the mask I.During conditional sampling with Algorithm 1 we replace, after each update step, the relevant tokens with the conditioned ones, i.e., X = I ⊙ Y + (1 -I) ⊙ X, where X is the current sample, Y is the condition, and I is the condition's mask.NFE bound. For mask modeling, i.e., p = δ m , we have seen that the probability denoiser is time-independent (see Proposition 6) . Consequently, when sampling with Algorithm 1 and u t from equation 24 without corrector sampling one is not required to recompute the forward passp 1|t (•|X t ) if X t is identical to X t-h (i.e., no m has been unmasked). This means that the NFE of Algorithm 1 in this case is bounded by the number of tokens N .Post training scheduler change. For a trained posterior ŵt (x i |z) of a conditional probability path as in equation 9 with a scheduler κ t , the velocity is given by equations 24 or 25, where ŵt (x i |z) is either p 1|t (x i |z) or p 0|t (x i |z) respectively. In this case, we can apply the velocities in equations 24 and 25 for sampling with any scheduler κ ′ t , using the change of scheduler formula for posteriors,EQUATIONwhere ŵ′ t (x i |z), is the posterior of the schedulerκ ′ t , t ′ = κ -1 κ ′ t, and κ -1 is the inverse of κ. The scheduler change formula in equation 32 is proved in Proposition 8. We note that by Proposition 6, for mask modeling, i.e., p = δ m , the posterior ŵt (x i |z) is time independent. Hence, in that case, the posterior is not affected by a scheduler change. compiles@1 pass@1 Figure 4 : Pass@1 and compiles@1 scores for the 1.5B parameter models as a function of the input masking rations on HumanEval.We additionally evaluate the proposed method considering the task of code infilling. In which, we are provided with an input prompt that contains various spans of masked tokens, and our goal is to predict them based on the unmasked ones. See Figure 1 (middle and right sub-figures) for a visual example. Notice, this evaluation setup is the most similar to the training process.For that, we randomly mask tokens with respect to several masking rations, p ∈ {0.0, 0.1, 0.2, . . . , 1.0}, from HumanEval and report both pass@1 and compiles@1 metrics. For the purpose of this analysis, we provide the oracle length for each masked span. In other words, the model predicts the masked tokens for already given maks length. Results for the 1.5B parameters models can be seen in Figure 4 . As expected, both pass@1 and compiles@1 keep improving as we decrease the level of input masking. Interestingly, when considering the fully masked sequence, providing the oracle prediction length significantly improves the pass@1 scores (6.7 vs. 11.6) .Train and sampling path scheduler choice (κ t ). We study how the choice of the probability path scheduler affects the model performance. For that, we consider a parametric family of cubic polynomial with parameters a, b:(0,0) (0,1) (0,2) (1,0) (1,1) (1,2) (2,0) (2,1) (2,2) (a)EQUATION)Note that κ 0 = 0 and κ 1 = 0 and a and b are setting the derivative of κ t at t = 0 and t = 1, respectively. We visualize this κ t with choices of a, b ∈ {0, 1, 2} in Figure 5a . Temperature 0.9 To test the effect of path schedulers in training we have trained 150M parameters models for all choices of a, b ∈ {0, 1, 2, 3}. We then generate 1000 samples from each model. The samples are computed using Algorithm 1 with the path scheduler the model was trained on, and with temperature levels τ ∈ {0.8, 0.9, 1}, where temperature is applied viaEQUATIONEQUATION(0, 0) (0, 1) (0, 2) (1, 0) (1, 1) (1, 2) (1, 3) (2, 0) (2, 1) (2, 2) (2, 3) (3, 0) (3, 1) (3, 2) (3, 3)EQUATIONWe then evaluate the generative perplexity of these samples with GPT-2. Figure 6 shows the results.The graphs indicate that, in the context of text modality, the cubic polynomial scheduler with a ≡ 0, b ≡ 2 (equivalent to a square function) achieves the highest performance. Consequently, we exclusively used this scheduler for the language models.Corrector scheduler. In our experiments we only applied corrector sampling to our large models (U-coupling and C-coupling; 1.7B parameters). We used the optimal path schedulers from previous section and considered the following parametric family of schedulers for the corrector sampling:EQUATIONwhere, we set β t = α t -1 and generate 1000 samples using Algorithm 1 with parameter values a, b ∈ {0, 0.25, 0.5} and α ∈ {10, 15, 20}. We then evaluated generative perplexity for these samples with Llama-2, showing results in Figure 7 . These plots indicate that smaller values of a and b result in lower perplexity values, albeit with somewhat reduced entropy. We therefore opted for setting a = b = 0.25 that strikes a good balance between perplexity and entropy.Temperature scheduling. For temperature sampling, we consider the following scheduler:τ t = τ (1 -t) 2 .(36) Model: C-Coupling, Temperature: 0.9, Alpha: 20.0 We present the computation of the discrete divergence in equation 18, i.e.,EQUATION)Computing the discrete divergence (equation 18) of the flux p t u t at a state x amounts to adding outgoing flux from x and subtracting the incoming flux into x. Using the fact that δ z (x ī) = 1 if and only if z = x or z differs from x only at the i-th token, gives:div x (p t u t ) = z∈D N i=1 δ x (z ī) p t (x)u i t (z i , x) -p t (z)u i t (x i , z) = p t (x) N i=1 z i =1   z ī δ x (z ī)   u i t (z i , x) - z∈D N i=1 δ x (z ī)p t (z)u i t (x i , z) = p t (x) N i=1 =0 z i u i t (z i , x) - z∈D N i=1 δ x (z ī)p t (z)u i t (x i , z) ▷ equation 13 = - z∈D N i=1 δ x (z ī)p t (z)u i t (x i , z),that gives equation 37 after noting that δ x (z ī) = δ z (x ī).We provide a simple proof for Theorem 2, originally proved in Campbell et al. (2024) : Theorem 2. Given a conditional probability velocity u i t (x i , X t |x 0 , x 1 ) generating a conditional probability path p t (x|x 0 , x 1 ), the marginal velocity defined byEQUATIONgenerates the marginal probability path p t (x), where by Bayes' ruleEQUATION)Proof (Theorem 2). We start by taking the time derivative of the marginal probability path, p t (x) = x0,x1 p t (x i |x 0 , x 1 )π(x 0 , x 1 ), as follows,ṗt (x) = x0,x1 ṗt (x|x 0 , x 1 )π(x 0 , x 1 ) = x0,x1 z p t (z|x 0 , x 1 ) N i=1 δ z (x ī)u i t (x i , z|x 0 , x 1 ) π(x 0 , x 1 ) ▷ Continuity Equation (16) = z p t (z) N i=1 δ z (x ī) x0,x1 u i t (x i , z|x 0 , x 1 ) p t (z|x 0 , x 1 )π(x 0 , x 1 ) p t (z) = z p t (z) N i=1 δ z (x ī)u i t (x i , z) = -div x (p t u t )Now since u i t (x i , z) is a convex combinations of u i t (x i , z|x 0 , x 1 ) and these satisfy equation 13 then also u i t (x i , X t ) satisfies equation 13.Equation 22 with the coefficients a i,j t and b i t are provided below,EQUATIONwhereEQUATION)Theorem 3 (Probability velocity of conditional paths). A generating probability velocity for the conditional paths p t (x|x 0 , x 1 ) defined in equations 7 and 8 isEQUATIONwith a i,j t = κi,j t -κ i,j t κi,ℓ t /κ i,ℓ t , and b i t = κi,ℓ t /κ i,ℓ t where ℓ = arg min j∈[m] κi,j t /κ i,j t .Proof (Theorem 3). First, let us show that equation 40 satisfies the conditions in equation 13: Fix X t ∈ D, andx i u i t (x i , X t |x 0 , x 1 ) = x i m j=1 κi,j t -κ i,j t κi,ℓ t κ i,ℓ t w j (x i |x 0 , x 1 ) + κi,ℓ t κ i,ℓ t δ Xt (x i ) = m j=1 κi,j t -κ i,j t κi,ℓ t κ i,ℓ t + κi,ℓ t κ i,ℓ t = m j=1 κi,j t + κi,ℓ t κ i,ℓ t   1 - m j=1 κ i,j t   ▷ j κ i,j t = 1, and j κi,j t = 0 = 0.and for x i ̸ = X i t we haveu i t (x i , X t |x 0 , x 1 ) = m j=1 κi,j t κ i,j t - κi,ℓ t κ i,ℓ t κ i,j t w j (x i |x 0 , x 1 ) ≥ 0 (43)since κ i,j t ≥ 0, ŵt (x i |z) ≥ 0, and κi,jt κ i,j t - κi,ℓ t κ i,ℓ t ≥ 0 since ℓ = arg min j∈[m] κi,j t κ i,j t. Second, we show that u t satisfies the Continuity Equation (equation 16). To that end we write equation 8 asEQUATIONwhere ℓ = arg min j∈[m] κi,j t κ i,j t . Now by differentiating p t (x|x 0 , x 1 ) we getp t (x|x 0 , x 1 ) = N i=1 p t (x i |x 0 , x 1 ) ṗt (x|x 0 , x 1 ) = N i=1 p t (x ī|x 0 , x 1 ) ṗt (x i |x 0 , x 1 ) = N i=1 p t (x ī|x 0 , x 1 )   m j=1 κi,j t w j (x i |x 0 , x 1 )   = N i=1 p t (x ī|x 0 , x 1 )   j̸ =ℓ κi,j t w j (x i |x 0 , x 1 ) + κi,ℓ t w ℓ (x i |x 0 , x 1 )   = N i=1 p t (x ī|x 0 , x 1 )        m j=1 a i,j t κi,j t -κ i,j t κi,ℓ t κ i,ℓ t w j (x i |x 0 , x 1 ) + b i t κi,ℓ t κ i,ℓ t p t (x i |x 0 , x 1 )        ▷ equation 44 = N i=1        m j=1 a i,j t =pt(x ī|x0,x1) z δ x (z ī)p t (z|x 0 , x 1 ) w j (x i |x 0 , x 1 ) + b i t =pt(x|x0,x1) z δ x (z ī)δ x (z i )p t (z|x 0 , x 1 )        = z p t (z|x 0 , x 1 ) N i=1 δ x (z ī) u i t (x i ,z|x0,x1)   m j=1 a i,j t w j (x i |x 0 , x 1 ) + b i t δ x (z i )   ▷ δ x (z i ) = δ z (x i ), δ x (z ī) = δ z (x ī) = -div x (p t (•|x 0 , x 1 )u t (•|x 0 , x 1 )),as required.Here we prove the equivalent of Theorem 3 for backward-time generating probability field. But first, let us justify the backward sampling formula,EQUATION)Similar to equation 20 we haveE Xt N i=1 δ Xt (x i ) -hu i t (x i , X t ) = E Xt δ Xt (x) -h N i=1 δ Xt (x ī)u i t (x i , X t ) + o(h) = p t (x) + hdiv x (p t u t ) + o(h) (16) = p t (x) -h ṗt (x) + o(h) = p t-h (x) + o(h).Therefore if the Continuity equation holds and -u t satisfies the conditions in equation 13 then given X t ∼ p t , equation 45 provides an approximation X t-h ∼ p t-h + o(h). The change to the generating probability velocity in equation 22 to accommodate reverse time sampling is to replace the argmin in equation 41 with argmax,EQUATION)An analogous result to Theorem 3 for backward-time sampling is therefore, Theorem 7 (Probability velocity of conditional paths, backward time). The probability velocity -u t , where u t defined in equation 21 with ℓ = arg max j∈[m] κi,j t /κ i,j t is a backward-time generating probability velocity for the conditional paths p t (x|x 0 , x 1 ) defined in equations 7 and 8.Proof (Theorem 7). We follow the proof of Theorem 3 and indicate the relevant changes. First, for arbitrary X t ∈ D,EQUATIONexactly using the same arguments as the forward-time case. Now, for x i ̸ = X i t we haveu i t (x i , X t |x 0 , x 1 ) = m j=1 κi,j t κ i,j t - κi,ℓ t κ i,ℓ t κ i,j t w j t (x i |x 0 , x 1 ) ≤ 0 (48)due to ℓ being now the argmax of κi,j t κ i,j t . Therefore -u t satisfies equation 13. Lastly, we notice that the proof of the Continuity Equation follows through exactly the same also in this case. E.5 Backward-time generating velocity for i.i.d. source p(x 0 ) and simple paths Here we consider the case of probability paths defined via the conditionals in equation 9 with independent coupling π(x 0 , x 1 ) = p(x 0 )q(x 1 ) and i.i.d. source distribution p(x 0 ) = N i=1 p(x i 0 ), where p(x i 0 ) is some PMF over [d] . In this case one can simplify the time-backward sampling formula in equation 25 by using the following one which is equivalent (i.e., their difference is divergence free and consequently generate the same probability path p t ):EQUATION)The benefit in this equation is that it does not require the posterior p 0|t , which needs to be learned in general cases.To show that equation 49 is indeed a generating probability velocity it is enough to show thatEQUATION)where ǔ⋆ t is the probability velocity in equation 25. Let us verify using equation 19:div x [p t (ǔ t -ǔ⋆ t )] = i,z p t (z)δ z (x ī) p(x i ) - x0,x1 δ x0 (x i ) p t (z|x 0 , x 1 )p(x 0 )q(x 1 ) p t (z) ▷ π(x 0 , x 1 ) = p(x 0 )q(x 1 ) = i,z δ z (x ī) p(x i )p t (z) - x0,x1 δ x0 (x i )p t (z|x 0 , x 1 )p(x 0 )q(x 1 ) = i,x0,x1 p(x i ) -δ x0 (x i ) z δ z (x ī)p t (z|x 0 , x 1 ) p(x 0 )q(x 1 ) = i,x0,x1 p(x i ) -δ x0 (x i ) p t (x ī|x 0 , x 1 )p(x ī 0 )p(x i 0 )q(x 1 ) = i,x ī 0 ,x1   x i 0 p(x i )p(x i 0 ) -δ x0 (x i )p(x i 0 )   p t (x ī|x 0 , x 1 )p(x ī 0 )q(x 1 ) = 0,where in the second to last equality we used the fact that the paths we are considering have the form:p t (x ī|x 0 , x 1 ) = j∈[N ]\i κ t δ x1 (x j ) + (1 -κ t )δ x0 (x j) , and therefore do not depend on the i-th source token, x i 0 .Theorem 4. For perfectly trained posteriors and α t , β t > 0, t ∈ (0, 1), ūt in equation 26 is a probability velocity, i.e., satisfies equation 13, and: (i) For α t -β t = 1, ūt provides a probability velocity generating p t ; (ii) For α t -β t = 0, repeatedly sampling with ūt at fixed t ∈ (0, 1) and sufficiently small h is guaranteed to converge to a sample from p t .Proof (Theorem 4). First let us write explicitly ūt from equation 26:EQUATION)Since equation 51 is a sum of PMFs with coefficients that sum up to zero the first condition in equation 13, i.e., x i ūi t (x i , X t ) = 0 holds. The second condition in equation 13 holds since for t ∈ (0, 1) we have αt κt 1-κt , βt κt κt ≥ 0. Now,EQUATION)For (i): Using equation 52 with α t -β t = 1 we get thatdiv x (p t ūt ) = -ṗt (x),i.e., ūt satisfies the continuity equation and therefore generates p t .For (ii): Setting α t -β t = 0 in equation 52 we get div x (p t ūt ) = 0 and therefore similar to equation 20 we haveEQUATIONwhere using equation 51 we havep(x|z) = h N i=1 α t κt 1 -κ t δ z (x ī)p 1|t (x i |z) + h N i=1 β t κt κ t δ z (x ī)p 0|1 (x i |z) + 1 -h N i=1 α t κt 1 -κ t + β t κt κ t δ z (x i ).For sufficiently small h > 0 therefore p(x|z) is a convex combination of PMFs (in red) x and consequently is itself a PMF in x, that is p(x|z) is a probability transition matrix, and p t (x) is its stationary distribution, i.e., it is an eigenvector of p(x|z) with eigenvalue 1, which is maximal. To prove convergence of the iterations in equation 53 we are left with showing that p(x|z) is irreducible and a-periodic, see Norris (1998) (Theorem 1.8.3) . Irreducibly of p(x|z) can be shown by connecting each two states z, z ′ by changing one token at a time, and assuming that p 1|t or p 0|t are strictly positive (which is usually the case since as at-least one of them is defined as soft-max of finite logits); a-periodicity is proved by showing p(x|x) > 0 which is true as the coefficient of δ z (x) is greater than zero for sufficiently small h > 0. Lastly, note that the iteration in equation 53 changes one token at a time. An approximation to this sampling can be achieved using our standard parallel sampling via equation 12, justified by equation 20.Proposition 5. The minimizer of L (equation 28) is ŵj t (x i |X t ) (equation 23).Proof (Proposition 5). It is enough to prove the claim for m = 1, with a single w(x i |x 0 , x 1 ).L(θ) = - 1 N N i=1 E t x0,x1,z,y i log ŵt (y i |z; θ)w(y i |x 0 , x 1 )p t (z|x 0 , x 1 )π(x 0 , x 1 ) = - 1 N N i=1 E t z p t (z)   y i log ŵt (y i |z; θ) x0,x1w(y i |x 0 , x 1 ) p t (z|x 0 , x 1 )π(x 0 , x 1 ) p t (z)  = -E t,Xt 1 N N i=1   y i log ŵt (y i |X t ; θ) ŵt (y i |X t )   ,that amounts to minimizing the Cross Entropy loss between ŵt (x i |X t ; θ) and ŵt (x i |X t ) for all i ∈ [N ], the minimizer of which satisfies ŵt (x i |X t ; θ) ≡ ŵt (x i |X t ).E.8 Time-independent posterior for masked modeling Proposition 6. For paths defined by equations 7 and 9 with source p(x) = δ m (x) the posteriorp t (x 0 , x 1 |z) = p(x 0 , x 1 |z) is time-independent. Consequently, the probability denoiser p 1|t (x i |z) = p 1 (x i |z) is also time-independent.Proof (Proposition 6). First,p t (z i |x 0 , x 1 ) = (1 -κ t )δ m (z i ) + κ t δ x1 (z i ) = (1 -κ t ) z i = m κ t δ x1 (z i ) z i ̸ = mand thereforep t (z|x 0 , x 1 ) =   i:z i =m (1 -κ t ) i:z i ̸ =m κ t   i:z i ̸ =m δ x1 (z i ).The posterior now gives ( ( ( ( ( ( ( ( ( ( ( i:z EQUATIONEQUATION:z j =m (1 -κ t ) j:z j ̸ =m κ t j:z j ̸ =m δ x1 (z j ) π(x 0 , x1 ) = p(x 0 , x 1 |z).showing that the posterior is time-independent for dummy source distributions and convex paths. Consequently also the probability denoiser,p 1|t (x i |z) = x0,x1 δ x1 (x i ) p t (z|x 0 , x 1 )π(x 0 , x 1 ) p t (z) = x0,x1 δ x1 (x i )p(x 0 , x 1 |z),is time-independent.For completeness we provide the formulas for denoiser (x-prediction) and noise-prediction (εprediction) parameterizations of the generating velocity field u :[0, 1] × R N → R N appearing in Table 1.In Continuous Flow Matching one can chose several ways to define the probability paths (Lipman et al., 2022; Liu et al., 2022; Albergo and Vanden-Eijnden, 2022; Pooladian et al., 2023; Tong et al., 2023) :EQUATION= p 1|t (x|x 1 )q(x 1 )dx 1 (55)EQUATIONDenoiser parameterization. The conditional generating velocity field u t (x|x 1 ) for p t (x|x 1 ), i.e., satisfy the Continuity Equation 16, takes the form (Lipman et al., 2022 )EQUATIONand the marginal generating velocity field is therefore given by the marginalization with the posterior p t (x 1 |x),EQUATIONThis shows the continuous Flow Matching denoiser parameterization of the generating velocity field in Table 1 .Noise-prediction parameterization. The conditional generating velocity field for p t (x|x 0 ) takes the formEQUATIONand the marginal generating velocity field in this case is given by marginalization with the posterior p t (x 0 |x),EQUATIONThis shows the continuous Flow Matching noise-prediction parameterization of the generating velocity field in Table 1 .Proposition 8. Assume a conditional probability path as in equation 9, then for any two schedulers κ t , κ ′ t , and ŵt (x i |z), ŵ′ t (x i |z) their corresponding posteriors as in equation 23,EQUATIONwheret ′ = κ -1 κ ′ t, and κ -1 is the inverse of κ.Proof (Proposition 8). For a conditional probability path as in equation 9,EQUATIONwhere in the 3rd equality we used κ t ′ = κ ′ t . Thus, also for the marginal probability path as in equation 7,EQUATIONwhere in the 2nd equality we used p t ′ (x|x 0 , x 1 ) = p ′ t (x|x 0 , x 1 ). Finally the change of scheduler for a posterior as defined in equation 23,EQUATIONwhere in the 3rd equality we used both p t ′ (z|x 0 , x 1 ) = p ′ t (z|x 0 , x 1 ) and p t ′ (z) = p ′ t (z).One potential benefit of non-autoregressive decoding is improved latency due to a significantly lower number of decoding steps. To demonstrate that, we measure the average latency of the proposed method compared with the autoregressive alternative using a single A100 GPU with 80 GB of RAM. We calculate the average latency time on the HumanEval benchmark using a batch size of 1. When considering 256 NFEs, the proposed method was found to be ∼2.5x faster than the autoregressive model (19.97 vs. 50.94 seconds on average per example). However, when considering 512 NFEs, both methods reach roughly the same latency. These results make sense as the number of tokens in most of the examples in HumanEval are below 512. Notice, that these results analyze latency and not model throughput. Due to the kv-caching mechanism following the autoregressive approach will result in significantly better throughput compared to the proposed approach Ziv et al. (2024) . We leave the construction of a kv-cache mechanism to the proposed approach for future research.G.1 Text Data. We use three splits of data. First is OpenWebText (Gokaslan and Cohen, 2019) . Second is the same mix used in Llama-2 (Touvron et al., 2023) , including textual and code data. For the code-focused models we use the same split used in CodeLlama (Roziere et al., 2023) . For the small models, we use OpenWebText. For the big models we use the Llama-2 and CodeLlama mixes.Models. We train two sizes of models: small (150M parameters) and large (1.7B parameters). For the small model we used a DiT transformer architecture (Peebles and Xie, 2022) with 12 layers, 12 attention heads, and hidden dimension of 768. We also used GPT2 tokenizer. The small models were trained on OpenWebText. For the large model, we use also used a DiT transformer architecture but with 48 layers, 24 attention heads, and hidden dimension of 1536 (Peebles and Xie, 2022) . For these models we used a tiktoken tokenizer. The large models were trained on the Llama-2 mix and the CodeLlama mix. For both models we used ROPE (Su et al., 2024) embedding with θ = 10000. Models are trained with Adam optimizer with β 1 = 0.9 and β 2 = 0.999. We use dropout rate of 0.1. Models are trained with a warm-up of 2500 steps, with a peak learning rate of 3e-4. We train the big models with batch size of 4096 for 1.3 million iterations and the big models with batch size of 512 for 400 thousand iterations.Entropy metric. We report the entropy of tokens within a sequence, averaged over all generated sequences. This intuitively quantifies the diversity of tokens within a given sequence. It's important to note that when computing sequence entropy, tokens not present in the sequence are excluded from consideration.Generative perplexity metric. The generative perplexity metric is the average likelihood of generated text evaluated with a second (usually stronger) model. We report the generative perplexity when averaged over 1000 samples. Double precision sampling. Zheng et al. (2024) demonstrated that sampling from a highdimensional distribution with full precision can lead to a similar affect as sampling with temperature. We evaluate our model using a categorical sampler in double precision. Table 5 presents the results of baselines compared to our method. Models. For all our experiments on CIFAR10 we use the U-Net architecture as in Dhariwal and Nichol (2021) , with following three changes to make it fully discrete and time independent (as we used mask modeling): (i) We replace the first layer with an embedding table of size 257 × 96, and we stack the channel features such that the input to the U-Net is of shape 288 × 32 × 32. (ii) We enlarge the size of the final layer to output a tensor of shape 3 × 32 × 32 × 257. (iii) We remove the time dependency from architecture. The hyper-parameters of the architecture: channels 96 , depth 5, channels multiple [3, 4, 4] , heads channels 64, attention resolution 16, dropout 0.4, which gives a total parameters count of 113M. We optimize the network using Adam optimizer with β 1 = 0.9 and β 2 = 0.999, a learning rate of 1e-4. We trained with an effective batch size pf 512 for roughly 300K iterations.Scheduler ablation. Figure 8 shows FID of our method with four different schedulers: Linear, Quadratic, Cubic, Cosine, both for training and evaluation. That is, for each scheduler we trained a model and evaluate FID with all four schedulers. We observe a high variance in FID between different schedulers, with the Cubic scheduler generally performing the best on both training and evaluation.Comparison with baselines. In the following, we provide implementation details for producing Figure 3a , that compares our schedulers and sampling algorithm with those employed by previous works.Cubic Scheduler (Ours). For the Cubic scheduler we set the corrector scheduler as above to,EQUATIONand we search over the parameters a, b ∈ {0, 0.25, 0.5, 1, 2, 2.5, 3}, and α ∈ {6, 8, 10, 12, 14}. Additionally, we search over the temperature ∈ {1, 0.9, 0.8}. We find that a = 2, b = 0.25, α = 12 give best FID.Linear Scheduler (Campbell et al., 2024) . For the linear scheduler we search over two additional hyper-parameters of the method: (i) For corrector scheduler as in equation 26, we set α t = 1 + tη, β t = α t -1, where η is the stochasticity parameter as in Campbell et al. (2024) , and search over η ∈ {0, 1, 2, 5, 10, 15}. (ii) We search over temperature in {1, 0.9, 0.8}. Finally, we find that the best FID is a achieved by η = 10 and temperature 0.9. MaskGIT (Chang et al., 2022) . For the MaskGIT we train and sample with the Cosine scheduler κ(t) = 1 -cos π 2 t which is reported to achieve best results by Chang et al. (2022) . For sampling we adjust the code from the re-implementation of Besnier and Chen (2023) . In addition, we also search over the temperature in {1, 0.9, 0.8, 0.7, 0.6, 0.5}, and we find the best FID is achieved by temperature 1.""" return sorted(set(l1) & set(l2)) def fibfib(n: int):"""The FibFib number sequence is a sequence similar to the Fibbonacci sequnece that's defined as follows: fibfib(0) == 0 fibfib(1) == 0 fibfib(2) == 1 fibfib(n) == fibfib(n-1) + fibfib(n-2) + fibfib(n-3 [ uv ] > dist [ cout ] + self.graph[ uv ][ v ]): dist [ uv ] = dist [ cout ] + self.graph[ uv ][ v ]self.printSolution(dist)We present below example generations for the proposed method together with several baseline methods. We provide both conditional and unconditional generations. For the conditional generations, we mark the prompt in gray.The United States on Wednesday asked the UN Security Council to slap an oil embargo on North Korea and freeze the assets of leader Kim Jong-un, in response to Pyongyang's response to the revelations it had restarted its nuclear work in March. "We will continue working to use maximum international pressure on North Korea to agree to the suspension of its nuclear program and reinstate sanctions," said John Bolton, who served as National Security Advisor and Secretary of State under US President Bill Clinton. "Here is North Korea's response to our sanctions," Bolton wrote in a letter to House Minority Leader Nancy Pelosi. "We want you to know that the international community is seriously monitoring North Korea at this time. North Korea is still complying with our requests from the past few days," Bolton said on Monday. "We have been working through the United Nations to provide the information that they gave us." Asked to whether any international pressure will be put in place for North Korea to give up its nuclear weapons, Bolton said the United States can use maximum pressure to get North Korea to abandon its nuclear weapons if it wants. "We've been working to use maximum pressure on North Korea, through the Security Council, and we will continue to do so," said White House Deputy Press Secretary Sarah Huckabee Sanders in Washington. "We're committed to taking any steps necessary to help North Korea pursue its only option for peace, including in this period," she added. The United States did not plan to produce any more oil at this time last year and had not planned to do so this year. "We believe that the North Korea approach is misguided in moving forward with its nuclear program to endanger peace and security in its homeland and neighbors in Asia," said Bolton, adding that the US supplies its nuclear weapons. "We don't want them to sell their nuclear weapons to other nations," he said. Bolton said the US would look for pressure on North Korea, which has been known to use nuclear weapons, as leverage to negotiations with the US. "I will reiterate what I have said before. So, the US has to put pressure on North Korea. But who else is going to hold the cards? Somebody else has to hold the cards," Mr Bolton said. Bolton described what the United States is prepared to do to get North Korea to agree to give up its weapons and asks for sanctions. "As far as I know, we have to use the pressure the reason for putting sanctions on North Korea," he said, adding that the US does not plan to ask the UN Security Council for sanctions alone.The defender is available for the Maribor first leg but his club believed he should be suspended. SNS Group Celtic made an administrative blunder which saw Efe Ambrose left behind in the midfield in the Maribor department and has given him a potential three-match ban today. Although Efe Ambrose will be suspended next Friday, according to reports in Scottish media, the Celtic defender will still be fit for the Champions League first leg at Celtic Stadium in the middle of August. However, the Celtic club wrongly thought that Efe should only receive a three-match ban because he is available for the first leg. Although Efe Ambrose may receive a three-match ban next Friday, Efe Ambrose was part of the Celtic squad for the last match against Liverpool. However, says SNS Group Celtic he was making a tactical error and was left behind in midfield. It is understood that Efe Ambrose did not make the final squad and only played 11 games for the club this season. Efe Ambrose made his professional debut for Celtic in 2008 but spent nine months looking for a new club to return to. With a career-high 72 Celtic appearances, Efe is among Celtic's most capped players ever.Carl Jara aka Grain Damaged is an award-winning, professional sand sculptor from Cleveland, Ohio. Jara says he has known since high-school that he wanted to be an artist. After studying English and Art History at the Northeastern University, Jara says one semester he started carving a custom sculpture into sand molds, but didn't know how to do it. With the help of an instructor, he found out and learned how to use rubber molds to make art. Later, he made the decision to learn how to use sand and sculpt himself. In addition to how he makes his own sculptures, Jara says he does special events for comics companies such as Stan Lee and also for institutions like local community colleges and universities. In November of this year, he won the WWHS, The Very Art Of The Very Things Cleveland competition. Afterward, he will continue carving for clients in the comics industry and looks forward to sand sculpting in Cleveland. The Artist is professional sculptor who has been making art, for over 25 years, in various shapes and sizes.The artist says art is all about relationships and the best way to go into the heart is to create art. The artist has taught in various high schools in the Cleveland area and has taught a full time Honors Studio for High School students in grades 6, 7, and 8 time for over 20 years. Since Art is a personal form of artistic expression, he works individually in a way that allows the student that his work engages their imagination and presents ideas in ways that inform and challenge their own paths. Miguel Romano is a professional artist who worked in 3D modeling and animation in the areas of web design and production. The artist currently works as a digital artist in the ad and mass communication industries. In coming to Concrete Cleveland, he is excited to apply the 3D development and production skills he have to his work. The artist has a BFA in sculpture from Ohio University, along with an MFA in sculpture.We look forward to seeing his work very soon! Designed and installed by Concrete This Week. He is a guy originally from Cleveland, Ohio where he pursued a career as a nurse. He then moved to the Atlanta, GA area where he returned to school with a BSN and a BS in nursing and is a licensed nurse. He is a proud sorority brother and still has extra-curricular, as well as taking music lessons and the occasional play. He is a lovely asset at Concrete Cleveland and looks forward to seeing concreteHere's how that strategy works for your job: 1) You now plan upon what you accomplish to fulfill your goals.Management cannot plan what happens to you. This may not be your ultimate personal decision, but it's perfectly fine to look at it. You just need to make sure you want to achieve this. Now, because you've worked at goal, you don't have to talk about your status tomorrow, after all, you have to do your job and take care of yourself.Next steps, there may be some work to do. There is a company down the road you right -literally millions of things that would have to be done. But of course it would have a different outlook. If you're going to do something, the customer might not be able to tell you.2) Between those two steps are the plan in step so that your actions will be executed. Although you might make a mistake with every single day to day plan, it still is a great opportunity to correct your mistake, become new and commit to working extra hours and meeting your goals promptly. The truth is, everything goes right for you no matter how quick a decision you become. The customer will never allow you to make the worst decisions. Otherwise, you make the very first decision.3) Take timing as part of action. If you don't feel like you can keep it, a plan without help of timing stops you from doing. When it's like your plan in action can lead to something such as this: Now that you know what to do. For example, you might live in a place in the building that serves every customer, has 3 employees per team, and 3 clients on one. You will get things done the next day. Change your performance is the first step towards greater success, for example. Your team, at this point in the Customer department, will know how the customer deal with a single employee, the level chain, and more. Make sure you take action now that you change it. As a company, it won't be hard but you will have lots of work to do when you change. 4) Make sure it's your night.Watch the humour but also the humanity behind the work we're doing. The truth is something very tragic and delicate in the middle of a very fractured world. It's the one thing that makes me proud. I feel like a singular individual has had to come together with this story. There's a lot of people who I've worked with for the very beginning, because I have got people, you've got people who have just had these eyes on this story, and this sense of what we are, that run through our final movie. It's the very beginning we're at. The very beginning, we're not there, we will get there but we won't need. This is a story and these amazing actors, these fantastic violence, violence, that was just are elements and a complex world of conflict. When something like that is set to build this narrative and you're directing the world of these characters based around their individual needs it's very, extremely confusing, very heartbreaking -it's really quite intense-this was all built within it, and what it is, it's a 35-year old period that was slavery and still was very strong, these guys were operating to the edge and going to the point where we ended up setting up a big narrative, OK, that's good, it's okay, in some ways heroism is a noble imperative that we are fighting against, and recognize maturity as the mercurial nature and these are all human and we've got to clean it up so that that stuff is there and we've got to restore it. And the project we're looking at here is our common goal is that anything can be done to make that happen and everybody can do whatever their want to do and do it as they please. That's the spirit of it. That's the movie I've made with Steven Wright in writing.Conclusions and future workWe introduce Discrete Flow Matching, a generalization of continuous flow matching and discrete flows that provides a large design space of discrete non-autoregressive generative models. Searching within this space we were able to train large scale language models that produce generated text with an improved generative perplexity compared to current non-autoregressive methods and able to solve coding tasks at rates not achievable before with non-autoregressive models, as far as we are aware. While reducing the number of network evaluations required to generate a discrete sample compared to autoregressive models, Discrete Flow Matching still does not achieve the level of sampling efficiency achieved by its continuous counterpart, flagging an interesting future work direction. Another interesting direction is to explore the space of probability paths in equation 8 (or a generalization of which) beyond what we have done in this paper. We believe discrete non-autoregressive models have the potential to close the gap and even surpass autoregressive models as well as unlock novel applications and use cases. As our work introduces an alternative modeling paradigm to discrete sequential data such as language and code, we feel it does not introduce significant societal risks beyond those that already exist with previous large language models. 
 
 Appendix:  


Decision: The paper introduces a novel variant of the flow-matching model for discrete data types. The authors provide solid theoretical foundations for their approach and conduct experiments in language modeling, code generation, and image generation tasks. The model can be seen as an alternative approach to autoregressive models and possibly opens new research directions in generative models for discrete data. 

The reviewers are pretty consistent in evaluating the presented approach's quality and novelty, and many concerns were solved during the rebuttal phase. The idea of discrete flow matching was initially introduced by Campbell et al., 2024, which may slightly lower the contribution. However, the authors clearly point out the differences and compare quantitatively with this approach considering the image generation task. Therefore, the paper should be accepted.

Reviews: review: paper_summary: paper_summary: The paper presents a discrete flow matching method for modeling discrete data with discrete state space. The paper presents unified frameworks for training and sampling from the discrete probabilistic model. Importantly, the paper also studied scaling up the model to 1.7B parameters and tested the model on code generation tasks. Comprehensive experiments shows that the proposed method outperform existing methods and also close the gap to autoregressive models.
strength_weakness: strength_weakness: The paper lacks some experiments on common benchmarks used in existing discrete diffusion models, e.g., LM1B and OWT. I understand the scaling-up experiments on HumanEval are more challenging, but I believe a comparison on the common benchmark would be important to justify the effectiveness over existing methods.
questions: questions: See weakness.
limitations: limitations: I didn't find limitations specifically discussed in the paper, though the author marked it discussed in the checklist. Please correct me if I missed it.
paper_summary: paper_summary: The paper introduces a novel approach called Discrete Flow Matching, which adapts continuous flow models to discrete sequential data. It extends prior work by integrating discrete state spaces and time-dependent schedulers into a unified framework for non-autoregressive generative modeling. Methodologically, Discrete FM employs generating probability velocities derived from learned posteriors and schedulers, enabling efficient sampling and correction processes. This approach facilitates the transformation of noise distributions into target data distributions with enhanced flexibility and performance. Experimental evaluations across language modeling, code generation, and image generation tasks demonstrate significant improvements over existing methods. Specifically, the model achieves state-of-the-art results on various benchmarks, including HumanEval and MBPP coding tasks, showcasing its efficacy in generating high-quality outputs.
strength_weakness: strength_weakness: - The method is heavily based on the Continuous-Time discrete Markov Chain (CTMC) paradigm from Campbell et al., although this paper proposed theoretical and empirical improvements such as the unified formulation for more general probabilistic paths and velocity as well as the scheduler designs.

- There are still performance gap in code generation evaluation when compared with autoregressive language models.
questions: questions: - I am a bit confused with Eq 10 especially the scheduler terms. Please explain or tell if there are mistakes.

- What are the main reasons for discrete flow matching to require significantly large number of evaluations?
limitations: limitations: The authors acknowledged their limitations such as the number of evaluation is high compared to continuous flow matching.
paper_summary: paper_summary: The paper proposes an approach to generative modelling for discrete data, i.e. multidimensional distributions where the variable along every dimension can take value in a finite set. This is an alternative approach to autoregressive generative modelling for discrete data which is currently actively studied for language and code generation.

The philosophy of the proposed method is heavily inspired by the Flow Matching algorithm [1] and Continuous Time Markov Chains (CTMC), which were previously used in [2] to propose a similar model. That is, for a given CTMC, the authors define the vector field generating samples from this CTMC by local updates of the samples (independently along every dimension). Based on the PMF (Probability Mass Function) of the CTMC, the authors derive a formula for the vector field. Furthermore, they introduce the continuity equation analogous to the continuous case, which allows for an easy validation that the change of the density given by CTMC corresponds to the vector field.

The authors perform an ablation study of some of their design choices and extensive empirical studies for generation of code, language, and discrete-valued images. The proposed model outperforms the competitors bridging the gap between flow-based models and autoregressive models.

[1] Lipman, Yaron, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. "Flow matching for generative modeling." *arXiv preprint arXiv:2210.02747* (2022).

[2] Campbell, Andrew, Jason Yim, Regina Barzilay, Tom Rainforth, and Tommi Jaakkola. "Generative flows on discrete state-spaces: Enabling multimodal flows with applications to protein co-design." arXiv preprint arXiv:2402.04997 (2024).
strength_weakness: strength_weakness: The paper presents a complete study and its methodological part does not raise any major concerns. However, I would like to ask the authors to clarify the following question.

Why do posteriors in eq. 15 define a proper distribution? Indeed, according to eq. 8, the linear combination of the conditional distributions with the schedulers should define a correct distribution (sum up to $1$). If we simply try to sum over all possible $x^i$ in eq. 15, we won’t get $1$, hence eq. 15 does not define a posterior. This is an important detail when defining the objective in eq. 25, which is the cross entropy between distributions.

The minor

- Descriptions of the schedulers on lines 96 and 98 are not complete, i.e. there are no conditions that schedulers have to define correct probability distributions in eqs. (9,10) and the scheduler $\kappa^3$ is missing a description.
- Figure 2. The plots presented in the figure have $d = 2$ unlike what is stated in the caption.
- At the beginning of the paper, the authors introduce the notation of random variables as capital letters (e.g. $X_t$). This creates confusion in equations 13-17 because there the authors clearly mean $X_t$ to be the value of a random variable.
- There is a typo in line 160 when describing the panel of Fig. 2.
- There is a typo in line 163 when defining the vector field $v(x,z)$.
- There is a type in line 582. Index $\ell$ is not a function of $j$ according to eq. (36).
- I haven’t checked thoroughly, but I think equations on the top of page 19 should have a summation over $j\neq \ell$ instead of the summation over all possible $j$.
questions: questions: I would suggest adding a discussion of the variable-length generation. It is an important difference with autoregressive modelling and I’m wondering how the authors handle this issue given that their model has two important properties:

1. Conditional generation for partially masked sequences.
2. Independence of the conditional distributions between dimensions.

The questions I would like to have answered are:

1. Does one have to define the length of the generated sequence before generation? If so how this can be decided?
2. What’s the computational cost if one continues generating the sequence in the autoregressive way?
limitations: limitations: NeurIPS Paper Checklist does not follow the required format.

The paper adequately discusses the limitations of the proposed approach. The only thing that is not cover enough in the paper is the variable-length generation.
paper_summary: paper_summary: This paper presents Discrete Flow Matching, a new method for generating discrete data, such as language. The approach uses a general family of probability paths between source and target distributions. It offers a formula for sampling from these paths using learned posteriors. By focusing on specific probability paths, it improves generative perplexity compared to previous similar models. When scaled up, the method achieves notable performance on benchmarks. This approach bridges the gap between autoregressive models and discrete flow models.
strength_weakness: strength_weakness: - In line 32, you mention one advantage of FM is its flexibility in handling non-Gaussian target distributions. Have you demonstrated this case?
- The method's description is unclear and confusing. I cannot distinguish which part pertains to Campbell's methods and which part is yours.
- In Table 2, it's unclear why the results from Austin et al. [2021a] are poor, as no explanation is provided.
- In Line 214, I'm not entirely sure what the difference is. Perhaps a table could clarify it?
- In Figure 3, the Inception Score (IS) needs to be included to demonstrate the diversity of image generation.
- Except the perplexity, how about the performance on BLEU, bertscore?
- Diversity is a significant concern for flow-based models. What about the result on diversity-related metrics?
- In F.1, in the context of conditional generation, it's unclear what the source (src) and target (tgt) samples are. Are they conditional prompts, target prompts, or a combination of Gaussian noise and target prompts?
- In section H, there is no qualitative analysis of unconditional generation.
- In Section H, I did not see the color red. I only observed the background colors: grey and yellow.
- It will be great that the code can be provided.
- Related works are missing.  it's important to note that flow matching has been utilized in various domains to capture the reader's interest. E.g., boosting diffusion[1],image generation[5], depth estimation[2], motiom[3], even text generation[4].

[1]. Boosting Latent Diffusion with Flow Matching

[2]. DepthFM: Fast Monocular Depth Estimation with Flow Matching

[3]. Motion Flow Matching for Human Motion Synthesis and Editing

[4]. Flow Matching for Conditional Text Generation in a Few Sampling Steps

[5], Latent Space Editing in Transformer-based Flow Matching
questions: questions: as above
limitations: limitations: as above

Comments: title: strong accept
response acknowledgment
thanks for your reply
comment: Thanks for the author's reply; my concerns are resolved.

I have one suggestion for the draft: emphasize more in the main paper about the instantiation of the scheduler, as it's critical for performance.
Thanks for the response that addressed my concerns. I would keep the rating to recommend acceptance.
As we approach the deadline for the authors-reviewers discussion, we would like to kindly inquire if there are any additional questions from the reviewers. We are happy to address any remaining clarifications or concerns.
We are planning to release the code around the time of publication. As we near the end of the discussion period, we would be happy to address any further clarifications or concerns regarding the other comments raised by the reviewer.
Thank you for your response! Everything is clear. Sorry for the confusion in some of the questions.
i am looking forwards to the code.
**Comment: In section H, there is no qualitative analysis of unconditional generation.** Per the reviewer's suggestion, we added unconditional samples, generated by our model (attached in the pdf in the main response). We will add these samples to the revised version.

**Commnet: In Section H, I did not see the color red. I only observed the background colors: gray and yellow.** This is a mistake, prompts are marked in gray, we will fix it in the revised version.

**Comment: It will be great that the code can be provided.** We are planning on releasing the code not much after the paper is published.

**Comment: Related works are missing.** We thank the reviewer for the suggestion, we will add the relevant related works to the revised version.